InfMasking: Unleashing Synergistic Information  by Contrastive Multimodal Interactions
Authors: Liangjian Wen
Qun Dai
Jianzhuang Liu
Jiangtao Zheng
Yong Dai
Dongkai Wang
zhao kang
Jun Wang
Zenglin Xu
Jiang Duan
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal contrastive learning，Self-supervised learning，Multimodal Interaction， Synergistic Information
Primary area: general_machine_learning

Abstract: In multimodal representation learning, synergistic interactions between modalities not only provide complementary information but also create unique outcomes through specific interaction patterns that no single modality could achieve alone. Existing methods may struggle to effectively capture the full spectrum of synergistic information, leading to suboptimal performance in tasks where such interactions are critical. This is particularly problematic because synergistic information constitutes the fundamental value proposition of multimodal representation. To address this challenge, we introduce InfMasking, a contrastive synergistic information extraction method designed to enhance synergistic information through an Infinite Masking strategy. InfMasking stochastically occludes most features from each modality during fusion, preserving only partial information to create representations with varied synergistic patterns. Unmasked fused representations are then aligned with masked ones through mutual information maximization to encode comprehensive synergistic information. This infinite masking strategy enables capturing richer interactions by exposing the model to diverse partial modality combinations during training. As computing mutual information estimates with infinite masking is computationally prohibitive, we derive an InfMasking loss to approximate this calculation. Through controlled experiments, we demonstrate that InfMasking effectively enhances synergistic information between modalities. In evaluations on large-scale real-world datasets, InfMasking achieves state-of-the-art performance across seven benchmarks. Code is released at https://github.com/brightest66/InfMasking.

InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions
Liangjian Wen 122 Qun Dai Jianzhuang Liu", Jiangtao Zheng', Yong Dai ,Dongkai
Wang Zhao Kang", Jun Wang', Zenglin Xu ,Jinng Duan
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics
Engineering Research Center of Intelligent Finance, Ministry of Education,,outtwwsste University of Finance and Economics
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
*X-Humanoid
Unniversity of Electronic Science and Technology of China
"Shanghai Academy of AI for Science
Artificial Intelligence Innovation and Incubation Institute, Fudan University Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province fw1j6816, dai qun 1243 @gmail. com com
Abstract
In multimodal representation learning, synergistic interactions between modalities not only provide complementary information but also create unique outcomes through specific interaction patterns that no single modality could achieve alone. Existing methods may struggle to effectively capture the full spectrum of syner- gistic information, leading to suboptimal performance in tasks where such interac- tions are critical. This is particularly problematic because synergistic information constitutes the fundamental value proposition of multimodal representation. To address this challenge, introduce InfMasking, contrastive synergistic infor- mation extraction method designed to enhance synergistic information through Infinite Masking strategy. InfMasking stochastically occludes most features from each modality during fusion, preserving only partial information to create representations with varied synergistic patterns. Unmasked fused representations are then aligned with masked ones through mutual information maximization to encode comprehensive synergistic information. This infinite masking strategy enables capturing richer interactions by exposing the model to diverse partial modality combinations during training. As computing mutual information es- timates with infinite masking is computationally prohibitive, we derive an Inf- Masking loss to approximate this calculation. Through controlled experiments, we demonstrate that InfMasking effectively enhances synergistic information be- tween modalities. In evaluations on large-scale achieves sfate-of-the-ale art performance across seven benchmarks, Code is released at htttps://githh wwt.booggtetttt////// Iww.attttttt//aaaannaaaaa http::///ggttuuu..mmm/rgggttette
datasets, InfMasking
'Equal contribution. Corresponding author. Project leader.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1 Introduction
Multimodal contrastive learning has revolutionized representation learning by enabling the integration of diverse data modalities, such as text, images, and audio, into a unified latent space. This paradigm leverages contrastive loss [37 9 align features from different modalities as pioneered by founda- tional works like CLIP 38 and ALIGN [23] in vision- lision-language tasks. These models demonstrate the power of aligning multimodal features to capture shared patterns across data sources, enabling versatile downstream applications. However, current approaches often rely on the restrictive multiview redundancy assumption 44 43 46], which posits that a modality is (approximately) sufficient for the prediction of downstream tasks and contains the same task-relevant information. This assumption derives from multi-view learning and is limited in real-world multimodal settings because many multimodal tasks involve minimal shared information.
The shortcomings of this redundancy-centin perspective become increasingly apparent when we examine the multifaceted nature of multimodal interactions. As illustrated in G 12] these inter- actions can be classified into three fundamental categories: redundancy, uniqueness, and synergy. Redundancy refers to scenarios where a modality can independently perform the task due to over- lapping, shared information. Uniqueness describes cases where only one modality possesses all the requisite information for task completion. Synergy, arguably the most significant yet elusive of the three. occurs when modalities provide complementary information that must be combined to achieve the desired outcome. These interaction types are not statie; their predominance varies depending on the specific task, adding a layer of complexity to multimodal learning. For instance, a task might rely heavily on redundant information in one context, while another demands the synergistic integration of modalities to succeed. A compelling example of this is hateful meme detection [26], where synergy emerges when seemingly neutral modalities (such as innocuous image and benign text) combine to create harmful content that neither conveys on its own. This highlights the critical importance of synergistic integration, as models must fuse information from different modalities cues to uncover implicit biases or offensive implications that are only apparent in their joint context, enabling more effective identification and mitigation of such content in real-world applications. Consequently, task- agnostic multimodal representations must necessarily encompass the full spectrum of multimodal interactions that transcend mere informational redundancy.
Recently, FactorCL (29] explicitly decomposes shared and unique representations, enabling the estimation of redundancy and unique interactions beyond multi-view redundancy. However, its factorized approximation of multimodal interactions is prone to cumulative errors and fails to capture synergistic information effectively. In contrast, [ integrates features from all modalities to derive a common representation and subsequently maximizes the mutual information between this representation and its augmented variants, as well between the common representation and its corresponding unimodal features. Although this approach facilitates the capture of redundant, unique, and synergistic information across modalities, primarily emphasizes enhanced redundant and unique interactions. Synergistic information is captured by maximizing the mutual information between the common representation and its augmented variants. Such handling may prove inadequate for tasks that heavily rely on complex inter-modal synergy. The complexity of synergistic interactions lies not merely in modalities providing complementary information but in how these modalities combine through specific interaction patterns to produce outcomes unattainable by any single modality alone. Such interactions may involve nonlinear dependencies or context- dependtde dynamics. Hence, effectively capturing the full spectrum of synergistic information remains significant challenge.
To address this challenge, introduce a contrastive synergistic information extraction method via infinite masking (denoted as InfMasking). Specifically, InfMasking stochastically occludes a substan- tial proportion of features from each modality during the fusion process. This masking preserves only partial information. creating fused representations with varied synergistic patterns. Subsequently, unmasked fused representations are aligned with these masked ones via mutual information maxi- mization to encode comprehensive synergistic information. The infinite masking strategy enables InfMasking to capture richer synergistic interactions by exposing the model to diverse combinations of partial modality information during training. However. the computation of mutual information estimates with infinite masking is computationally prohibitive. To address this issue, we derive an InfMasking loss to approximate the calculation of this loss function. Empirically, InfMasking effectively captures synergistic information between modalities in controlled environments with known interaction types. When tested real-world datasets across diverse domains (healthcare,

---

robotics) and data types (image, text, audio), InfMasking achieves state-of-the-arr results on seven multimodal tasks involving two or three modalities.
2 Preliminary: Contrastive Multimodal Interactions
Consider X1, (1,....... Xn, as random variables, each representing a distinct modal data (e.g.. image, text, audio, or tabular data), alongside a downstream task Y. The objective is to derive an effective multimodal latent variable Zø fX()), where X = (X Xn,) and @ detmotes the parameter of multimodal encoder. Multimodal interactions refer to the process of extracting and integrating information from multiple data sources, such text, image, audio, or tabular data, to form cohesive representation for downstream tasks. These interactions can be categorized into three types: redundancy (R), where information is shared across modalities; uniqueness (U), where information is specific to a single modality; and synergy (S), where complementary information emerges only when modalities are combined.
To understand multimodal interactions, we can leverage Partial Information Decomposition (PID) [49 6). theoretical framework that dissects the mutual information between multiple variables. For analytical tractability, we focus on the case of n = 2. Consider two modalities X1 and X₂ and a task Y. The mutual information I [(X1 X2; Y) quantifies the total task-relevant information provided by both modalities jointly. According to PID, this can be decomposed as: I X2; X2; where R represents redundant information, common both X1 and X2, S represents synergistic information, emerging only from the combination of X, and X2, and U1 and U2 represent unique information specific to X2 and X2, respectively. This decomposition is supported by consistency equations derived from the chain rule of mutual information:
= R+ S U1 U2.
(X);Y = R U1, 1(X2; Y) = R U2, X(;1; = R S.
Effectively capturing these interactions is fundamental multimodal learning, as they collectively enhance a model's ability to generalize across diverse tasks. Therefore, an effective multimodal representation Zo must capture information relevant a downstream task Y, preserving the mutual information such that Y) = T(X; Y). In self-supervised learning, however, Y remains unspeci- fied during the representation learning phase, presenting a unique challenge. To address this, [12] extends the contrastive learning principle of multiview redundancy to multimodal contexts:
Assumption (Minimal label-preserving multimodal augmentations) A set T of multimodal transformations exists, such that for any E T and X' 1(X), the mutual information I(X;'') I(X;Y) holds, preserving the information with label Y.
Assumption [ is justified within the framework of multimodal representation learning. It enables broader range of augmentations, as it is not limited the set T [((X) = (4, (X1), (X2)) ..... Ln (X.)). We define Z = fo(X'), where X'= ((X) with T representing multimodal augmentation. Considering the data processing inequalities for the Markov chains X + X' - Z/ and % + X + Zeg, we can establish the following mutual information bounds:/ (Za; 2%) < 1 (X; 43) I 1 (X; According to these inequalities and Assumption I we
can prove the following lemmas:
Lemma 1 When optimizing the function fe to maximize mutual information T (7g;; Zz), and under the assumption that the network fe possesses sufficient expressivity, we observe that in the optimal parameter configuration: 1[Ze-, Z.g) I(X,X') I(X,Y).
For Zg to serve as an effective representation of X, it must adequately preserve and encode all task-relevant information inherent in X. We note that [(;; Y) = [(x, X2; Y) = R+$ S+S+US UU + U2. Based on Lemmal I we can learn common multimodal representations Za and quantify all multimodal interactions beyond redundancy by maximizing the mutual information I (Zo; Z%).
Lemma Suppose fo- is optimal, meaning it maximizes I(Zo-, Zg-)
Then, the equality
1(Z Y) = (X',Y) YY) holds. For the special case where T = {14) such that X' (xX)= X; and 2g. = So- (X)= Z; for E (1,2), the following holds: I (Z,; Y();;)))= =
R+
For unimodal representations Z; where E (1,2) to effectively represent X,, each representation must encode the information 7(X,; Y) R+U, contained in the corresponding modality. According to Lemma 2 we can learn optimal unimodal representations Z, and quantify both redundant and unique multimodal interactions by maximizing the mutual information I (Z,; Z%) and I (Z,; Zo).
3

---

X2
feature
Multimodal InfoNCE loss forward propagation
m
mK
Masked token
Unimodal InfoNCE forward propagation
Representation
InfMasking loss forward propagation
Latent Feature
X
X1 x
Modality-speci
X2
random mask
X2 augmentation
X2
X
Enoeder
k G2
concatenation
concatenation
X Xn
X
concatenation
Transformer blocks
x2
x2
2
2
6.
Z2
IIfONCE
LInfONCE
FootMasking
2 d
Z,
2
Unimodal Features
Multimodal Features
Masked View Features
Figure 1:
The overall pipeline of InfMasking. Given modalities X (X1,X2, ,,,..., Xn),
augment them to obtain X' and X", which are then encoded independently by modality-speciinn encoders to extract latent features. These features are processed in three ways: (1) All modality features are concatenated and input into a Transformer block, yielding fused features Z' and Z": (2) Each modality feature is individually input into Transformer block, producing unimodal features Z1, Zz..... Zn (3) Features of each modality are randomly masked, concatenated, and input into a Transformer block, repeated times obtain Zmask 2 .... Zk musk.
Unleashing Synergistic Information through Infinite Masking
3.1 The General Framework
The proposed framework, termed InfMasking, is a multimodal contrastive interaction method designed enhance synergistic information across modalities by leveraging infinite masked views. The overall pipeline of InfMasking is illustrated in Fig. and consists of two primary stages: modality-specific latent feature encoding and multimodal feature fusion via Transformer. Given input set of modalities X =
Xx), we obtain X' and X" through an augmentation process. Subsequently. X' and X" are processed by modality-speeiine encoded independently to extract latent features. As shown in Fig. []] these modality features are then processed in parallel through three distinct ways: (1) All modality features are concatenated and input into a Transformer block, yielding fused features Z' and Z"; (2) Each modality feature is individually input into Transformer block, producing unimodal features Z Z!...., Z' and Z Z (3) Features of each modality are randomly masked, then concatenated and input into a Transformer block obtain fused features. This process is executed & times, resulting Maakk 2 Zaask and 7#1
encoders, where each modality is
Z
2
7
Based on Lemma I and Lemma
proposes a contrastive multimodal (CoMM) learning approach
learn lask-agnostic multimodal representations by modeling multimodal interactions, including redundancy, uniqueness, and synergy. CoMM estimates the mutual information using the InfoNCE
loss: ÍN Z Z') = E2, ~2(,2") log exp(=" "io==//////+++ +/,/T)+E.L Ei,, exp(= / where a temperature
exp(s" 2////r)
parameter. Hence, its training objective is formulated as follows:
n
LCAMM
(Ncr(2',
(xer Zi, Z') + iNCF (Zi 2nr)
(1)
2
*R+S+27 U.
RR+U,

---

While the first term theoretically quantifies redundancy, synergy, and the unique information across modalities, empirical evidence indicates that its practical performance exhibits notable limitations. Hence, the second term constitutes the fundamental component of CoMM, specifically designed to strengthen both unique and redundant interaction patterns. However, enhancing synergistic interactions remains a substantial challenge in this framework.
Synergy is complex interaction that arises when different modalities provide complementary infor- mation, necessitating their integration for effective task performance. We aim to learn a multimodal representation that captures all three types of interactions, with particular emphasis on enhancing synergistic information. To achieve this, we introduce InfMasking novel approach that leverages infinite masking enhance the modeling of multimodal interactions. Our training objective formulated as follows:
Cooal loss INCE(Z', (Z'
[iNCE(Z,, 22) iNCE Z")
*1+ +
xR+UU
Emask
[NCP( maaak Z + INCA ("maskk Zn
Stuhhtsiing
where LiniMasking represents novel masking-based regularization term designed specifically enhance synergistic interactions, as detailed in Section 3.2
3.2 Contrastive Synergistic Information via Infinite Masking
In multimodal learning, capturing synergistic information where different modalities provide complementary insights-is essential for tasks requiring integrated understanding. We propose contrastive synergistic Information method via infinte masking enhance synergistic interactions. It's core idea to randomly mask a significant portion of the features from each modality during the fusion process. As shown in Fig. [1 we fuse all masked features from different modalities to obtain a fused representation Zymask via a Transformer. Each time features from each modality are randomly masked, only partial information from each modality is retained. Consequently, after each masking operation, Z'mask contains distinct synergistic information. Then, by aligning Z'nask with its unmasked counterparts Z' through maximizing their mutual information, Z' are encouraged to capture distinct synergistic information. This process is repeated for K times of masking, we can obtain the final training loss: 인 Σ İNCE (Zyki Z') + INCE (Z" 2"). To enable the model to learn diverse forms synergistic information, we allow K to approach infinity through infinite masking, ultimately obtaining the masking loss LintMusking as follows:
LiniMuskig = lim
1
Σ
INC (7 7") + INCE ( z")
K-000
K
1=1
=-Emusk
[INCE (Zmuer Z') + INCIE( (" Zm Z"))
(3)
This infinite masking strategy enables InfMasking to capture richer synergistic interactions by exposing the model to diverse combinations of partial modality information during training. However, the estimation of Emusk [iNCe( Z via random mask samples.
} and Fyaask [[NCE 7 Z")] is computationally expensive
5

---

To address this issue, we derive lower bound for Emask [NCE mask z7] optimize the Infaaskin
loss function Eq. 3 The detailed derivation is follows:
exp(2" (T </mas//T)
Emask| [NCEE Ziaas : Z')] = Bmask (Bxr log exp(2"2'"maa smack// /T) + Ex thpg
(4)
expl 2222 kneg TT)
Eetwo((z)) [Emuak
(e mmack//) log/exp(22> x </mas//T) + Σ exp(2 Zneg! [hog//T]]
(5)
Assa
2 Be
x' T "mmack (oansk//mmsk//// log Bmask [mms(expp(=" m
Σ
exp(= F T)]
Freg
The inequality Eq. (6) merges from the application of Jensen inequality on concave functions i.c., Eg (Xg(X) S log Ex[x], mask denotes the integrated representation derived from the fusion of all masked features across diverse modalities via the Transformer architecture.
Inspired by [7]. posit that </mask follows Gaussian distribution, formally expressed a
NHH: "ook Snima): where = and Σx i.a.k denote the mean vector and covariance matrix of "mask* respectively. This assumption is well-founded for two principal reasons. First, the masked embeddings tend to cluster around central value in the embedding space, as they all inherently reflect aspects of the query's 's semantic nature, Second, the variance observed across feature dimensions can be interpreted as representation of semantic differentiation in the ambient space, which aligns with established principles in distributional semanties. Under this assumption, we can derive:
Lemma Let Praae and Enenss be the Gaussian mean vector and covariance matrix of maskk respectively. The lower bound of Emark [iNCE( (mmax: z3)] can be obtained as follows:
Emmast (NNCE( (N ?marks? ?!))
IT +
2)
2 Ex)x
loglexp(=" pht /T
Eexp(="T2nee
(7)
27
This allows to approximate the mutual information between the masked and unmasked represen- tations without requiring exhaustive sampling of all possible masks. A detailed proof is given in Appendix G
4
Experiments
We perform experiments on both synthetic benchmarks and multiple large-scale real-world datasets to verify the effectiveness InfMasking in learning representations from diverse modalitie.. To evaluate InfMasking's is capacity to capture three essential aspects of multimodal interactions (i.e., uniqueness, redundancy, and synergy), we generate synthetic data controlled environment based on the Trifeature dataset [22] Furthermore, we assess the generalizability of InfMasking on several widely used multimodal benchmark datasets involving diverse modality combinations real-world scenarios. These tasks span various domains (e.g., healthcare, robotics, elc.) allowing for thorough assessment of the model's representation capabilities across diverse modalities. Detailed experimental settings provided in Appendix A For evaluation,
linear probing, i.e., freezing the pre-trule pre-trained feature extractor and training a linear classifier (or regressor, depending on the task) on top of the learned representations. The downstream task performance of the linear model serves an indicator of the quality of the learned multimodal representations.
4.1 Synthetic Experiments on Trifeature Datasets
Following the experimental design of the Trifeature dataset in CoMM 12 trolled experiments on a synthetic dataset derived from Trifeature. We assess the model's ca- pacity to learn uniqueness, redundancy and synergy through two separate experiments. In terms
we conduct con-

---

of uniqueness and redundancy, we define shapes as redundant features and textures as unique- ness features. And the task involves two subtasks: (1) identifying the shared shape between two images (redundancy) and (2) predicting the texture of the first image (or second image) (unique- ness). The random-guessing baselines in both cases corresponds to 10%. for synergy, we artificially introduce a strong correlation between textures and colors by defining mapping M in the training set (e.g., blue maps to triangles, stripes to red), resulting in 50% baseline for random guessing. The model is trained on image pairs that follow this mapping. The task is to determine whether a given image pair satisfies the mapping Y 1 [(eetture color (X2) M), thereby evaluating the model's s ability to capture synergistic interactions across modalities.
Experimental results are illustrated in Tab. 0 Cross- modality constraints based on the InfoNCE loss [38
Table I: Linear probing accuracy (in %) of redundancy (shape),
uniqueness (texture) and synergy (color and texture) on Trifeature dataset. + denotes results are from [2]
("Cross" model) achieve the best performance at capturing redundant information but struggle with uniqueness and synergy. FactorCL [29]. self-supervised constraints each encoder ("Cross Self" (51) and MAE [19 (implementation details are provided in Appendix [.I]]
Model
redundancyt
uniquenesst
synergyt
Cross
38
100.0
11.6
50.0
Selss+Self
51
99.7
86.9
50.0
FactorCL +
99.8
62.5
46.5
MAE 19
99.8
82.4433.09
50.140.24
CoMM 12
99.9
86.812.99
71.4
InfMasking (ours)
99.9 10.09
90.6
77.0 77.0
improve uniqueness but remain limited in modeling synergy. CoMM 112 performs well across all three interactions. However, still has considerable room for improvement, particularly in capturing synergistic information. In comparison, InfMasking achieves superior performance in capturing both redundancy and synergy, outperforming CoMM by 3.8% and 5.6%, respectively.
4.2 Experiments Real-world Datasets
We further evaluate the performance of our model on several real-world multimodal datasets provided by Multibench [30]. These datasets span diverse modality combinations and task types, providing comprehensive benchmark to assess the model's ability to learn effective multimodal representations. Further dataset details are provided in Appendix B
4.2.1 Experiments with 2 Modalities on Multibench
Following the data preprocessing procedure of previous work 22 12 we conduct our experiments using the same encoders, modality configurations and train models based encoded inputs with diverse modalities. We consider "Cross". "Cross+Self", FactorCI and CoMM as haselines for comparison. As presented in Tab. InfMasking consistently achieves the best performance across all
Table 2: Linear probing MSE(x10-1 for regression task and top-1 accuracy (in %) for classification
tasks on Multibench. denotes results are from 22 denotes average is only selected from the results of classification tasks.
Regression
Classification
Model
V&T EE
MIMIC
MOSIT
UR-FUNNYt
MUSTARD1
Average"
Cross
33. 43.00943.77
66.7
47.8+18
21.9
53.5+29
54.52
Sell
5
7. 0.56
65.4910.0
49.0+1.1
59.920.
53.944.
57.07
FactorCI.
29
10.8244.56
67.3
51.2+1.6
60.5
55.8040.9
58.7
CoMM
12
7.9642.13
66.440.41
63. 22..5
63.3
64.441.1
64.45
InfMasking (ours)
4.23
68. 68.140.42
69.0+1.2
64.3
66.8
67.05
benchmark datasets. In the binary classification tasks, InfMasking outperforms CoMM- the strongest baseline-by 1.7%, 5.3%, 1.0%, and 2.4% on the MIMIC, MOSI, UR-FUNNY, and MUSTARD datasets, respectively. For regression tasks, InfMasking also delivers superior results, achieving
7

---

IMDb) (2] is a real-world multimodal, multi-label dataset designed for movie genre classification. poses two major challenges: significant class imbalance with genres such as comedy and drama dominating the label distribution, and substantial semantic discrepancy between visual (poster) and textual (plot's description) modalities. Since genre prediction based single modality lead of 3 x 10-4 in terms of MSE compared to the second-best best model. These experimental results demonstrate the effectiveness of InfMasking in capturing bimodal interactions. Furthermore, its consistently strong performance across diverse datasets highlights the generalizability and robustness of InfMasking in real-world bimodal scenarios.
4.2.2 Experiments with 3 Modalities on Multibench
We evaluate the generalizability of Inf- Masking in learning multimodal rep- resentations beyond two modalities. Specifically, we conduct experiments two datasets: Vision&Touchs (for the con- tact prediction task, with visual, force, and proprioceptive modalities) and UR- FUNNY (with visual, text, and audio modalities). CMC [41] and CoMM are selected baselines for comparison in the three-modality setting.
Table 3: Linear probing top-1 accuracy (in %) for clas- sification tasks on Vision&Touch and UR-FUNNY. * denotes results are from [12].
Model
#Mod.
V&T CPT
UR-FUNNY
Cross
2
86.3=a.25
50.1
Cross+Self
2
87.6=0.26
59.9*
CoMM
2
87.0
63.3
InfMasking (ours)
88. 88.520.33 2
64.3
The results are summarized in Tab. 3 more intuitively assess the information gain introduced by incorporating third modality, we additionally report results from bi-modal training scenarios using
To
CMC 41
3
94.1
59.2
CoMM
3
94.1
64.821.13
InfMasking (ours)
3
94.120.00
65.6
COMM, "Cross" and "Cross + Self". Specifically, we train these baselines on (1) the image and proprioceptive modalities of the Vision& Touch dataset, and (2) the image and text modalities of the UR-FUNNY dataset. Our experiments reveal that adding a third modality significantly enhances the performances of CoMM and InfMasking. CoMM as strong baseline shows performance gains of 7.1% and 1.5 1.5% on Vision& Touch and UR-FUNNY, respectively. Although InfMasking' 's performance gain from adding the third modality is relatively modest compared to CoMM, it still matches CoMM's performance on the Vision& Touch dataset. On the UR-FUNNY dataset, InfMasking achieves the best result (+0.8%).
4.2.3 Experiments with 2 Modalities on Multimodal IMDb
Multimodal IMDD(MM-
Table 4: Linear probing F1-score (weighted and macro) (in %) for MM-IMDB. indicates further training unlabeded data. denotes results are from [12].
Model
Modalities
weighted-fft
macro-flt
SimCLR
4
40.35
27.99
V
51.5
40.8
CLIP
38
L.
51.0
43.0
V+L
58.9
50.9
SLIP4
351
V+L
56.5420.19
47 17.35
CLIP
38
V+L
54.49+0..19
44. 44.9440.30
CoMMcLLP heMMccu
V+L 61.29
53. $3..992..22
InfMaskingoous backhome)
V+L
62.60
55.93
often unreliable and the combination of both modalities can perform better (2). this underscores the need for effective modeling of multimodal interactions. We select both single-modal and multi-modal as baselines. For unimodal, we choose SimCLR (image-only) [ and CLIP (pretrained on image-text pairs) 38 For multimodal, we include CLIP, SLIP [35), and CoMM.
Tab. summarizes the experimental results. Models trained on both modalities consistently outper- formed their single-modality counterparts, further validating the importance of optimizing multimodal representation learning. InfMasking achieves the best overall performance, improving upon CoMM by 1.31 % in weighted F1-score and 2. 14% in macro Fl-score. It is also worth noting that CLIP with

---

its original public weights achieves 58.9% on weighted Fl-score, outperforming CLIP fine-tuned on MM-IMDb (54.59%). This suggests that redundant information learning may not always benefit complex tasks such as genre prediction, which require complementary modality alignment [[2]. These results demonstrate the robustness and generalizability of InfMasking handling imbalanced, semantically heterogeneous, and multi-label multimodal classification tasks.
Ablation Studies
To examine the effectiveness of the design of InfMasking, conduct comprehensive ablation studies on the bimodal Trifeature dataset focusing on three critical components: the loss function formulation, the optimal number of masked views, and the masking ratio parameter.
Table 5: Linear probing accuracy (in %) of redundancy R, uniqueness U and synergy S on Trifcature Dataset under different combinations of loss functions. λ1 A2, and A3 denote the weights for Lmasks E; fis and L, respectively, where & and Σ; Li correspond to the first and second terms in Eq.
loss weights
R
U1
U2
S
Average
A1
X2
11.91
85.912.11
83.8
58.717.11
80.1
0
99.940.00
87.14
86.5+2.60
71.443.47
86.0
0
99.9+0.08
90.7+2.10
91.4+3.03
69. 69.246.20
87.8
99.940.09
90.3+1.52
90.8
77.0
89.5
Loss function. We conducted an ablation study on the Trifeature dataset evaluate different loss combinations for capturing multimodal interactions. As shown in Tab. 5 the full objective (A1 =2 A2 A3 = InfMasking) achieves the highest synergy at 77.0% while maintaining balanced performance across other metrics. Using only CoMM loss (A1 = 0, X2 = 1, A3 = 1) yields 71.4 71.4% synergy, while using only (A1 0, 22 0, A = 1) further decreases to 58.7%, indicating that CoMM loss alone is insufficient without the view diversity from masking. Excluding the L(ie., INCE Z'' )))) drops synergy to 69.2%, despite marginal improvements in redundancy and uniqueness scores. As noted in CoMM [12]. minimizing enables the model to capture all information terms, albeit a slower rate. When this loss is removed, the model learns redundancy and uniqueness more efficiently, achieving higher scores within the same epoch, but the cost of diminished synergy performance.
Number of masked views. According to Section 3.2 increasing the number of masked views yields
closer approximation to Emask [NCE( Zymmsk
Z') and Emask ÍNCU [Z Z")] albeit higher
computational cost. As observed in Fig. 2(a) the synergy score improves progressively with an increasing number of views. Notably, performance sufficiently robust when the number is in the
range of [6, 10), demonstrating practical feasibility for GPU implementation.
78
Ascurcy@@uup155
Accurcy@/lup
75
72
69
2
6
10
40
50
60
70
80
(a) Number of masked views.
(b) Ratio of masking (%).
Figure Synergy accuracy changes with different masked setting on Trifeature datasets.
Masking ratio. Fig. 2(b) illustrates the impact of varying the masking ratio. At lower ratios (S 50%), although the model can capture synergistic information, the overall performance remains

---

unsatisfactory. In contrast, maintaining higher masking ratio enables the model to generate superior multimodal representations that effectively leverage complementary information across modalities. Furthermore, a higher masking ratio can also provide greater speedup benefit [9]
6
Related Work
Multimodal learning. Multimodal learning integrates diverse data sources-suue such as text. image.
audio, and tactile inputs-to enhance information understanding across modalities [3] 36
17 32]
10]. Traditional approaches rely on simple fusion techniques like feature concatenation || or
modality-speciee prediction averaging [14]. Transformer architectures revolutionized this field through dynamic cross-modal attention mechanisms [42] [5]. Contemporary approaches typically follow two-stage framework: training specialized encoders for each modality, then projecting these representations into unified embedding space [4] 12 This paradigm has been applied across
representation learning [5 38 cross-modal alignment 28 and generative modeling [1]...
Self-Supervised multimodal representation learning, Self-supervised learning generates supervi- sion signals from data's 's inherent structure 13 24 48 47). In multimodal contexts, these approaches leverage cross- modal correspondences [53 45). Prior methods have explored generative approaches, such reconstructing one modality from another (39], and masked prediction strategies for joint modality modeling 3 Contrastive learning has emerged as particularly effective for multimodal constructing positive pairs through data augmentation [9] and intro- ducing both intra-modal and cross-modal objectives 33]. Some approaches incorporate regularization
representation learning 1
38
terms align representations across modalities [4
optimize cross-modal contrastive loss but emphasize redundant information while neglecting unique and synergistic information requiring joint consideration, with FactorCL [29] addressing this through explicit modeling despite practical implementation challenges. CoMM [12] advances the field using multimodal augmentations and information theory- grounded losses to capture various interaction
Contrastive Multimodal Interactions. Contrastive multimodal approaches 38
23
patterns, though enhancing synergistic interactions remains challenging.
7 Conclusion
This paper introduces InfMasking, a contrastive method that effectively captures synergistic informa- tion in multimodal representation learning by stochastically occluding features during fusion and aligning representations through mutual information maximization. We derive computationally efficient approximation for infinite masking patterns and demonstrate that our approach not only enhances synergistic information extraction in controlled settings but also achieves state-of-the- the performance across seven diverse multimodal benchmarks. InfMasking has some limitations. lacks a rigorous theoretical framework to systematically analyze the mechanisms of synergistic interactions. Future research will prioritize developing comprehensive theoretical foundations to formally characterize and measure the synergistic information.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. U24A20323), the Major Science and Technology Special Project of the Sichuan Provincial Department of Sci- ence and Technology (Grant No. 2024ZDZX0002), the Sichuan Provincial Innovation Group Project (Grant No. 2024NSFTD0054), Fundamental Research Funds for the Central Universities (JBK20251108) Blockchain Research Center of China, Natural Science Foundation of China under Grant No. 62502397, National Natural Science Foundation of China (NSFC) (72471197), and the Sichuan Provincial Philosophy and Social Science Fund (Grant No. SCJJ25ND091) We thank anonymous reviewers for their insightful feedback and Benoit Dufumier, the first author of CoMM [12], for his contributions to open sourcing the code.
10

---

References
[1] Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for co-speech gesture animation: A multi-speaker conditional-mixtus approach. In Computer EisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 248-265. Springer, 2020.
[2] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multi- modal units for information fusion. In International Conference on Learning Representations Workshops (ICLR-W) W), 2017.
[3] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision (ECCV), pages 348-367, 2022.
[4] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE Transactions on Puttern Analysis and Machine Intelligence, 41(2):423-443, 2018.
[5] George Barnum, Sabera Talukder, and Yisong Yue. On the benefits of early fusion in multimodal representation learning. arXiv preprint arXiv:201 1.07191, 2020.
[6] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161-2183, 2014.
[7] Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. Joint contrastive learning with infinite possibilities. Advances in Neural Information Processing Systems, 33:12638- 2020.
[8] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an obvioosly_ppertiiie perfect paper). In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages
2019.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597-1607, 2020.
[10] Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126 2022.
[11] Sidney K D' mello and Jacqueline Kory. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR), 47(3):133,, 2015.
[12] Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? In International Conference Learning Representations, 2025.
[13] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 39(3):42-6) 2022.
[14] Jerome H Friedman and Bogdan E Popescu. Predictive learning via rule ensembles. 2008
[15] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180-15190, 2023.
[16] GroupLens research. MovieLens dataset, 2015.
[17] Zhaochen Guo, Zhixiang Shen, Xuanting Xie, Liangjian Wen, and Zhao Kang. Disentangling homophily and heterophily multimodal graph clustering. arXiv preprint arXiv: 15253, 2025.
11

---

[18] Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, al. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM,,,) pages 2046-2056, 2019.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 16000-16009, 2022.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[21] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, -aaniStpe Alayrac, and Aida Ne- matzadeh. Decoupling the role of data, attention, and losses multimodal transformers. Trunsuctions of the Association for Computational Linguistics, 9:570-585, 2021.
[22] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 33:9995-10006, 2020.
[23] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuar Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), pages 1904-4916 2021.
[24] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural net- works: A survey, IEEE transactions on pattern analysis and machine intelligence, 43(11):4037- 4058, 2020.
[25] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[26] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neurul information processing systems, 33:2611-2624 2020.
[27] Michelle Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582- 596, 2020.
[28] Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16420 16429, 2022.
[29] Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis Phili-P Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-vies view redundancy. Advances in Neural Information Processing Systems (NeurIPS), 36:32971--3399000 2023.
[30] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. MultiBench: Multiscale benchmarks for multimodal representation learning. In Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, volume 1, 2021.
[31] Paul Pu Liang, Amir Zadeh, and Louis- Philippe Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.0444 2022.
12

---

[32] Che Liu, Yingji Zhang, Dong Zhang, Weijie Zhang, Chenggong Gong, Haohan Li, Yu Lu, Shilin Zhou, Yue Lu, Ziliang Gan, al. Nexus: An amni-perceptive and-interactive model for language, audio, and vision. arXiv preprint arXiv:250300187, 2025.
[33] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICIR), 2019.
[34] Li Mi, Chang Xu, Javiera Dastillo-Navarro, Syrielle Montariol, Wen Yang, Antoine Bosselut, and Devis Tuia. ConGeo; Robust view gco-localization across ground view variations. arXiv preprint arXiv:2403. 13965, 2024.
[35] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets language-image pre-training. In European Conference on Computer Vision (ECCV), pages 529-544, 2022.
[36] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In International Conference on Machine Learning (ICML), pages 689 189-696.22 2011.
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0377 037788, 2018.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), pages 8748-8763, 2021.
[39] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML), pages 8821 1 1-881, 2021.
[40] Karthik Sridharan and Sham M. Kakade. An information theoretic framework for multi-view learning. In Annual Conference on Learning Theory, 2008.
[41] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[42] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference, Association for computational linguistics. Meeting, volume 2019, page 6558, 2019.
[43] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In International Conference on Learning Representutions (ICLR), 2021.
[44] Haoqing Wang, Xun Guo, Zhi- Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient repre- sentation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16041-16050, 2022.
[45] Shuo Wang. Shunyang Huang, Jinghui Yuan, Zhixiang Shen, and Zhao Kang. Cooperation of experts: Fusing heterogeneous information with large margin. arXiv preprint arXiv:2505.20853, 2025.
[46] Liangjian Wen, Xiasi Wang, Jianzhuang Liu, and Zenglin Xu. Mveb: Self-supervised learning with multi-view entropy bottleneck. IEEE Transactions on Pattern Anulysis and Machine Intelligence, 46(9):6097-6108, 2024.
[47] Liangjian Wen, Xiasi Wang, Jianzhuang Liu, and Zenglin Xu. MVEB: self- learning with vilwi-view entropy bottleneck. IEFE Trans. Puttern Anal. Mach. Intell., 46(9)6097--608,, 2024.
13

---

[48] Liangjian Wen, Yiji Zhou, Lirong He, Mingyuan Zhou, and Zenglin Xu. Mutual information gradient estimation for representation learning. International Conference on Learning Representations, ICLR, Addis Ababu, Ethiopia, April 26-30, 2020.
[49] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:
2010.
[50] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(10):12113-1211) 2023.
[51] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faicta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1995-7004, 2021.
[52] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 3 31(6):82-88, 2016.
[53] Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1 1-20, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We mainly claim 'synergistic information constitutes the fundamental value proposition of multimodal representation' and we propose a contrastive synergistic infor- mation method designed to enhance synergistic information through an Infinite strategy. To verify the effectiveness, we tested on real-world datasets across diverse domains (healthcare, robotics) and data types (image, text, audio). InfMasking achieves state-of-the of-the-art results seven multimodal tasks involving two or three modalities.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We claim the need for further application to pertaining and post-training, which will benefit more, which is presented in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theorems, formulas, and proofs in the main text properly numbered and reffeeeefer....d. The key proofs have been included in the appendix G for clarity and completeness.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This study introduces InfMasking, multimodal contrastive interaction method that enhances synergistic Information by utilizing Infinite Masking. InfMasking strategically masks substantial portions of features from each modality during the fusion process, and then maximizes the mutual information between these masked fused features and their unmasked counterparts. Through controlled experiments, we demonstrate that InfMasking effectively enhances synergistic information between modalities. In evaluations on large-scale real-world datasets, InfMasking achieves state-of-the-art performance across seven benchmarks. Detailed analysis and results can be found in Sec. 4 key hyperparameters in Appendix
We also present the
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
16

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: After the anonymity period, we will open-source our training code and training data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guide wwweseooooSSmmssooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctttps.cc/pubbll wwtppppwwpppppepppppppppppaaaaep Stw.:iddde///eSSdmbmeeeeeee
ssioioolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As shown in Appendix A, we provide comprehensive details of both training and evaluation procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For all experiments, report the mean and standard deviation across five independent runs with random seeds in the range [42, 46).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix providees detailed information about the computational resources utilized in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics :tttps//neuurps.. wwtp////wwwttttccccuuuuuueeeeeee
Answer: [Yes]
Justification: This study develops representation learning model (InfMasking) using publicly available datasets (compliant with their respective licenses) and does not involve human participants or personally identifiable information, thus requiring no IRB approval. We have provided detailed experimental descriptions and plan to open-source the training code and data after the anonymity period ensure reproducibility. The paper discusses the method's ss limitations and potential socictal impacts, including bias risks and mitigation
18

---

strategies. However, due to the absence of a formal IRB process in the research location, we conducted an internal peer review address ethical considerations, as suggested by the NeurlPS guidelines. Additionally, local data privacy regulations prevent immediate data sharing during the anonymity period, but we commit to full transparency post-anonymity. After reviewing the NeurlPS Code of Ethics, we confirm that the study complies with most requirements, with the noted exceptions being addressed through alternative measures.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We utilize the pre-processed datasets provided by MultiBench 30 which have been anonymized to safeguard personal privacy. For the MM-IMDb experiments, we employ publicly available open-source pre-trained models that adhere to established security guidelines.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided detailed descriptions and clearly marked the sources and citations for all models and frameworks involved the paper within the experimental section. For open-source code, we have included comprehensive comments and explanations for all imported packages and foundational code used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, wwpprswittceeeooo thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces new assets, including the training code and dataset used for the InfMasking representation learning model. These assets are thoroughly documented in the experimental section, which provides detailed descriptions of the model architecture, training procedures, and dataset characteristics, along with citations to all foundational frameworks and packages used. For the open-source code, we include comprehensive comments explaining the functionality of imported packages and the structure of the founda- tional code. A structured documentation template, detailing training configurations, dataset licensing (compliant with applicable open-source licenses), and model limitations, will be provided alongside the assets.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
20

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: this work, the LLM is utilized solely for writing and editing purposes. They do not influence the core methodology, scientific rigor, or the originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy for what should or should not be described.
21

---

Appendix
A Experimental Details
Training protocol. All experiments are conducted using five independent runs with random seeds in the range (42, 46). We report the mean and standard deviation of performance metrics (i.e., accuracy, mean squared error) to account for variability across runs. Early stopping based on validation accuracy is systematically applied to prevent overfitting. The best-performing checkpoint the validation set selected for final evaluation on the test set.
For dataset-specific encoder architectures, modality-specifie data augmentation and latent converters, we follow the same configurations as CoMM [2]
Training details. We use Adam W 33 as the optimizer in all experiments. Detailed hyperparameters are listed in Tab. 6 Following [12 on MM- MM-IMDb, we also use a cosine scheduler with final value 10-8 and a warmup over 10 epochs. And all models are trained for 100 epochs except for MM-IMDb which is trained for 70 epochs. All experiments are conducted on a single NVIDIA 4090 GPU with 24GB memory.
Table 6: Hyperparameters for InfMasking. Masking ratio is the ratio of masking for each masked view. The V&T CP and V&T EE the contact prediction and end-effector position prediction tasks on Vision& Touch dataset respectively.
dataset
learning rate
masking ratio
number of masked views
Trifeature
3 X 10
0.7
MIMIC
10-4
0.8
MOSI
x 10
0.8
UR- (2-FUUN modalities)
x 10
0.5
MUSTARD
10-3
0.5
V&T CP(2 modalities)
1 X 10-4
0.7
V&T EE
X 10
0.5
MM-IMDb
10 10-3
0.8
UR-FUNNYC modalities)
10-3
0.5
V&T CP(3 modalities)
1 X 10
0.8
Fusion module configuration. For all experiments involving InfMasking, we employ the fusion module similar to that used in CoMM [2], which operates on a sequence of modality-specife embeddings and is implemented as Transformer-basee encoder layer. Specifically. the architecture consists of multi-head self-attention followed by feed-forwer forward network, with residual connections and layer normalization. In the bimodal setting. we use 1-layer Transformer with 8 attention heads, while in the trimodal setting, 2-layer Transformer with the same number of heads is adopted. In addition, learnable [CLS] token is appended the input sequence, which serves global representation aggregating information across modalities.
B
Dataset Details
B.1
Trifeature
The Trifeature dataset (22) is designed to investigate the properties of visual neural networks and comprises three distinct features: shape, color, and texture. Each feature consists of 10 categories, resulting in 1,000 unique combinations. Of these, 800 are used for training and 200 for testing. Each training combination is instantiated three times with random rotations applied to both shape and texture components. Shapes are rendered within a 128 x 128 bounding box, with rotation angles uniformly sampled from |-450, 45%]. and then randomly placed within a 224 x 224 image canvas while ensuring full visibility. Texture and color are independently applied in the same manner. Image pairs are constructed from these instances, resulting 10,000 training pairs and 4, 4,096 test pairs, both sampled from the same underlying distribution.
22

---

B.2 Multibench
According [30]. all datasets below have been pre-pie reeprrcesssed ensure the removal of any personally identifiable information and to safeguard user privacy (some datasets don't include any personal information, e.g. Vision&Touch and MM-IMDb).
MIMIC [25] comprises 53,423 hospital admissions from 38,597 distinct patients, spanning the years 2001 2012. It includes two modalities: hourly clinical measurements over 24-hour period (represented as 12-dimensional vectors, times series modality) and static patient information such as age and gender (represented as 5-dimensional vectors, tabu- lar modalities). The task is a binary classification problem aiming to predict whether patient belongs to (ID--(Interaational Statistical Classification of Diseases and Related Health Problems) code group 7 (460-519), which is commonly used in studies on disease classification (29
MOSI 52 consists of 2,199 video clips collected from YouTube, designed for sentiment analysis tasks. Each sample includes video, audio signals, and corresponding text transcrip- tions. The original annotations range continuously from -3 to 3; following the approach in [29], these labels are binarized into positive and negative classes. The model trained based on textual and visual modalities.
UR-FUNNY 18] constructed from 1,866 TED talk videos and comprises 16,514 samples for the task of humor detection. Each sample contains video, audio, and corresponding text transcriptions. The objective is to determine whether a given sequence is humorous, formulated binary classification problem. For the bi-modal setting, we use the textual and visual modalities.
MUSTARD 8 is designed for sarcasm detection and is sourced from television shows such as Friends. It contains 690 balanced utterances, each comprising video, audio, and text transcriptions, annotated as either sarcastic or non-sarcastic. our experiments, we utilize the textual and visual modalities.
Vision& Touch [27] comprises data from robotic manipulation tasks, consisting of 150 trajectories, each with 1,000 time steps. The dataset includes RGB images, depth maps, force measurements, and end-effector positions and velocities. The benchmark tasks are (1) binary classification predict whether contact will occur in the next step and (2) regression to predict the end-effector position, evaluated using mean squared error (MSE). For the bi-modal setting, the visual and proprioceptive modalities.
B.3 MM-IMDb
Multimodal IMDb (MM-IMDb) is designed for movie genre prediction and comprises 25,959 films, each annotated with posters. plot summaries, genre labels, and metadata. Derived from the Movielens 20M dataset [16), this benchmark focuses on 23-way multi-label classification. In our experiments, we utilize the image modality (movie posters) and the text modality (plot summaries). While MM-IMDD is part of the Multibench benchmark 30,, we present separately our experi- ments, as our model is trained directly on the raw data instead of relying on the pre-processed features offered by Multibench.
C Broader Impact
This study aims to enhance the modelling of cross-modal synergy to generate more informative multimodal representations. To ensure that InfMasking can be deployed responsibly in real-world scenarios, we highlight several key considerations.
Computational complexity. InfMasking introduces infinite masking strategy that maximizes the mutual information between masked fused views and their unmasked counterparts, strengthening the complementarity of different modalities. However, this method inevitably increases GPU memory usage, as each additional masked view amplifies the memory footprint. We encourage future work to explore lightweight variants that can alleviate the associated computational demands.
Privacy and security. As discussed in Section 4.2.1 the datasets used in this study span multiple domains, including healthcare, sentiment analysis and multimedia. According to (30], all instances
23

---

containing personal information have been rigorously anonymized and de- de-identified. And the Vision&Touch and MM-IMDb datasets do not contain any personally identifiable information. All experiments are conducted using irreversible, pre-extracted features, except for MM-IMDb, which is processed directly from raw data; no raw or reconstructable user data is accessed, thereby minimizing privacy risks.
Future work. Future research will focus on establishing rigorous theoretical frameworks to quantity and formally characterize the synergistic information extracted by InIMasking. Such frameworks would provide mathematical guarantees on information preservation while elucidating the funda- mental limits of multimodal representation learning. Additionally, we aim to develop adaptive masking strategies that dynamically optimize masking patterns based on task requirements and modality characteristics, potentially employing reinforeement learning to fine-tune these configurations. These advancements would significantly enhance our capacity to model complex syn- ergistic relationships in multimodal data, advancing the field toward more generalizable multimodal intelligence.
D
Additional Experiments
D.1 Difference with MAE
19 achieve self-supervised learning through reconstruction of masked image patches, which consist of two parts. For the encoder, it encodes randomly masked image patches into latent features. The decoder is trained to predict the masked patches using reconstruction loss, thereby enhancing semantic relationships between them in a single-modal (vision) setting. InfMasking adapts and extends this masking paradigm a contrastive multimodal context. It focuses on aligning and extracting synergistic information from multimodal tokens (e.g., features from text, images, audio, or tabular data) through infinite masking, emphasizing cross-modal interactions like redundancy, uniqueness, and synergy.
Masked (MA))
Table 7: Linear probing accuracy (%) on three datasets from MultiBench 30 for MAE, CoMM, and InfMasking models.
Dataset
MIMIC
UR-FUNNY
MOSI
average
CoMM
66.420.41
63.3
63. 63.742.
64.47
MAE
67.440.3
62.5
65.441.6
65.1
InfMasking
68.1
64.340.9
69.0
67.12
Unlike MAE, our masking approach does not mask the raw input of each modality but rather masks the features of each modality before fusion. Furthermore, we aim to maximize mutual information between masked and unmasked multimodal representations without reconstruction. derives lower bound approximation for the InfMasking loss assuming Gaussian distributions for masked features, making it computationally feasible for infinite views. This makes InfMasking natural evolution for handling diverse modalities, addressing limitations in traditional contrastive learning (e.g., over-reliance on multiview redundancy) while preserving MAE's core idea of using masking to create challenging, informative views.
We further compare InfMasking loss with MAE reconstruction loss multiple datasets from MultiBench [30]. The results are illustrated in Tab. Under the same experimental conditions, we randomly mask tokens across modalities, encoder forward pass with masked iuput, and rrcodderbbsse reconstruction focused solely on masked tokens (using MSE loss averaged over masked positions). This creates generative baseline analogous MAE but extended multimodal tokens. The MAE variant replaces the InfMasking loss component terms in Eq. 2 with reconstruction loss.
D.2 Ablation Studies Data Augmentation on Trifeature Datasets
The InfoMin Principle [1 plays pivotal role in self-supervised learning. It demonstrates that data augmentation is an effective strategy for adhering to this principle, as stronger data augmentations reduce mutual information to an optimal level. In our work, we adopt the same settings for modality- specific data augmentation as outlined in CoMM [12. To extend this investigation, we further explore the influence of data augmentation strategies on the bimodal Trifeature dataset [22].
24

---

Table 8: Impact of data augmentation on linear probing accuracy (%) for multimodal interactions. The term "All" refers to SimCLR [9] augmentations. InfMasking applies "All" augmentations to both modalities, consistent with CoMM.
Augmentations
R
U1
U2
Average
Modality
Modality 2
[AII]
Ø
99.7810.0 10.08
85.28
49.89 18.73
50.0
71.24
Ø
(AII}
99.85+0.06
49.08 65
87.4443.59
50.0+0.0
71.59
[All] Mcrop)
[All]
97.70+0.84
58.07 #2.68
87.15
50.0+0.0
73.23
[All]
[All) Mcrop)
96.91
85.42+4.01
57.8546.63
50.0+0.0
72.54
IniMasking
99.8640.10
90.30
90.80
77.02+4.22
89.5
As shown in Tab. omitting data augmentation leads to significant degradation in model perfor- mance, particularly in uniqueness. Notably, cropping as a critical transformation in self-supervised learning for vision tasks [9 20]. is vital for learning synergistic representations in the Trifeature dataset. When modality-specifff cropping augmentation omitted, the model struggles to capture the uniqueness of the corresponding modality, resulting in inability effectively learn synergy.
E Analysis of Gaussian Approximation Assumption via Visualization
Based on the theoretical framework of InfMasking dis- cussed in Sec. 3.2 further analyze the robustness of Gaussian approximation assumption through visualiza- tion.
Multimodal Peoture
We employ dimensionality reduction to project the high- dimensional embeddings of multimodal features and their masked view features from the Trifeature dataset into a two-dimensional space, which visualized us- ing I-SNE. As shown in Fig. the masked embeddings cluster around central point in the projected space. Notably, this central point aligns closely with the multi- modal features, indicating that despite the perturbations introduced by masking, the masked embeddings inher- ently preserve core aspects of the synergistic semantic nature. It uncovers the stability of synergistic integra- tion: even when parts of modalities are obscured, the fused representations converge toward a shared seman- tic manifold, reflecting the emergent properties that arise from modal complementarity rather than redun- dancy alone. Furthermore, the dispersion of masked embeddings similarly indicates semantic differentiation within the ambient embedding space. The observed variance highlights subtle nuances in how masking af- fects the differences of representations combined from masked view features, potentially corresponding to variations in semantic granularity such as implicit biases or contextual implications that become apparent only through joint modal analysis.
Figure 3: Visualization of the distribution of multimodal fusion embeddings and its masked counterpars.
F Pseudo-Code
Algorithm Notlines the training procedure of InfMasking, formulated in the general case with modalities (e.g., image, text, audio, etc).
The key input components are as follows: T* denotes a set of pabee-eres transformations used for data augmentation. The fusion transformer y integrates latent features from diverse modalities. The masked view number M' indicates how many masked instances are generated per modality. The random masking operator M stochastically obscures portions the embedding features. And the temperature parameter 7 controls the sharpness of the total loss.
25

---

Algorithm Multimodal contrastive interaction learning with InfMasking
1: Input:
Multi-modal dataset (X1, X2: Xn), label-preserving transformations T*, set of projection transformations P {pI,....,'' PP J. batch size N, masked view number M', uni-modal encoders (J)) fusion transformer g. random mask operator M, temperature parameter
2: for sampled mini-batch {xx [xEEIE[(..N)] (x),... x [EEI.[. do
3: for k E [1..N] do T 5: x' x", + (((xx),((xx,)
4: draw E,,
6: for j E [1.MM'] do
z mack + (...(
7:
8:
"j 2 mask - 9 (M(Ji
9:
end for z'x < (xf((x')) k): [n(x)))
10:
11:
z"x +- g(fl(x('' k)..... ..... fn (xxx))
k
for i [L..]] do
13:
x px( z - (x((()))
14:
end for
16:
end for
expl= tk T (2///T)
LiniMasking
M
ma of par parpp i z)) 21)
log exp(x" musk 4ya (/a/t)-2i exp(= teee
exp(=" mark
7 ("a/T)
18:
+I mark spo mpl 2 mark 2")
log exp(=" "pp/ (//tt)2inn explx" #meg/>)
19:
for i E [1...] do
expl=? s/maT))
20:
C. +
E Ben x 22.22))
log exp(= (///t)++ Eion exp( ZnggT
exp(= /T) log exp(z! 7/oal /// +++
21:
+E +E ~p(2..z")
Em exp( -
22:
end for
exp(=" s//// E. Be p'''p 2") log expl=" spal / En exp(=" _
23:
L
LTotal loss + Σi= Li LintMasking
25:
update (fr)eEj..m) to minimize Lrotal loss end for 27: return (fo)ke(1..n)..)
26

---

G Proof
Proof (lemma
Emask INCE( (maak:+ Z')]
(8)
expl= >/mask/T) TT
-Emaok [Ex'
log
(9)
exp(22 (xpp("''> r m T) + E:. exp(2 T #neg/
exp(2" smmot//)
= =Eg(np(21) Bmosk
log exp(=' maakk / T) + Exin exp(z T)
(10)
= =E Emest
(z 2y (/mas//T) - logloxp(=" " //wrr// /T) + Σ exple /nng/T]]]
(11)
2 Ext=o(Z))
S (mmas) log Emassk [cxp(2 ST / r)+
(T)
Σ exp(2" mnnggTT]]
(12)
The inequality Eq. (12) merges from the application of Jensen inequality on concave functions i.e., Eg (Xg log E,[[X] Zimax denotes the integrated representation derived from the fusion of all masked features across diverse modalities via the Transformer architecture.
Lemma 4 Consider random variable that follows multivariate Gaussian distribution, denoted as x ~ N(,, Σ), where E R represents the mean vector and E Rnx* is the covariance matrix The moment generating function (MGF) of this random variable is given by the following expression:
Ex
aa
=aa" u+ 3+t
(13)
where a E IR is arbitrary constant vector.
ST 2 According to Lemma we can derive the MGF of the inequality Eq. as follows:
Emask INCEe (Z Z')]
(14)
2 m. Ex(np((')
/T Me.
log log[exp(2" "T u. mmo 272
/T 2 TT
mmat Eg 2) + Σ exp(x neg [T]
(15)
27


Reviews:
[{"review_id": "CvtQuB0Fsh", "reviewer": "Reviewer_S2Su", "summary": "This paper introduces InfMasking, a novel method to capture synergistic information in multimodal learning, which traditional methods often miss. By contrasting heavily masked representations with their complete versions, InfMasking forces the model to learn deep, complementary relationships between modalities. The approach achieves state-of-the-art results on seven diverse benchmarks, demonstrating its effectiveness in modeling complex multimodal interactions.", "questions": "1. How does InfMasking's contrastive objective fundamentally differ from MAE's reconstructive approach in learning synergistic multimodal information?\n2. Can you provide clearer guidance on selecting optimal masking hyperparameters for new tasks based on their inherent modality characteristics?\n3. Beyond indirect performance metrics, can you propose a more direct method to quantify or visualize the synergistic information captured by your model?\n4. How does the computational cost of InfMasking scale with an increasing number of modalities, and is there a point of diminishing returns?\n5. How robust is InfMasking to extreme scenarios where one modality is completely noisy or irrelevant to the downstream task?\n\nI will consider raising the score if the authors can address my concerns.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper pinpoints a critical yet underexplored challenge in multimodal learning—capturing synergistic information—moving beyond the conventional focus on redundancy.\n2. The paper provides compelling evidence through a combination of controlled synthetic experiments and state-of-the-art results across seven diverse real-world datasets, demonstrating both effectiveness and generalizability.\n3. InfMasking consistently outperforms previous leading methods, validating its practical value and establishing a new benchmark for multimodal interaction modeling.\n\nWeaknesses:\n1. The model's performance is sensitive to newly introduced hyperparameters, namely the masking ratio and the number of masked views. This adds complexity to the model tuning process and may require careful adjustment for optimal performance on new tasks.\n2. While the method is designed to enhance synergy, the paper lacks a rigorous theoretical framework to formally measure or guarantee the capture of synergistic information, relying primarily on empirical performance as evidence.\n3. Synergistic information in multimodal learning is both important and challenging. However, this paper resembles a multimodal version of the well-known MAE [1]. While MAE enhances the relevant information between patches, this paper focuses on multimodal tokens.\n\n[1] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.", "was_revised": true, "final_justification": "I understand the differences between this paper and MAE, but the advantages are primarily based on experience rather than a solid analytical foundation. Therefore, I may keep my score but remain open to accepting this paper.", "rebuttal": "**Response W.1 and Q.2**: Small multimodal datasets like MOIS are highly sensitive to hyperparameters, such as the seed, resulting in significant performance variance. However, larger datasets like IMDB and V&T CP exhibit less sensitivity to hyperparameters like the seed. As shown in Table 6, a masking ratio of 0.7 or 0.8 typically performs well, with values between 4 and 6 being viable.\n\n**Response W.2 and Q.3**: Measuring multimodal interactions as defined by PID is challenging since it involves estimating information-theoretic measures. The problem is even harder for high-dimensional data distributions. Currently, there is no well-established metric for measurement. We referenced the ICLR paper Comm[1] and conducted synthetic experiments on trifeature datasets in Section 4.1 to validate that our method more effectively learns synergistic information.\n\n**Response W.3 and Q.1**:\n\n(1) Although MAE's reconstructive approach can also learn collaborative information, downstream task prediction considers all tokens, leading to a gap between pre-training and downstream tasks. InfMasking's contrastive objective aligns unmasked fused representations with masked ones via mutual information maximization to encode synergistic information.\n\n(2) MAE's reconstructive approach typically converges slowly, with high pre-training costs. We adopt the infinite masking strategy and derive an InfMasking loss to approximate the calculation of this loss function.\n\n(3) Our masking approach does not mask the raw input of each modality but rather masks the features of each modality before fusion.\n\nFor the purpose of comparison with MAE, under the same experimental conditions, we conducted experiments using the loss function from MAE on multiple 2 modalities datasets. The average results across five seeds (42–46) are as follows:\n\n|    Dataset     |     UR-FUNNY     |      MOSI       |\n| :------------: | :--------------: | :-------------: |\n|      MAE       | $62.5_{\\pm1.43}$ | $65.4_{\\pm1.6}$ |\n|      CoMM      | $63.3_{\\pm0.51}$ | $63.7_{\\pm2.5}$ |\n| **Infmasking** | $64.3_{\\pm0.9}$  | $69.0_{\\pm1.2}$ |\n\n**Response Q.4**: As illustrated in Figure 1, our masking approach does not mask the raw input of each modality, but rather masks the features of each modality before fusion. The repeated sampling process incurs negligible computational cost. The only computational overhead comes from fusing the masked views, where the fusion network typically consists of a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we set the masking ratio to 0.8, meaning only 20% of tokens remain unmasked. With 5 and 4 masked views respectively, the computational increase is minimal, and each sampling can be processed in parallel during fusion.\n\n**Response Q.5**: In the 3 modalities experiments on the UR-FUNNY dataset, we conducted a zeroing operation on each individual modality (i.e., vision, text, and audio) respectively, and conducted experiments on both the CoMM and InfMasking models. The average results across five seeds (42–46) are as follows: \n\n| Modality       | 3 modalities <br>with noise | 3 modalities <br>without noise |      |\n| -------------- | :-------------------------: | :----------------------------: | :--: |\n| CoMM           |           $60.6$            |             $64.8$             |      |\n| **Infmasking** |           $61.2$            |             $65.6$             |      |\n\n[1] Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align  in multimodal contrastive learning? In International Conference on Learning Representations,  2025."}, {"review_id": "li19cTJTXQ", "reviewer": "Reviewer_Ht5g", "summary": "The paper studied how to enhance synergistic information in multimodal learning. Synergistic interactions occur when modalities combine to produce information unavailable by any single modality alone. The paper introduces InfMasking, a contrastive learning method designed to enhance synergy. InfMasking employs an infinite masking strategy: during fusion, it stochastically occludes most features from each modality, preserving partial information to create diverse synergistic patterns. Masked fused representations are aligned with unmasked ones via mutual information maximization. To mitigate the computational cost of infinite masking, the authors derive a tractable lower bound to the original objective. Experiments on synthetic and real-world datasets show SOTA results on seven benchmarks, with gains in synergy and downstream tasks.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ The infinite masking strategy is theoretically grounded.\n+ Extensive experiments across 8 benchmarks with diverse modalities demonstrate broad applicability.\n\nWeaknesses:\n- Synergy is vaguely described as \"complementary information emerged only when … combined\" without formal measures beyond PID. No real-world examples illustrate tasks where synergy is critical and redundancy/uniqueness is insufficient. One example that comes to my mind is hateful meme detection. A neutral image combined with neutral text generates harmful content that neither modality conveys alone. Including such real-world examples would strengthen the motivation.\n- In section 3.2, while the derived lower bound makes the loss calculation more tractable, the training still involves repeatedly sampling and fusing masked views. This approach may face scalability challenges with three or more modalities, where the combinatorial complexity increases substantially.\n- In Table 5, it’s interesting that setting $\\lambda_{3}$ to zero increases redundancy and uniqueness while reducing synergy. A brief discussion of why this behavior occurs might offer additional insight.\n- The evaluation lacks comparisons masking baselines (e.g., MultiMAE [1]) or methods explicitly modeling interactions (e.g., attention bottlenecks [2]).\n- Typos: Section 3.2 title: “Ininite” → “Infinite” \n\n[1] Bachmann, Roman, et al. \"Multimae: Multi-modal multi-task masked autoencoders.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Nagrani, Arsha, et al. \"Attention bottlenecks for multimodal fusion.\" Advances in neural information processing systems 34 (2021): 14200-14213.", "was_revised": true, "final_justification": "Thanks to the authors for their detailed rebuttal. My concerns regarding the real-world examples, scalability, and comparisons masking baselines have been addressed. Accordingly, I have raised my rating.", "rebuttal": "**W.1** Synergy is vaguely described as \"complementary information emerged only when … combined\" without formal measures beyond PID. No real-world examples illustrate tasks where synergy is critical and redundancy/uniqueness is insufficient. One example that comes to my mind is hateful meme detection. A neutral image combined with neutral text generates harmful content that neither modality conveys alone. Including such real-world examples would strengthen the motivation.\n\n**Response:**  Thank you for this valuable suggestion. We agree that concrete real-world examples would significantly strengthen our motivation for studying synergy. In the revision, we will add the hateful meme detection example you mentioned in the introduction, as it perfectly illustrates how synergy emerges when seemingly neutral modalities combine to create harmful content that neither conveys alone.\n\nAdditionally, we will include the following content to provide empirical evidence for the importance:\n\n\"Liang et al. [1] conducted a comprehensive human evaluation study across 30 multimodal datasets to determine the dominant information-theoretic interactions (Table 1). While their findings show that redundancy predominates in many VQA tasks, they identified 8 datasets—including IRFL, MM-IMDb, and MAGIC BRUSH—where synergy is the dominant interaction. This empirical evidence demonstrates that synergy is not merely a theoretical concept but a measurable phenomenon critical for understanding multimodal learning in real applications.\n\n**W.2**  In section 3.2, while the derived lower bound makes the loss calculation more tractable, the training still involves repeatedly sampling and fusing masked views. This approach may face scalability challenges with three or more modalities, where the combinatorial complexity increases substantially.\n\n**Response:**  As illustrated in Figure 1, our masking approach does not mask the raw input of each modality, but rather masks the features of each modality before fusion. The repeated sampling process incurs negligible computational cost. The only computational overhead comes from fusing the masked views, where the fusion network typically consists of a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we set the masking ratio to 0.8, meaning only 20% of tokens remain unmasked. With 5 and 4 masked views, respectively, the computational increase is minimal, and each sampling can be processed in parallel during fusion.\n\nFurthermore, our method serves as an unsupervised pre-training approach. During downstream task inference and fine-tuning, it introduces no additional computational overhead compared to standard methods.\n\n**W.3**  In Table 5, it’s interesting that setting to zero increases redundancy and uniqueness while reducing synergy. A brief discussion of why this behavior occurs might offer additional insight.\n\n**Response:**  $R$ represents redundancy, $U_i$ denotes uniqueness for view $i$, and $S$ indicates synergy.\n\nSetting $\\lambda_3 = 0$ eliminates the term $\\hat{I}_{\\text{NCE}}(Z', Z'')$ \n\nfrom the loss function. This results in an increase in $U$ but a reduction in synergy $S$ compared to including this term. Theoretically, this term should enhance redundancy, uniqueness, and synergy simultaneously. However, in practice, uniqueness decreases when this term is included. This behavior likely occurs because the $\\hat{I}_{\\text{NCE}}(Z', Z'')$ term prioritizes learning redundancy and synergy across views, which can constrain the model's ability to capture view-specific information (uniqueness). When this term is removed, the model focuses more on view-specific features, boosting uniqueness at the expense of synergy.\n\n**W.4** The evaluation lacks comparisons masking baselines (e.g., MultiMAE [1]) or methods explicitly modeling interactions (e.g., attention bottlenecks [2]).\n\n**Response:** \n\n(1) For the purpose of comparison with MultiMAE, under the same experimental conditions, we conducted experiments using the loss function from MultiMAE on multiple 2-modal datasets. The average results across five seeds (42–46) are as follows:\n\n|    Dataset     |     UR-FUNNY     |      MOSI       |\n| :------------: | :--------------: | :-------------: |\n|    MultiMAE    | $62.5_{\\pm1.43}$ | $65.4_{\\pm1.6}$ |\n|      CoMM      | $63.3_{\\pm0.51}$ | $63.7_{\\pm2.5}$ |\n| **Infmasking** | $64.3_{\\pm0.9}$  | $69.0_{\\pm1.2}$ |\n\n(2) Attention bottlenecks is a multimodal fusion network that can be directly applied to CoMM and our method as fusion networks. For a fairer comparison with its baseline, we only use a simple Transformer for fusion.\n\n\n[1] Liang, P. P., Goindani, A., Chafekar, T., Mathur, L., Yu, H., Salakhutdinov, R., & Morency, L. P. (2024). Hemm: Holistic evaluation of multimodal foundation models,  NeurIPS 2024"}, {"review_id": "XDQMAklaeH", "reviewer": "Reviewer_P89E", "summary": "**InfMasking** addresses a critical gap in multimodal learning: existing methods primarily capture redundant information across modalities while neglecting synergistic information that emerges only when modalities are combined. The core innovation is an Infinite Masking strategy that randomly occludes features from each modality during fusion, creating diverse synergistic patterns aligned through mutual information maximization. A computationally efficient approximation makes this approach practical. Experiments show significant improvements in synergy capture (77.0% vs 71.4% on synthetic data) and state-of-the-art performance across seven real-world benchmarks, demonstrating that explicitly modeling complementary cross-modal interactions is crucial for effective multimodal representation learning.", "questions": "1. How robust is the Gaussian approximation assumption in practice? \n\nThe method relies on assuming that masked representations $z'\\_{mask}$ follow a Gaussian distribution to derive the computationally tractable lower bound. However, in real multimodal scenarios, the distribution of fused features after masking could be highly complex, multimodal, or skewed. What happens when this assumption breaks down? Have the authors tested the sensitivity of their approximation quality across different data distributions, and how does deviation from Gaussianity affect the theoretical guarantees and practical performance?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The random masking approach is both novel and intuitive. By stochastically occluding features from different modalities during fusion, the model naturally learns to discover synergistic information that emerges only from cross-modal interactions.\n2. The paper systematically decomposes multimodal interactions into redundancy (R), uniqueness (U), and synergy (S). Grounded in Partial Information Decomposition (PID) theory and mutual information maximization, it provides a rigorous theoretical foundation for addressing synergistic information capture.\n3. The evaluation combines controlled synthetic experiments (Trifeature dataset) to explicitly measure interaction types with extensive real-world evaluations across seven diverse benchmarks. This dual approach provides convincing evidence of effectiveness and generalizability.\n4. The transformation of theoretically infinite masking into a computationally tractable Gaussian approximation elegantly bridges theoretical optimality with practical feasibility, making the approach viable for real-world applications.\n\n**Weaknesses**\n1. The method requires k masking operations which might resulting in higher computational cost than traditional approaches. This might not suitable for large-scale datasets. \n2. This random masking method has shortcomings. It is difficult to take into account the fact that different data sets have different characteristics.\n3. There is a problem of hyperparameter sensitivity. Different data sets may require completely different parameter settings. There is also a lack of automatic parameter adjustment mechanism.\n4. The method primarily benefits tasks where synergistic information is critical. For scenarios where redundant information is more valuable, InfMasking may provide limited improvements, lacking adaptive mechanisms to adjust to different task characteristics and information interaction patterns.", "was_revised": true, "final_justification": "My questions have been answered and I will keep my rating.", "rebuttal": "**W.1** The method requires k masking operations which might resulting in higher computational cost than traditional approaches. This might not suitable for large-scale datasets.\n\n**Response W.1:**  As shown in Figure 1, our masking approach targets the features of each modality before fusion, rather than the raw input, minimizing computational overhead. The repeated sampling process adds negligible cost. The primary computational increase stems from fusing masked views, using a lightweight fusion network with a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we apply a masking ratio of 0.8, retaining 20% of tokens unmasked. With 5 and 4 masked views, respectively, the computational impact remains minimal, as sampling can be parallelized during fusion.\n\nFurthermore, our method serves as an unsupervised pre-training approach. During downstream task inference and fine-tuning, it introduces no additional computational overhead compared to standard methods.\n\n**W.2** This random masking method has shortcomings. It is difficult to take into account the fact that different data sets have different characteristics.\n\n**Response W.2:**  The random masking method has limitations, as it struggles to adapt to the unique characteristics of different data sets. In contrast, our infinite mask strategy overcomes this by leveraging the Infmasking loss to estimate mean and variance, effectively capturing the distinct properties of various data sets without requiring specific adjustments for each.\n\n**W.3** There is a problem of hyperparameter sensitivity. Different data sets may require completely different parameter settings. There is also a lack of an automatic parameter adjustment mechanism.\n\n**Response W.3:**  Small multimodal datasets like MOSI are highly sensitive to hyperparameters, such as the seed, resulting in significant performance variance. However, larger datasets like IMDB and V&T CP exhibit less sensitivity to hyperparameters like the seed. As shown in Table 6, a masking ratio of 0.7 or 0.8 typically performs well, with values between 4 and 6 being viable.\n\n**W.4**  The method primarily benefits tasks where synergistic information is critical. For scenarios where redundant information is more valuable, InfMasking may provide limited improvements, lacking adaptive mechanisms to adjust to different task characteristics and information interaction patterns.\n\n**Response W.4**: Liang et al. [1] conducted an extensive human evaluation study across 30 multimodal datasets to identify the dominant information-theoretic interactions (Table 1). Their results indicate that redundancy is prevalent in many VQA tasks, but they identified 8 datasets—including IRFL, MM-IMDb, and MAGIC BRUSH—where synergistic interactions dominate.\n\nOur method is a general-purpose pretraining approach. Currently, determining dataset-specific interaction patterns relies on human evaluation. As shown in Table 2, in datasets like MIMIC [1] where redundancy predominates, InfMasking yields limited improvements but does not hinder model performance. For known tasks, we can design a mixture-of-experts approach to adaptively handle varying information interaction types.  This method can serve as our future research work.\n\n**Q.1** How robust is the Gaussian approximation assumption in practice?\n\nThe method relies on assuming that masked representations follow a Gaussian distribution to derive the computationally tractable lower bound. However, in real multimodal scenarios, the distribution of fused features after masking could be highly complex, multimodal, or skewed. What happens when this assumption breaks down? Have the authors tested the sensitivity of their approximation quality across different data distributions, and how does deviation from Gaussianity affect the theoretical guarantees and practical performance?\n\n**Response Q.1**: This assumption is well-founded for two principal reasons. First, the masked embeddings tend to cluster around a central value in the embedding space, as they all inherently reflect aspects of the synergist’s semantic nature. Second, the variance observed across feature dimensions can be interpreted as a representation of semantic differentiation in the ambient space. In the revised version, we have included visualizations to support this argument.\n\nIn practice, the Gaussian assumption holds reasonably well for high-dimensional representations after feature fusion (e.g., via a Transformer), as the aggregated features from multiple modalities tend to exhibit central limit theorem-like behavior due to the averaging effects of fusion. Our experiments on diverse real-world datasets, such as those in MultiBench (Section 4.2), spanning domains like healthcare, robotics, and multimedia (e.g., MIMIC, MOSI), demonstrate that InfMasking achieves state-of-the-art performance across seven benchmarks. This suggests that the Gaussian approximation is sufficiently robust for a wide range of practical multimodal scenarios, even when the underlying data distributions may deviate from strict Gaussianity."}, {"review_id": "WxqAshTHvm", "reviewer": "Reviewer_Z2T2", "summary": "This article proposes a novel multimodal contrastive learning method named InfMasking, aiming to enhance the synergistic information in multimodal representations through an \"infinite masking strategy\". The main idea of InfMasking is to randomly mask most features during the multimodal fusion process and retain only part of the information, and maximizing the mutual information between the masked and unmasked representations through an innovative InfMasking loss function. In this way, the model generates diverse collaborative representations.", "questions": "1. I noticed that in the ablation experiment(Figure 2), more masked views are not necessarily better. Can author explain the cause of this phenomenon?\n\n2. Could the author provide some synergistic information metrics in actual datasets under different number of masked view Settings?\n\n3. I don't quite understand the role of data augmentation in the entire method. The author set the number of data augmentation methods to 2 perhaps for the convenience of comparison with CoMM. However, I am still curious about the impact of using different numbers(0,1, or more) of augmentation methods.", "limitations": "1. The ablation experiment results in Figure2 indicate that more masked views are not necessarily better, which does not conform to the theoretical derivation in the paper.\n2. The paper claims that the method can \"enhance synergistic information across modalities by leveraging infinite masked views\". However, this claim lacks effective verification on experimental data.\n3. I noticed that the number of data augmentation methods in the paper was set to 2, yet the author did not provide a relatively objective explanation of the methods of data augmentation and the number of methods applied each time.\n\ntypo:\n\nsection3.2 title \"Contrastive Synergistic Information via **Ininite** Masking\" -> \"Contrastive Synergistic Information via **Infinite** Masking\"\n\nPlease refer to **Questions**.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The overall quality is good. The article provides theoretical proof of the method and verification through specific experiments.\n\nThe article is of average significance.\n\nThe method in the paper involves many stages, the clarity of expression can be further improved.\n\nThe article has a certain degree of originality.", "was_revised": true, "final_justification": "Thank you to the authors for the response; most of my concerns have been addressed. I have accordingly raise the score.\nHowever, for Q3&L3, my real concern is: **can additional data augmentation techniques further improve performance?** As an exploration and validation of the method's scalability and performance upper bound, I would greatly appreciate it if the authors could supplement ablation experiments in this regard.", "rebuttal": "**Response Q.1 and L.1**: The phenomenon observed in the ablation study (Figure 2(a)), where more masked views do not necessarily lead to better performance, can be attributed to the balance between capturing synergistic information and avoiding representational collapse. As described in the paper (Page 9, Section 5), the InfMasking approach relies on randomly masking a significant portion of features from each modality during fusion to create diverse synergistic patterns. However, increasing the number of masked views excessively may lead to insufficient information retention, causing the model to struggle with learning robust representations.\n\nSpecifically, the paper notes that the optimal performance on the MIMIC and MOSI datasets is achieved at a masking ratio of 0.5 or 0.6 (Page 9, Figure 2(b)). Beyond this range, performance declines because excessive masking can result in representational collapse, where the model fails to retain enough complementary information across modalities to form generalizable representations. With too many masked views, the model may overly rely on limited subsets of features, reducing the diversity of synergistic interactions and hindering the alignment of masked and unmasked representations through mutual information maximization.\n\nThus, the key is to maintain a controlled number of masked views to ensure sufficient information is preserved while still encouraging the model to infer complementary, synergistic information across modalities. This balance is critical for enhancing the robustness and generalizability of the learned representations, as discussed in Section 3.2 (Page 5).\n\nWe can draw from [1] and add uniform loss to solve this representational collapse. In the revision version, we further analyze this issue and conduct experiments for comparison.\n\n**Response Q.2 and L.2**: Measuring multimodal interactions as defined by PID is challenging since it involves estimating information-theoretic measures. The problem is even harder for high-dimensional data distributions. Currently, there is no well-established metric for measurement. We referenced the ICLR paper Comm[2] and conducted synthetic experiments on trifeature datasets in Section 4.1 to validate that our method more effectively learns synergistic information.\n\n**Response Q.3 and L.3**: Data augmentation plays a critical role in our unsupervised pre-training method, as outlined in Section 2. For Contrastive Multimodal Interactions (CoMM), data augmentation is essential to create two distinct yet mutually supervised multimodal views, enabling effective contrastive learning. The authors chose two augmentation methods, likely to align with CoMM for fair comparison.\n\n[1] Zhang, Q., Wang, Y., and Wang, Y. How mask matters: Towards theoretical understandings of masked autoencoders.Advances in Neural Information Processing Systems, 35: 27127–27139, 2022b. 1, 3, 4, 7, 8\n\n[2] Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? In International Conference on Learning Representations,  2025."}]

Author rebuttal:

Meta-review: The paper faces some concerns pre-rebuttal, namely, computational overhead, the masking strategies (as in similarity to MAE, infmasking vs random masking) and synergy vs representation collapse. Reviewers seem well appeased by the rebuttal on these concerns.

There was a concern that the paper is primarily based on empirical results rather than a solid analytical foundation. While the AC understood the reviewer's concern about this, it is nevertheless a non-trivial effort to discover such an observation.

Further, AC appreciates that by stochastically occluding features from different modalities during fusion, the model naturally learns to discover synergistic information that emerges only from cross-modal interactions. AC thinks this idea is quite novel and decides to accept the paper.
Decision: Accept (spotlight)