Self-Weighted Contrastive Learning among Multiple Views for Mitigating Representation Degeneration
Authors: Jie Xu
Shuo Chen
Yazhou Ren
Xiaoshuang Shi
Heng Tao Shen
Gang Niu
Xiaofeng Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-view learning
Contrastive learning
Representation degeneration
Self-supervised learning
TL;DR: This paper proposes a simple but effective framework of self-weighted multi-view contrastive learning to mitigate representation degeneration in multi-view contrastive learning.

Abstract: Recently, numerous studies have demonstrated the effectiveness of contrastive learning (CL), which learns feature representations by pulling in positive samples while pushing away negative samples. Many successes of CL lie in that there exists semantic consistency between data augmentations of the same instance. In multi-view scenarios, however, CL might cause representation degeneration when the collected multiple views inherently have inconsistent semantic information or their representations subsequently do not capture sufficient discriminative information. To address this issue, we propose a novel framework called SEM: SElf-weighted Multi-view contrastive learning with reconstruction regularization. Specifically, SEM is a general framework where we propose to first measure the discrepancy between pairwise representations and then minimize the corresponding self-weighted contrastive loss, and thus making SEM adaptively strengthen the useful pairwise views and also weaken the unreliable pairwise views. Meanwhile, we impose a self-supervised reconstruction term to regularize the hidden features of encoders, to assist CL in accessing sufficient discriminative information of data. Experiments on public multi-view datasets verified that SEM can mitigate representation degeneration in existing CL methods and help them achieve significant performance improvements. Ablation studies also demonstrated the effectiveness of SEM with different options of weighting strategies and reconstruction terms.

Self- Weighted Contrastive Learning among Multiple
Views for Mitigating Representation Degeneration
Jie Xu Shuo Chen Yazhou Ren' Xiaoshuang Shi!
Gang Niu Xiaofeng Zhu University of Electronic Science and Technology of China, China RIKEN Center for Advanced Intelligence Project, Japan
Heng Tao Shen'
Abstract
Recently, numerous studies have demonstrated the effectiveness of contrastive learning (CL), which learns feature representations by pulling in positive samples while pushing away negative samples. Many successes of CL lie in that there exists semantic consistency between data augmentations of the same instance. In multi- view scenarios, however, CL might cause representation degeneration when the collected multiple views inherently have inconsistent semantic information or their representations subsequently do not capture sufficient discriminative information. To address this issue, propose a novel framework called SEM: SElf weff-weighted Multi-view contrastive learning with reconstruction regularization. Specifically, SEM is a general framework where we propose to first measure the discrepancy be- tween pairwise representations and then minimize the corresponding self-weighted contrastive loss, and thus making SEM adaptively strengthen the useful pairwise views and also weaken the unreliable pairwise views. Mcanwhile, we impose a self-supervised reconstruction term to regularize the hidden features of encoders, to assist CL. in accessing sufficient discriminative information of data. Experiments on public multi-view datasets verified that SEM can mitigate representation de- generation in existing CL methods and help them achieve significant performance improvements. Ablation studies also demonstrated the effectiveness SEM with different options of weighting strategies and reconstruction terms.
1 Introduction
Contrastive learning (CL. explicitly enlarges the feature representation similarity between semantic- relevant samples, and it is adept at capturing high-level semantics while discarding irrelevant infor- mation. This learning paradigm has facilitated many research and application fields, such visual representation [1. 2], text understanding [3, 4], and cross cross-modal agreement [5, 6, 7J. Samples with consistent semantics are typically constructed as positive sample pairs for CL loss (e.g., InfoNCE [8]), which motivates multi-view learning scenarios [9, 10] where researchers focus on exploring common semantics among multi-vier data. However, this kind of data usually is with heterogeneous views and thus cannot be directly processed by previous CL methods with two shared network branches.
To handle this situation, many multi-view contrastive learning (MCL) methods [11, 12, 13, 14] have been proposed, which treats multiple views as positive sample pairs and achieves important progresses in exploring multi-view common semantics (see Sec. 2 for details). Nevertheless, we find that CL might cause representation degeneration that the representations of high-quality views tend to degenerate. This may make the MCL methods perform worse than the optimal single view (see Sec. 3.1 and Sec. 4),,, and thus heavily limiting the usability of MCL practical scenarios. Although
"Corresponding Author (seanzhuxf@ gmail.com). Code link: http://ghhaa..comsssiimiisssss/s
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

several CL work [15, 16] proposed different CL losses aiming at increasing robustness to noise and made important advances on vision and graph data, our experiments discover that these CL losses are still fragile in multi-view scenarios as multi-view data are with more diversity than single-view data. Different from changing CL loss, recent MCL. methods [14, 17] focused on changing model structures and successfully improved the effectiveness of clustering the learned representations. Nevertheless, representation degeneration still exists in many cases and it requires further solutions.
We find that there could be two reasons leading to representation degeneration in MCL. 1) The quality difference among multiple views. The success of CL based on the priori condition that the constructed positive sample pair has semantic consistency, which generally holds in previous CL applications [1, 5, 8]. Unfortunately, for multi-view learning, the collected views usually have quality difference and the semantic of positive sample pairs might be inconsistent due to view diversity. Consequently, CL. causes the representation degeneration of high-quality views due to the existence of low-quality views. III Losing discriminative information during data processing. Multi-view data typically involve heterogeneous data forms [9. 18], e.g., different dimensions, modalities, and sparsity. For achieving MCL, the model needs to transform heterogeneous multi-view data into the same form with different encoders. However, data transformation could lose discriminative information as this process has no supervised signals for maintaining information. As result, CL might miss multiple views' common semantics and focus on cemantic-irrelevan information due to inductive bias
To this end, we propose SElf-weighted Multi-view con- trastive learning with reconstruction regularization (SEM) as shown in Figure 1 that takes the m, n, o-th views in V views as an example (where Wm,, denotes the pairwise weight, LCL is the contrastive loss, and zm is the learned representations). Specifically, SEM minimizes self-weighted contrastive losses Wmmh Lej. and Wr. Lei, after measuring the discrepancy be- tween pairwise views' representations, i.e. (Zmh Z²) and (Z" " .Z",, respectively. This makes SEM adaptively strengthen CL between the useful pairwise views and also weaken CL between the unreliable pairwise views. Meanwhile, SEM takes self- supervisdd reconstruction objectives as regularization terms (Rm, R", and Rº) on the hidden features (Hm H and HD) of encoders for individual views, respectively. This reconstruction regularization assists CL in accessing sufficient dis- criminative information hidden in raw input data (X"., X", and Xº), which could be implemented by exist- ing information encoder- decoder models, e.g., AE [19], DAE [20], and MAE [21]. In SEM, the representations and pairwise weights are alternatively updated mutually enhance one another.
m.n
strengshon
7
w
2
W
2"
encoder
encoder
encoder
32
encoder
encoder
encoder
x
X
X"
aseetll patrwise views
anrellable pairwire siews
The framework of SEM. It leverages different networks to extract information of dif- ferent views and conducts the proposed self- weighted multi-view contrastive learning with reconstruction regularization.
Figure 1:
In summary, our contributions are: I) We propose a novel general framework SEM that leverages seff-weighting and information reconstruction address representation degeneration in MCL. II) We provide three options with different advantages to implement the weighting strategy of SEM including class mutual information, JS divergence, and maximum mean discrepancy. IIII Theoretical and experimental analysis verified the effectiveness of SEM. It helps many CL methods (e.g., InfoNCE [8], RINCE [15), and PSCL [16]) achieve significant performance improvements in multi-view scenarios.
2 Related Work
Contrastive learning (CL) As a popular self-supervised learning paradigm, CL focuses on learning semantically informative representations for downstream tasks [22, 23, 24, 25J. The most widely used loss function is InfoNCE [8] which pulls in the representations between positive sample pairs while pushing away that between negative sample pairs. Some work have attempted to explain the reasons for the success of applying InfoNCE, e.g., from perspectives of mutual information [8, 26]. task-dependent view [27], or deep metric learning [28, 29]. Furthermore, [30, 31] pointed out to conduct CL with reconstruction regularization to achieve robust representations for downstream tasks. RINCE [15] (a short name of Robust InfoNCE) is a variant of InfoNCE contrastive loss that considers noise in false positive sample pairs. The recent work [16] investigates CL without
2

---

6.2
60
Loos vallees
6.0
Similariee
ACC (8)
View
InoNCI (Viee 1:
60
Vicw wio
5.8
Costre simiboit (Viw 1: View
95
View wo C1
20
0,7
50
45
0.8
40
View
Vlow
View
View
View
100
200
300
400
100
200
300
400
Different views
Training epochs
Training epochs
(a) ACC us. Views
(b) Loss and similarity us. Epoch
(c) ACC us. Epoch
Figure 2: (a) Clustcring accuracy of individual views on Caltech dataset. (b) Contrastive loss and representation
similarity between view and view 4. (c) Clustering accuracy of view and view during contrastive learning.
conditional independence assumption positive sample pairs and proposes population spectral contrastive loss (we call PSCL for short). Despite important progresses have been made, this work, we discover that these CL losses still fragile in multi-view scenarios where data qualities are hard to guaranteed, and even the reconstruction regularized CL is not enough.
Multi-view contrastive learning (MCL) Different from many CL methods that usually generate two inputs by data augmentation [32], MCL aims handle multi-view data widely exiting in real-world applications. Multi-view data often contain more than two views/modalities and they naturally form multiple inputs [33, 34, 35]. Since the semantic consistency among multiple views not guaranteed, it challenging to capture the useful information in multi-view data, while considering the side effects of harmful information. Therefore, MCL attracts increasing attention in recent years [36, 37, 38]. For example, CMC [11] empirically shows that MCL performed with more scene views obtains the better representations with semantic information. DCP [39] leverages the maximization of mutual information to conduct consistency learning across different views and aims to achieve provable sufficient and minimal representation. MFLVC [14] observes the conflict between consistency and reconstruction objectives in encoder-decoder frameworks and proposes to learn multi-level features for multiple views. DSIMVC [17] establishes a theoretical framework to reduce the risk of clustering achieved
performance degradation from semantic inconsistent views. Although satisfactory results in many cases, the representation degeneration caused by CL is still not well considered and addressed. In this paper, we point out that the representation degeneration could seriously limit the application of CL in multi-view scenarios, and propose the discrepancy- based selghweighted MCL to address it.
Notations
This paper leverages bold uppercase characters and bold lowercase characters to denote matrices and vectors, respectively. Operator || ||2 denotes vector f2-norm and operator || || 1.2..... matrix F-norm. {x," E XX) 1,2,..,N denotes the multi-view dataset with N samples in V views.
Methodology
This section first illustrates the phenomenon of representation degeneration in multi-view contrastive learning. To address this issue, we then establish general framework of SEM: SELf-weighed Multi- view contrastive learning with reconstruction regularization. To implement the SEM framework, we further provide different options of weighing strategy, contrastive loss, and reconstruction term.
3.1 Motivation: Representation Degeneration in Multi-View Contrastive Learning
Researchers proposed many contrastive lcarning approaches and also achieved plenty of progress in multi-view learning. However, multi-view contrastive learning might result in the representation degeneration of high-quality views (i.e., those views contain rich semantic information) due to the diversity of multi- multi-view data. Specifically, we illustrate it in Figure 2 that takes a popular multi-view dataset Caltech [40] (6 views)
example. We leverage unsupervised linear clustering accuracy obtained by K-Means [41] to evaluate the representation quality of containing class-level semantics.
Firstly, we leverage self-superviie autoencoders (the setting is shown in Appendix B) to pretrain the representations of each view's data. Figure 2(a), one can find that different views inherently have different levels of discriminative information and exhibit different qualities, where the worst (view 1)

---

and the best (view 4) have large gap. Then, we adopt InfoNCE loss to perform contrastive learning between view and view 4 in Figure 2(b), and record the clustering accuracy of their representations in Figure 2(c). We can observe that InfoNCE loss is well-minimized, which makes the representation similarity (evaluated by cosine) between view and view 4 converge 1.0. The performance on view 1 gradually increases. Nevertheless, the cost is that the representations of view 4 degenerate, on which the useful discriminative information reduces and thus the performance gradually decreases.
In multi-view learning, quality difference among multiple views is common phenomenon. However, the representation degeneration in multi-view contrastive learning might make the representations of some high-quality views tend to be mediocre and thus miss their useful discriminative information.
3.2 Self' WeighWeigh Multi-Viee Contrastive Learning with Reconstruction Regularization
To mitigate representation degeneration in multi-view contrastive learning, we propose simple but effective framework called SEM: SElf-weighted Multi-view contrastive learning with reconstruction regularization shown in Figure 1. Specifically, given view-specific data X E N we let Z" E N denote the corresponding new representations learned by a view view-specific encoder. Between X and Z" record precursor state of representations hidden features), and the encoder is partitioned into two parts (the front and back parts are stacked and denoted as f" and g" sequentially). For the e-th view, let di and q" denote the network parameters of f and g", respectively, and then the view-specific model can be formulated as follows:
H" R Nxh,, (termed as
Z" ===g 9" (H" :0)) (f (X"; Ø"):
(I)
In SEM, we leverage Ler" (Z Z denote contrastive loss?, and let À 0 denote trade-off coefficient on regularization terms. Then, SEM is trained by minimizing the following objective:
Σ WTT. Cer (Zm Z") (")
+ E,
(²(X",H"))
where Wm,, is the pairwise weight between the m-th and the n-th views, and Rd (X",, HD) denotes the reconstruction regularization H". We define D(Z" Zz) as the discrepancy between ZTd and z" and denote F as a negative correlation function. Then, in SEM, the pairwise weight is updated by Wm. F(D(Z"), (Z)))
(3)
J==1 Self- Self-weighting In unsupervised settings, it is hard to know which representations within {Z")V contain useful semantic information and which are with more noise. To mitigate the representation degeneration caused by contrastive learning, SEM needs to be adaptive to quality difference among multiple views. Therefore, different from using equal-sum manner [11, 14, 17] (e.g., Em, CCL") )
we propose to use the pairwise weighted multi-view contrastive loss, i.e., Em,n Wmm. E Here, Wmn. leverages the discrepancy to achieve the adaptive self-weighting. Concretely, if two views are useful pairwise views and both with informative semantics, contrastive learning between them is adaptively strengthened; if two views are unreliable pairwise views (for example, one or two of them are with less informative semantics), contrastive learning between them is adaptively weakened.
Reconstruction regularization In Eq. (2), (X",, H") acts as self-supervised objective to transfer much discriminative information possible from X" to H". When record H the hidden features in encoder networks, the information transfer path can be described as X" - H" + Z", E 21,2, ... V). However, information losing might occur in the processing of X H such that discriminative information from some views' data is lost, and thus making contrastive learning among [2') focus harmful noise instead of common semantics across multiple views. To this end, on hidden features H", our SEM leverages X" to build the reconstruction regularization R" (X", H") to assist contrastive learning in accessing sufficient discriminative information from raw data.
"Cm.a Z²) can be casily replaced by previous contrastive losses, egg. InfoNCE [8], RINCE [15), and PSCL [16]. Let P denote the set of positive sample pairs and A' the set of negative sample pairs the m, n-th views, & and a are hyper-parameters of RINCE, then the three contrastive losses could be formulated as follows:
LIfoNCE = -EstEP
log (e**
Ex=EN
CmL CRNCE -ESEE
vat
9 (a
(cet + + ))=
9 st
--E, EP
+ B, eN [(s))7]
where st (s denotes the cosine distance between the representations of positive (negative) sample pair.

---

3.3 Different Options for Implementing the SEM Framework
The crucial components of our proposed SEM as Eq. (2) include the weighting strategy Wm contrastive loss _m. , and regularization term R". Next, we concentrate on the implementations of Wmn. (including JSD, MMD, and CMJ) and briefly introduce the implementations of CCL and R".
Discrepancy measurements of weighting strategy When implementing Wm = F(D(Z ))) in Eq. (3), many methods measure the discrepancy D(Z", Z²). Firstly, we can transfer represen- tations to a probability distribution and leverage lensen-Shannon divergence (ISD) to compute the discrepancy DJSD (Z", ). The advantages of JSD are its symmetry and simplicity, but might be inapplicable when two distributions are non-overlapping. Furthermore, we can leverage maximum mean discrepancy (MMD) as the second method obtain the discrepancy DMMD(Zi (Z Z²). MMD can effectively measure non-overlapping two distributions, but it has higher complexity than JSD'.
Actually, both JSD and MMD leverage all information of representations 2 and Z". However, the semantic- inreeiiitieeeeaant information or random noise might also be embedded in 7 and Z", Moreover, what we expect to obtain is the mutual relation of their most representative semantic information. To this end, we propose Class Mutual Information (CMI) as the third method to obtain the discrepancy DCMI Zm, 7.). To be specific, since it is difficult to accurately estimate the mutual information (denoted as 1) for multi-dimensional continuous variables 1 (Z",; Z"), we denote y and y" as 1-dimensional discrete vectors and change estimating / [Z"; Z") ) to computing I (Ly";"") such that:
Wm. = F(1/I(Z") )) = (y" y )), s.t. argmax ,ym) + yy" (4)
y"
Intuitively, discrete class information in Z" (v E {m, n)) is I-dimensional as well the most repre- sentative information. Hence, we can optimize K-Means objective to extract the class information:
Y argmax I ,.t s.t.Y" Y",c"
s.t.
IN, Y E [00
(5)
where C" IR xx* denotes the K cluster centers of Z" Y E {0. 1 is the indicator matrix that can be further transformed to 1 1-dimensional discrete vector y" by defining y := argmax; Voj where yf" E y", Vij" E Y In this way, the class information in n and Z" can be compressed into y and y", respectively. Then, the class mutual information ((y))""))) ) normalized and the discrepancy measurement DC Z") between pairwise views is defined as follows:
Hy"") + H(y") DCMI 2n) 2. y")
(6)
where
N - p() p(y.") the cross-entropy of y This design of CMI has at least two advantages: 1) It is conducive to maintaining the representative class information while filtering out noise information; 2) Calculation is easy and owns better physical meaning.
Finally, it is also flexible to implement the negative correlation function F. Considering Wm.. 2 0, base on the three different discrepancies and simply give the following weighting strategies:
Wast =
FcMi (Dcu(DeMI) (Zm )²)) e¹ (Z" zz") 1,
Wm. FJSD(DDSSD(2 (ZT 2")) c1 D-Dss m2 2²)) 1,
Wm.
MMD = FMMDDDMMD((Z )))) = e PMMMD(""."")
Compatibility for contrastive learning When implementing the contrastive loss m... it should be pointed out that multi-view contrastive learning usually has handle more than two views (i.e.,
We write क."h E 2m Softmaz(( (Z"). k(z,, x,) denotes the inner product of ((z,) and Φ(z,), where x6-) denotes the mapping (e.g.. by Gaussian kemeel) to project representations into Reproducing Kernel Hilbert Space (RKHS). Then, DJsp (Z",, ZZ" and D (Z" IZ") can be formulated as follows:
2 p(2)))
2 2.p(2))
(z" 2 Σ p(2"") log
p(2.) log
p(2TT) + +p(2))
2
p(()) + +p(2T))
N Σ
N
N
DMM ) N²
k(z 2,
Σ
k(z 2 22 Σ Σ k(z)" 2,¹)
5

---

[Z" > 2), which is different from two-view setting (e.g., (Z', (2 in traditional contrastive learning. To make our SEM framework be compatible with previous contrastive learning methods, we construct positive/negative sample pairs as follows. Specifically, for two views {z." E 7 2 E )" the positive sample pairs are {z." 2 >i=.....N:: for any z,", its negative sample pairs are {zm", , "jjjfi " Cosine with temperature parameter T is leveraged to measure the representation distance between pairs, i.e., s = 1/T (2.", z²) //1,," [a|z ll2. Then, we compute the contrastive loss between views and sum all combinations Eq. (2). We formulated three contrastive losses in Sec. 3.2, and the experiments in Sec. 4.1 will verify the compatibility of our SEM framework to them.
Reconstruction regularization When implementing the regularization term (X",", ,H) in Eq. (2), we are motivated by the information saccoding-decodiin process [14, 19, 30], and stack view-specific decoder f" with network parameter Q" on each view's s H" to perform data recovery of X In this way, the regularization term in SEM can be implemented with the reconstruction loss of autoencoders*, whose encoder-decoder models can make hidden features preserve discriminative information of data. When decoder generally rebuilds X" with H information X", for promoting contrastive learning fully access discriminative information of data.
believe that HD compresses the sufficient
Algorithm 1: Self- multi view contrastive learning with reconstruction regularization
Input: Dataset [X"]V v=1- Training epochs E, Step size S, Batch size n. Hyper--aramede À H
Initialize {vr", [2 1==1 by Eq. (8) and initialize (Wm,..) Im,== with (H") J==1 like Eq. (7)
for e E {1. 2........ F) do
for E f1,2...,,N/A do Pick mini-batch data (xx)) (b==-1)n++11 1 from (X")
Compute the gradient of loss via Eq. (2) on the mini- batch data
Update {Ø" 4", , via Adam [42] optimizer
if mod(e, == 0 then
L
Update (Wmm, V
Jm,n=1 with {Z") by Eq. (7)
Output: Model parameters (Φb", (P"
The training steps of SEM is summarized in Algorithm 1, where representations and weights are updated alternatively to make them promote each other. E denotes total training epochs, and the step size S denotes the number of training epochs after each update of pairwise weights. As we cannot obtain meaningful {y") J==1 before we start training neural networks, we first obtain meaningful (H") V
1==1 1 by pre- -eeiiiriiing the model with Eq. (8), and then initialize (Wm..) 1V Jm,n=1 with { To=1
3.4 Theoretical Analysis
In this part, we theoretically analyze the mechanism of SEM in exploring mutual information among multiple views while mitigating representation degeneration. All proofs are given in Appendix A.
Considering SEM with InfoNCE loss and CMI weighting strategy, have the following theorem in- dicating that minimizing the self-weighted contrastive loss keeps maximizing the mutual information between useful pairwise views, as well as avoiding the effects between unreliable pairwise views.
*We borrow core ideas information reconstruction applied in vanilla autoencoder (AE [19]), denoising autoencoder (DAE (20]), and masked autoencoder (MAE [21]) and provide three reconstruction regularization options. In same form, the three kinds of reconstruction loss functions could be formulated as follows:
(X, H |X" (H ;0") X (x(f" V"))
R X", H") x" f"" (2")
IX" - fX'' +
(8)
(X", H = x" - f- (iT): (i"";
IX" (x' @ 2.
where X" + denotes the data disturbed by random Gaussian noise E Nxd, in DAE. X" is the data masked by random 0- - 1 matrix A E {0, 1pxxds in MAE. H and H denote the representations inferred from data X + and X" @ A in DAE and MAE, respectively.

---

Theorem 1. For any three views (v E {m. n. o)), class mutual information only exists in two views.
e.g. Iyy",y') + 0, I(y"; + 0, and I [yy "yy":y" :y") = 6. 8 > O. we have minimizing the weighted LInfoNCE (Ztr 7d + cm.
InfoNCE losses Wmm. L
InfoNCE (7 + L LInfNNCE (Z", Z0) is equivalent to maximizing the mutual information between the two views (e N - 1) Z7).
Combining with the information losing of each layer through encoder networks, the following theorem further reveals that reconstruction regularization on the hidden features H" is conducive to alleviating the losing of discriminative semantic information through data transformation. Hence, treat the layer output closest to Z" in encoders as hidden features to maximize I1 (1 )("))
and I1 (1 aiming at maintaining useful semantic information for contrastive learning.
Theorem 2. For any two views (v E {m,n) with positive class mutual information, denoting L" as the total layer number of the v-th view's encoder network before representation Z", the 1-th layer has the information losing rate "i" 2 O. If S is an oracle variable that contains and only contains multiple views' discriminative semantic information, and H is the t"-th layer's features have minimizing the regularized loss Wmm. Lm..
IInfoNCE (Z 7") + X Σ, R R"XX", " H is expected II - >;"",, (S) II - -.
to obtain [(; m 2²) S min(
4 Experiments
This section validates the effectiveness of our SEM. Specifically, we first conduct comparison experi- ments on state-of-the-ar contrastive learning baselines and SEM with three options of contrastive losses (i.e., LIafoNCE, Crsci, CRINCE). We then conduct ablation studies with three options of weighting strategies (i.., WCMI, WJSD, War M D), as well as with three options of reconstruction terms (i.e., RAE RDAE, RE).). Evaluation is built on the concatenation of all views' representa- tions learned by methods. Finally, we show SEM's training process and its hyper-paramen We provided more experimental results as well as all implementation details of SEM in Appendix.
analysis.
Our experiments employ five open-source multi-view datasets. Their information is shown in Ta- ble 1, where DHA [43] is a depth- -ncthdid human ac- tion dataset where each action has RGB and depth fea- tures; CCV [44] refers to the columbia consumer video database whose samples are described with SIFT, STIP, and MFCC features; NUSWIDE [45] collects web im- ages with multiple views (color histogram, block-wise color moments, color correlogram, edge direction histogram, and wavelet texture); Caltech [40] is widely-used image dataset which leverages six views (Gabor, Wavelet moments, CENTRIST, HOG, GIST, and LBP) to represent samples; Youtube [46] is large-scale dataset where each sample has three views including cuboids histogram, HOG, and vision mise. These datasets are diverse in forms and are often organized to comprehensively evaluate the performance of multi-view methods.
Datasets
Table 1: Information of datasets
Name
View
Size
Class
DHAA
2
483
23
CCV
3
6,773
20
NUSWIDE
5
5,000
5
Caltech
6
1,400
7
Youtube
3
101,499
31
4.1 Comparison Experiments on Contrastive Learning
Baselines K-Means-BSY denotes K-Means clustering results on the best single-view of raw data, and we leverage this baseline to investigate the representation degeneration in comparison methods. InfoNCE [8], PSCL [16], and RINCE [15] are three kinds of CL methods. Since their original versions are designed to handle single views, we extended them to multi-view scenarios as did in [11, 17]. CMC [11], DCP [39], MFLVC [14], and DSIMVC [17] are four kinds of MCL methods. We evaluate our SEM with different contrastive losses (i.c., SEM+InfNNCE, SEM+PSCL, and SEM+RINCE), where the weighting strategy and reconstruction term are fixed to WCMI and RAE. respectively.
We leverage the linear clustering method K-Means to evaluate the performance of learning represen- tations and report the average results of 10 runs in Table The results indicate that: 1) Our SEM framework is compatible with different contrastive losses (e.g.. InfoNCE, PSCL, and RINCE) and we can clearly observe that SEM+IIfoNNE//SCCL/RII large margins. For instance, SEM+InfONE, respectively outperforms InfoNCE by about 25%, 13%, 4%, 7%, 1 22 ACC on the five datasets. II) MCL approaches could access the semantic information from multiple views, and thus outperforming that from single views. However, side effect is
successfully improve the baselines for
7

---

Table 2: Linear clustering performance evaluated by ACC and NMI (mean-Lstd%)
DHA
CCV
NUSWIDE
Calloch
VouiebeVidso
Method
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMT
K-Means-BSV
66.6=2.6
1..021.3
19.540.3
17.8=0.3
39.7=0.0
11.620.0
35.140.1
75.6=0.1
16.520.6
15.9=0.4
InfoNCE [8]
54.9=3.8
77.742.1
25.8+0.9
25.9+0.7
56.3+2.2
10
79.9+10
71.4=0.9
19.6+0.1
19.7=0.0
PSCL [16]
39.8+2288
72.942.00
21.54:1.6
24.3+:10
53.4404
27.440.5
67.8+-.88
69.5-2.8
15440..3
14.3:0.5
RINCE [15]
49.9116.2
76.3412.6
22.510.4
23.51002
56.6114
30.811.3
80.312.
72.0-222
14.7.0.3
13.6.10.2
CMC [11]
65,0=2.1
79.2+1.3
21.320.4
21,82066
56.2=155
24.7#0,9
72.7=1.4
60.3=14
19.420.3
19.620.1
DCP [39]
69.8-2.2
82.9+1.6
24.1+112
20.6=099
48.1=14
24.5+1.1
09.6+666
66.2-5.3
14.0=0.3
12.3+0.4
MFLVC [14]
70.7+114
8144+08
31,6+0.00
31.3+0000
55.9+0.0
27440.0
77.1+0.55
67.1-0.6
18.3+0.1
18.7+0.2
SEM+InloNCE
80.92199
84.1
39.4100.7
35.520.4
60.4204
34.910.7
87.22033
80.3
31.32111
31.12099
SEM+PSCL
69.7=42
81441.6
39.3+1.1
35,9=0.6
57,8=14
32.6+0.9
86.3+1.7
78.6-2.1
32.2+0.7
32,2+0.6
SEM+RINCE
76.3=113
82.8+0.7
38.9+0.9
34.6+00.5
60.6+00.7
35.6+1.3
85.4+1.4
76.7-22
29.8+00.5
29.5+0.5
that contrastive learning directly increases the feature representation similarity of multiple views, which might obseure useful discriminative information hidden high-quality views and lead to the representation degeneration. For example, on DHA and Caltech, results on many MCL methods (e.g., PSCL, RINCE, CMC, and MFLVC) are worse than the single-view baseline K-Means-BSV. IIII Our SEM not only outperforms all these MCL methods but also mitigates the representation degeneration in MCL, e.g., SEM+PSCL respectively outperforms K-Means BSV by about 3%, 20%. 18%. 1% 16% ACC on the five datasets. This is because the framework of SEM is adaptive to multiple views' qualities, which can reduce the side effect between unreliable views with inconsistent information, for better extracting discriminative information and consistent semantics among useful views.
Additionally, we lever- age the linear classifica- tion method SVM [47] evaluate the perfor- mance of MCL methods learn representations, where we only use 30% of the learned representa- tions for the training set and the rest for the test set. Figures 3 and 4 show the classification perfor- mance on DHA and CCV, respectively. Our SEM im- proves the baseline meth- ods (especially for PSCL and RINCE) and consistently outperforms other MCL methods (such as CMC, DCP, MFLVC, and DSIMVC), Since contrastive learning usually discards the information which irrelevant to optimization objectives, the results further indicate that the representations learned by MCL dassificationn-fiennte friendly, which generally focus on catching class-level semantics among multiple views.
SEMORRENCE SEMISCE
Figure 3: Classification performance on DHA.
ACC Becal
Figure Classification performance on CCV.
4.2 Ablation Experiments on Seightiig Strategy and Reconstruction Regularization
This part presents the ablation experiments investigate the effectiveness of different weighting strategies WeMIJJSD/MMD and reconstruction terms RAE/DAE/MAB in our SEM framework.
Table Clustering accuracy (%) SEM with differ- ent options of weighting strategy W on two datasets
Table 4: Clustering accuracy (%) SEM differ-
ent options of reconstruction term R on two datasets
DHA
CCV
DHA
CCV
SEM w/o
71.3
33.5
SEM w/ R
60.5
28.7
SEM w/
WCMI
80.9 (f 9.6) 51
SEM w/ RAB
80.9 20.4)
39.4 (1 10.7)
SEM w/ WISD
80.5 (f 9.2) (1.6
SEM w/ RDAE
81.5 (1 21.0)
38.4 (f 9.7)
SEM w/ WMMD
84.4 (1 13.1) (3.44
SEM w/ RMAE 83.0 22.5)
39.5 (1 10.8)
Table reports the linear clustering performance (evaluated by ACC) of our SEM framework without self-weighting strategy (i.e., SEM w/o W) and that with three weighting strategies (i.e.,

---

1.6
6.2
InfoNUIc loss (Now 1: View
InRANCE losa (Neew 4, View 5)
80
12
2055 values
70
5.8
ACC
60
808
50
0.4
10h-
5.4
40
View 5 CL
View wi CL
0
100
200
300
400
0
3
lleration
Training epochs
lleration
(a) Weight 28. Iteration
(b) Loss us. Epoch
(c) ACC 018. Iteration
Figure 5: (a) The change trend of weights W1,4 and W in SEM. (b) Loss values L1.4 during contrastive learning. (c) Clustering accuracy on the learned representations of view view 4, and view 5.
InfoNCE and C4.6
IIfeNNE
WCMI, WJSD, WMMD). where the contrastive loss and reconstruction term are fixed to LInfoNCE and RAE, respectively. Compared with SEM w/o W (this setting is the reconstruction regularized multi-view contrastive learning) that equally treats contrastive learning between any two views, SEM w/ WCMI/JSD D can adaptively weight the contrastive learning according to specific two views and thus all these three variants of SEM obtain significant improvements. For example, SEM w/ WMMD has a 13.1% improvement on DHA and SEM w/ WeMr has a 5.9% improvement on CCV Results on more datasets and time costs are shown in Appendix C, where we find that the proposed weighting strategy of class mutual information WCM1 generally achieves the best performance on accuracy and time consumption among the three options of weighting strategy.
Table reports the linear clustering performance (evaluated by ACC) of our SEM framework without reconstruction regularization (i.e., SEM w/o R) and that with three reconstruction terms (i.e., RAE, RDAE, RM AE), where the contrastive loss and weighting strategy are fixed to LiefoNCE and WCMI, respectively. We easily find that the proposed SEM with reconstruction terms obviously outperforms that without reconstruction terms. For instance, compared with SEM w/o R. SEM DHA and CCV, respectively. This is because
w/ WMAE has 22.5% and 10.8% improvements the reconstruction regularization makes the hidden features {H") " avoid losing discriminative information, which promotes the multi-view contrastive learning performed on subsequent (Z") " Meanwhile, SEM w/ RAS AE and SEM w/ R.D perform better than SEM w/ RAF.. This is because, compared with vanilla AE, DAE MAE (by adding noise masking
data) can make our model more conducive to removing semantie-irrelevant noise as well as capturing hidden patterns.
4.3 Experimental Analysis on Mechanism of SEM
This part presents the visualization and analysis on SEM to give an intuition of its behavior and mechanism, where the combination LInoNNE+WeMI++R AE is taken as an example.
Let's first recall the views of Caltech dataset in Figure 2(a), we can consider that view 4 and view 5 are high-quality views, while view I is low-quality view. The performance relation among them is ACCview In Figure 2(c), view 4's representation degeneration occurs.
ACCview 4 > ACCview
Figure 5 shows the pairwise weights, losses, and clustering accuracy Caltech dataset during SEM's training process, where I iteration corresponds to 100 epochs, i.e., the step size is set to 100 epochs. Our SEM is self-weighted multi-view contrastive learning framework that automatically infers different weights for different pairwise views as shown in Figure 5(a), where we can observe that weights W1. > WL4 and they were dynamically updated for 4 times. As result, contrastive learning between view 4 and view 5 is strengthened by W4.5 while contrastive learning between view 1 and view 4 is weakened by W/1,4 Meanwhile, loss 44.5
InfoNCCE is minimized earlier than loss
C1.4 InfoNCE as shown in Figure 5(b). other words, since the mutual effect between view and view is strengthened, the effect of view on view 4/view 5 is weakened such that view 4/view does not degenerate. At the same time, the effect of view 4/view 5 on view 1 remains and promotes the views" performance Figure 5(c) increases
representation learning of view 1. Consequently, through our SEM, and the representation degeneration of view 4 occurring in Figure 2(c) is mitigated.

---

Hyper-parameter analysis Since different datasets have different levels of reconstruction er- rors, the trade-off coefficient is introduced balance the contrastive learning and infor- mation recovery in our SEM framework. In Figure 6(a), we change À within the range of [10-s, 10-22 10-1, 10", 10 102, 10 and report the clustering accuracy tested on representations. The experimental results indicate that SEM is not sensitive to À in [10-1 ,1'). In our experiments, À consistently set to 1 for all the five datasets. Regarding self-supervised learning, frameworks with fewer manually set hyper-parametent might be more convenient for their applications.
Additionally, we investigate the effect of cluster number when the weight strategy of our SEM framework is selected as WCM1 which needs to pre-define the cluster number when applying K K-Means algorithm. As shown in Figure 6(b), when comput- ing the class mutual informa- tion, we change the number of clusters within the range of [K/2, K, 2K, 4K] where K de- notes the truth class number multi-view datasets. Compared with K. K/2 leads more coarse grasingr class mutual information, while 2K and K come in more fine- fine-grained class mutual information. The experimental results demonstrate that SEM with WCMI is not sensitive to the choices of cluster number.
100
Calch
CCV
YouubeViddo
suvvie
NUSWIDE
K2
2K
4K'
Number clusters
Figure 6: (a) ACC es. A.
(b) ACC us. K.
5
Conclusion
In this paper, we showcase that the representation degeneration could seriously limit the application of contrastive learning multi-view scenarios. To mitigate this issue, we propose self-weighted multi- view contrastive learning with reconstruction regularization (SEM), which is general framework is is compatible with different options of the contrastive loss, weighting strategy, and reconstruction term. Theoretical and experimental analysis verified the effectiveness of SEM, and it can significantly improve many existing contrastive learning methods in multi-view scenarios. Moreover, ablation studies indicated that SEM is effective with different weighting strategies and reconstruction terms.
Our future work is extend the proposed SEM be useful only for multi-view scenarios, but also for other contrastive learning based domains, such as contrastive learning in sequences. Conceptually, the limitation of the self-weighting strategy is that it is more effective when there are over two views. When there are only two views, the self-weighted multi-view contrastive learning framework transforms into traditional contrastive learning but with reconstruction regularization. Therefore, another future work is to extend the view-level weighting of SEM to sample-level weighting.
Acknowledgment
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA 1004100, in part by the EngiieerEngnn Cooperation Funds from University of Electronic Science and Technology of China under Grant ZYGX2022YGRHO09 and Grant ZYGX2022GRRH014 in part by the National Natural Science Foundation of China under Grant 62276052.
10

---

References
[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pages 1597-1607, 2020.
[2] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0422 2020.
[3] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence
embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1894-6910, 2021.
[4] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
pages
[6] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
[7] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang. Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEESCVF Conference on Computer Vision and Pallern Recognition, pages 6995-7004, 2021.
[8] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: arXiv:1807.00 1807.03748, 2018.
[9] Chang Xu, Dacheng Tao, and Chao Xu. A survey multi-view learning. arXiv preprint 13445:130 2013.
[10] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206, 2021.
[11] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. European Conference Computer Vision, pages 776-794, 2020.
[12] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116-4126, 2020.
[13] Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. Advances in Neural Information Processing Systems, 34:2148-2159, 2021.
[14] Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi-view clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16051-16060, 2022.
[15] Ching-Yas Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16670-16681, 2022.
[16] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
[17] Huayi Tang and Yong Liu. Deep safe incomplete multi-view clustering: Theorem and algorithm. In International Conference on Machine Learning, pages 21090-21110, 2022.
11

---

[18] Yang Yang, Chubing Zhang, Yi-Chu Xu, Dianhai Yu, De-Chuan Zhan, and Jian Yang. Rethink- ing label-wise cross-modal retrieval from a semantic sharing perspective. Proceedings of the International Joint Conference Artificial Intelligence, pages 3300-3306, 2021.
[19] Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in Neural Information Processing Systems, 6:3-10, 1993.
[20] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference Machine Leariin, pages 1096-1103, 2008
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[22] Yunfan Li, Mouxing Yang, Dezhong Peng. Taihao Li, Jiantao Huang, and Xi Peng. Twin con- trastive learning for online clustering. International Journal of Computer Vision, 130(9):2205- 2221, 2022.
[23] Ching Yao Chuang, Joshua Robinson, Yen- Ven-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in Neural Information Processing Systems, 33:8765- 8775, 2020.
[24] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances Neural Information Processing Systems, 33:5812-5823,, 2020.
[25] Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng. Unsuper- vised contrastive modal hashing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3877-3889 2022.
[26] R Devon Hjelm, Alex Fedorov, Samuel Marchild,,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
[27] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020.
[28] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, 29:1857-1865. 2016.
[29] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. International Conference on Learning Representationss 2020.
[30] Shuo Chen, Chen Gong, Jun Li, Jian Yang, Gang Niu, and Masashi Sugiyama. Learning contrastive embedding in oow-dimensional space. Advances in Neural Information Processing Systems, 35:6345-6336 2022.
[31] Haoqing Wang, Xun Guo, Zhi- thi-Hong Deng, and Yan Lu. Rethinking minimal sufficient repre- sentation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 16041-16050, 2022.
[32] A Jaiswal, A Ramesh Babu, M Zaki Zadeh, D Banerjee, and F Makedon. A survey on contrastive self-supervised learning. Machine Learning, 12:4182-4192 2020.
[33] Changqing Zhang, Zongbo Han, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, et al. CPM- Nets: Cross partial multi-view networks. Advances in Neural Information Processing Systems, 32:559-566 2019.
[34] Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning, pages 17782-17800, 2022.
12

---

[45] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. NUS- WIDE: a real-world web image database from national university of singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval, pages 1-9, 2009. feature [35] Yang Yang, Jingshuai Zhang, Fan Gao, Xiaoru Gao, and Hengshu Zhu. DOMFN: A divergence- orientated multi-modal fusion network for resume assessment. Proceedings of the ACM International Conference on Multimedia, pages 1612-1620, 2022.
[36] Mouxing Yang. Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 1134-1143, 2021.
[37] Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 754-763, 2021.
[38] Jiyuan Liu, Xinwang Liu, Yuexiang Yang. Qing Liao, and Yuanqing Xia. Contrastive multi-view kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):5552- 9566, 2023.
[39] Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning, IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4147-4461, 2022.
[40] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshop, pages 178 178, 2004.
[41] John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):100-10) 1979.
[42] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1.... 69800, 2014.
[43] Yan-Ching Lin, Min-Chur Hu, Wen-Huang Cheng, Yung-Huan Hsieh, and Hong-Ming Chen. Human action recognition and retrieval using sole depth information. In Proceedings of the ACM International Conference on Multimedia, pages 1053- 1056, 2012.
[44] Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, and Alexander Loui. Consumer video understanding: A benchmark database and an evaluation of human and machine perfor- mance. In Proceedings of the ACM International Conference on Multimedia Retrieval, pages
1-8, 2011.
[46] Omid Madani, Manfred Georg, and David A. Ross. On using families for high precision and confidence. Machine Learning, 92:457-477, 2013.
[47] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20:273-297.
1995.
13


Reviews:
[{"review_id": "QyPdl0g0kh", "reviewer": "Reviewer_gird", "summary": "In summary, this paper investigates an important question about how to mitigate representation degradation in multi-view contrastive learning. Considering the quality difference of views and losing useful information during nets, the proposed method uses adaptive weighted contrastive learning and adds information reconstruction to improve the performance of contrastive learning in multi-view learning.", "questions": "Please see weaknesses.", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I) The inductive bias of contrastive learning may allow the newly learned representation to capture trivial information, thus causing representation degradation. So, it is meaningful to propose an effective framework to alleviate this issue in this paper.\n\nII) The authors proposed a novel multi-view contrastive learning framework called SEM with multiple implementations. For example, the paper provides different weighting strategies with different advantages and experiments show their effectiveness.\n\nIII) Comparison experiments on different datasets show that SEM+infoNCE,RINCE,PSCL has a significant improvement over infoNCE,RINCE,PSCL themself, indicating the effectiveness.\n\nIV) The supplementary material is well organized, including detailed appendices as well as code.\n", "weaknesses": "I) Due to the high complexity of MMD computation, it seems difficult to obtain MMD results on YoutubeVideo dataset (over 100,000 samples).\n\nII) Introducing weighting strategies may increase the number of hyper-parameters. It might be better to add more descriptions, e.g., MMD.\n\nIII) Some grammatical mistakes need to be corrected, e.g., transfers is transfer in Line 155; adaptively weighting is adaptively weight in Line 304.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response To Reviewer gird:\n\n>Q1: Due to the high complexity of MMD computation, it seems difficult to obtain MMD results on YoutubeVideo dataset (over 100,000 samples).\n\nYes, it is indeed difficult to obtain weights of the MMD weighting strategy on YoutubeVideo. Therefore, for reducing the computation complexity of MMD, we leveraged partial instead of whole samples when applying the MMD weighting strategy. We already discussed this in complexity analysis in Appendix A (Page 7). Furthermore, Appendix B (Page 7) provides the implementation details when applying the MMD weighting strategy. That is, the weights of the MMD weighting strategy are computed by only leveraging the first 2,000 samples on YoutubeVideo, and thus the computation complexity of our method is controllable in practice uses.\n\n>Q2: Introducing weighting strategies may increase the number of hyper-parameters. It might be better to add more descriptions, e.g., MMD.\n\nIn our experiments, for the CMI weighting strategy, the cluster number is pre-defined to the truth class number of a dataset. For the MMD weighting strategy, the bandwith and number of kernels are set to 4 on all datasets used in this paper.\n\n>Q3: Some grammatical mistakes need to be corrected, e.g., transfers is transfer in Line 155; adaptively weighting is adaptively weight in Line 304.\n\nThanks for these comments. We will correct the grammatical mistakes and further polish our draft in the final version."}, {"review_id": "Cwr8qpqTtt", "reviewer": "Reviewer_tvpR", "summary": "This work proposed the SEM: SElf-weighted Multi-view contrastive learning framework, which first performs discrepancy measures on representations, and then obtains weights to assist adaptive contrastive learning. Meanwhile, the decoders are leveraged to avoid losing discriminative information. Extensive experiments are conducted to verify the effectiveness of SEM and its different implementation options.", "questions": "Can the authors show the performance of representation learning relying solely on autoencoders to understand the individual contributions between contrastive learning and reconstruction objectives?", "limitations": "This research is not expected to introduce new negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In unsupervised environments, it is hard but crucial for multi-view learning to automatically know which view’s features are with useless noise and which view’s features contain useful semantic information. I believe the idea of pair-wise self-weighted contrastive learning is novel. The proposed method obtains the adaptive ability to quality differences among multiple views, and it doesn't require much prior knowledge.\n\nMoreover, multi-view representation learning is important for working with multi-view data as raw views usually have inconsistent semantic meaning when one considers them in practical applications. In this paper, the authors present a robust multi-view contrastive learning method, whose effectiveness is verified by extensive experiments.\n", "weaknesses": "(1) To reduce losing information, the proposed method treats the last layer of encoders as hidden features. It's not clear exactly which layer is as H, and which layer is as Z.\n\n(2) When using the weighting strategies, the framework needs reconstruction pretraining to obtain meaningful representations. The decoder is added to the framework as an auxiliary module. As far as I known, many self-supervised multi-view methods also use autoencoders as the main representation learning module. Some ablation experiments are expected.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response To Reviewer tvpR:\n\n>Q1: To reduce losing information, the proposed method treats the last layer of encoders as hidden features. It's not clear exactly which layer is as H, and which layer is as Z.\n\nThanks for this valuable comment. We present the network setting in Appendix B (Page 7). Specifically, if we think of the network from $\\mathbf{X}^v$ to $\\mathbf{Z}^v$ as an encoder, then $\\mathbf{Z}^v$ is the output of the last layer of the encoder, and $\\mathbf{H}^v$ is the output of the penultimate layer of the encoder. We will further clarify this descriptions in the final paper.\n\n>Q2: As far as I known, many self-supervised multi-view methods also use autoencoders as the main representation learning module. Some ablation experiments are expected.\n\nWe conduct additional ablation experiments and report the clustering accuracy tested on the learned representations in the following table.\n\n&nbsp; |DHA |  CCV | NUSWIDE | Caltech | YoutubeVideo\n----|----|----|----|----|----\nvanilla AE|69.2  |14.3  |38.7  |86.0  |20.0\nDCP       |69.8  |24.1  |48.1  |69.6  |14.0\nMFLVC     |70.7  |31.6  |55.9  |77.1  |18.3\nDSIMVC    |63.8  |31.8  |56.7  |76.9  |18.9\nSEM w/ AE |80.9  |39.4  |60.4  |87.2  |31.3\n\nWe can observe that, previous AE-based methods (i.e., DCP, MFLVC, and DSIMVC) do not obtain better results than the vanilla AE on some datasets (e.g., Caltech and YoutubeVideo). In comparison, our proposed SEM w/ AE achieves better performance, as it incorporates the self-weighted contrastive learning to handle the representation degeneration.\n\n\n>Q3: Can the authors show the performance of representation learning relying solely on autoencoders to understand the individual contributions between contrastive learning and reconstruction objectives?\n\nYes. We conduct ablation experiments on three kinds of autoencoders and report clustering accuracy tested on the learned representations as follows.\n\n\n&nbsp; |DHA |  CCV | NUSWIDE | Caltech | YoutubeVideo\n----|----|----|----|----|----\nAE w/o SEM  |69.2  |14.3  |38.7  |86.0  |20.0\nDAE w/o SEM |70.4  |12.7  |39.5  |86.4  |21.7\nMAE w/o SEM |70.0  |14.6  |35.8  |86.2  |22.8\nSEM w/o AEs  |60.5  |28.7  |57.7\t |79.4  |32.7 \nSEM w/ AE   |80.9  |39.4  |60.4  |87.2  |31.3 \nSEM w/ DAE  |81.5  |38.4  |59.5  |86.6  |38.8\nSEM w/ MAE  |83.0  |39.5  |60.9  |86.7  |33.3 \n\n\nFirstly, the autoencoders have shown their basic representation abilities. They obtain good results on DHA and Caltech but fail on CCV and NUSWIDE (AE/DAE/MAE w/o SEM). Secondly, contrastive learning is conducive to exploring useful mutual information among multiple views. It obtains improvements on CCV and NUSWIDE but fails on DHA and Caltech (SEM w/o AEs). Thirdly, our framework leverages self-weighted contrastive learning while using reconstruction objectives to avoid losing discriminative information. Therefore, SEM with AE/DAE/MAE can obtain significant performance improvements compared with the above results."}, {"review_id": "TWDtDGdGP6", "reviewer": "Reviewer_USXC", "summary": "In this paper, the authors show that the representation degradation could limit the application of contrastive learning in multi-view scenarios. To mitigate this issue, they propose the self-weighted multi-view contrastive learning, a general framework that has different options in the contrastive loss, weighting strategy, and reconstruction term. In my opinion, this paper does a good job of showing what is emphasized, with many strengths. I've listed them below, along with some possible weaknesses.", "questions": "See Weaknesses.", "limitations": "The authors have discussed social impacts and limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "#1. The manuscript flows smoothly and is easy to understand. The analysis in the manuscript is well-thought-out and facilitates understanding the motivation and methodology.\n#2. Authors show a new multi-view contrastive learning framework that considers handling representation degeneration via self-weighting and information reconstruction.\n#3. The framework is technically sound. The manuscript provides three variants of weighting strategy including class mutual information, Jensen-Shannon divergence, and maximum mean discrepancy, and they are with different advantages.\n#4. The framework helps existing contrastive learning methods, like InfoNCE, achieve significant performance improvements in multi-view scenarios.\n#5. Sufficient ablation experiments and implementation details are provided.", "weaknesses": "#1. There is a technical detail that needs to be clarified. SEM adaptively strengthen contrastive learning between the useful pairwise views and also weaken contrastive learning between the unreliable pairwise views. This process can be conducted without supervision by the designed self-weighting framework. But after training, we still don't know which of the representations of multiple views is a good representation. Are the representations learned from useful pairwise views artificially selected to evaluate the performance of downstream clustering tasks?\n#2. It would be better for related work to include some recent jobs of contrastive learning. \n#3. The abbreviation should be interpreted, such as SIFT, STIP, and MFCC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response To Reviewer USXC:\n\n>Q1: Are the representations learned from useful pairwise views artificially selected to evaluate the performance of downstream clustering tasks?\n\nNo. To comprehensively evaluate the performance, our experiments use the concatenation of all learned representations from different views.\n\n\n>Q2: It would be better for related work to include some recent jobs of contrastive learning.\n\nThanks for this comment. We will further improve the manuscript by discussing more recently published work.\n\n>Q3: The abbreviation should be interpreted, such as SIFT, STIP, and MFCC.\n\nThanks for the suggestion. We add the following new table to interpret the abbreviations used in this paper.\n\nabbr. | Meaning\n----|----|\nSIFT | Scale-Invariant Feature Transform\nSTIP | Space-Time Interest Points\nMFCC | Mel Frequency Cepstral Coefficents\nCENTRIST | Census transform histogram\nHOG | Histogram of Oriented Gradient\nLBP | Local Binary Pattern\n"}, {"review_id": "qlOLWsX5E8", "reviewer": "Reviewer_dnwM", "summary": "This paper researches the representation degeneration of multi-view contrastive learning. To address it, this paper proposes a simple but effective framework of self-weighted multi-view contrastive learning.", "questions": "In Figure 5(a), are the weights incrementally and linearly increased or are they only updated 4 times?", "limitations": "The authors adequately addressed the limitations of this work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "++The manuscript is well-written and self-consistent. For example, the visualization analysis makes it easy for the reader to understand that considering view differences is necessary for multi-view data.\n\n++The motivation of this work is reasonable, and there are sufficient ablation experiments and theoretical analyses to verify the effectiveness of the proposed SEM algorithm.\n\n++SEM is a general framework that can be adapted to a variety of existing contrastive learning losses, as well as to a variety of autoencoder models.\n", "weaknesses": "--Figure 5(a) shows that weights are updated dynamically in different iterations. Here, are the weights incrementally and linearly increased or are they only updated 4 times?\n\n--The class mutual information weighting strategy is an interesting method that aims to easily measure the discrepancy between views, which compresses the representative class-semantic information of features into one-hot labels. As thus, Eq. (5) seems to be missing some constraints of $\\mathbf{Y}^{v*}$ of being the labels.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response To Reviewer dnwM:\n\n>Q1: Figure 5(a) shows that weights are updated dynamically in different iterations. Here, are the weights incrementally and linearly increased or are they only updated 4 times?\n\nWe are sorry that the illustration of Figure 5(a) is not clear enough. The weights are only updated 4 times during training. The line between the two points just shows their change tendency.\n\n>Q2: The class mutual information weighting strategy is an interesting method that aims to easily measure the discrepancy between views, which compresses the representative class-semantic information of features into one-hot labels. As thus, Eq. (5) seems to be missing some constraints of $\\mathbf{Y}^{v*}$ of being the labels.\n\nThanks for this valuable comment. We will complete the constraints of Eq. (5) that make $\\mathbf{Y}^{v*}$ represent one-hot labels, i.e., $s.t. \\sum_{j}^{K} y_{ij}^v = 1, y_{ij}^v \\in \\mathbf{Y}^{v}$."}, {"review_id": "oawa7aujoc", "reviewer": "Reviewer_tV6r", "summary": "This paper discusses the limitations of contrastive learning (CL) in multi-view scenarios and proposes a novel framework called SElf-weighted Multi-view contrastive learning (SEM) to address these limitations. The contributions of SEM framework are as follows:\n- Alleviating representation degeneration: In multi-view scenarios, CL can lead to representation degeneration when the collected views have inconsistent semantic information or lack sufficient discriminative information. SEM aims to mitigate this issue by adaptively strengthening useful pairwise views and weakening unreliable pairwise views through a self-weighted contrastive loss.\n- Regularizing hidden features: SEM introduces a self-supervised reconstruction term to regularize the hidden features of encoders. This regularization assists CL in accessing sufficient discriminative information from the data.\n- Extensive experimental validation: Experiments on public multi-view datasets demonstrate that SEM effectively mitigates representation degeneration in existing CL methods and leads to significant performance improvements. Ablation studies further verify the effectiveness of SEM with different options of weighting strategies and reconstruction terms.\n", "questions": "This method utilizes a number of models equal to the scale of the views, and it also requires calculating pairwise weights, which significantly increases the memory usage and training time as the number of views increases. The authors seem to lack detailed comparative figures with previous methods in this regard.\n", "limitations": "This paper does not include sufficient discussions about the computational cost and memory overhead, which are suggested to be included.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposal of SEM is well-motivated: SEM tackles the issue of representation degeneration in multi-view scenarios where inconsistent semantic information and insufficient discriminative information exist.\n- Flexibility in weighting strategies: SEM provides three options for implementing the weighting strategy, including class mutual information, JS divergence, and maximum mean discrepancy.\n- Significant performance improvements and thorough ablation studies: Experimental results on 5 public multi-view datasets, component ablation studies and hyper-parameter analysis demonstrate that SEM effectively mitigates representation degeneration in existing contrastive learning (CL) methods.\n", "weaknesses": "- I have some concerns regarding the rationale behind introducing the reconstruction module. If the aim of incorporating reconstruction is to enhance the discriminative information of the representation, why does the process involve an additional encoder after obtaining the representation H, instead of directly reconstructing the representation z? It would be beneficial to provide more details in the experimental results regarding the position of the reconstruction within the overall encoder and further analyze the relationship between discriminative information and the resulting performance improvement in both the method and experimental sections, thus solidifying the framework. \n- (minor) Based on the experimental findings, it appears that the enhancement from the reconstruction regularization is more pronounced compared to the improvement stemming from the self-weighted module. Why the reconstruction leads to a greater improvement needs a more detailed explanation in the experimental section. Besides, in terms of the method's naming, it would be beneficial to highlight the concept of the reconstruction component.\n- (minor) The presentation and captions of Figures 1 and 5 can be further improved, and it would be ideal to summarize the conclusions and connections among the data displayed in the three figures.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response To Reviewer tV6r:\n\n>Q1: I have some concerns regarding the rationale behind introducing the reconstruction module. If the aim of incorporating reconstruction is to enhance the discriminative information of the representation, why does the process involve an additional encoder after obtaining the representation $\\mathbf{H}^v$, instead of directly reconstructing the representation $\\mathbf{Z}^v$?\n\nThanks for raising this concern. As we all know, data in different views usually contain useful discriminative information for common semantics as well as semantic-irrelevant information. We introduce the reconstruction module on $\\mathbf{H}^v$ to avoid that $\\mathbf{H}^v$ loses the useful discriminative information of data (here, $\\mathbf{H}^v$ also maintains some semantic-irrelevant information due to the information reconstruction). Then, contrastive learning on $\\mathbf{Z}^v$ can make $\\mathbf{Z}^v$ access sufficient discriminative information from $\\mathbf{H}^v$ to explore the common semantics of multiple views. Nevertheless, if the reconstruction module is punished on $\\mathbf{Z}^v$, $\\mathbf{Z}^v$ also retains the semantic-irrelevant information which might disturb $\\mathbf{Z}^v$ to explore the common semantics of multiple views. In this regard, we conduct experiments to investigate the different reconstruction positions and report the clustering accuracy as follows.\n\n&nbsp; |DHA |  CCV | NUSWIDE | Caltech | YoutubeVideo\n----|----|----|----|----|----\nReconstruction on $\\mathbf{H}^v$  |80.9  |39.4  |60.4  |87.2  |31.3\nReconstruction on $\\mathbf{Z}^v$  |72.4  |27.8  |60.1  |86.6  |30.9\n\nWe can observe that the results of the reconstruction on $\\mathbf{Z}^v$ have some performance degeneration (especially on DHA and CCV datasets), compared with the results of the reconstruction on $\\mathbf{H}^v$. Therefore, the reconstruction of our framework is punished on $\\mathbf{H}^v$ instead of $\\mathbf{Z}^v$, for reducing the interference of semantic-irrelevant information to the contrastive learning on $\\mathbf{Z}^v$.\n\n>Q2: Why the reconstruction leads to a greater improvement needs a more detailed explanation in the experimental section. Besides, in terms of the method's naming, it would be beneficial to highlight the concept of the reconstruction component.\n\nIf there are no supervisory signals during data processing, useful discriminative information might lose before contrastive learning. The reconstruction regularization makes the hidden features maintain the discriminative information of data. Then, contrastive learning can access the discriminative information only when they are not lost. Information is well transmitted before it is used and thus the reconstruction leads to great improvements. The self-weighted module and the reconstruction regularization are both important for our method. Hence, we would like to rename our method as \"Self-weighted multi-view contrastive learning with reconstruction regularization\".\n\n\n>Q3: The presentation and captions of Figures 1 and 5 can be further improved, and it would be ideal to summarize the conclusions and connections among the data displayed in the three figures.\n\nWe will improve the presentation and captions of figures. To be specific, the visualization of Figures 2 and 5 are both carried out on the Caltech dataset. Figure 2 shows the representation learning process of traditional contrastive learning, and Figure 5 displays that of our proposed SEM whose framework is illustrated in Figure 1. For example, the useful pairwise views could be view 4 and view 5 and their contrastive learning is strengthened, and the unreliable pairwise views could be view 1 and view 4 and their contrastive learning is weakened.\n\n>Q4: This method utilizes a number of models equal to the scale of the views, and it also requires calculating pairwise weights, which significantly increases the memory usage and training time as the number of views increases. This paper does not include sufficient discussions about the computational cost and memory overhead, which are suggested to be included.\n\nThanks for this suggestion. To achieve multi-view contrastive learning, previous methods (such as CMC, DCP, MFLVC, and DSIMVC) also need different models to transform different views into the same form, as multi-view data typically involve heterogeneous data forms. Since the mini-batch optimization is adopted, the computational cost of our method (and comparison methods) is linear to the sample size. Additionally, the number of views for multi-view data is often no more than 6 in practical scenarios, which will not increase the unaffordable memory usage and computational burden when calculating pairwise weights. We have put the complexity analysis in Appendix A (Page 7). Additionally, the practical time cost of our method is also shown in Table 3 in Appendix C (Page 8)."}]

Author rebuttal:

Meta-review: This paper proposes a self-weighted multi-view contrastive learning framework that enables the model to adaptively enhance good pair-wise view representations by measuring the difference between representations. The results show that this method effectively alleviates the representation degradation problem of contrastive learning and significantly improves the performance of contrastive learning on public multi-view datasets. The paper is clearly written and well-analyzed. The novelty of this method mainly lies in the adaptive multi-view contrastive learning framework. The author's response successfully addresses some of the reviewers' concerns and provides additional (supportive) experimental results. The detailed settings of the experiment and grammatical errors need to be improved, and the related work of the paper can be further supplemented. Authors are encouraged to consider the reviewers' detailed comments in the final version. For the above reasons, the AC suggested accepting this paper.
Decision: Accept (poster)