Title: Self-Weighted Contrastive Learning among Multiple Views for Mitigating Representation Degeneration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-view learning

Contrastive learning

Representation degeneration

Self-supervised learning
Abstract: Recently, numerous studies have demonstrated the effectiveness of contrastive learning (CL), which learns feature representations by pulling in positive samples while pushing away negative samples. Many successes of CL lie in that there exists semantic consistency between data augmentations of the same instance. In multi-view scenarios, however, CL might cause representation degeneration when the collected multiple views inherently have inconsistent semantic information or their representations subsequently do not capture sufficient discriminative information. To address this issue, we propose a novel framework called SEM: SElf-weighted Multi-view contrastive learning with reconstruction regularization. Specifically, SEM is a general framework where we propose to first measure the discrepancy between pairwise representations and then minimize the corresponding self-weighted contrastive loss, and thus making SEM adaptively strengthen the useful pairwise views and also weaken the unreliable pairwise views. Meanwhile, we impose a self-supervised reconstruction term to regularize the hidden features of encoders, to assist CL in accessing sufficient discriminative information of data. Experiments on public multi-view datasets verified that SEM can mitigate representation degeneration in existing CL methods and help them achieve significant performance improvements. Ablation studies also demonstrated the effectiveness of SEM with different options of weighting strategies and reconstruction terms.

Self-Weighted Contrastive Learning among Multiple Views for Mitigating Representation Degeneration

Abstract

Recently, numerous studies have demonstrated the effectiveness of contrastive learning (CL), which learns feature representations by pulling in positive samples while pushing away negative samples. Many successes of CL lie in that there exists semantic consistency between data augmentations of the same instance. In multiview scenarios , however, CL might cause representation degeneration when the collected multiple views inherently have inconsistent semantic information or their representations subsequently do not capture sufficient discriminative information. To address this issue, we propose a novel framework called SEM: SElf-weighted Multi-view contrastive learning with reconstruction regularization . Specifically, SEM is a general framework where we propose to first measure the discrepancy between pairwise representations and then minimize the corresponding self-weighted contrastive loss, and thus making SEM adaptively strengthen the useful pairwise views and also weaken the unreliable pairwise views. Meanwhile, we impose a self-supervised reconstruction term to regularize the hidden features of encoders, to assist CL in accessing sufficient discriminative information of data. Experiments on public multi-view datasets verified that SEM can mitigate representation degeneration in existing CL methods and help them achieve significant performance improvements. Ablation studies also demonstrated the effectiveness of SEM with different options of weighting strategies and reconstruction terms.

1 Introduction

Contrastive learning (CL) explicitly enlarges the feature representation similarity between semanticrelevant samples, and it is adept at capturing high-level semantics while discarding irrelevant information. This learning paradigm has facilitated many research and application fields, such as visual representation [1, 2], text understanding [3, 4], and cross-modal agreement [5, 6, 7]. Samples with consistent semantics are typically constructed as positive sample pairs for CL loss ( e.g. , InfoNCE [8]), which motivates multi-view learning scenarios [9, 10] where researchers focus on exploring common semantics among multi-view data. However, this kind of data usually is with heterogeneous views and thus cannot be directly processed by previous CL methods with two shared network branches.

To handle this situation, many multi-view contrastive learning (MCL) methods [11, 12, 13, 14] have been proposed, which treats multiple views as positive sample pairs and achieves important progresses in exploring multi-view common semantics (see Sec. 2 for details). Nevertheless, we find that CL might cause representation degeneration that the representations of high-quality views tend to degenerate. This may make the MCL methods perform worse than the optimal single view (see Sec. 3.1 and Sec. 4.1), and thus heavily limiting the usability of MCL in practical scenarios. Although

several CL work [15, 16] proposed different CL losses aiming at increasing robustness to noise and made important advances on vision and graph data, our experiments discover that these CL losses are still fragile in multi-view scenarios as multi-view data are with more diversity than single-view data. Different from changing CL loss, recent MCL methods [14, 17] focused on changing model structures and successfully improved the effectiveness of clustering the learned representations. Nevertheless, representation degeneration still exists in many cases and it requires further solutions.

We find that there could be two reasons leading to representation degeneration in MCL. I) The quality difference among multiple views. The success of CL is based on the priori condition that the constructed positive sample pair has semantic consistency, which generally holds in previous CL applications [1, 5, 8]. Unfortunately, for multi-view learning, the collected views usually have quality difference and the semantic of positive sample pairs might be inconsistent due to view diversity. Consequently, CL causes the representation degeneration of high-quality views due to the existence of low-quality views. II) Losing discriminative information during data processing. Multi-view data typically involve heterogeneous data forms [9, 18], e.g. , different dimensions, modalities, and sparsity. For achieving MCL, the model needs to transform heterogeneous multi-view data into the same form with different encoders. However, data transformation could lose discriminative information as this process has no supervised signals for maintaining information. As a result, CL might miss multiple views' common semantics and focus on semantic-irrelevant information due to inductive bias.

To this end, we propose SElf-weighted Multi-view contrastive learning with reconstruction regularization (SEM) as shown in Figure 1 that takes the m,n,o -th views in V views as an example (where W m,n denotes the pairwise weight, L m,n CL is the contrastive loss, and Z m is the learned representations). Specifically, SEM minimizes self-weighted contrastive losses W m,n L m,n CL and W n,o L n,o CL after measuring the discrepancy between pairwise views' representations, i.e. , ( Z m , Z n ) and ( Z n , Z o ), respectively. This makes SEM adaptively strengthen CL between the useful pairwise views and also weaken CL between the unreliable pairwise views. Meanwhile, SEM takes self-supervised reconstruction objectives as regularization terms ( R m , R n , and R o ) on the hidden features ( H m , H n , and H o ) of encoders for individual views, respectively. This reconstruction regularization assists CL in accessing sufficient discriminative information hidden in raw input data ( X m , X n , and X o ), which could be implemented by exist-

m,n

n,o

CL

CL

strengthen

weaken

zm

wm,n

←z—

wn,o

←Z°

encoder

encoder

encoder

个

Hm

Hn

oH

Rm

Rn

R°

encoder

encoder

encoder

Xm

Xn

Xo

usefulpairwiseviews

unreliablepairwiseviews

ing information encoder-decoder models, e.g. , AE [19], DAE [20], and MAE [21]. In SEM, the representations and pairwise weights are alternatively updated to mutually enhance one another.

In summary, our contributions are: I) We propose a novel general framework SEM that leverages self-weighting and information reconstruction to address representation degeneration in MCL. II) We provide three options with different advantages to implement the weighting strategy of SEM including class mutual information, JS divergence, and maximum mean discrepancy. III) Theoretical and experimental analysis verified the effectiveness of SEM. It helps many CL methods ( e.g. , InfoNCE [8], RINCE [15], and PSCL [16]) achieve significant performance improvements in multi-view scenarios.

2 Related Work

Contrastive learning (CL) As a popular self-supervised learning paradigm, CL focuses on learning semantically informative representations for downstream tasks [22, 23, 24, 25]. The most widely used loss function is InfoNCE [8] which pulls in the representations between positive sample pairs while pushing away that between negative sample pairs. Some work have attempted to explain the reasons for the success of applying InfoNCE, e.g. , from perspectives of mutual information [8, 26], task-dependent view [27], or deep metric learning [28, 29]. Furthermore, [30, 31] pointed out to conduct CL with reconstruction regularization to achieve robust representations for downstream tasks. RINCE [15] (a short name of Robust InfoNCE) is a variant of InfoNCE contrastive loss that considers noise in false positive sample pairs. The recent work [16] investigates CL without

(a) ACC

vs.

Views

(b) Loss and similarity

vs.

Epoch

(c) ACC

vs.

Epoch

80

6.4

1.0

80

70

75

6.2

60

0.9

70

(%)

50

values

6.0

Similarities

ACC (%)

65

View 1 w/o CL

ACC

C

40

30

LoSS

5.8

Cosine similarity (View 1;View 4)

0.8

55

60

View 4w/o CL

View 1 w/CL

View 4 w/ CL

20

5.6

0.7

50

45

10

5.4

0.6

40

0

View1View2View3View4View5View6

0

100

200

300

400

0

100

200

300

400

Trainingepochs

Trainingepochs

conditional independence assumption on positive sample pairs and proposes a population spectral contrastive loss (we call it PSCL for short). Despite important progresses have been made, in this work, we discover that these CL losses are still fragile in multi-view scenarios where data qualities are hard to be guaranteed, and even the reconstruction regularized CL is not enough.

Multi-view contrastive learning (MCL) Different from many CL methods that usually generate two inputs by data augmentation [32], MCL aims to handle multi-view data widely exiting in real-world applications. Multi-view data often contain more than two views/modalities and they naturally form multiple inputs [33, 34, 35]. Since the semantic consistency among multiple views is not guaranteed, it is challenging to capture the useful information in multi-view data, while considering the side effects of harmful information. Therefore, MCL attracts increasing attention in recent years [36, 37, 38]. For example, CMC [11] empirically shows that MCL performed with more scene views obtains the better representations with semantic information. DCP [39] leverages the maximization of mutual information to conduct consistency learning across different views and aims to achieve a provable sufficient and minimal representation. MFLVC [14] observes the conflict between consistency and reconstruction objectives in encoder-decoder frameworks and proposes to learn multi-level features for multiple views. DSIMVC [17] establishes a theoretical framework to reduce the risk of clustering performance degradation from semantic inconsistent views. Although satisfactory results are achieved in many cases, the representation degeneration caused by CL is still not well considered and addressed. In this paper, we point out that the representation degeneration could seriously limit the application of CL in multi-view scenarios, and propose the discrepancy-based self-weighted MCL to address it.

Notations This paper leverages bold uppercase characters and bold lowercase characters to denote matrices and vectors, respectively. Operator ∥ · ∥ 2 denotes vector ℓ 2 -norm and operator ∥ · ∥ F is matrix F -norm. { x v i ∈ X v } v =1 , 2 ,...,V i =1 , 2 ,...,N denotes the multi-view dataset with N samples in V views.

3 Methodology

This section first illustrates the phenomenon of representation degeneration in multi-view contrastive learning. To address this issue, we then establish a general framework of SEM: SElf-weighed Multiview contrastive learning with reconstruction regularization . To implement the SEM framework, we further provide different options of weighing strategy, contrastive loss, and reconstruction term.

3.1 Motivation: Representation Degeneration in Multi-View Contrastive Learning

Researchers proposed many contrastive learning approaches and also achieved plenty of progress in multi-view learning. However, multi-view contrastive learning might result in the representation degeneration of high-quality views ( i.e. , those views contain rich semantic information) due to the diversity of multi-view data. Specifically, we illustrate it in Figure 2 that takes a popular multi-view dataset Caltech [40] (6 views) as an example. We leverage unsupervised linear clustering accuracy obtained by K-Means [41] to evaluate the representation quality of containing class-level semantics.

Firstly, we leverage self-supervised autoencoders (the setting is shown in Appendix B) to pretrain the representations of each view's data. In Figure 2(a), one can find that different views inherently have different levels of discriminative information and exhibit different qualities, where the worst (view 1)

and the best (view 4) have a large gap. Then, we adopt InfoNCE loss to perform contrastive learning between view 1 and view 4 in Figure 2(b), and record the clustering accuracy of their representations in Figure 2(c). We can observe that InfoNCE loss is well-minimized, which makes the representation similarity (evaluated by cosine) between view 1 and view 4 converge to 1 . 0 . The performance on view 1 gradually increases. Nevertheless, the cost is that the representations of view 4 degenerate, on which the useful discriminative information reduces and thus the performance gradually decreases.

In multi-view learning, quality difference among multiple views is a common phenomenon. However, the representation degeneration in multi-view contrastive learning might make the representations of some high-quality views tend to be mediocre and thus miss their useful discriminative information.

3.2 Self-Weighted Multi-View Contrastive Learning with Reconstruction Regularization

To mitigate representation degeneration in multi-view contrastive learning, we propose a simple but effective framework called SEM: SElf-weighted Multi-view contrastive learning with reconstruction regularization as shown in Figure 1. Specifically, given view-specific data X v ∈ R N × d v , we let Z v ∈ R N × z denote the corresponding new representations learned by a view-specific encoder. Between X v and Z v , we record a precursor state of representations as H v ∈ R N × h v (termed as hidden features), and the encoder is partitioned into two parts (the front and back parts are stacked and denoted as f v and g v sequentially). For the v -th view, we let Ψ v and Φ v denote the network parameters of f v and g v , respectively, and then the view-specific model can be formulated as follows:

In SEM, we leverage L m,n CL ( Z m , Z n ) to denote a contrastive loss 2 , and let λ > 0 denote a trade-off coefficient on regularization terms. Then, SEM is trained by minimizing the following objective:

where W m,n is the pairwise weight between the m -th and the n -th views, and R v ( X v , H v ) denotes the reconstruction regularization on H v . We define D ( Z m , Z n ) as the discrepancy between Z m and Z n and denote F as a negative correlation function. Then, in SEM, the pairwise weight is updated by

Self-weighting In unsupervised settings, it is hard to know which representations within { Z v } V v =1 contain useful semantic information and which are with more noise. To mitigate the representation degeneration caused by contrastive learning, SEM needs to be adaptive to quality difference among multiple views. Therefore, different from using equal-sum manner [11, 14, 17] ( e.g. , ∑ m,n L m,n CL ), we propose to use the pairwise weighted multi-view contrastive loss, i.e. , ∑ m,n W m,n L m,n CL . Here, W m,n leverages the discrepancy to achieve the adaptive self-weighting. Concretely, if two views are useful pairwise views and both with informative semantics, contrastive learning between them is adaptively strengthened; if two views are unreliable pairwise views (for example, one or two of them are with less informative semantics), contrastive learning between them is adaptively weakened.

Reconstruction regularization In Eq. (2), R v ( X v , H v ) acts as a self-supervised objective to transfer as much discriminative information as possible from X v to H v . When we record H v as the hidden features in encoder networks, the information transfer path can be described as X v → H v → Z v , v ∈ { 1 , 2 , . . . , V } . However, information losing might occur in the processing of X v → H v such that discriminative information from some views' data is lost, and thus making contrastive learning among { Z v } V v =1 focus on harmful noise instead of common semantics across multiple views. To this end, on hidden features H v , our SEM leverages X v to build the reconstruction regularization R v ( X v , H v ) to assist contrastive learning in accessing sufficient discriminative information from raw data.

where s + ( s -) denotes the cosine distance between the representations of positive (negative) sample pair.

3.3 Different Options for Implementing the SEM Framework

The crucial components of our proposed SEM as Eq. (2) include the weighting strategy W m,n , contrastive loss L m,n CL , and regularization term R v . Next, we concentrate on the implementations of W m,n (including JSD, MMD, and CMI) and briefly introduce the implementations of L m,n CL and R v .

Discrepancy measurements of weighting strategy When implementing W m,n = F ( D ( Z m , Z n )) in Eq. (3), many methods can measure the discrepancy D ( Z m , Z n ) . Firstly, we can transfer representations to a probability distribution and leverage Jensen-Shannon divergence (JSD) to compute the discrepancy D JSD ( Z m , Z n ) . The advantages of JSD are its symmetry and simplicity, but it might be inapplicable when two distributions are non-overlapping. Furthermore, we can leverage maximum mean discrepancy (MMD) as the second method to obtain the discrepancy D MMD ( Z m , Z n ) . MMD can effectively measure non-overlapping two distributions, but it has higher complexity than JSD 3 .

Actually, both JSD and MMD leverage all information of representations Z m and Z n . However, the semantic-irrelevant information or random noise might also be embedded in Z m and Z n . Moreover, what we expect to obtain is the mutual relation of their most representative semantic information. To this end, we propose Class Mutual Information (CMI) as the third method to obtain the discrepancy D CMI ( Z m , Z n ) . To be specific, since it is difficult to accurately estimate the mutual information (denoted as I ) for multi-dimensional continuous variables I ( Z m ; Z n ) , we denote y m and y n as 1 -dimensional discrete vectors and change estimating I ( Z m ; Z n ) to computing I ( y m ; y n ) such that:

Intuitively, discrete class information in Z v ( v ∈ { m,n } ) is 1 -dimensional as well as the most representative information. Hence, we can optimize K-Means objective to extract the class information:

where C v ∈ R K × z denotes the K cluster centers of Z v . Y v ∗ ∈ { 0 , 1 } N × K is the indicator matrix that can be further transformed to 1 -dimensional discrete vector y v by defining y v i := argmax j y v ∗ ij where y v i ∈ y v , y v ∗ ij ∈ Y v ∗ . In this way, the class information in Z m and Z n can be compressed into y m and y n , respectively. Then, the class mutual information I ( y m ; y n ) is normalized and the discrepancy measurement D CMI ( Z m , Z n ) between pairwise views is defined as follows:

where H ( y m ) = -∑ N i =1 p ( y i m ) log p ( y i m ) is the cross-entropy of y m . This design of CMI has at least two advantages: 1) It is conducive to maintaining the representative class information while filtering out noise information; 2) Calculation is easy and owns better physical meaning.

Finally, it is also flexible to implement the negative correlation function F . Considering W m,n ≥ 0 , we base on the three different discrepancies and simply give the following weighting strategies:

Compatibility for contrastive learning When implementing the contrastive loss L m,n CL , it should be pointed out that multi-view contrastive learning usually has to handle more than two views ( i.e. ,

̸

{ Z v } V v =1 , V > 2 ), which is different from two-view setting ( e.g. , { Z 1 , Z 2 } ) in traditional contrastive learning. To make our SEM framework be compatible with previous contrastive learning methods, we construct positive/negative sample pairs as follows. Specifically, for two views { z i m ∈ Z m , z n j ∈ Z n } , the positive sample pairs are { z i m , z n i } i =1 ,...,N ; for any z i m , its negative sample pairs are { z i m , z v j } v = m,n j = i . Cosine with a temperature parameter τ is leveraged to measure the representation distance between pairs, i.e. , s = 1 /τ · ⟨ z i m , z n j ⟩ / ∥ z i m ∥ 2 ∥ z n j ∥ 2 . Then, we compute the contrastive loss between two views and sum all combinations as Eq. (2). We formulated three contrastive losses in Sec. 3.2, and the experiments in Sec. 4.1 will verify the compatibility of our SEM framework to them.

Reconstruction regularization When implementing the regularization term R v ( X v , H v ) in Eq. (2), we are motivated by the information encoding-decoding process [14, 19, 30], and stack a view-specific decoder f v -with network parameter Ω v on each view's H v to perform data recovery of X v . In this way, the regularization term in SEM can be implemented with the reconstruction loss of autoencoders 4 , whose encoder-decoder models can make hidden features preserve discriminative information of data. When decoder generally rebuilds X v with H v , we can believe that H v compresses the sufficient information of X v , for promoting contrastive learning fully access discriminative information of data.

The training steps of SEM is summarized in Algorithm 1, where representations and weights are updated alternatively to make them promote each other. E denotes total training epochs, and the step size S denotes the number of training epochs after each update of pairwise weights. As we cannot obtain meaningful { y v } V v =1 before we start training neural networks, we first obtain meaningful { H v } V v =1 by pre-training the model with Eq. (8), and then initialize {W m,n } V m,n =1 with { H v } V v =1 .

3.4 Theoretical Analysis

In this part, we theoretically analyze the mechanism of SEM in exploring mutual information among multiple views while mitigating representation degeneration. All proofs are given in Appendix A.

Considering SEM with InfoNCE loss and CMI weighting strategy, we have the following theorem indicating that minimizing the self-weighted contrastive loss keeps maximizing the mutual information between useful pairwise views, as well as avoiding the effects between unreliable pairwise views.

Theorem 1. For any three views ( v ∈ { m,n,o } ), if class mutual information only exists in two views, e.g., I ( y m ; y o ) → 0 , I ( y n ; y o ) → 0 , and I ( y m ; y n ) = δ , δ > 0 , we have minimizing the weighted InfoNCE losses W m,n L m,n InfoNCE ( Z m , Z n )+ W m,o L m,o InfoNCE ( Z m , Z o )+ W n,o L n,o InfoNCE ( Z n , Z o ) is equivalent to maximizing the mutual information between the two views ( e δ/ log N -1) I ( Z m ; Z n ) .

Combining with the information losing of each layer through encoder networks, the following theorem further reveals that reconstruction regularization on the hidden features H v is conducive to alleviating the losing of discriminative semantic information through data transformation. Hence, we treat the layer output closest to Z v in encoders as hidden features to maximize ∏ L m l = t m +1 (1 -γ l m ) and ∏ L n l = t n +1 (1 -γ n l ) , aiming at maintaining useful semantic information for contrastive learning.

Theorem 2. For any two views ( v ∈ { m,n } ) with positive class mutual information, denoting L v as the total layer number of the v -th view's encoder network before representation Z v , the l -th layer has the information losing rate γ v l ≥ 0 . If S is an oracle variable that contains and only contains multiple views' discriminative semantic information, and H v is the t v -th layer's features, we have minimizing the regularized loss W m,n L m,n InfoNCE ( Z m , Z n ) + λ ∑ v R v ( X v , H v ) is expected to obtain I ( S ; Z m ; Z n ) ≤ min { I ( S ; X m ) · ∏ L m l = t m +1 (1 -γ l m ) , I ( S ; X n ) · ∏ L n l = t n +1 (1 -γ n l ) } .

4 Experiments

This section validates the effectiveness of our SEM. Specifically, we first conduct comparison experiments on state-of-the-art contrastive learning baselines and SEM with three options of contrastive losses ( i.e. , L InfoNCE , L PSCL , L RINCE ). We then conduct ablation studies with three options of weighting strategies ( i.e. , W CMI , W JSD , W MMD ), as well as with three options of reconstruction terms ( i.e. , R AE , R DAE , R MAE ). Evaluation is built on the concatenation of all views' representations learned by methods. Finally, we show SEM's training process and its hyper-parameter analysis. We provided more experimental results as well as all implementation details of SEM in Appendix.

Datasets Our experiments employ five open-source multi-view datasets. Their information is shown in Table 1, where DHA [43] is a depth-included human action dataset where each action has RGB and depth features; CCV [44] refers to the columbia consumer video database whose samples are described with SIFT, STIP, and MFCC features; NUSWIDE [45] collects web images with multiple views (color histogram, block-wise

color moments, color correlogram, edge direction histogram, and wavelet texture); Caltech [40] is a widely-used image dataset which leverages six views (Gabor, Wavelet moments, CENTRIST, HOG, GIST, and LBP) to represent samples; YoutubeVideo [46] is a large-scale dataset where each sample has three views including cuboids histogram, HOG, and vision misc. These datasets are diverse in forms and are often organized to comprehensively evaluate the performance of multi-view methods.

4.1 Comparison Experiments on Contrastive Learning

Baselines K-Means-BSV denotes K-Means clustering results on the best single-view of raw data, and we leverage this baseline to investigate the representation degeneration in comparison methods. InfoNCE [8], PSCL [16], and RINCE [15] are three kinds of CL methods. Since their original versions are designed to handle single views, we extended them to multi-view scenarios as did in [11, 17]. CMC[11], DCP [39], MFLVC [14], and DSIMVC [17] are four kinds of MCL methods. We evaluate our SEM with different contrastive losses ( i.e. , SEM+InfoNCE, SEM+PSCL, and SEM+RINCE), where the weighting strategy and reconstruction term are fixed to W CMI and R AE , respectively.

We leverage the linear clustering method K-Means to evaluate the performance of learning representations and report the average results of 10 runs in Table 2. The results indicate that: I) Our SEM framework is compatible with different contrastive losses ( e.g. , InfoNCE, PSCL, and RINCE) and we can clearly observe that SEM+InfoNCE/PSCL/RINCE successfully improve the baselines for large margins. For instance, SEM+InfoNCE respectively outperforms InfoNCE by about 25% , 13% , 4% , 7% , 12% ACC on the five datasets. II) MCL approaches could access the semantic information from multiple views, and thus outperforming that from single views. However, a side effect is

that contrastive learning directly increases the feature representation similarity of multiple views, which might obscure useful discriminative information hidden in high-quality views and lead to the representation degeneration. For example, on DHA and Caltech, results on many MCL methods ( e.g. , PSCL, RINCE, CMC, and MFLVC) are worse than the single-view baseline K-Means-BSV. III) Our SEM not only outperforms all these MCL methods but also mitigates the representation degeneration in MCL, e.g. , SEM+PSCL respectively outperforms K-Means-BSV by about 3% , 20% , 18% , 1% , 16% ACC on the five datasets. This is because the framework of SEM is adaptive to multiple views' qualities, which can reduce the side effect between unreliable views with inconsistent information, for better extracting discriminative information and consistent semantics among useful views.

Additionally, we leverage the linear classification method SVM [47] to evaluate the performance of MCL methods to learn representations, where we only use 30% of the learned representations for the training set and the rest for the test set. Figures 3 and 4 show the classification performance on DHA and CCV, respectively. Our SEM improves the baseline methods (especially for PSCL

80

ACC

Recall

0

PSCL

RINCE

CMC

DCP

MFLVC

DSIMVCSEM+InfoNCESEM+PSCLSEM+RINCE

Classfication accuraey (%)

50

ACC

40

Recall

30

20

10

0

PSCL

RINCE

CMC

DCP

MFLVC

DSIMVCSEM+InfoNCESEM+PSCLSEM+RINCE

and RINCE) and consistently outperforms other MCL methods (such as CMC, DCP, MFLVC, and DSIMVC). Since contrastive learning usually discards the information which is irrelevant to optimization objectives, the results further indicate that the representations learned by MCL are classification-friendly, which generally focus on catching class-level semantics among multiple views.

4.2 Ablation Experiments on Self-Weighting Strategy and Reconstruction Regularization

This part presents the ablation experiments to investigate the effectiveness of different weighting strategies W CMI/JSD/MMD and reconstruction terms R AE/DAE/MAE in our SEM framework.

DHA

71.3

↑

80.9 (

↑

80.5 (

↑

SEM w/o

SEM w/

SEM w/

SEM w/

W

W

W

W

CMI

JSD

MMD

84.4 (

9.6)

9.2)

13.1)

CCV

33.5

↑

39.4 (

↑

35.6 (

33.9 (

↑

5.9)

2.1)

0.4)

Table 3 reports the linear clustering performance (evaluated by ACC) of our SEM framework without self-weighting strategy ( i.e. , SEM w/o W ) and that with three weighting strategies ( i.e. ,

(a) Weight

vs.

Iteration

(b) Loss

vs.

Epoch

(c) ACC

vs.

Iteration

90

1.6

6.2

1.4

80

6.0

1.2

values

70

1.0

5.8

0.8

Loss

C

60

0.6

5.6

50

View 1 w/ CL

0.4

二

5.4

40

View 4 w/ CL

Wl,4

View5w/CL

0.2

1

2

3

4

0

100

200

300

400

0

2

3

4

Iteration

Trainingepochs

Iteration

W CMI , W JSD , W MMD ), where the contrastive loss and reconstruction term are fixed to L InfoNCE and R AE , respectively. Compared with SEM w/o W (this setting is the reconstruction regularized multi-view contrastive learning) that equally treats contrastive learning between any two views, SEM w/ W CMI/JSD/MMD can adaptively weight the contrastive learning according to specific two views and thus all these three variants of SEM obtain significant improvements. For example, SEM w/ W MMD has a 13.1 % improvement on DHA and SEM w/ W CMI has a 5.9 % improvement on CCV. Results on more datasets and time costs are shown in Appendix C, where we find that the proposed weighting strategy of class mutual information W CMI generally achieves the best performance on accuracy and time consumption among the three options of weighting strategy.

Table 4 reports the linear clustering performance (evaluated by ACC) of our SEM framework without reconstruction regularization ( i.e. , SEM w/o R ) and that with three reconstruction terms ( i.e. , R AE , R DAE , R MAE ), where the contrastive loss and weighting strategy are fixed to L InfoNCE and W CMI , respectively. We can easily find that the proposed SEM with reconstruction terms obviously outperforms that without reconstruction terms. For instance, compared with SEM w/o R , SEM w/ W MAE has 22.5 % and 10.8 % improvements on DHA and CCV, respectively. This is because the reconstruction regularization makes the hidden features { H v } V v =1 avoid losing discriminative information, which promotes the multi-view contrastive learning performed on subsequent { Z v } V v =1 . Meanwhile, SEM w/ R MAE and SEM w/ R DAE perform better than SEM w/ R AE . This is because, compared with vanilla AE, DAE or MAE (by adding noise or masking on raw data) can make our model more conducive to removing semantic-irrelevant noise as well as capturing hidden patterns.

4.3 Experimental Analysis on Mechanism of SEM

This part presents the visualization and analysis on SEM to give an intuition of its behavior and mechanism, where the combination of L InfoNCE + W CMI + R AE is taken as an example.

Let's first recall the views of Caltech dataset in Figure 2(a), we can consider that view 4 and view 5 are high-quality views, while view 1 is a low-quality view. The performance relation among them is ACC view 4 > ACC view 5 > ACC view 1 . In Figure 2(c), view 4's representation degeneration occurs.

Figure 5 shows the pairwise weights, losses, and clustering accuracy on Caltech dataset during SEM's training process, where 1 iteration corresponds to 100 epochs, i.e. , the step size is set to 100 epochs. Our SEM is a self-weighted multi-view contrastive learning framework that automatically infers different weights for different pairwise views as shown in Figure 5(a), where we can observe that weights W 4 , 5 > W 1 , 4 and they were dynamically updated for 4 times. As a result, contrastive learning between view 4 and view 5 is strengthened by W 4 , 5 , while contrastive learning between view 1 and view 4 is weakened by W 1 , 4 . Meanwhile, loss L 4 , 5 InfoNCE is minimized earlier than loss L 1 , 4 InfoNCE as shown in Figure 5(b). In other words, since the mutual effect between view 4 and view 5 is strengthened, the effect of view 1 on view 4/view 5 is weakened such that view 4/view 5 does not degenerate. At the same time, the effect of view 4/view 5 on view 1 remains and promotes the representation learning of view 1. Consequently, all views' performance in Figure 5(c) increases through our SEM, and the representation degeneration of view 4 occurring in Figure 2(c) is mitigated.

Hyper-parameter analysis Since different datasets have different levels of reconstruction errors, the trade-off coefficient λ is introduced to balance the contrastive learning and information recovery in our SEM framework. In Figure 6(a), we change λ within the range of [10 -3 , 10 -2 , 10 -1 , 10 0 , 10 1 , 10 2 , 10 3 ] and report the clustering accuracy tested on representations. The experimental results indicate that SEM is not sensitive to λ in [10 -1 , 10 1 ] . In our experiments, λ is consistently set to 1 for all the five datasets. Regarding self-supervised learning, frameworks with fewer manually set hyper-parameters might be more convenient for their applications.

Additionally, we investigate the effect of cluster number when the weight strategy of our SEM framework is selected as W CMI which needs to pre-define the cluster number when applying K-Means algorithm. As shown in Figure 6(b), when computing the class mutual information, we change the number of clusters within the range of [ K/ 2 , K, 2 K, 4 K ] where K denotes the truth class number of

Figure 6: (a) ACC

vs. λ

.

(b) ACC

vs. K

.

90

80

70

100

80

60

ACC (%)

60

50

40

ACC

40

20

30

0

20

DHA

Caltech

Caltech

NUSWIDE

DHA

101

102

103

10

0K/2

CCV

NUSWIDE

YoutubeVideo

YoutubeVideo

CCV

103

10-2

10-1

100

入

Numberofclusters

K

2K

4K

multi-view datasets. Compared with K , K/ 2 leads to more coarse-grained class mutual information, while 2 K and 4 K come in more fine-grained class mutual information. The experimental results demonstrate that SEM with W CMI is not sensitive to the choices of cluster number.

5 Conclusion

In this paper, we showcase that the representation degeneration could seriously limit the application of contrastive learning in multi-view scenarios. To mitigate this issue, we propose self-weighted multiview contrastive learning with reconstruction regularization (SEM), which is a general framework that is compatible with different options of the contrastive loss, weighting strategy, and reconstruction term. Theoretical and experimental analysis verified the effectiveness of SEM, and it can significantly improve many existing contrastive learning methods in multi-view scenarios. Moreover, ablation studies indicated that SEM is effective with different weighting strategies and reconstruction terms.

Our future work is to extend the proposed SEM to be useful not only for multi-view scenarios, but also for other contrastive learning based domains, such as contrastive learning in sequences. Conceptually, the limitation of the self-weighting strategy is that it is more effective when there are over two views. When there are only two views, the self-weighted multi-view contrastive learning framework transforms into traditional contrastive learning but with reconstruction regularization. Therefore, another future work is to extend the view-level weighting of SEM to sample-level weighting.

Acknowledgment

This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA1004100, in part by the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China under Grant ZYGX2022YGRH009 and Grant ZYGX2022YGRH014, in part by the National Natural Science Foundation of China under Grant 62276052.

References

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607, 2020.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 6894-6910, 2021.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763, 2021.

Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12475-12486, 2021.

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6995-7004, 2021.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint arXiv:1304.5634 , 2013.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory , pages 1179-1206, 2021.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision , pages 776-794, 2020.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning , pages 4116-4126, 2020.

Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. Advances in Neural Information Processing Systems , 34:2148-2159, 2021.

Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi-view clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16051-16060, 2022.

Ching-Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16670-16681, 2022.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Huayi Tang and Yong Liu. Deep safe incomplete multi-view clustering: Theorem and algorithm. In International Conference on Machine Learning , pages 21090-21110, 2022.

Yang Yang, Chubing Zhang, Yi-Chu Xu, Dianhai Yu, De-Chuan Zhan, and Jian Yang. Rethinking label-wise cross-modal retrieval from a semantic sharing perspective. In Proceedings of the International Joint Conference on Artificial Intelligence , pages 3300-3306, 2021.

Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in Neural Information Processing Systems , 6:3-10, 1993.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning , pages 1096-1103, 2008.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision , 130(9):22052221, 2022.

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in Neural Information Processing Systems , 33:87658775, 2020.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems , 33:5812-5823, 2020.

Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng. Unsupervised contrastive cross-modal hashing. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3877-3889, 2022.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations , 2019.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems , 33:6827-6839, 2020.

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems , 29:1857-1865, 2016.

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations , 2020.

Shuo Chen, Chen Gong, Jun Li, Jian Yang, Gang Niu, and Masashi Sugiyama. Learning contrastive embedding in low-dimensional space. Advances in Neural Information Processing Systems , 35:6345-6357, 2022.

Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16041-16050, 2022.

AJaiswal, A Ramesh Babu, M Zaki Zadeh, D Banerjee, and F Makedon. A survey on contrastive self-supervised learning. Machine Learning , 12:4182-4192, 2020.

Changqing Zhang, Zongbo Han, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, et al. CPMNets: Cross partial multi-view networks. Advances in Neural Information Processing Systems , 32:559-569, 2019.

Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning , pages 17782-17800, 2022.

Yang Yang, Jingshuai Zhang, Fan Gao, Xiaoru Gao, and Hengshu Zhu. DOMFN: A divergenceorientated multi-modal fusion network for resume assessment. In Proceedings of the ACM International Conference on Multimedia , pages 1612-1620, 2022.

Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1134-1143, 2021.

Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 754-763, 2021.

Jiyuan Liu, Xinwang Liu, Yuexiang Yang, Qing Liao, and Yuanqing Xia. Contrastive multi-view kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(8):95529566, 2023.

Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4447-4461, 2022.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop , pages 178-178, 2004.

John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 28(1):100-108, 1979.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yan-Ching Lin, Min-Chun Hu, Wen-Huang Cheng, Yung-Huan Hsieh, and Hong-Ming Chen. Human action recognition and retrieval using sole depth information. In Proceedings of the ACM International Conference on Multimedia , pages 1053-1056, 2012.

Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, and Alexander C Loui. Consumer video understanding: A benchmark database and an evaluation of human and machine performance. In Proceedings of the ACM International Conference on Multimedia Retrieval , pages 1-8, 2011.

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. NUSWIDE: a real-world web image database from national university of singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval , pages 1-9, 2009.

Omid Madani, Manfred Georg, and David A. Ross. On using nearly-independent feature families for high precision and confidence. Machine Learning , 92:457-477, 2013.

Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning , 20:273-297, 1995.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the limitations of contrastive learning (CL) in multi-view scenarios and proposes a novel framework called SElf-weighted Multi-view contrastive learning (SEM) to address these limitations. The contributions of SEM framework are as follows:\n- Alleviating representation degeneration: In multi-view scenarios, CL can lead to representation degeneration when the collected views have inconsistent semantic information or lack sufficient discriminative information. SEM aims to mitigate this issue by adaptively strengthening useful pairwise views and weakening unreliable pairwise views through a self-weighted contrastive loss.\n- Regularizing hidden features: SEM introduces a self-supervised reconstruction term to regularize the hidden features of encoders. This regularization assists CL in accessing sufficient discriminative information from the data.\n- Extensive experimental validation: Experiments on public multi-view datasets demonstrate that SEM effectively mitigates representation degeneration in existing CL methods and leads to significant performance improvements. Ablation studies further verify the effectiveness of SEM with different options of weighting strategies and reconstruction terms.\n\nSTRENGTHS:\n- The proposal of SEM is well-motivated: SEM tackles the issue of representation degeneration in multi-view scenarios where inconsistent semantic information and insufficient discriminative information exist.\n- Flexibility in weighting strategies: SEM provides three options for implementing the weighting strategy, including class mutual information, JS divergence, and maximum mean discrepancy.\n- Significant performance improvements and thorough ablation studies: Experimental results on 5 public multi-view datasets, component ablation studies and hyper-parameter analysis demonstrate that SEM effectively mitigates representation degeneration in existing contrastive learning (CL) methods.\n\nWEAKNESSES:\n- I have some concerns regarding the rationale behind introducing the reconstruction module. If the aim of incorporating reconstruction is to enhance the discriminative information of the representation, why does the process involve an additional encoder after obtaining the representation H, instead of directly reconstructing the representation z? It would be beneficial to provide more details in the experimental results regarding the position of the reconstruction within the overall encoder and further analyze the relationship between discriminative information and the resulting performance improvement in both the method and experimental sections, thus solidifying the framework. \n- (minor) Based on the experimental findings, it appears that the enhancement from the reconstruction regularization is more pronounced compared to the improvement stemming from the self-weighted module. Why the reconstruction leads to a greater improvement needs a more detailed explanation in the experimental section. Besides, in terms of the method's naming, it would be beneficial to highlight the concept of the reconstruction component.\n- (minor) The presentation and captions of Figures 1 and 5 can be further improved, and it would be ideal to summarize the conclusions and connections among the data displayed in the three figures.\n\nQUESTIONS:\nThis method utilizes a number of models equal to the scale of the views, and it also requires calculating pairwise weights, which significantly increases the memory usage and training time as the number of views increases. The authors seem to lack detailed comparative figures with previous methods in this regard.\n\nLIMITATIONS:\nThis paper does not include sufficient discussions about the computational cost and memory overhead, which are suggested to be included.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper researches the representation degeneration of multi-view contrastive learning. To address it, this paper proposes a simple but effective framework of self-weighted multi-view contrastive learning.

STRENGTHS:
++The manuscript is well-written and self-consistent. For example, the visualization analysis makes it easy for the reader to understand that considering view differences is necessary for multi-view data.

++The motivation of this work is reasonable, and there are sufficient ablation experiments and theoretical analyses to verify the effectiveness of the proposed SEM algorithm.

++SEM is a general framework that can be adapted to a variety of existing contrastive learning losses, as well as to a variety of autoencoder models.

WEAKNESSES:
--Figure 5(a) shows that weights are updated dynamically in different iterations. Here, are the weights incrementally and linearly increased or are they only updated 4 times?

--The class mutual information weighting strategy is an interesting method that aims to easily measure the discrepancy between views, which compresses the representative class-semantic information of features into one-hot labels. As thus, Eq. (5) seems to be missing some constraints of $\mathbf{Y}^{v*}$ of being the labels.

QUESTIONS:
In Figure 5(a), are the weights incrementally and linearly increased or are they only updated 4 times?

LIMITATIONS:
The authors adequately addressed the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors show that the representation degradation could limit the application of contrastive learning in multi-view scenarios. To mitigate this issue, they propose the self-weighted multi-view contrastive learning, a general framework that has different options in the contrastive loss, weighting strategy, and reconstruction term. In my opinion, this paper does a good job of showing what is emphasized, with many strengths. I've listed them below, along with some possible weaknesses.\n\nSTRENGTHS:\n#1. The manuscript flows smoothly and is easy to understand. The analysis in the manuscript is well-thought-out and facilitates understanding the motivation and methodology.\n#2. Authors show a new multi-view contrastive learning framework that considers handling representation degeneration via self-weighting and information reconstruction.\n#3. The framework is technically sound. The manuscript provides three variants of weighting strategy including class mutual information, Jensen-Shannon divergence, and maximum mean discrepancy, and they are with different advantages.\n#4. The framework helps existing contrastive learning methods, like InfoNCE, achieve significant performance improvements in multi-view scenarios.\n#5. Sufficient ablation experiments and implementation details are provided.\n\nWEAKNESSES:\n#1. There is a technical detail that needs to be clarified. SEM adaptively strengthen contrastive learning between the useful pairwise views and also weaken contrastive learning between the unreliable pairwise views. This process can be conducted without supervision by the designed self-weighting framework. But after training, we still don't know which of the representations of multiple views is a good representation. Are the representations learned from useful pairwise views artificially selected to evaluate the performance of downstream clustering tasks?\n#2. It would be better for related work to include some recent jobs of contrastive learning. \n#3. The abbreviation should be interpreted, such as SIFT, STIP, and MFCC.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed social impacts and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed the SEM: SElf-weighted Multi-view contrastive learning framework, which first performs discrepancy measures on representations, and then obtains weights to assist adaptive contrastive learning. Meanwhile, the decoders are leveraged to avoid losing discriminative information. Extensive experiments are conducted to verify the effectiveness of SEM and its different implementation options.\n\nSTRENGTHS:\nIn unsupervised environments, it is hard but crucial for multi-view learning to automatically know which view’s features are with useless noise and which view’s features contain useful semantic information. I believe the idea of pair-wise self-weighted contrastive learning is novel. The proposed method obtains the adaptive ability to quality differences among multiple views, and it doesn't require much prior knowledge.\n\nMoreover, multi-view representation learning is important for working with multi-view data as raw views usually have inconsistent semantic meaning when one considers them in practical applications. In this paper, the authors present a robust multi-view contrastive learning method, whose effectiveness is verified by extensive experiments.\n\nWEAKNESSES:\n(1) To reduce losing information, the proposed method treats the last layer of encoders as hidden features. It's not clear exactly which layer is as H, and which layer is as Z.\n\n(2) When using the weighting strategies, the framework needs reconstruction pretraining to obtain meaningful representations. The decoder is added to the framework as an auxiliary module. As far as I known, many self-supervised multi-view methods also use autoencoders as the main representation learning module. Some ablation experiments are expected.\n\nQUESTIONS:\nCan the authors show the performance of representation learning relying solely on autoencoders to understand the individual contributions between contrastive learning and reconstruction objectives?\n\nLIMITATIONS:\nThis research is not expected to introduce new negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
In summary, this paper investigates an important question about how to mitigate representation degradation in multi-view contrastive learning. Considering the quality difference of views and losing useful information during nets, the proposed method uses adaptive weighted contrastive learning and adds information reconstruction to improve the performance of contrastive learning in multi-view learning.

STRENGTHS:
I) The inductive bias of contrastive learning may allow the newly learned representation to capture trivial information, thus causing representation degradation. So, it is meaningful to propose an effective framework to alleviate this issue in this paper.

II) The authors proposed a novel multi-view contrastive learning framework called SEM with multiple implementations. For example, the paper provides different weighting strategies with different advantages and experiments show their effectiveness.

III) Comparison experiments on different datasets show that SEM+infoNCE,RINCE,PSCL has a significant improvement over infoNCE,RINCE,PSCL themself, indicating the effectiveness.

IV) The supplementary material is well organized, including detailed appendices as well as code.

WEAKNESSES:
I) Due to the high complexity of MMD computation, it seems difficult to obtain MMD results on YoutubeVideo dataset (over 100,000 samples).

II) Introducing weighting strategies may increase the number of hyper-parameters. It might be better to add more descriptions, e.g., MMD.

III) Some grammatical mistakes need to be corrected, e.g., transfers is transfer in Line 155; adaptively weighting is adaptively weight in Line 304.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
The authors addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05