SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathcal{D}_{JSD}(\mathbf{Z}^m,\mathbf{Z}^n) = \frac{1}{2}\sum_{i = 1}^{N}p(\hat{\mathbf{z}}_i^m)\log \left(\frac{2\cdot p(\hat{\mathbf{z}}_i^m)}{p(\hat{\mathbf{z}}_i^m) + p(\hat{\mathbf{z}}_i^n)}\right) + \frac{1}{2}\sum_{i = 1}^{N}p(\hat{\mathbf{z}}_i^n)\log \left(\frac{2\cdot p(\hat{\mathbf{z}}_i^n)}{p(\hat{\mathbf{z}}_i^n) + p(\hat{\mathbf{z}}_i^m)}\right),$
$\mathcal{D}_{MMD}(\mathbf{Z}^m,\mathbf{Z}^n) = \frac{1}{N^2}\left[\sum_{i = 1}^{N}\sum_{j = 1}^{N}k(\mathbf{z}_i^m,\mathbf{z}_j^m) + \sum_{i = 1}^{N}\sum_{j = 1}^{N}k(\mathbf{z}_i^n,\mathbf{z}_j^n) - 2\sum_{i = 1}^{N}\sum_{j = 1}^{N}k(\mathbf{z}_i^m,\mathbf{z}_j^n)\right].$
$\{\mathbf{Z}^v\}_{v=1}^V, V > 2)$ , which is different from two-view setting (e.g.,  $\{\mathbf{Z}^1, \mathbf{Z}^2\}$ ) in traditional contrastive learning. To make our SEM framework be compatible with previous contrastive learning methods, we construct positive/negative sample pairs as follows. Specifically, for two views  $\{\mathbf{z}_i^m \in \mathbf{Z}^m, \mathbf{z}_j^n \in \mathbf{Z}^n\}$ , the positive sample pairs are  $\{\mathbf{z}_i^m, \mathbf{z}_i^n\}_{i=1,\dots,N}$ ; for any  $\mathbf{z}_i^m$ , its negative sample pairs are  $\{\mathbf{z}_i^m, \mathbf{z}_j^v\}_{j \neq i}^{v=m,n}$ . Cosine with a temperature parameter  $\tau$  is leveraged to measure the representation distance between pairs, i.e.,  $s = 1 / \tau \cdot \langle \mathbf{z}_i^m, \mathbf{z}_j^n \rangle / \| \mathbf{z}_i^m \|_2 \| \mathbf{z}_j^n \|_2$ . Then, we compute the contrastive loss between two views and sum all combinations as Eq. (2). We formulated three contrastive losses in Sec. 3.2, and the experiments in Sec. 4.1 will verify the compatibility of our SEM framework to them.
Reconstruction regularization When implementing the regularization term  $\mathcal{R}^v (\mathbf{X}^v,\mathbf{H}^v)$  in Eq. (2), we are motivated by the information encoding-decoding process [14, 19, 30], and stack a view-specific decoder  $f_{-}^{v}$  with network parameter  $\Omega^v$  on each view's  $\mathbf{H}^v$  to perform data recovery of  $\mathbf{X}^v$ . In this way, the regularization term in SEM can be implemented with the reconstruction loss of autoencoders<sup>4</sup>, whose encoder-decoder models can make hidden features preserve discriminative information of data. When decoder generally rebuilds  $\mathbf{X}^v$  with  $\mathbf{H}^v$ , we can believe that  $\mathbf{H}^v$  compresses the sufficient information of  $\mathbf{X}^v$ , for promoting contrastive learning fully access discriminative information of data.
Algorithm 1: Self-weighted multi-view contrastive learning with reconstruction regularization
Input: Dataset  $\{\mathbf{X}^v\}_{v = 1}^V$  , Training epochs  $E$  , Step size  $S$  , Batch size  $n$  , Hyper-parameter  $\lambda$  Initialize  $\{\Psi^v,\Omega^v\}_{v = 1}^V$  by Eq. (8) and initialize  $\{\mathcal{W}^{m,n}\}_{m,n = 1}^V$  with  $\{\mathbf{H}^v\}_{v = 1}^V$  like Eq. (7) for  $e\in \{1,2,\ldots ,E\}$  do for  $b\in \{1,2,\dots ,N / n\}$  do Pick mini-batch data  $\{\{\mathbf{x}_i^v\}_{i = (b - 1)n + 1}^{bn}\}_{v = 1}^V$  from  $\{\mathbf{X}^v\}_{v = 1}^V$  Compute the gradient of loss via Eq. (2) on the mini-batch data Update  $\{\Phi^v,\Psi^v,\Omega^v\}_{v = 1}^V$  via Adam [42] optimizer if mod(e,S)  $= = 0$  then Update  $\{\mathcal{W}^{m,n}\}_{m,n = 1}^{V}$  with  $\{\mathbf{Z}^v\}_{v = 1}^V$  by Eq. (7) Output: Model parameters  $\{\Phi^v,\Psi^v\}_{v = 1}^V$
The training steps of SEM is summarized in Algorithm 1, where representations and weights are updated alternatively to make them promote each other.  $E$  denotes total training epochs, and the step size  $S$  denotes the number of training epochs after each update of pairwise weights. As we cannot obtain meaningful  $\{\mathbf{y}^v\}_{v=1}^V$  before we start training neural networks, we first obtain meaningful  $\{\mathbf{H}^v\}_{v=1}^V$  by pre-training the model with Eq. (8), and then initialize  $\{\mathcal{W}^{m,n}\}_{m,n=1}^V$  with  $\{\mathbf{H}^v\}_{v=1}^V$ .
In this part, we theoretically analyze the mechanism of SEM in exploring mutual information among multiple views while mitigating representation degeneration. All proofs are given in Appendix A.
Considering SEM with InfoNCE loss and CMI weighting strategy, we have the following theorem indicating that minimizing the self-weighted contrastive loss keeps maximizing the mutual information between useful pairwise views, as well as avoiding the effects between unreliable pairwise views.
$$
\mathcal {R} _ {A E} ^ {v} \left(\mathbf {X} ^ {v}, \mathbf {H} ^ {v}\right) = \left\| \mathbf {X} ^ {v} - f _ {-} ^ {v} \left(\mathbf {H} ^ {v}; \Omega^ {v}\right) \right\| _ {F} ^ {2} = \left\| \mathbf {X} ^ {v} - f _ {-} ^ {v} \left(f ^ {v} \left(\mathbf {X} ^ {v}; \Psi^ {v}\right); \Omega^ {v}\right) \right\| _ {F} ^ {2},
$$
$$
\mathcal {R} _ {D A E} ^ {v} \left(\mathbf {X} ^ {v}, \tilde {\mathbf {H}} ^ {v}\right) = \left\| \mathbf {X} ^ {v} - f _ {-} ^ {v} \left(\tilde {\mathbf {H}} ^ {v}; \Omega^ {v}\right) \right\| _ {F} ^ {2} = \| \mathbf {X} ^ {v} - f _ {-} ^ {v} \left(f ^ {v} \left(\mathbf {X} ^ {v} + \epsilon ; \Psi^ {v}\right); \Omega^ {v}\right) \| _ {F} ^ {2}, \tag {8}
$$
$$
\mathcal {R} _ {M A E} ^ {v} (\mathbf {X} ^ {v}, \ddot {\mathbf {H}} ^ {v}) = \left\| \mathbf {X} ^ {v} - f _ {-} ^ {v} (\ddot {\mathbf {H}} ^ {v}; \Omega^ {v}) \right\| _ {F} ^ {2} = \| \mathbf {X} ^ {v} - f _ {-} ^ {v} (f ^ {v} (\mathbf {X} ^ {v} \odot \mathbf {A}; \Psi^ {v}); \Omega^ {v}) \| _ {F} ^ {2},
$$
<sup>4</sup>We borrow the core ideas of information reconstruction applied in vanilla autoencoder (AE [19]), denoising autoencoder (DAE [20]), and masked autoencoder (MAE [21]) and provide three reconstruction regularization options. In a same form, the three kinds of reconstruction loss functions could be formulated as follows:
where  $\mathbf{X}^v +\epsilon$  denotes the data disturbed by random Gaussian noise  $\epsilon \in \mathbb{R}^{N\times d_v}$  in DAE.  $\mathbf{X}^v\odot \mathbf{A}$  is the data masked by random  $0 - 1$  matrix  $\mathbf{A}\in \{0,1\}^{N\times d_v}$  in MAE.  $\ddot{\mathbf{H}}^v$  and  $\ddot{\mathbf{H}}^v$  denote the representations inferred from data  $\mathbf{X}^v +\epsilon$  and  $\mathbf{X}^v\odot \mathbf{A}$  in DAE and MAE, respectively.
Theorem 1. For any three views  $(v\in \{m,n,o\})$ , if class mutual information only exists in two views, e.g.,  $I(\mathbf{y}^m;\mathbf{y}^o)\to 0$ ,  $I(\mathbf{y}^n;\mathbf{y}^o)\to 0$ , and  $I(\mathbf{y}^m;\mathbf{y}^n) = \delta$ ,  $\delta >0$ , we have minimizing the weighted InfoNCE losses  $\mathcal{W}^{m,n}\mathcal{L}_{InfoNCE}^{m,n}(\mathbf{Z}^m,\mathbf{Z}^n) + \mathcal{W}^{m,o}\mathcal{L}_{InfoNCE}^{m,o}(\mathbf{Z}^m,\mathbf{Z}^o) + \mathcal{W}^{n,o}\mathcal{L}_{InfoNCE}^{n,o}(\mathbf{Z}^n,\mathbf{Z}^o)$  is equivalent to maximizing the mutual information between the two views  $(e^{\delta /\log N} - 1)I(\mathbf{Z}^m;\mathbf{Z}^n)$ .
Combining with the information losing of each layer through encoder networks, the following theorem further reveals that reconstruction regularization on the hidden features  $\mathbf{H}^v$  is conducive to alleviating the losing of discriminative semantic information through data transformation. Hence, we treat the layer output closest to  $\mathbf{Z}^v$  in encoders as hidden features to maximize  $\prod_{l = t^{m} + 1}^{L^{m}}(1 - \gamma_{l}^{m})$  and  $\prod_{l = t^n +1}^{L^n}(1 - \gamma_l^n)$ , aiming at maintaining useful semantic information for contrastive learning.
Theorem 2. For any two views  $(v\in \{m,n\})$  with positive class mutual information, denoting  $L^v$  as the total layer number of the  $v$ -th view's encoder network before representation  $\mathbf{Z}^v$ , the  $l$ -th layer has the information losing rate  $\gamma_l^v\geq 0$ . If  $\mathbf{S}$  is an oracle variable that contains and only contains multiple views' discriminative semantic information, and  $\mathbf{H}^v$  is the  $t^v$ -th layer's features, we have minimizing the regularized loss  $\mathcal{W}^{m,n}\mathcal{L}_{InfoNCE}^{m,n}(\mathbf{Z}^m,\mathbf{Z}^n) + \lambda \sum_v\mathcal{R}^v (\mathbf{X}^v,\mathbf{H}^v)$  is expected to obtain  $I(\mathbf{S};\mathbf{Z}^{m};\mathbf{Z}^{n})\leq \min \{I(\mathbf{S};\mathbf{X}^{m})\cdot \prod_{l = t^{m} + 1}^{L^{m}}(1 - \gamma_{l}^{m}),I(\mathbf{S};\mathbf{X}^{n})\cdot \prod_{l = t^{n} + 1}^{L^{n}}(1 - \gamma_{l}^{n})\}$ .
This section validates the effectiveness of our SEM. Specifically, we first conduct comparison experiments on state-of-the-art contrastive learning baselines and SEM with three options of contrastive losses (i.e.,  $\mathcal{L}_{InfoNCE}$ ,  $\mathcal{L}_{PSCL}$ ,  $\mathcal{L}_{RINCE}$ ). We then conduct ablation studies with three options of weighting strategies (i.e.,  $\mathcal{W}_{CMI}$ ,  $\mathcal{W}_{JSD}$ ,  $\mathcal{W}_{MMD}$ ), as well as with three options of reconstruction terms (i.e.,  $\mathcal{R}_{AE}$ ,  $\mathcal{R}_{DAE}$ ,  $\mathcal{R}_{MAE}$ ). Evaluation is built on the concatenation of all views' representations learned by methods. Finally, we show SEM's training process and its hyper-parameter analysis. We provided more experimental results as well as all implementation details of SEM in Appendix.
Datasets Our experiments employ five open-source multi-view datasets. Their information is shown in Table 1, where DHA [43] is a depth-included human action dataset where each action has RGB and depth features; CCV [44] refers to the columbia consumer video database whose samples are described with SIFT, STIP, and MFCC features; NUSWIDE [45] collects web images with multiple views (color histogram, block-wise
Table 1: Information of datasets
Name	View	Size	Class
DHA	2	483	23
CCV	3	6,773	20
NUSWIDE	5	5,000	5
Caltech	6	1,400	7
YoutubeVideo	3	101,499	31
color moments, color correlogram, edge direction histogram, and wavelet texture); Caltech [40] is a widely-used image dataset which leverages six views (Gabor, Wavelet moments, CENTRIST, HOG, GIST, and LBP) to represent samples; YoutubeVideo [46] is a large-scale dataset where each sample has three views including cuboids histogram, HOG, and vision misc. These datasets are diverse in forms and are often organized to comprehensively evaluate the performance of multi-view methods.
4.1 Comparison Experiments on Contrastive Learning
Baselines K-Means-BSV denotes K-Means clustering results on the best single-view of raw data, and we leverage this baseline to investigate the representation degeneration in comparison methods. InfoNCE [8], PSCL [16], and RINCE [15] are three kinds of CL methods. Since their original versions are designed to handle single views, we extended them to multi-view scenarios as did in [11, 17]. CMC [11], DCP [39], MFLVC [14], and DSIMVC [17] are four kinds of MCL methods. We evaluate our SEM with different contrastive losses (i.e., SEM+InfoNCE, SEM+PSCL, and SEM+RINCE), where the weighting strategy and reconstruction term are fixed to  $\mathcal{W}_{CMI}$  and  $\mathcal{R}_{AE}$ , respectively.
We leverage the linear clustering method K-Means to evaluate the performance of learning representations and report the average results of 10 runs in Table 2. The results indicate that: I) Our SEM framework is compatible with different contrastive losses (e.g., InfoNCE, PSCL, and RINCE) and we can clearly observe that SEM+InfoNCE/PSCL/RINCE successfully improve the baselines for large margins. For instance, SEM+InfoNCE respectively outperforms InfoNCE by about  $25\%$ ,  $13\%$ ,  $4\%$ ,  $7\%$ ,  $12\%$  ACC on the five datasets. II) MCL approaches could access the semantic information from multiple views, and thus outperforming that from single views. However, a side effect is
Table 2: Linear clustering performance evaluated by ACC and NMI (mean±std%)
Method	DHA	CCV	NUSWIDE	Caltech	YoutubeVideo
ACC	NMI	ACC	NMI	ACC	NMI	ACC	NMI	ACC	NMI
K-Means-BSV	66.6±2.6	78.0±1.3	19.5±0.3	17.8±0.3	39.7±0.0	11.6±0.0	85.1±0.1	75.6±0.1	16.5±0.6	15.9±0.4
InfoNCE [8]	54.9±3.8	77.7±2.1	25.8±0.9	25.9±0.7	56.3±2.2	30.3±1.4	79.9±1.0	71.4±0.9	19.6±0.1	19.7±0.0
PSCL [16]	39.8±2.8	72.9±2.0	21.5±1.6	24.3±1.0	53.4±0.4	27.4±0.5	67.8±2.8	69.5±2.8	15.4±0.3	14.3±0.5
RINCE [15]	49.9±6.2	76.3±2.6	22.5±0.4	23.5±0.2	56.6±1.4	30.8±1.3	80.3±2.2	72.0±2.2	14.7±0.3	13.6±0.2
CMC [11]	65.0±2.1	79.2±1.3	21.3±0.4	21.8±0.6	56.2±1.5	24.7±0.9	72.7±1.4	60.3±1.4	19.4±0.3	19.6±0.1
DCP [39]	69.8±2.2	82.9±1.6	24.1±1.2	20.6±0.9	48.1±1.4	24.5±1.1	69.6±6.6	66.2±5.3	14.0±0.3	12.3±0.4
MFLVC [14]	70.7±1.4	81.4±0.8	31.6±0.0	31.3±0.0	55.9±0.0	27.4±0.0	77.1±0.5	67.1±0.6	18.3±0.1	18.7±0.2
DSIMVC [17]	63.8±3.0	77.2±1.7	31.8±0.9	30.8±0.6	56.7±2.3	28.0±1.6	76.9±1.7	67.3±1.3	18.9±0.3	18.7±0.2
SEM+InfoNCE	80.9±1.9	84.1±0.9	39.4±0.7	35.5±0.4	60.4±0.4	34.9±0.7	87.2±0.3	80.3±0.5	31.3±1.1	31.1±0.9
SEM+PSCL	69.7±4.2	81.4±1.6	39.3±1.1	35.9±0.6	57.8±1.4	32.6±0.9	86.3±1.7	78.6±2.1	32.2±0.7	32.2±0.6
SEM+RINCE	76.3±1.3	82.8±0.7	38.9±0.9	34.6±0.5	60.6±0.7	35.6±1.3	85.4±1.4	76.7±2.2	29.8±0.5	29.5±0.5
that contrastive learning directly increases the feature representation similarity of multiple views, which might obscure useful discriminative information hidden in high-quality views and lead to the representation degeneration. For example, on DHA and Caltech, results on many MCL methods (e.g., PSCL, RINCE, CMC, and MFLVC) are worse than the single-view baseline K-Means-BSV. [III] Our SEM not only outperforms all these MCL methods but also mitigates the representation degeneration in MCL, e.g., SEM+PSCL respectively outperforms K-Means-BSV by about  $3\%$ ,  $20\%$ ,  $18\%$ ,  $1\%$ ,  $16\%$  ACC on the five datasets. This is because the framework of SEM is adaptive to multiple views' qualities, which can reduce the side effect between unreliable views with inconsistent information, for better extracting discriminative information and consistent semantics among useful views.
Additionally, we leverage the linear classification method SVM [47] to evaluate the performance of MCL methods to learn representations, where we only use  $30\%$  of the learned representations for the training set and the rest for the test set. Figures 3 and 4 show the classification performance on DHA and CCV, respectively. Our SEM improves the baseline methods (especially for PSCL
Figure 3: Classification performance on DHA.
Figure 4: Classification performance on CCV.
and RINCE) and consistently outperforms other MCL methods (such as CMC, DCP, MFLVC, and DSIMVC). Since contrastive learning usually discards the information which is irrelevant to optimization objectives, the results further indicate that the representations learned by MCL are classification-friendly, which generally focus on catching class-level semantics among multiple views.
4.2 Ablation Experiments on Self-Weighting Strategy and Reconstruction Regularization
This part presents the ablation experiments to investigate the effectiveness of different weighting strategies  $\mathcal{W}_{\text{GMI/JSD/MMD}}$  and reconstruction terms  $\mathcal{R}_{AE/DAE/MAE}$  in our SEM framework.
Table 3: Clustering accuracy  $(\%)$  of SEM with different options of weighting strategy  $\mathcal{W}$  on two datasets
DHA	CCV
SEM w/o W	71.3	33.5
SEM w/ WCMI	80.9 (↑ 9.6)	39.4 (↑ 5.9)
SEM w/ WJSD	80.5 (↑ 9.2)	35.6 (↑ 2.1)
SEM w/ WMMD	84.4 (↑ 13.1)	33.9 (↑ 0.4)
Table 4: Clustering accuracy  $(\%)$  of SEM with different options of reconstruction term  $\mathcal{R}$  on two datasets
DHA	CCV
SEM w/o R	60.5	28.7
SEM w/ RAE	80.9 (↑ 20.4)	39.4 (↑ 10.7)
SEM w/ RDAE	81.5 (↑ 21.0)	38.4 (↑ 9.7)
SEM w/ RMAE	83.0 (↑ 22.5)	39.5 (↑ 10.8)
Table 3 reports the linear clustering performance (evaluated by ACC) of our SEM framework without self-weighting strategy (i.e., SEM w/o  $\mathcal{W}$ ) and that with three weighting strategies (i.e.,
(a) Weight vs. Iteration
(b) Loss vs. Epoch
Figure 5: (a) The change trend of weights  $\mathcal{W}^{1,4}$  and  $\mathcal{W}^{4,5}$  in SEM. (b) Loss values  $\mathcal{L}_{InfoNCE}^{1,4}$  and  $\mathcal{L}_{InfoNCE}^{4,5}$  during contrastive learning. (c) Clustering accuracy on the learned representations of view 1, view 4, and view 5.
(c) ACC vs. Iteration
$\mathcal{W}_{CMI}, \mathcal{W}_{JSD}, \mathcal{W}_{MMD})$ , where the contrastive loss and reconstruction term are fixed to  $\mathcal{L}_{InfoNCE}$  and  $\mathcal{R}_{AE}$ , respectively. Compared with SEM w/o  $\mathcal{W}$  (this setting is the reconstruction regularized multi-view contrastive learning) that equally treats contrastive learning between any two views, SEM w/  $\mathcal{W}_{CMI / JSD / MMD}$  can adaptively weight the contrastive learning according to specific two views and thus all these three variants of SEM obtain significant improvements. For example, SEM w/  $\mathcal{W}_{MMD}$  has a  $13.1\%$  improvement on DHA and SEM w/  $\mathcal{W}_{CMI}$  has a  $5.9\%$  improvement on CCV. Results on more datasets and time costs are shown in Appendix C, where we find that the proposed weighting strategy of class mutual information  $\mathcal{W}_{CMI}$  generally achieves the best performance on accuracy and time consumption among the three options of weighting strategy.
Table 4 reports the linear clustering performance (evaluated by ACC) of our SEM framework without reconstruction regularization (i.e., SEM w/o  $\mathcal{R}$ ) and that with three reconstruction terms (i.e.,  $\mathcal{R}_{AE},\mathcal{R}_{DAE},\mathcal{R}_{MAE}$ ), where the contrastive loss and weighting strategy are fixed to  $\mathcal{L}_{InfoNCE}$  and  $\mathcal{W}_{CMI}$ , respectively. We can easily find that the proposed SEM with reconstruction terms obviously outperforms that without reconstruction terms. For instance, compared with SEM w/o  $\mathcal{R}$ , SEM w/  $\mathcal{W}_{MAE}$  has  $22.5\%$  and  $10.8\%$  improvements on DHA and CCV, respectively. This is because the reconstruction regularization makes the hidden features  $\{\mathbf{H}^v\}_{v=1}^V$  avoid losing discriminative information, which promotes the multi-view contrastive learning performed on subsequent  $\{\mathbf{Z}^v\}_{v=1}^V$ . Meanwhile, SEM w/  $\mathcal{R}_{MAE}$  and SEM w/  $\mathcal{R}_{DAE}$  perform better than SEM w/  $\mathcal{R}_{AE}$ . This is because, compared with vanilla AE, DAE or MAE (by adding noise or masking on raw data) can make our model more conducive to removing semantic-irrelevant noise as well as capturing hidden patterns.
4.3 Experimental Analysis on Mechanism of SEM
This part presents the visualization and analysis on SEM to give an intuition of its behavior and mechanism, where the combination of  $\mathcal{L}_{InfoNCE} + \mathcal{W}_{CMI} + \mathcal{R}_{AE}$  is taken as an example.
Let's first recall the views of Caltech dataset in Figure 2(a), we can consider that view 4 and view 5 are high-quality views, while view 1 is a low-quality view. The performance relation among them is  $\mathrm{ACC}_{\mathrm{view}4} > \mathrm{ACC}_{\mathrm{view}5} > \mathrm{ACC}_{\mathrm{view}1}$ . In Figure 2(c), view 4's representation degeneration occurs.
Figure 5 shows the pairwise weights, losses, and clustering accuracy on Caltech dataset during SEM's training process, where 1 iteration corresponds to 100 epochs, i.e., the step size is set to 100 epochs. Our SEM is a self-weighted multi-view contrastive learning framework that automatically infers different weights for different pairwise views as shown in Figure 5(a), where we can observe that weights  $\mathcal{W}^{4,5} > \mathcal{W}^{1,4}$  and they were dynamically updated for 4 times. As a result, contrastive learning between view 4 and view 5 is strengthened by  $\mathcal{W}^{4,5}$ , while contrastive learning between view 1 and view 4 is weakened by  $\mathcal{W}^{1,4}$ . Meanwhile, loss  $\mathcal{L}_{InfoNCE}^{4,5}$  is minimized earlier than loss  $\mathcal{L}_{InfoNCE}^{1,4}$  as shown in Figure 5(b). In other words, since the mutual effect between view 4 and view 5 is strengthened, the effect of view 1 on view 4/view 5 is weakened such that view 4/view 5 does not degenerate. At the same time, the effect of view 4/view 5 on view 1 remains and promotes the representation learning of view 1. Consequently, all views' performance in Figure 5(c) increases through our SEM, and the representation degeneration of view 4 occurring in Figure 2(c) is mitigated.
Hyper-parameter analysis Since different datasets have different levels of reconstruction errors, the trade-off coefficient  $\lambda$  is introduced to balance the contrastive learning and information recovery in our SEM framework. In Figure 6(a), we change  $\lambda$  within the range of  $[10^{-3}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}, 10^{2}, 10^{3}]$  and report the clustering accuracy tested on representations. The experimental results indicate that SEM is not sensitive to  $\lambda$  in  $[10^{-1}, 10^{1}]$ . In our experiments,  $\lambda$  is consistently set to 1 for all the five datasets. Regarding self-supervised learning, frameworks with fewer manually set hyper-parameters might be more convenient for their applications.
Additionally, we investigate the effect of cluster number when the weight strategy of our SEM framework is selected as  $\mathcal{W}_{CMI}$  which needs to pre-define the cluster number when applying K-Means algorithm. As shown in Figure 6(b), when computing the class mutual information, we change the number of clusters within the range of  $[K / 2,K,2K,4K]$  where  $K$  denotes the truth class number of
Figure 6: (a) ACC vs.  $\lambda$ .
(b) ACC vs.  $K$
multi-view datasets. Compared with  $K$ ,  $K/2$  leads to more coarse-grained class mutual information, while  $2K$  and  $4K$  come in more fine-grained class mutual information. The experimental results demonstrate that SEM with  $\mathcal{W}_{CMI}$  is not sensitive to the choices of cluster number.
In this paper, we showcase that the representation degeneration could seriously limit the application of contrastive learning in multi-view scenarios. To mitigate this issue, we propose self-weighted multi-view contrastive learning with reconstruction regularization (SEM), which is a general framework that is compatible with different options of the contrastive loss, weighting strategy, and reconstruction term. Theoretical and experimental analysis verified the effectiveness of SEM, and it can significantly improve many existing contrastive learning methods in multi-view scenarios. Moreover, ablation studies indicated that SEM is effective with different weighting strategies and reconstruction terms.
Our future work is to extend the proposed SEM to be useful not only for multi-view scenarios, but also for other contrastive learning based domains, such as contrastive learning in sequences. Conceptually, the limitation of the self-weighting strategy is that it is more effective when there are over two views. When there are only two views, the self-weighted multi-view contrastive learning framework transforms into traditional contrastive learning but with reconstruction regularization. Therefore, another future work is to extend the view-level weighting of SEM to sample-level weighting.
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA1004100, in part by the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China under Grant ZYGX2022YGRH009 and Grant ZYGX2022YGRH014, in part by the National Natural Science Foundation of China under Grant 62276052.