Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization
Authors: Junlin He
Jinxiao Du
Susu Xu
Wei Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-view representation learning; Canonical Correlation Analysis; Deep Canonical Correlation Analysis; Noise regularization; Model collapse
Primary area: deep_learning_architectures
TL;DR: A novel noise regularization approach is developed to prevent deep canonical correlation analysis-based methods from model collapse.

Abstract: Multi-View Representation Learning (MVRL) aims to learn a unified representation of an object from multi-view data.
Deep Canonical Correlation Analysis (DCCA) and its variants share simple formulations and demonstrate state-of-the-art performance. However, with extensive experiments, we observe the issue of model collapse, i.e., the performance of DCCA-based methods will drop drastically when training proceeds. The model collapse issue could significantly hinder the wide adoption of DCCA-based methods because it is challenging to decide when to early stop. To this end, we develop NR-DCCA, which is equipped with a novel noise regularization approach to prevent model collapse. Theoretical analysis shows that the Correlation Invariant Property is the key to preventing model collapse, and our noise regularization forces the neural network to possess such a property. A framework to construct synthetic data with different common and complementary information is also developed to compare MVRL methods comprehensively. The developed NR-DCCA outperforms baselines stably and consistently in both synthetic and real-world datasets, and the proposed noise regularization approach can also be generalized to other DCCA-based methods such as DGCCA.

Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization
Junlin He
Jinxiao Du
The Hong Kong Polytechnic University Hong Kong SAR, China junlinspeed. he@connect. helcomeect.poly..ht
The Hong Kong Polytechnic University Hong Kong SAR. China j jinxiao... www.onnectt....mmmmmmmmmmm polyu.
Susu Xu Johns Hopkins University Maryland, USA
Wei Ma"
The Hong Kong Polytechnic University Hong Kong SAR, China wei, W. www...amll....mmmmmmmmmmmmmmmaa. edu. edu. hk
sxu830jhu.c edu
Abstract
Multi- View Representation Learning (MVRL) aims learn unified represen- tation of an object from multi-view data. Deep Canonical Correlation Analysis (DCCA) and its variants share simple formulations and demonstrate state-of-the- art performance. However, with extensive experiments, we observe the issue of model collapse, i.e., the performance of DCCA-bases methods will drop drasti- cally when training proceeds. The model collapse issue could significantly hinder the wide adoption of DCCA-based methods because challenging to decide when to early stop. To this end, we develop NR-DCCA, which is equipped with a novel noise regularization approach to prevent model collapse. Theoretical anal- ysis shows that the Correlation Invariant Property is the key to preventing model collapse, and our noise regularization forces the neural network to possess such property. A framework to construct synthetic data with different common and complementary information is also developed to compare MVRL methods com- prehensively. The developed NR-DCCA outperforms baselines stably and con- sistently in both synthetic and real-world datasets, and the proposed noise regu- larization approach also be generalized to other DCCA-based methods such as DGCCA. Our code will released at https://github.com. http://tttubccommmmmmmmcch//a// Umaruchain/NR- DCCA.git.
Introduction
In recent years, multi-view representation learning (MVRL) has emerged as a core technology for learning from multi-source data and providing readily useful representations to downstream tasks (Sun et al. 2023, Yan et al. 2021), and it has achieved tremendous success in various ap- plications, such as video surveillance (Guo et al. 2015, Feichtenhofer al. 2016, Deepak, et al. 2021), medical diagnosis (Wei et al. 2019, Xu et al. 2020) and social media (Srivastava & Salakhut- dinov 2012, Karpathy Fei-Fei 2015, Mao et 2014, Fan et 2020). Specifically, multi-source data can be collected from the same object, and each data source can be regarded one view of the object. For instance, an object can be described simultancously through texts, videos, and au- dio, which contain both common and complementary information of the object (Yan et al. 2021, Zhang, Liu & 2019, Hwang et al. 2021, Geng et al. 2021), and the MVRL aims to learn a unified representation of the object from the multi-view data.
"Conesponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The key challenge of MVRL is to learn the intricate relationships of different views. The Canoni- cal Correlation Analysis (CCA), which one of the early and representative methods for MVRL, transforms all the views into a unified space by maximizing their correlations (Hotelling 1992, Horst 1961, Hardoon et al. 2004, Lahat et al. 2015, Yan et al. 2023, Sun et al. 2023). Through correlation maximization, CCA can identify the common information between different views and extract them to form the representation of the object. On top of CCA, Linear CCA, and DCCA maximize the correlation defined by CCA through gradient descent, while the former uses affine transforma- tion and the latter uses Deep Neural Networks (DNNs). (Andrew et al. 2013). Indeed, there are quite few variants of DCCA, such as DGCCA (Benton et al. 2017), DCCAE (Wang et al. 2015), DVCCA (Wang et al. 2016), DTCCA (Wong et al. 2021) and DCCA.GHA (Chapman et al. 2022).
However, extensive experimentation reveals that DCCA-based methods typically excel during the initial stages of training but suffer a significant decline in performance as training progresses. This phenomenon is defined as model collapse within the context of DCCA. Notably, our definition grounded in the performance of the learned representations on downstream tasks. Previous studies found that the representations (i.e., final output) of both Linear CCA and DCCA are full-rank (An- drew et al. 2013, De Bie & De Moor 2003). Nevertheless, they did not further explore whether merely guaranteeing that the full-rank representations can guarantee that the weight matrices are full-rank.
Though early stopping could be adopted to prevent model collapse (Prechelt 1998, Yao et al. 2007), it remains challenging when to stop. The model collapse issue of DCCA- methods prevents the adoption in large models, and currently, many applications still use simple concatenation to combine different views (Yan et al. 2021, Zheng et al. 2020, Nie et al. 2017). Therefore, how to develop a DCCA-based MVRL method free of model collapse remains an interesting and open question.
In this work, we demonstrate that both representations and weight matrices of Linear CCA are full-rank whereas DCCA only guarantees that representations are full-rank but not for the weight matrices. Considering that Linear CCA does not show the model collapse while DCCA does, we conjecture that the root cause of the model collapse in DCCA is that the weight matrices in DNNs tend be low-rank. wealth of research supports this assertion, both theoretically and empiri- cally, demonstrating that over- wver-parameterized DNNs are predisposed to discovering low-rank solu- tions (Jing et al. 2021, Saxe et al. 2019, Soudry et al. 2018, Dwibedi al. 2021). the weight matrices in DNNs tend to be low-rank, it means that the weight matrices are highly self-related and redundant, which limits the expressiveness of DNNs and thus affects the quality of representations.
Therefore, this paper develops NR-DCCA, a DCCA-based method equipped with a generalized noise regularization (NR) approach. The NR approach ensures that the correlation with random data is invariant before and after the transformation, which we define as the Correlation Invariant Prop- erty (CIP). It is also verified that the NR approach can be applied other DCCA-based methods. Comprehensive experiments using both synthetic datasets and real-world datasets demonstrate the consistent outperformance and stability of the developed NR-DCCA method.
From a theoretical perspective, we derive the equivalent conditions between the full-rank property and CIP of the weight matrix. By forcing DNNs possess CIP and thus mimicking the behavior of Linear CCA, we introduce random data to constrain the weight matrices in DNNs and expect to avoid them being redundant and thus prevent model collapse.
In summary, our contributions are four-fold:
The model collapse issue in DCCA- methods for MVRL identified, demonstrated, and explained.
A simple yet effective noise regularization approach is proposed and NR-DCCA is devel- oped to prevent model collapse. Comprehensive experiments using both synthetic datasets and real-world datasets demonstrate the consistent outperformance and stability of the de- veloped NR-DCCA.
Rigorous proofs are provided to demonstrate that CIP is the equal condition of the full-rank weight matrix, which justifies the developed NR approach from thcoretical perspective.
A novel framework is proposed to construct synthetic data with different common and complementary information for comprehensively evaluating MVRI. methods.

---

2 Related Works
2.1 Multi-view representation learning
MVRL aims to uncover relationships among multi-viww data in an unsupervised manner, thereby obtaining semantically rich representations that can be utilized for various downstream tasks (Sun et al. 2023, Yan et al. 2021). Several works have been proposed to deal with MVRL from differ- ent aspects. DMF-MVC (Zhao et al. 2017) utilizes deep matrix factorization to extract a shared representation from multiple views. MDcR (Zhang, Fu, Hu, Zhu & Cao 2016) maps each view to a lower-dimensional space and applies kernel matching to enforce dependencies across the views. CPM-Nets (Zhang, Han, Fu, Zhou, Hu et al. 2019) formalizes the concept of partial MVRL and many works have been proposed for such issue (Zhang et al. 2020, Tao al. 2019, Li et al. 2022, Yin & Sun 2021). AEt-Nets (Zhang, Liu & Fu 2019) utilizes a two-level autoencoder framework to obtain a comprehensive representation of multi-view data. DUA-Nets (Geng et al. 2021) takes a generative modeling perspective and dynamically estimates the weights for different views. MVT- CAE (Hwang et al. 2021) explores MVRL from an cormatioontthoreete perspective, which can capture the shared and view-specific factors of variation by maximizing minimizing specific total correlation. Our work focuses on CCA as a simple, classic, and theoretically sound approach as can still achieve ofte--o-ttheart art performance consistently.
2.2 CCA and its variants
Canonical Correlation Analysis (CCA) projects the multi-view data into a unified space by maxi- mizing their correlations (Hotelling 1992, Horst 1961, Hardoon et al. 2004, Lahat et al. 2015, Yan et al. 2023, Sun et al. 2023). has been widely applied in various scenarios that involve multi- view data, including dimension reduction (Zhang, Zhang, Pan & Zhang 2016, Sun, Ceran & Ye 2010, Avron et al. 2013), classification (Kim et al. 2007, Sun, Ji & Ye 2010), and clustering (Fern et al. 2005, Chang & Lin 2011). To further enhance the nonlinear transformability of CCA, Kernel CCA (KCCA) uses kernel methods, while Deep CCA (DCCA) employs DNNs. Since DNNs is parametric and take advantage of large amounts of data for training, numerous DCCA-based methods have been proposed. Benton et al. (2017) utilizes DNNs to optimize the objective of Gen- eralized CCA, reveal connections between multiple views more effectively. To better preserve view-specific information, Wang et al. (2015) introduces the reconstruction errors of autoencoders to DCCA. Going step further, Wang et al. (2016) proposes Variational CCA and utilizes dropout and private autoencoders to project common and view-specific information into two distinct spaces. Furthermore, many studies are exploring efficient methods for computing the correlations between multi-view data when dealing with more than two views such as MCCA, GCCA, and TCCA (Horst 1961, Nielsen 2002, Kettenring 1971, Hwang et al. 2021). Some research focuses on improving the efficiency of computing CCA by avoiding the need for singular value decomposition (SVD) (Chang al. 2018, Chapman et al. 2022). However, the model collapse issue of DCCA-based methods has not been explored and addressed.
2.3 Noise Regularization
Noise regularization is a pluggable approach to regularize the neural networks during train- ing (Bishop 1995, An 1996, Sietsma & Dow 1991, Gong et al. 2020). In supervised tasks, Sietsma & Dow (1991) might be the first to propose that, by adding noise to the train data, the model will gener- alize well on new unseen data. Moreover, Bishop (1995), Gong et al. (2020) analyze the mechanism of the noise regularization, and He al. (2019), Gong al. (2020) indicate that noise regularization can also be used for adversarial training to improve the generalization of the network. In unsuper- vised tasks, Poole et al. (2014) systematically explores the role of noise injection at different layers in autoencoders, and distinct positions of noise perform specific regularization tasks. However, how to make use of noise regularization for DCCA-based methods, especially for preventing model collapse, has not been studied.
3

---

3 Preliminaries
In this section, we will explain the objectives of the MVRL and then introduce Linear CCA and DCCA as representatives of the CCA-based methods and DCCA-based methods, respectively. Lastly, the model collapse issue in DCCA is demonstrated.
3.1 Settings for MVRL
Suppose the set of datasets from K different sources that describe the same object is represented by X, and we define X (X1,... "',,".. ..,,.. x view (k-th data source), n the sample size, and dy represents the feature dimension for the k-th view. And we use X denote the transpose of Xg. We take the Caltech101 dataset an example and the training set has 6400 images. One image has been fed to three different feature extractors producing three features: 1984-d HOG feature, 512-d GIST feature, and a 928-d SIFT feature. Then for this dataset, we have X1 E IR 1984x6400 X2 E R 522x20000 .3 E R
Xx ,x),,
R where Iyk represents the k-th
The objective of MVRL is to learn transformation function y that projects the multi-vie view data X to unified representation E pmxa, where m represents the dimension of the representation space, shown below:
z= v = VX,, ",,"",,
(1)
After applying for representation learning, we expect that the performance of using would be better than directly using X for various downstream tasks.
3.2 Canonical Correlation Analysis
Among various MVRL methods, CCA projects the multi-view data into a common space by maxi- mizing their correlations. We first define the correlation between the two views as follows:
Co W2Xx) (r((ET1)
1/2 ( 12 1/////2
where tr denotes the matrix trace, En, Σ22 represent the self-covariance matrices of the projected views, and E1 is the cross-covariance matrix between the projected views (D' Agosttni 1994, An- drew et al. 2013). The correlation between the two projected views can be regarded as the sum of all singular values of the normalized cross-covariance (Hotelling 1992, Anderson et al. 1958).
For multiple views, their correlation defined as the summation of all the pairwise correla- tions (Nielsen 2002, Kettenring 1971), which is shown as follows:
Con(W, W, (WXXk. Σ Xa,
k<j
Essentially, Linear CCA searches for the linear transformation matrices (W+)R that maximize cor- relation among all the views. Mathematically, it can be represented as follows (Wang et al. 2015):
(WE } = arg max Cor(WW, ... WR .... WK)
(4)
(W2t=
Once WE obtained by backpropagation, the multi-view data are projected into unified space. Lastly, all projected data are concatenated to obtain Z = [i W X WWx for downstream tasks.
As an extension of linear CCA, DCCA employs neural networks to capture the nonlinear relationship among multi-view data. The only difference between DCCA and Linear CCA is that the linear transformation matrix Wq replaced by multi-layer perceptrons (MLP). Specifically, each W, is replaced by neural network fk. which can be viewed nonlinear transformation. Similar to Linear CCA, the goal of DCCA is to solve the following optimization problem:
h = arg max Corr (fi, fx(()), ......fKK (K()))) (faha
(5)
The parameters in Linear CCA and DCCA are both updated through backpropagation (An- drew et al. 2013, Wang al. 2015). Again, the unified representation obtained by (T(())::.. (X(x)) : (Xx) x)) for downstream tasks.

---

Model Collapse of DCCA
Despite exhibiting promising performance, DCCA shows significant decline in performance the training proceeds. We define this neciin-in-performance phenomenon as the model collapse of DCCA.
Previous studies found that the representations (i.e., final output) of both Linear CCA and DCCA full-rank (Andrew et al. 2013, De Bie & De Moor 2003). However, further demonstrate that both representations and weight matrices of Linear CCA are full-rank whereas DCCA only full-rank but not for the weight matrices. Given that Linear CCA
guarantees that representations has only a single layer of linear transformation Wi and the representations & are constrained to be full-rank by the loss function, Wx in Linear CCA is full-rank (referred to Lemma 4 and assume that Wa is a square matrix and X3 is full-rank). As for DCCA, we consider a simple case when (Xx) Relu(W,,xx). and fa is single-layer network and uses clement-wise Relu activation function. Only the representations (ealaWaaai are constrained to be full-rank, and hence we cannot guarantee that X is full-rank. For example, when Re)u(W;,Aa) = (d: 9). it is clear that this is a matrix rank 2, but in fact W can be (-i, 7'), and this not full-rank. This reveals that the neural network fe is overfitted on Xs. i.e., making representations Relu(We to be full-rank with the constraint of its loss function, rather than W5 itself being full-rank (verified in Appendix A.5.
Thus, we hypothesize that model collapse in DCCA arises primarily due to the low-rank nature of the DNN weight matrices. To investigate this, analyze the eigenvalue distributions of the first linear layer's weight matrices in both DCCA and NR-DCCA across various training epochs on synthetic datasets. Figure illustrates that during the initial training phase (100th epoch), the cigenvalues decay slowly for both DCCA and NR-DCCA. However, by the 1200th epoch, DCCA exhibits a markedly faster decay in eigenvalues compared NR-DCCA. This observation suggests synchronization between model collapse in DCCA and increased redundancy of the weight matrices. For more details on the experimental setup and results, please refer Section 6.2.
y-12
Nornaised eigervalus
Nomallsed eigenvaluv
6.1
0.1
0.01
0.01
0.001
10
100028
0.001
10
200028
Eigenvalue rank
Egenvvilue rank
(a) 100-th epoch (DCCA)
(b) 100-th epoch (NR-DCCA)
Nornalised elçenvalus
Nornalised eigervalue
6.1
0.1
001
Layer 1_126
0.011
100128
200028
Eigenvvlue rank
gennvallue rank
(c) 1200-th epoch (DCCA)
(d) 1200-th epoch (NR-DCCA)
Figure 1: Eigenvalue distributions the first linear layer's weight matrices in the encoder of 1-st view.
5

---

DCCA with Noise Regularization (NR-DCCA)
5.1
Method
Based on the discussions in previous sections, we present NR-DCCA, which makes use of the noise regularization approach prevent model collapse in DCCA. Indeed, the developed noise regular- ization approach can be applied to variants of DCCA methods, such as Deep Generalized CCA (DGCCA) (Benton et al. 2017). An overview of the NR-DCCA framework presented in Figure 2.
Multi-siww
Nitlow
NR loss
CCAles
[.0AJ
LDFT
FAAJ
F-06.)
Figure 2: Illustration of NR-DCCA. We take the CUB dataset example: similar to DCCA, the k-th view X₃ is transformed using S& to obtain new representation fx(Xx) and then maximize the correlation between new representations. Additionally, for the k-th view, incorporate the proposed NR loss to regularize Sk.
The key idea in NR-DCCA is to generate set of i.idd Gaussian white noise, denoted as A {AI.... Ak..... A E gdaxn, with the same shape as the multi-view data X In Lin- CCA, the correlation with noise is invariant the linear transformation WR: Corr(X,: A&) CorrWW,X,, W& (rigorous proof provided Theorem 1). However, for DCCA, Corr(X, A&) might not equal Corr(fa (Xxx), fo (AA))) because the powerful neural networks f& have overfitted to the maximization program in DCCA and the weight matrices have been highly self-related. Therefore, enforce the DCCA to mimic the behavior of Linear CCA by adding NR loss Ço = [Corr(]e(Xx), Jx Corr(X, Xx, Ax)]. and hence the formulation of NR-DCCA is:
K
(f arg max Corr (fi(X),, fx(xk))) - (fa)
CA.
(6)
where a is the hyper-parameter weighing the NR loss. NR-DCCA can trained through backprop- agation with the randomly generated A in each epoch, and the unified representation is obtained directly using (fī h in the same manner as DCCA.
5.2 Theoretical Analysis
In this section, provide the rationale for why the developed noise regularization can help to prevent the weight matrices from being low-rank and thus model collapse. Moreover, we prove the effect of full-rank weight matrices on the representations, which provides a tool to empirically verify the full-rank property of weight matrices by the quality of representations.
Utilizing a new Moore-Penrose Inverse (based (Petersen et al. 2008) form of Corr in CCA, discover that the full-rank property of Wa is equal to CIP:
Theorem (Correlation Invariant Property (CIP) of Wi) Given Wt is square matrix for any
k and mp = [C, Xx. AR Corr( Ax)], we have 7} = (L.e CIP) = Wt full-rank.

---

Similarly, we say f& possess CIP if C = O. Under Linear CCA, it is redundant to introduce the NR approach and force Wz to possess CIP, since forcing W & be full-rank is sufficient to ensure that We is full-rank. However, in DCCA, f& is overfitted on Xx. i.e., making representations fa (X&) to be full-rank, rather than weight matrices in fk being full-rank. By forcing fa to possess CIP and thus mimicking the behavior of Linear CCA, the NR approach constrains the weight matrices to be full-rank and less redundant and thus prevents model collapse.
Next, we show that full-rank weight matrices (i.e., CIP) can greatly affect the quality of representa- tions.
Theorem (Effects of CIP on the obtained representations) For any k. if Wi is a square matrix and CIP holds for Wy (i.e. We is ralk-rank), W holds that:
min P& Xxllr= =0
Pk
min 11Q. + A²) - W.l. VTllW Qu
= 1|
(8)
where I1- denotes the Frobenius norm and is a small positive threshold. Ps and Qk are searched weight matrices of k-th view to recover the input and discard noise, respectively. And we refer W X&IIF and &[Q&W& + As) WxX& as reconstruction loss and denosing loss.
Theorem 2 suggests that the obtained representation is of low reconstruction loss and denoising loss. Low reconstruction loss suggests that the representations can be linearly reconstructed to the inputs. This implies that We preserves distinct and essential features of the input data, which is a desirable property to avoid model collapse since it ensures that the model captures and retains the whole modality of data (Zhang, Liu & Fu 2019, Tschannen et al. 2018, Tian & Zhang 2022). Low denoising loss implies that the model's representation is robust to noise, which means that small perturbations in the input do not lead to significant changes the output. This condition can be seen form of regularization that prevents overfitting the noise in the data (Zhou & Paffenroth 2017, Yan et al. 2023, Staerman et al. 2023). Additionally, the theorem also suggests that the rank of weight matrices is a good indicator to assess the quality of representations. which coincides with existing literature (Kornblith et al. 2019, Raghu et al. 2021, Garrido et al. 2023, Nguyen et al. 2020, Agrawal al. 2022).
6 Numerical Experiments
We conduct extensive experiments on both synthetic and real-world datasets to answer the following research questions:
RQI: How can we construct synthetic datasets to evaluate the MVRL methods compre- hensively?
RQQ2: Does NR-DCCA avoid model collapse across all synthetic MVRL datasets?
RQQ3 Does NR-DCCA perform consistently in real-world datasets?
We follow the protocol described in Hwang et (2021) for evaluating the MVRL methods. For each dataset, we construct training dataset and a test dataset. The encoders of all MVRL methods are trained on the training dataset. Subsequently, we encode the test dataset to obtain the representation, which will be evaluated in downstream tasks. We employ Ridge Regression (Hoerl & Kennard 1970) for the regression task and use R2 as the evaluation metric. For the classification task, we use a Support Vector Classifier (SVC) (Chang & Lin 2011) and report the average Fl scores. All tasks are evaluated using 5-fold vross-validation, and the reported results correspond to the average values of the respective metrics.
For fair comparison, we use the same architectures of MLPs for all D(G)CCA methods. To be spe- cific, for the synthetic dataset, which is simple, we employ only one hidden layer with dimension of 256. For the real-world dataset, we use MLPs with three hidden layers, and the dimension of the middle hidden layer is 1024. We further demonstrate that increasing the depth of MLPs further ac- celerates the mod collapse of DCCA, while NR-DCCA maintains a stable performance in Appendix A.7.
7

---

Baseline methods include CONCAT, PRCCA (Tuzhilina et al. 2023), KCCA (Akaho 2006), Linear CCA Wang et al. (2n15),Linear GCCA,DCCA (Andrew et al. DCCA),DCCA EY, DCCA GHA (Chapman et al. 2022), DGCCA (Benton et al. 2017), DCCAE/DGCCAE, (Wang al. 2015), DCCA PRIVATT/DDGCA. PRIVATE (Wang et al. 2016), and MVTCAE (Hwang et al. 2021).
important to note that our proposed NR approach requires the noise matrix employed to be full-rank, which is compatible with several common continuous noise distributions. In our primary experiments, we utilize Gaussian white noise. Additionally, as demonstrated in Appendix A.6, uniformly distributed noise is also effective in our NR approach.
Details of the experiment settings including datasets and baselines are presented in Appendix A.3. Hyper-parameter setrings, including ridge regularization of DCCA, a of NR, are discussed in Ap- pendix A.5. We also analyze the computational complexity of different DCCA-based methods in Appendix A.12 and the learned representations are visualized in Appendix A.9. In the main paper, mainly compare Linear CCA, DCCA-basel methods, and NR-DCCA while other MVRL meth- ods are discussed in Appendix A.11. The results related to DGCCA and are similar and presented in Appendix A0
Data Generation (Commo Fate: 0%)
Multi vlew Data
God Embedding
Task Generation
Dowrnstrwam Tasks
Figure 3: Construction of a synthetic dataset. This example consists views and n objects, and the common rate is 0%.
6.1 Construction of synthetic datasets (RQ1)
We construct synthetic datasets assess the performance of MVRL methods, and the framework is illustrated in Figure 3. We believe that the multi-view data describes the same object, which represented by a high-dimensional embedding Gd where the feature dimension and is the size of the data, and we call it God Embedding. Each view of data is regarded as non- linear transformation of part (or all) of For example, we choose K 2, d 100, and then X1 = o (G(0 50 + CR/2,:)) X2 &x (G(50 CR/2: 100).:), where Φ1 and Φ are non-linear transformations, and C referred to as common rate. The common rate defined as follows:
Definition 1 (Common Rate) For two view data X = X2), common rate is defined as the percentage overlap of the features in X1 and X2 that originate from G.
One can see that the common rate ranges from 0% to 100%. The larger the value, the greater the correlation between the views, and value of indicates that the two views do not share any common dimensions in G. Additionally, we construct the downstream tasks by directly transforming the God Embedding G. Each task T, (G),, where thy is a transformation, and T7 represents the j-th task. By setting different G, common rates, dx. and Bhy we can create various synthetic datasets evaluate the MVRL methods. Finally, X₂ are observable to the MVRL methods for learning the representation, and the learned representation will be used to classify/regress T3 to examine the performance of each method. Detailed implementation is given in Appendix A.4.
6.2 Performance on Synthetic Datasets (RQ2)
We generate the synthetic datasets with different common rates, and the proposed NR-DCCA and other baseline methods are compared. As shown in Figure 4, one see that the DCCA-based

---

0.3
0.2
R2 pe.ppfofrraacc
0.1
0.0
LINEAR DOCA
DCCAE
-0.1
NR-DCCALOUSS)
200
400
600
800
1000
1200
Epoch
(a) Performance
16
NESiIm ot drms
Noloo COT
0.25
0.20
10
D000
0.15
0004
0004 WANTT
200
400
600
800
1000
1200
200
500
800
1200
Epoch
Epoch
(b) Correlation
NESum of weight matrices
0.5
20.000225
0.000100
Dengiiing
0.00075
0.1
0004
5331
0.000000
(0000A
0.0
Epocs
Epoch
(d) Reconstruction
(c) Denoisng
Figure 4: (a) Mean and standard deviation of the (D)CCA-based method performance across syn- thetic datasets in different training epochs. (b) The mean correlation between noise and real data after transformation varies with epochs.
(c) Average NESum across all weights within the trained encoders. (d.e) The mean of reconstruction and denoising loss on the test set.
methods (e.g. DCCA, DCCAF, DCCA_PRIVATEE will encounter model collapse during training, and the variance of accuracy also increases. Linear CCA demonstrates stable performance, while the best accuracy is not good as DCCA-based methods. Our proposed NR-DCCA achieves state- of-the-art performance as well as training stability to prevent model collapse. The results at the final epoch for all common rates also presented in Table 3 in Appendix A.
Considering that we believe that the low-rank property (i.e. highly self-related and redundant) the weight matrices is the root cause of the model collapse, we utilize NESum to measure the correlation among filters in weight matrices. As shown in (b) of Figure 4, our findings demonstrate that the NR approach effectively reduces filter redundancy, thereby preventing the emergence of low-rank weight matrices and thus averting model collapse.
weight matrices defined in A.8). Higher NESum represents lower redundancy

---

Moreover, according to our analysis. the correlation should be invariant if neural networks have CIP. Therefore, after training DCCA, DCCAE, and NR-DCCA, we utilize the trained encoders project the corresponding view data and randomly generated Gaussian white noise and then compute their correlation, as shown (c) of Figure It can be observed that, except for our method (NR-DCCC as training progresses, other methods increase the correlation between unrelated data. It should be noted that this phenomenon always occurs under any common rates.
Given that the full-rank weight matrix not only produces features that are linearly reconstructed but also discriminates noise the inputs, we also present the mean value pf Reconstruction and Denoising Loss across different common rates in (d) of Figure 4. Notably, NR-DCCA achieves markedly lower loss, comparable that observed with Linear CCA, whereas alternative DCCA- based approaches generally lose the above properties.
6.3 Consistent Performance on Real-world Datasets (RQ3)
We further conduct experiments on three real-world datasets: PolyMnist (Sutter et al. 2021), CUB (Wah et al. 2011), Caltech (Deng et al. 2018). Additionally. we use a different number of views in PolyMnist. The results are presented in Figure and the performance of the final epoch in the figure is presented in Table 3 in the Appendix A. Generally, the proposed NR-DCCA demon- strates competitive and stable performance. Different from the synthetic data, the DCCA-basel methods exhibit varying degrees of collapse on various datasets, which might be due to the complex nature of the real- real-world views.
Pry Mer.317
Ady Nar
205
Tonch
Figure 5: Performance of different methods real-world datasets. Each column represents the performance on specific dataset. The number of views in the dataset is denoted in the parentheses next to the dataset name.
Conclusions
We propose novel noise regularization approach for DCCA in the context of MVRL, and it can prevent model collapse during the training, which is an issue observed and analyzed this paper for the first time. Specifically, we theoretically analyze the CIP in Linear CCA and demonstrate that it is the key to preventing model collapse. To this end, we develop a novel NR approach to equip DCCA with such property (NR-DCCA). Additionally, synthetic datasets with different common rates are generated and tested, which provide a benchmark for fair and comprehensive comparisons of different MVRL methods. The NR-DCCA developed in the paper inherits the merits of both Linear CCA and DCCA to achieve stable and consistent outperformance in both synthetic and real-world datasets. More importantly, the proposed noise regularization approach can also be generalized to other DCCA-basee methods (e.g., DGCCA).
In future studies, we wish explore the potential of noise regularization in other representation learning tasks, such as contrastive learning and generative models. It is also interesting to further investigate the difference between our developed NR and other neural network regularization ap- proaches, such as orthogonality regularization (Bansal et al. 2018, Huang et al. 2020) and weight decay (Loshchilov & Hutter 2017, Zhang et al. 2018, Krogh & Hertz 1991). Our ultimate goal is to make the developed noise regularization pluggable and useful module for neural network regularization.
10

---

Acknowledgments
The work described in this paper was supported by the Research Grants Council the Hong Kong Special Administrative Region, China (Project No. PolyU/25209221 and PolyU/15206322), and grants from the Otto Poon Charitable Foundation Smart Cities Research Institute (SCRI) at the Hong Kong Polytechnic University (Project No. P0043552). The contents of this article reflect the views of the authors, who are responsible for the facts and accuracy the information presented herein.
References
Agrawal, K. K., Mondal, A. K., Ghosh, A. & Richards, B. (2022), 'a-req: Assessing representa- tion quality in self-supervised learning by measuring eigenspectrum decay', Advances in Neural Information Processing Systems 35, 17626-17638.
Akaho, S. (2006), 'A kernel method for canonical correlation analysis', arXiv preprint cs/0609071.
An, G. (1996), 'The effects adding noise during backpropagation training on a generalization performance' Neural computation 8(3), 643-674.
Anderson, T. W., Anderson, T. W., Anderson, T. W. & Anderson, W. (1958), An introduction to multivariate statistical analysis, Vol. 2, Wiley New York.
Andrew, G., Arora, R., Bilmes, J. & Livescu, K. (2013). Deep canonical correlation analysis, in "International conference on machine learning', PMLR, pp. 1247-1255.
Avron, H., Boutsidis, C., Toledo, S. & Zouzias, A. (2013), Efficient dimensionality reduction
for canonical correlation analysis, 'International conference on machine learning', PMLR, pp. 347-355.
Bansal, N., Chen, X. Wang, Z. (2018), 'Can we gain more from orthogonality regularizations in training deep networks?', Advances in Neural Information Processing Systems 31.
Belitskii, G. et al. (2013), Matrix norms and their applications, Vol. 36, Birkhäuser.
Benton, A., Khayrallah, H., Gujral, B., Reisinger, D. A., Zhang, S. & Arora, R. (2017), *Deep generalized canonical correlation analysis', arXiv preprint arXiv:1702.02519)
Bishop, C. M. (1995), 'Training with noise is equivalent to tikhonov regularization' Neural Com- putation 7(1), 108-116.
Chang, C.-C. & Lin, C.-. (2011), 'Libsvm: library for support vector machines', ACM transac- tions on intelligent systems and technology (TIST) 2(3), 1-27.
Chang, X., Xiang, & Hospedales, T. M. (2018), Scalable and effective deep cea via soft decor- relation, in "Proceddings of the IEEE Conference on Computer Vision and Pattern Recognition', pp. 1488-1497.
Chapman, J., Aguila, A. L. & Wells, L. (2022), 'A generalized eigengame with extensions multi- view representation learning', arXiv preprint arXiv:2211.11323.
Chapman, & Wang, H.-T. (2021), 'Cca-zoo: A collection of regularized, deep learning based, kernel, and probabilistic cea methods in a scikit-learn style framework'. Journal of Open Source Software 6(68), 3823.
Journall
D' Agostini, G. (1994), 'On the use of the covariance matrix to fit correlated data', Nuclear Instru- ments and Methods Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 346(1-2), 306-311.
De Bie, T. & De Moor, (2003), the regularization of canonical correlation analysis', Int. Sympos. ICA and BSS pp. 785-790.

---

Deepak, K., Srivathsan, G., Roshan, S. & Chandrakala, S. (2021), 'Deep multi-view representation learning for video anomaly detection using spatiotemporal autoencoders' Circuits Systems and Signal Processing 40(2).
Deng, C., Chen, Z., Liu, X., Gao, X. & Tao, (2018), "Triplet-based deep hashing network for cross-modal retrieval', IEEE Transactions on Image Processing 27(8), 3893-3903.
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P. Zisserman, A. (2021), With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, in 'Proceedings of the IEEE/CVF International Conference on Computer Vision', pp.
Fan, W., Ma, Y., Xu, H., Liu, X., Wang, J.. Li, Q. & Tang, J. (2020), Deep adversarial canonical cor- relation analysis, in 'Proceedings of the 2020 SIAM International Conference on Data Mining', SIAM, pp. 352-360.
Feichtenhofer, C., Pinz, A. & Zisserman, A. (2016), Convolutional two-stream network fusion for video action recognition, in 'Proceedings of the IEEE conference computer vision and pattern recognition', pp. 1933-1941.
Fern, X. Z., Brodley, C. E. & Friedl, M. (2005), Correlation clustering for learning mixtures of canonical correlation models, in 'Proceedings of the 2005 SIAM International Conference on Data Mining', SIAM, pp. 439-448.
Garrido, Q., Balestriero, R., Najman, L. & Lecun, Y. (2023), Rankme: Assessing the downstream performance of pretrained supervseee representations by their rank, in 'International Confer- ence on Machine Learning', PMLR, pp. 10929-10999 10974.
Geng, Y., Han, Z., Zhang, & Hu, Q. (2021), Uncertainty-awar multi-vies representation learn- ing, in Prooeedinng of the AAAI Conference on Artificial Intelligence', Vol. 35, pp. 7545-7553.
Gong, C., Ren, T., Ye, M. & Liu, Q. (2020), 'Maxup: simple way improve generalization of neural network training', arXiv preprint arXiv:2002.09024
Guo, H., Wang, J., Xu, M., Zha, Z.-. Lu, H. (2015), Learning multi-view deep features for small object retrieval in surveillance scenarios, in 'Proceedings of the 23rd ACM international conference on Multimedia', pp. 859-862.
Hardoon, D. R., Szedmak, S. & Shawe- Taylor, J. (2004), 'Canonical correlation analysis: An overview with application to learning methods' Neural computation 16(12), 2639-2664.
He, Z., Rakin, A. S. & Fan, D. (2019), Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack, in 'Proceedings the IEEE/CVF Con- ference Computer Vision and Pattern Recognition', pp. 588-597.
Hoerl, A. E. & Kennard, R. W. (1970), 'Ridge regression: applications to nonorthogonal problems', Technometrics 12(1), 69-82.
Horst, P. (1961), 'Generalized canonical correlations and their applications to experimental data', Journal of Clinical Psychology 17(4), 331-347.
Hotelling, H. (1992), 'Relations between two sets of variates', Breakthroughs in statistics: method- ology and distribution pp. 162-190.
Huang, L., Liu, L., Zhu, F., Wan, D., Yuan, Z., Li, B. & Shao, L. (2020), Controllable orthogonal- ization in training dnns, in 'Proceddings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition', pp.
Hwang, H., Kim, G.-H., Hong, S.& & Kim, K.-E. (2021), 'Multi-view representation learning via total correlation objective', Advances in Neural Information Processing Systems 34, 12194-12207,
Jing, I.., Vincent, P., LeCun, Y. Tian, (2021), 'Understanding dimensional collapse in con- trastive self-supervised learning'. arXiv preprint arx1v:2110000338..
12

---

Karpathy, A. & Fei-Fei, L. (2015), Deep
alignments for generating image de- scriptions, in PProceedings the IEEE conference on computer vision and pattern recognition', pp. 3128-333..
Kettenring, J. R. (1971), 'Canonical analysis of several sets of variables', Biometrika 58(3), 433- 451.
Kim, T.-K., Wong, S.-F. & Cipolla, R. (2007), Tensor canonical correlation analysis for action classífication, in '2007 IEEE Conference on Computer Vision and Pattern Recognition', IEEE, pp. 1-8.
Kornblith, S., Norouzi, M., Lee, H. & Hinton, G. (2019), Similarity of neural network representa- tions revisited, in 'International conference machine learning', PMLR, pp. 3519-3529.
Krogh, A. & Hertz, J. (1991), 'A simple weight decay can improve generalization' Advances in neural information processing systems 4.
Lahat, D., Adali, T. & Jutten, C. (2015), 'Multimodal data fusion: an overview of methods, chal- lenges, and prospects', Proceedings of the IEEE 103(9), 1449-1477.
Le, Q & Mikolov, T. (2014), Distributed representations of sentences and documents, in 'Interna- tional conference on machine learning', PMLR, pp. 1188-1196.
Li, Z., Tang, C., Zheng, X., Liu, X., Zhang, W. & Zhu, E. (2022), 'High-ordet correlation preserved incomplete multi-view subspace clustering'. IEEE Transactions on Image Processing 31, 2067- 2080.
Loshchilov, I. & Hutter, F. (2017), 'Decoupled weight decay regularization', arXiv preprint arXiv:1711.05101
Mao, Xu, W., Yang, Y., Wang, J., Huang, Yuille, (2014), 'Deep captioning with multi- modal recurrent neural networks (m-rnn', arXiv preprint arXiv:1412.660
Morcos, A. S., Barrett, D. G., Rabinowitz, N. C. & Botvinick, M. (2018), 'On the importance of single directions for generalization', arXiv preprint 1803.06900
Nguyen, T., Raghu, M. & Korbllith, (2020), 'Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth', arXiv preprint arXiv:2010.15327.
Nie, F., Li, J., Li, X. et al. (2017), Self-weighted multiview clustering with multiple graphs., in 'ICCAI' pp. 2564 2570.
Nielsen, A. A. (2002), 'Multiset canonical correlations analysis and multispectral, truly multitem- poral remote sensing data', IEEE transactions on image processing 11(3), 293-305.
Petersen, K. B., Pedersen, M. S. et al. (2008), 'The matrix cookbook', Technical University of Denmark 7(15),
Poole, B., Sohl-Dickstein, & Ganguli, S. (2014), 'Analyzing noise in autoencoders and deep networks', arXiv preprint vrxxiv14406 1831
Prechelt, L. (1998), 'Automatic early stopping using cross validation: quantifying the criteria', Neu- ral networks 11(4), 761-767.
Raghu, M.. Unterthiner, T., Kornblith, S., Zhang, C. & Dosovitskiy, A. (2021), 'Do vision transform- ers see like convolutional neural networks?' Advances in neural information processing systems 34, 12116-12128.
Saxe, A. M., McClelland, ment in deep neural networks', Proceedings of the National Academy of Sciences 116(23), 11537- 11546.
Ganguli, (2019), 'A mathematical theory of semantic develop-
Sietsma, & Dow, J. (1991), 'Creating artificial neural networks that generalize', Neural net- works 4(1), 67-79.
13

---

Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S. & Srebro, N. (2018), "The implicit bias of gradient descent on separable data', Journal of Machine Learning Research 19(70), 1-57.
Srivastava, N. & Salakhutdinov, R. R. (2012), 'Multimodal learning with deep boltzmann machines', Advances in neural information processing systems 25.
Staerman, G., Adjakossa, E., Mozharovskyi, P., Hofer, V., Sen Gupta, J. & Clémençon, S. (2023), 'Functional anomaly detection: a benchmark study', International Journal of Data Science and Analytics 16(1), 101-117.
Sun. J., Xiu, X., Luo, Z. & Liu, W. (2023), 'Learning high-order multi-view representation by new tensor canonical correlation analysis', IEEE Transactions on Circuits and Systems for Video Technology.
Sun. L., Ceran, B. & Ye, J. (2010). A scalable two-stage approach for class of dimensionality reduction techniques, in 'Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining', pp. 313-322.
Sun, L., Ji, S. & Ye, J. (2010). 'Canonical correlation analysis for multilabel classification: A least- squares formulation, extensions, and analysis', IEEE Transactions on Pattern Analysis and Ma- chine Intelligence 33(1), 194-200.
Sutter, T. M., Daunhawer, & Vogt, J. E. (2021), 'Generalized multimodal elbo', arXiv preprint arXiv:2105.02470.
Tao, H., Hou, C., Yi, D., Zhu, J. & Hu, D. (2019), 'Joint embedding learning and low-rank ap- proximation: framework for incomplete multiview learning'. IEEE transactions on cybernetics 51(3), 1690-1703.
Tian, Y. & Zhang, Y. (2022), 'A comprehensive survey regularization strategies in machine learn- ing", Information Fusion 80, 146-166.
Tschannen, M., Bachem, O. & Lucie, M. (2018), 'Recent advances autoencoder-based represen- tation learning', arXiv preprint arXiv: 1820080089
Tuzhilina, E., Tozzi, L. & Hastie, T. (2023), 'Canonical correlation analysis in high dimensions with structured regularization', Statistical modelling 23(3), 203-227.
Wah, C., Branson, S., Welinder, P., Perona, P. & Belongie, S. (2011), 'The caltech-ucsd birds-200- 2011 dataset', california institute of technology.
Wang, W., Arora, R., Liveseu, K. & Bilmes, J. (2015), On deep multi-view representation learning, in 'International conference on machine learning', PMLR, pp. 1083-1092.
Wang, W., Yan, X., Lee, H. & Livescu, K. (2016), 'Deep variational canonical correlation analysis',
arXiv preprini 1rXiv:1610.0345.
Wang, Z., Xiang, C., Zou, W. & Xu, C. (2020), 'Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles", Advances in Neural Information Processing Systems 33, 19099-19110.
Wei, Xia, Y. Zhang, Y. (2019), 'M3net: multi-model, multi-size, and multi-view deep neural network for brain magnetic resonance image segmentation', Pattern
Patter Recognition 91, 366-378.
Wong, H. S., Wang, L., Chan, R. & Zeng. T. (2021). 'Deep tensor cca for multi-view learning', IEEE Transactions on Big Data 8(6). 1664-1677.
Xu, J., Zheng, H., Wang, J., Li,
Fang, (2020), 'Recognition of eeg signal motor imagery intention based on deep multi-view feature learning', Sensors 20(12), 3496.
Yan, H., Cheng, L.,, Ye, Q., Yu, D.J. & Qi, Y. (2023), 'Robust generalized canonical correlation analysis', Applied Intelligence pp. 1-16.
Yan, X., Hu, S., Mao, Y., Ye, Y. Yu, H. (2021), 'Deep multi-view learning methods: A review", Neurocomputing 448, 106-129.
14

---

Yang, M., Li. Y., Huang, Z. Liu, Z., Hu, P. & Peng. X. (2021), Partially view-aligned representá- tion learning with noise-robust contrastive loss, in PProceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 1134-1143.
Yao, Y., Rosasco, L. & Caponnetto, A. (2007), 'On early stopping in gradient descent learning', Constructive Approximation 26, 289-315.
Yin, & Sun, S. (2021), 'Incomplete -iew clustering with reconstructed views', IEEE Trans- actions Knowledge and Data Engineering
Zhang, C., Cui, Y., Han, Z., Zhou, J. T., Fu, H. & Hu, Q. (2020), 'Deep partial multi-view learning', IEEE transactions on pattern analysis and machine intelligence 44(5), 2402-2415.
Zhang, C., Fu, H., Hu, Q. Zhu, P. & Cao, X. (2016). 'Flexible mieti-view dimensionality co- reduction', IEEE Transactions on Image Processing 26(2), 648-639
Zhang, C., Han, Z., Fu, H., Zhou, J. T., Hu, Q. et al. (2019), 'Cpm-nets: Cross partial multi-view networks' Advances in Neural Information Processing Systems 32.
Zhang, C., Liu, Y. & Fu, H. (2019), Ae2-nets: Autoencoder in autoencoder networks, in 'Proceed- ings of the IEEE/CVP conference on computer vision and pattern recognition', pp. 2577-2585.
Zhang, G., Wang, C., Xu, B. & Grosse, R. (2018), 'Three mechanisms of weight decay regulariza- tion', arXiv preprint arXiv: 1810. 12281.
Zhang, H., Wang, S., Ioamnidis, V. N., Adeshina. S., Zhang, J., Qin, X., Faloutsos, C., Zheng, D., Karypis, G. & Yu, P. S. (2023), 'Orthoreg: Improving graph-regularized mips via orthogonality regularization' arXiv preprint arXiv:2302.00//09
Zhang, Y., Zhang, J., Pan, Z. Zhang, D. (2016), 'Multi-viww dimensionality reduction via canon-
ical random correlation analysis', Frontiers of Computer Science 10, 856-869.
Zhao, H., Ding, Z. & Fu, Y. (2017), Multi-view clustering via deep matrix factorization, in 'Pro- ceedings of the AAAI conference on artificial intelligence', Vol. 31.
Zheng, Q. Zhu, J., Li, Z., Pang, S., Wang, J. & Li, Y. (2020), 'Feature concatenation multi-view subspace clustering'. Neurocomputing 379, 89-102.
Zhou, Paffenroth,
(2017), Anomaly detection with robust deep autoencoders, 'Pro- ceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining', pp. 665-6744
Zhu, X., Zhou, W. Li, H. (2018), Improving deep neural network sparsity through decorrelation regularization., in Tiecai' pp. 3064-322270
15

---

A Appendix
A.1 Proof of Theorem
A.1.1 Preparations
To prove Theorem 1, need first prove the following Lemmas.
Lemma Given specific matrix and a zero-centered with respect to rows, the product BC also zero-centered with respect rows.
Proof of Lemma 1:
Let Bi, and Ca2 denote the (i.j)-th entry of B and C, respectively. Then have:
(BC),s Σ Cr.
Since each of C' has mean of 0, we have EØ=1 Cr.j 0, Vr. For the mean value of i-th row of BC, we can write:
n
n
BC)ss
11
1
j=1
Bir Cri Cri
(9)
Bir
Σ Cru
B
n
0
The Moore-Penross Inverse (MPI) (Petersen et al. 2008) will be used for analysis, and the MPI is defined as follows:
Definition 2 Given specific matrix Y, its Moore- Pooor-Pennooee Inverse (MPI) is denoted as Y Y
satisfies: YY+Y = Y+ Y YY+ is symmetric, and + symmetric.
The MPI + is unique and always exists for any Y. Furthermore, when matrix Y is invertible, its inverse matrix Y- is exactly Y+. Using the definition MPI, we can rewrite the formulation of CCA. In particular, Corr(..) can be derived by replacing the inverse with MPL Using Corr A&) as an example, the following Lemma holds:
Lemma (MPI-based CCA) For the k-th view data X and the Gaussian while noise Ag, we have
Corr(Xx, Ak) (n 11-1 trA A 1/2 1/k
(10)
Proof of Lemma 2:
16

---

Corr(Xx: As) ) tr((En) 1/2 -12 "
-tr(2ing
=tr(E22 1/2 22
E12)
-tr(2z2 11 Σ12) 1/2
1
un( (r((Ak
(n 1)2
(1I)
(n 1)
(Ax (X+44))
(n 1)2 (AAA A (X, xx
(aA Ax Xx)
(n 102
The first row based on the definition Corr, the second row is because the trace is invariant under cyclic permutation, the fifth row is to replace matrix inverse by MPI and the ninth row is due to Y- -Y-Y++)) (Petersen et al. 2008).
Lemma Given specific matrix Y and its MPI Y+, let Rank(Y) and Rank(Y++)) be the ranks Y and Y+ respectively. true that:
Rank (Y) = Rank(Y
Rank(Y (onkkY+YY++.. iY
Proof of Lemma 3:
Firstly, the column space of y+Y is subspace of the column space of Y. Therefore, Rank(Y+Y) S Rank(Y). On the other hand, according to the definition of MPI (Petersen et al. 2008), we know that Y Y(Y+Y)) Since the rank of product of matrices is at most the min- imum of the ranks of the individual matrices, we have Rank(Y) < Rank(Y+ Combining the two inequalities, we have Rank(Y) Rank(Y++)) Furthermore, since (Y+ (it holds that Y+ = ++YY+ according to the definition of MPI (Petersen et al. 2008)), Y+Y is an idempotent and symmetric matrix, and thus its eigenvalues must be or So the sum of its eigenvalues is exactly its rank. Considering matrix trace is the sum of eigenvalues of matrices, we have Rank(Y tr(Y+)).
y+Y
Lemma 4 Xa, < Rank(X*). when We is not full-rank matrix and X is full-rank matrix.
Proof of Lemma 4: Since the rank of a product of matrices is at most the minimum of the ranks of individual matrices, we have Rank Rank(W,, (B) < minn (W&), Rank(Xg) (Xx)). Considering X is full-rank, Rank(Xx) = min(dg, n)) and then Rank(W; Xx) < min (ain(RRn((W))) Rank(X;)) min(Ran (Rankaa, (W&), min(d,, n)). Since W. not full-rank, we have Rank(W&) < dg. conclusion, Rank(W, < min(d,, min(dy, n)) and then (ank(W, < dy RRank(x)).
A.1.2 Main proofs of Theorem 1
We prove the two directions of Theorem in the following two Lemmas. First, we prove CIP holds if We is square and full-rank matrix.
Lemma For any k. if & is a square and full-rank matrix, the correlation between Xx and Ag remains unchanged before and after the transformation by We (i.e. CIP holds for Wy). Mathemati- cally, we have Corr(Xx, A&) = Corr(WyX,, WgAs).
Proof of Lemma 5:
Firstly, we have the k-th view data X5 to be full-rank, as we can always delete the redundant data, and the random noise Aş is full-rank as each column is generated independently. Without loss
17

---

of generality, we assume that all the datasets X are zero-centered with respect to row (Hotelling 1992), which implies that Ar and W are both zero-centered matrices 1. When computing the covariance matrix, there is no need for additional subtraction of the mean of row, which simplifies our subsequent derivations. And We is always full-rank since Linear CCA seeks full-rank W₂ Xg. Then by utilizing Lemma 2, we derive that the correlation between X and Ag remains unchanged before and after the transformation:
Corr(W;,aa, Wx
tr (WW))))) (WAAA)"WAAA W Wx
(n 1)
W&.AA) WW W (W....) Wax))1
(n 172
(r(( Wx,4)) WWw.a) (W W.X.)
(n 1)2
(r(A
A. X/
1)2
Corr(Xe.
(12) The first row is based on Lemma 2, the second row is because given two matrices B and C, (BC)+ = (B+ (BC++ always holds (Petersen et al. 2008), and the third row utilizes the properties of full-rank and square matrix Wa: W = W which means W W = W&W, Ia, (Petersen et al. 2008).
Then we prove that W is full-rank matrix if CIP holds and W is square. Lemma 6 For any k, if Corr(X,, As) = Con A&As) and W₂ is a square matrix, then W& must be a full-rank matrix.
Proof of Lemma 6:
This Lemma is equivalent its contra-positive proposition: if Wa is not full-rank matrix, there exists random noise data A₁ such that 7 = Corr(W, (A&) Corr A&) is not O. And we find that when WR is not full-rank, there exists As Xg such that np # We have the following derivation:
CoorrW, X2, W.Ae)) Corr(Xe.AA)
((( (WLA) (WX.)) (n- tr(A AX A xyy2
172 M[(W (w) AW) (W
W4 X2
m xxx X/
1)2 M((WWWW Ax)* *W W. W. W.
Ir( Ax X
(13)
The first row is the definition of NR loss with respect to Wi. the second row is based on the new form of CCA, the third row is because given two specific matrices B and C, it holds the equality (BC)+ BC)* (BC+C)** (Petersen et al. 2008), and the fourth row utilizes the properties of full-rank matrix: for full-rank matrices X₁ and Ag, whose sample size is larger than dimension size, they fulfill: Xx Id., A& Id, (given a specific full-rank matrix Y, if its number of rows smaller than that of cols, it holds that Y- Y'(YY")- which means that YY+ I) (Petersen et al. 2008).
Let us analyze the case when A X5:
(n 1)s (r( We WxXx) + W. W. Wx W Wx.X (n 1)2 (xx X,X, X/2
m
1) 1.W Wx) WW W (n 1)2 xxx
(14)
The first row is to replace A with Xę. the second row is because X X (W, WWW (W, Xx - W,KKA which are based on the definition of MPI that given specific matrix Y, ++yy+ + (Petersen et al. 2008).
- and
18

---

As result, we can know that when the random noise data A₁ is exactly X2 and Wi is not full-rank, ?e can not be zero:
1
tr( Wi, x
W XxX)
(n 1)2 tr( tr(x Xx)
7h
1)2
1
Rank W Wx.Xx)
Rank (
(n 1)2
(n 1)2
(15)
1
1
* (n
Rank(Xx)
Rank(Xx)
1)2
(n 112
# 0
The first row is due to Equation 14, the second row is based on Lemma 3 that tr((We) W WxXx)) = Rank(W, Xa) and tr(xt Xx) = Rank(X), and the third row because of Lemma 4.
Finally, if Mh is always constrained to 0 for any Ag. then We must be a full-rank matrix.
Combining Lemma 5 and 6, we complete the proof.
A.2 Proof of Theorem 2
For linear regression problem R arg ming |RB ClIF, where R pout imput div dim is the weight matrix, and B E IR inippt.diimx, C E Rouupar..dmxx are the input and target matrix (imput dim < n), respectively. R' has closed-form solution: R CBBB and therefore, it holds that:
min |BB C J
- |CB'(BB')= B Cl
(16)
_ (CB((BB)+B-CC C'IFF
- |[BBB--C
Equation a the use of the closed-form solution and Equation b is to replace the matrix inverse by MPI. Equation because Y'(YY')+ Y+ (Petersen al. 2008).
Given k-th view, when Wy is square possesses CIP, W₁ is full-rank. Using the above equation, we have:
min P&W&& - Xxlla - (Xx() - XAllF
b (VX Xxlle
$ |Xx(W W W. WaXx Xxlle
(17)
4 |XxX Xxlle
0
Equation is due Equation 16 (Petersen al. 2008). Equation holds because given two matrices B and C, (BC)+ = (B- BCC C)+ always holds and Equation e is because for full-rank matrix Xx gd, (d < n). X = Iae- Equation e utilizes the properties of full-rank and square matrix W5: W = W which means W W = W&Wt Id, (Petersen et al. 2008). Equation d is based on the definition of MPI: given specific matrix Y and its V+, it holds that +Y+Y = =. We show the first property in Theorem 2.
19

---

As for the second property:
min [|Q; (.G. + As) W.Xx, |P
(WL (IA(X) A.)) W.xelle
A) A.)) Walle
IW (OC - Welle
4 [W,(X, + + +44)) WLAL - -
4 [Wa[X, + Ab) - WxA,(X Ax) + Ab) -exxee
[W2,, WxA,(X, AA) Ad)l
4 [W (Xx + 4,)'(x A.)II
WWLAAl IK4, - (X2 A6)* (X+ A:)IIIF
WWLdell Vu(I. (Xe Ab) (XCx + Az))
WaAdllee RRnk(( (X2 + A. (x)
VTl WaAdlle
(18)
Equation a because of Equation 16. Equation holds because given two matrices B and C,
(BC)+ (B+ BCY+ (BC+C)++++++ always holds and Equation c is because we assume X + Ag is
full-rank matrix. Equation utilizes the properties of full-rank and square matrix Wa: W Wi
W. = Tde Equation is based on the definition of MPI: (X + Ax) +
(Xx Ac)
(Xx + A+). Equation j holds because given two specific matrices B and C, ||C[]| F [[]][
| (Belitskii et al. 2013). Equation k and is because given specific matrix B, - B+ is an
idempotent matrix and ||z B Bl|F Vir((I B+ B) B B)) Vir(I B)
Now, we use (s to donate the (f, entry of WRAx and the expected value of square of the Frobenius norm of Ag is:
E W A 11
E
ΣΣ ((
(19)
Expanding the product (W,. we have:
(WaAs)is= = Σ (Wa))
(20)
Substituting back into the expectation, we get:
1 ΣΣ
E [ = E ΣΣ Σ ((Wa)r
Σ [(Wa)s
(21)
Since the elements of A are i.i.d. with zero mean and unit variance, the expectation of their squares and the cross terms vanish due to the zero mean. Therefore, we have:
ΣΣ E E(Wa)ar(Aanss
ΣΣ Wk) 1|
(22)
Hence, we have shown that
E [I ||W112
(23)
This completes the proof.
A.3 Details of Datasets and Baselines
Synthetic datasets:
All the datasets used in the paper are either provided or open datasets. Detailed proofs of all the Theorems the main paper can be found in the Appendix. Both source codes and appendix can be downloaded from the supplementary material.
20

---

We make 6 groups of multi-view data originating from the same G E R (we set 4000, d 100). Each group consists of tuples with 2 views (2000 tuples for training and 2000 tuples for testing) and distinct common rate. Common rates of these sets are from (0%, 20%, 40%, 60%, 80%, 100%] and there are 50 downstream regression tasks. We report the mean and standard deviation of R2 score across all the tasks.
Real-world datasets:
PolyMnist (Sutter et al. 2021): A dataset consists of tuples with 5 different MNIST images (60, 000 tuples for training and 10, 000 tuples for testing). Each image within tuple possesses distinct backgrounds and writing styles, yet they share the same digit label. The background of each view is randomly cropped from image and is not used in other views. Thus, the digit identity represents the common information, while the background and writing style serve as view-specific factors. The downstream task is the digit classification task. CUB (Wah et al. 2011): A dataset consists of tuples with deep visual features (1024-d) extracted by GOOGLENET and text features (300-d) obtained through DOC2VEC (Le & Mikolov 2014) (480 tuples for training and 600 tuples for testing). This MVRL task utilizes the first 10 categories of birds in the original dataset and the downstream task is the bird classification task. Caltech (Deng al. 2018): A dataset consists of tuples with traditional visual features extracted from images that belong to 101 object categories, including an additional background category (6400 tuples for training and 9144 tuples for testing). Following Yang et al. (2021), three features are used as views: 1, 984-d d HOG feature, a 512-d GIST feature, and 928-d SIFT feature.
Baselines: All of our experiments are conducted with fixed random seeds and all the performance of downstream tasks is the average value of 5-fold cross-validation. We use single 3090 GPU. The CCA-zoo package is adopted as the implementation of various CCA/GCCA-base and the original implementation of MVTCAE is employed. Both baselines and our developed NR- ACCA/NR-DGCCC are implemented in the same PyTorch environment (see requirements.tx in the source codes).
methods,
Direct method:
* CONCAT straightforwardly concatenates original features from different views.
CCA methods:
PRCCA Tuzhilina et al. (2023) preserves the internal data structure by grouping high- dimensional data features while applying an 12 penalty to CCA,
Linear CCA (Wang et 2015) employs individual linear layers to project multi-view data and then maximizes their correlation defined in (Hotelling 1992).
Linear GCCA uses linear layers to maximize the correlation of multi-view data defined in (Benton et al. 2017).
Kernel CCA Methods:
KCCA (Akaho 2006) employs CCA methods through positive-defiite kernels.
DCCA-based methods:
DCCA (Andrew et al. 2013) employs neural networks to individually project multiple sets of views, obtaining new representations that maximize the correlation between each pair of views.
DGCCA (Benton et al. 2017) constructs a shared representation and maximizes the corre- lation between each view and the shared representation.
DCCA_EY (Chapman et al. 2022) optimizes the objective of CCA via a sample-based EigenGame.
DCCA_GRE (Chapman et al. 2022) solves the objective of CCA by sample-based gen-
eralized Hebbian algorithm.
DCCAE/DGCCAR (Wang et al. 2015) introduces reconstruction objectives to DCCA, which simultaneously optimize the canonical correlation between the learned represen- tations and the reconstruction errors of the autoencoders.
21

---

DCCA PRIVATE/DGCCA PRIVATE (Wang et al. 2016) incorporates dropout and pri- vate autoencoders, thus preserving both shared and view-specific information.
Information theory-based methods:
MVTCCE (Hwang et al. 2021) maximizes the reduction in Total Correlation to capture both shared and view-specific factors of variations.
All CCA-based methods leverage the implementaiion of CCA-Zoo (Chapman Wang 2021). To ensure fairness, the official implementation of MVTCAE while replacing the strong CNN backbone with MLP.
A.4 Implementation Details of Synthetic Datasets
We draw n random samples with dim from Gaussian distribution as G E Rdx= to represent complete representations of n objects. We define the non-linear transformation dy as the addition of noise the data, followed by passing through a randomly generated MLP. To generate the data for the k-th view, we select specific feature dimensions from G' based on a given common rate and then apply O& to those selected dimensions. And we define dh as linear layer, and task Ty is generated by directly passing G through 4i..
A.5 Hyper-parameter Settings
To ensure a fair comparison, we tune the hyyer-parameen of all baselines within the ranges sug- gested in the original papers, including papper-prraeter of ridge regularization, except for the following fixed settings:
The embedding size for the real-world datasets is set as 200, while the size for synthetic datasets is 100. Batch size is min full-size). The same MI. architectures are used for D(G)CCA- based methods. The hyper-parameter r of ridge regularization is set as 0 in our NR-DCCA and NR-DGCCA. And the best for Linear (G)CCA and D(G)CCA-based methods tuned on the validation data (synthetic datasets and PolyMnist: le-3, CUB and Caltech101: 0).
In the synthetic datasets, Linear CCA and Linear GCCA DCCA, DGCCA, DCCAE, and DGCCAE methods utilize a bigger learning rate of 5e 3. DCCA PRIVATE/DGCCA PRIVATE employ slightly higher learning rate of le 2. In contrast, our proposed methods, DR-DCCCINN-DGCCA, utilize the maximum learning rate of 1.5e 2. And the regularization weight is set 200.
minimum learning rate of 1e 4,
In the roaldworld datasets, the learning rates for all deep methods are set to 1e while that of Linear CCA and Linear GCCA are 1e - 5. To expedite the computation of Corr(X;; As), in the real-world datasets, we simply employ Xx(: outdim, :] and Axl: outdim, :] to compute of Corr(X,, A&). The optimal values of NR-DCCA for the CUB, PolyMnist, and Caltech datasets are found to be 1.5, 2, and 10, respectively.
A.5.1 Hyper-parameter Ridge Regularization
In this section, we discuss the effects of hyper-parameter in ridge regularization. Ridge regulariza- tion is commonly used across almost all (D)CCA methods, which improves numerical stability. works by adding an identity matrix / the estimated covariance matrix. However, ridge regulariza- tion mainly regularizes the features, rather than the transformation (i.c., W2 in Linear CCA and fa DCCA) and it cannot prevent the weight matrices in DNNs from being low-rank or redundant. To further support our arguments, we provide the experimental results with different ridge parameters on real-world dataset CUB as shown in Figure 6. One can see that the ridge regularization even damages the performance of DCCA and also leads to an increase in the internal correlation within the feature and the correlation between the feature and noise. our NR-DCCA, we set the ridge parameter to zero. We conjecture the reason is that the large ridge parameter could make the neural network even "lazier" to actively project the data into better feature space, as the full-rank property of features and covariance matrix are already guaranteed.
22

---

Figure 6: The effects of hyper-parammterr of DCCA in the CUB dataset.
A.5.2 Hyper-parameter a of NR-DCCA
is essential in NR-DCCA. Different from the conventional hyper-parameter tuning procedures, the determination of a is simpler, we can search for the smallest a that can prevent the model collapse, and the model collapse can be directly observed on the validation data. Specifically, we increase the a adaptively until the model collapse issue is tackled, i.e., the correlation with noise will not increase or the performance of DCCA will not drop with increasing training epochs, then the optimal is found. To further illustrate the influence of in NR-DCCA, we present performance curves of NR-DCCA CUB under different a. As shown in Figure 7, if ar is too large, the convergence of the training becomes slow; if a is too small, model collapse remains. Additionally, one can see the NR-DCCA outperforms DCCA robustly with a wide range of a.
The choice of the hyper-parametts
D023 Co.A
Figure 7: The effects of Hyper-parameter of NR-DCCA the CUB dataset.
A.6 Effects of the Distribution of Noise
From our theoretical analysis, the most important feature of noise in NR is that the sampled noise matrix is full-rank matrix. Therefore, continuous distributions such the uniform distribution also he applied to NR, which demonstrates the robustness of the proposed NR method. We compare NR-DCCA with different noise distributions on synthetic datasets, and both noises are effective in suppressing model collapse as shown in Table 1.
Table 1: Effect of noise on DCCA and NR-DCCA
Epoch
DCCA
NR-DCCA (Gaussian Noise)
NR-DCCA (Uniform Noise)
100
0.284 + 0.012
0.295 + 0.005
0.291 + 0.004
800
0.137 0.028
0.313 0.004
0.313 0.005
1200
0.106 + 0.027
0.312 + 0.005
0.316 + 0.005
A.7 Effects of Depths of Encoders
In this section, we test effects depths of encoders (i.e. MLPs) on model collapse and NR.
Specifically, we increase the depth of MLPs to observe the variation in the performance of DCCA and NR-DCCA on synthetic datasets. As shown Table 2, The increase network depth results in faster decline in DCCA performance, while NR-DCCA still maintains stable performance.
23

---

Table 2: Performance comparison of DCCA and NR- BR-DCCA across different network depths.
Epoch/R2
hidden layer
hidden layers
3 hidden layers
DCCA
NR-DCCA
DCCA
NR-DCCA
DCCA
NR-DCCA
100
0.284 = 0.012
0.295 + 0,005
0.161 =.011 0.013
0.304 + 0.006
0.071 + 0,084
0.299 = 0.010
800
0.137 = 0.028
0.313 + 0.004
-0.072 = 0.071
0.307 = 0.005
-0.975 + 0.442
0.309 = 0.005
1200
0.106 1 0.027
0.312 1 0.005
-0.154 1 0.127
0.303 1 0.006
-1.412 1 0.545
0.308 1 0.006
A.8
Effects of NR Loss on Filter Redundancy
Extensive research has established a significant correlation between the redundancy of neurons or filters and the compromised generalization capabilities of neural networks, indicating a propensity for overfitting (Wang et al. 2020, Morcos et al. 2018, Zhu et al. 2018). Considering the fully con- nected layer with 1024 units in MLP network as a paradigm, the initial layer's weights, denoted by W E 10104x3x228 28228 can be interpreted as 1024 discriminative filters. These filters operate on images with channels, each of size 28 x 28, with every filter representing a vector in a 3 28 28 dimensional space. Subsequently, a similarity matrix S is constructed, wherein each element Sij quantifies the cosine similarity between the tth and 5th filters, with higher values indicating greater redundancy. further assess filter redundancy in W, we employ NESum, a metric designed for evaluating redundancy and whiten degrees of features (Zhang et al. 2023).
Definition (NESum of Weight) Given weight matrix W Rootput xinput with an accompa- nying output-wise similarity matrix E x and eigenvalues { scending order, the normalized eigenvalue sum is defined as follows:
postput sorted in de-
output
NESum(W)
output
A1
In Figure 8, we present the evolution of the average NESum across all weights within the trained encoders. Notably, we observe sustained increase in NESum exclusively in NR-DCCA throughout prolonged training epochs. This phenomenon underscores the efficacy of the loss of NR in reducing filter redundancy, crucially preventing low-rank solutions.
0.35
0.30
NESum of dnns
10.25
0.20
DCA
0.15
DCCA
OCCAE
DCCA DCCACCC(u
0.10
400
600
1000
1200
Epoch
Figure 8: Average NESum across all weights within the trained encoders.
A.9 Visualization of Learned Representations
To further demonstrate the effectiveness of our method, we employ 2D-ISNE visualization depict the learned representations of the CUB dataset (test set) under different methods. Each data point colored based on its corresponding class, as illustrated Figure 9. There are a total of 10 cat- egories, with 60 data points in each category. A reasonable distribution of learned representations entails that data points belonging to the same class are grouped in the same cluster, which distin- guishable from clusters representing other classes. Additionally, within each cluster, the data points
24

---

should exhibit an appropriate level of dispersion, indicating that the data points within the same class can be differentiated rather than collapsing into single point. This dispersion is indicative of the preservation of as many distinctive features of the data possible.
From Figure. 9, we can observe that CCA, DCCA/ DGCCA have all confused the data from dif- ferent categories. Specifically, CCA completely seatters the data points as it cannot handle non- linear relationships. By incorporating autoencoders, DCCAE / DGCCAE and DCCA.PRIVATE / DGCCA.PRIVATE have partially separated the data; however, they have not fully separated the green and orange categories. NR-DCCA NR-DGCCA is the only method that successfully sepa- rates all categories.
It is worth noting that approach not only separates the data into different clusters but also main- tains dispersion within each cluster. Unlike DCCA PRIVATE DGCCA PRIVATE, where the data points within cluster form strip-like distribution, method ensures that the data points within each cluster remain appropriately scattered.
(a) CONCAT
(b) CCA
(c) MVTCAE
(d) DCCA
(e) DCCAE
(1) NR-DCCA
(g) DRCV.ERIVATE
(h) DGCCA
ii) DGCCAE
i NR-DGCCA
(k) DGCCA PRIVATE
Figure 9; Visualization of the learned representations with t-SNE in the CUB dataset.
25

---

A.10 DGCCA and NR-DGCCA
This section presents the experimental results for DGCCA and NR- BR-DGGCA, which supplement the results of GCCA and NR-DCCA presented in the main paper. In general, DGCCA is a variant of DCCA, and hence the proposed noise regularization approach can also be applied. We repeat the experiments in Figures 4,and 5, and hence we have the results for DGCCA in Figure 10, and 11. One can see that the proposed noise regularization approach can also help DGCCA prevent model collapse, proving its generalizability.
0.325
0.300
R2 perfomance
0.275
0.250
0.2255
0.200
LINEAR OCCA
DGCCA
0.175
DGECAE
NR-DGCCA(our)
0.150
200
400
600
800
1000
1200
Epoch
(a) Performance
0.30
232000
0.25
231000
230000
0.20
229000
228000
USSA SECA
00004
226000
200
400
000
Epech
800
1000
1200
200
400
500
Epoch
000
1000
1200
(b) Correlation
(c) NESum of weight matrices
0.05
Recongrucc mcconaruciinn Lass
Locs
0.03
0.02
JULLA
24118 RENT
0.00
Epoch
Spoch
(d) Reconstruction
(c) Denoisng
Figure 10: (a) Mean and standard deviation of the (D)GCCA- -base method performance across synthetic datasets in different training epochs.() The mean correlation between noise and real data after transformation varies with epochs. (c) Average NESum across all weights within trained encoders. (d.e) The mean of Reconstruction and Denoising Loss on the test set.
26

---

Ny
AJy No-N2
82
bach
Figure 11: Performance DGCCA- methods in real-world datasets. Each column represents the performance on a specific dataset. The number of views in the dataset is denoted in the paren- theses next the dataset name.
27

---

A.11 Additional Experimental Results
Table 3 and present the model performance of various MVRL methods in synthetic and real-world datasets, and both tables correspond the final epoch of the results presented in Figure 4and 5. It should be noted that the values in Table represent the mean and standard deviation the methods across different tasks, indicating their performance and variability.
Table 3: Performance in synthetic datasets.
R2/Common Rate
0%
20%
40%
60%
80%
100%
CONCAT
0,253++0.038
0.255=0.039
0.250=0.0040
0.254=0.040
0.256 +0.042
0.264 0.033
Linear_CCA
0.179+0.030
0.184+0.035
0.172++0.35
0.182+00033
0.182++0.034
0.188+000031
Linear_GCCA
0.177=00030
0.18440.036
0.171=00033
0.182:00034
0.182=#0.33
0.182::0031
KCCA
0.243+000477
0.261
0.260=00044
0.272++0.045
0.276=00004
0.288=0.033
PROCA
0.212:10.053
0.24910.046
0.21611.055
0.267110.046
0.256110.052
0.284110.039
MVTСAE
0.065+0.015
0.071
0.006
0.069+0.016
0.071
0.069+00015
DCCA
0.053:0.044
0.094
0.123=00047
0.107=0.046
0.125=0:052
0.133=0.044
DCCAE
0.063100044
0.09010.009
0.126.10.047
0.104.10.045
0.09000.000
0,139 10.041
DOCA PRIVATE
0.264
0.141
0.176:10.042
0.168:10.039
0.172:10.041
0.181::0.03
DCCA GHA
0.251::0.049
0.249
0.243
0.252::0.052
0.268+00005
0.245
DCCALEY
0.195.10.050
0.205100.04
0.220.100041
0.21411.0446
0.21510.046
0.234.100041
NR-DCCA (ours)
0.311+0.043
0.314+0.046
0.306+0.043
0.309+0.042
0.313+0.049
0.322++0.040
DGCCA
0.23110.042
0.24410.040
0.24210.040
0,211=000399
0.24011.0040
0.25610.033
DGCCAE
0.209900.039
0.2054.0.042
0..40
0.2141100038
0..041
0.254:10.036
DGCCA PRIVATE
0.264_10.039
0.171110.040
1..20200042
0.168
0.172.10.042
0.181100.03
NR-DGCCA (ours)
0.314+00.04
0.317+0.045
0.305+00043
0.308++0.044
0.315+0.049
0.322+0.040
Table 4: Performance in real-world datasets
FI Score/Data
PolyMnist (2)
PolyMnist (3)
PolyMnist (4)
PolyMnist (5)
CUB
Caltech101
CONCAT
0.828
0.937
0.964
0.962
0.878
0.599
Linear CCA
0.818
0.929
0.955
0.957
0.878
0.599
Linear GCCA
0.818
0.828
0.956
0.958
0.879
0.596
PRCCA
0.712
0.849
0.899
0.918
MVTCAE
0.852
0.964
0.964
0.900
DCCA
0.865
0.957
0.964
0.938
0.805
0.604
DCCAE
0.868
0.956
0.961
0.987
0,850
0.605
DCCA PRIVATE
0.915
0.959
01.688
0.955
0.853
0.480
NR-DCCA (ours)
0,910
0.970
0.991
0.993
0.921
0.625
DGCCA
0.875
0.964
0.986
0.941
0.790
0.617
DGCCAE
0.879
0.960
0.988
0.934
0.814
0.612
AGCCA_PRIVATE
0.907
0.965
0.969
0,969
0.864
0.476
AR-DGCCA(OUrS)
0.903
0.971
0.991
0.994
0.917
0.621
A.12 Complexity Analysis
In this section, compare the computational complexity of different DCCA-based methods. As- suming that we have data from K views, with each view containing N samples and D) feature dimensions, then we have the computational complexity of each method in Table
Table 5: Comparisons computational complexity against baselines
IXCA
DCCAE
DRIVATE
DR-A OIR O N
Generation of Noise
MI.P Encoder
O(K L H²)
OCK N H2) OKK N HP) OIK + N O((M+ K)))
K N H. O(K IP)) O(K N OQM*K99)
MLP Decoder
Reconstruction Loss
Correlation Maximization
O((MK)9)
O(M KY 012 K (M+K)))
Noise Regularization
We will use neural networks with the same MLP structure, consisting of L hidden layers, each with H neurons. Therefore, the computational complexity of one pass of the data through the neural networks can be expressed as O + (D D L H2)). To simplify, we use O(N L 2
Complexity of MLP:
28

---

Complexity of Corr:
During the process of calculating Cor among K views, three main computations are involved. The calculation complexity of the covariance is O(N (M
Ky². Second, the complexity of the inverse matrix and the eigenvalues are O(M K)3. As a result, the computational complexity of calculating Cor can considered as ((M K)))).
Complexity of reconstruction loss: The reconstruction loss, also known as the mean squared error (MSE) loss, has complexity of O(N D).
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we explain that we have found that model collapse occurs in DCCA, and we propose the NR approach to solve this phenomenon, and the NR approach generalizes well in DGCCA. In the introduction, we list our 4 contributions in detail.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Our approach requires the introduction of additional computational overhead, which is discussed in Appendix. Our theory needs to assume that 1) feature matrice is full- rank and 2) the input and output dimensions are the same. The first assumption is fairly common because the DCCA itself requires large batchsize. The second assumption is the same the theoretical assumption that Linear CCA gets full-rank weight matrix, and our experiments verify that our NR approach is still effective even if the dimensions are different.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
30

---

judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We declare our theorem in the main paper and provide a complete proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:[Yes]
Justification:
All of our experimental codes including how to divide the generated data, training and testing are provided in the supplementary materials. And we've fixed random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- reasonable avenue for reproducibility, which may depend
missions to provide on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
31

---

(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include instructions in the appendix, including how download and divide the data set, how to execute training and test scripts, and how to visualize the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wttpccpppppcccceee.ccccccc.ccltt wwwp//gg///eeeeeee/oooooo Seueee/ ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have explained data division and hyperparameters in detail in both Ap- pendix and code, and discussed the impact of hyperparameters on results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
32

---

Answer: [Yes]
Justification: We randomly generated dozens of downstream tasks and reported the mean and standard deviation of their performance. And we used a 5-fold cross-validation to report the average of the 5-fold performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We compared the time cost of different methods in Appendix, and explained that our experiments were all completed on a single 3090 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: :
Answer: [Ye]]
Justification: The data sets we use are all public data sets, and we strictly follow previous studies, which will not have bad impact on the society.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
33

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Multi-view data is very common in society, and method can more stably represent multi-view data, so that it can be applied to a wide range of tasks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We found the shortcomings of the original DCCA and tried to improve it so that it would not be applied to the wrong places.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
34

---

Answer: [Yes]
Justification: We referenced the framework used and the data set we used. In the code, we listed the URL of each data set and the code environment we needed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
com/ datasets has
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
35

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "lIajD327Ya", "reviewer": "Reviewer_21Qa", "summary": "This work focuses on multi-view learning. Specifically, it studies the deep canonical correlation analysis and its variants. This study observes the issue of model collapse and proposes a regularization learning strategy to release the problem, then solve the early stop challenging.", "questions": "Please refer to the weaknesses section for reference.", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Multi-view learning is a critical research topic in machine learning field, which is valuable to explore.\n2. The model collapse issue matters in multi-view learning due to the challenging early stop decision.\n3. Overall, the writing is easy to follow.", "weaknesses": "1. Adding regularization has been fully explored in different machine learning scenarios. To this end, the proposed method is lack of research novelty, which may diminish the paper contribution.\n2. The compared methods are relatively old, adding more recent publication for comparison helps to support the draft.\n3. Numerical results only contain some small datasets, using more large-scale ones will be helpful, especially in current large-scale learning era.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>Adding regularization has been fully explored in different machine learning scenarios. To this end, the proposed method is lack of research novelty, which may diminish the paper contribution.\n\n\nIn the machine learning area, adding regularization is a common setup. However, the exploration of new regularization approaches is always needed. To be specific,  \n\n**Our motivation is new**, we first time demonstrate the difference in behaviors between DCCA and Linear CCA, and our regularization forces DCCA to mimic the behavior of Linear CCA in order to take on the properties of CIP, thus mitigating the problem of model collapse of DCCA. \n\n**Our approach is new** in that noise regularization previously had to rely on the Autoencoder architecture to implicitly regularize the network, but we now rely on (G)CCA loss to regularize the network from a different perspective using noise, which opens up the possibility of wider use of noise regularization. \n\n>The compared methods are relatively old, adding more recent publication for comparison helps to support the draft.\n\n\nThank you for your suggestion. We add two more methods: DCCA\\_EY and DCCA\\_GHA, the latest two DCCA-based methods for comparisons \\[2\\]. Due to time constraints, we conducted experiments on synthetic datasets (mean and variance are from multiple datasets (different common rates)):\n\nDCCA\\_EY and DCCA\\_GHA use an efficient algorithm for matrix eigenvalue decomposition in CCA loss that can be replaced by inter-matrix operations. This algorithm is fast and does not depend on a large batchsize (no gradient bias).   \nIn terms of effectiveness, they are no better than DCCA in terms of generating feature quality (we all used large batchsize 2000). There is still model collapse, but they collapse slower than DCCA.\n\n| Epoch/R2 | DCCA | NR-DCCA  | DDCA\\_EY | DDCA\\_GHA |\n| :---- | :---- | :---- | :---- | :---- |\n| 100 | 0.284 \\+/- 0.012 | **0.295 \\+/- 0.005** | 0.187 \\+/- 0.005 | 0.209 \\+/- 0.005 |\n| 400 | 0.206 \\+/- 0.025 | **0.310 \\+/- 0.005** | 0.266 \\+/- 0.008 | 0.273 \\+/- 0.010 |\n| 800 | 0.137 \\+/- 0.028 | **0.313 \\+/- 0.004** | 0.248 \\+/- 0.009 | 0.272 \\+/- 0.010 |\n| 1200 | 0.106 \\+/- 0.027 | **0.312 \\+/- 0.005** | 0.214 \\+/- 0.012 | 0.256 \\+/- 0.011 |\n\n\\[1\\] Hwang, HyeongJoo, et al. \"Multi-view representation learning via total correlation objective.\" *Advances in Neural Information Processing Systems* 34 (2021): 12194-12207.\n\n\\[2\\] Chapman, James William Harvey, Ana Lawry Aguila, and Lennie Wells. \"A Generalized EigenGame With Extensions to Deep Multiview Representation Learning.\"\n\n\\[3\\] Ke, Guanzhou, et al. \"Rethinking Multi-view Representation Learning via Distilled Disentangling.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024\\.\n\n>Numerical results only contain some small datasets, using more large-scale ones will be helpful, especially in the current large-scale learning era.\n\n\nThanks to your suggestion, the largest dataset used for the current MVRL method is the PolyMnist dataset with 5 views, 60,000 images per view, and 10 categories.   \nHowever, it is really necessary to test MVRL in larger and more complex scenarios. To this end, we constructed 2 views of data in Cifar100 (100 categories, 60,000 images) by CLIP and BLIP pre-training image feature encoders respectively. Using the same PolyMnist experimental settings as in the paper, except that the embedding dimension was changed to 400, the 5-fold F1 score results are as follows:\n\n| Epoch/F1\\_score | DCCA | NR-DCCA  | Concat |\n| :---- | :---- | :---- | :---- |\n| 50 | 0.749 | 0.752 | 0.733 |\n| 500 | 0.672 | 0.753 |  |\n\nWe can see that although DCCA starts with better results than concat, it quickly collapses, while NR-DCCA shows stable performance. As to whether DCCA can be pre-trained for large-scale multimodal data like CLIP, this is an interesting question that we leave for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q5OqcXSvno", "reviewer": "Reviewer_g1Nh", "summary": "The paper proposes noise regularization term to prevent collapse issues found in deep canonical correlation analysis (DCCA) methods. The term makes DCCA to behave in a way similar to linear CCA, which is robust to collapse by definition, thereby making DCC with the regularization robust against collapse issue.", "questions": "1. In Line 137, the setting assumes the same sample size for all the datasets X_k. Is it necessary, and how about other CCA methods?\n2. What are the practical application of the CCA methods? Despite the highly theoretical nature of the work, I think it would be better if there is at least one line of comment that explains a practical use case of the CCA methods in the introduction or appendix for the readers who are not familiar with the field.\n3. In line 137, what are the actual examples of X_k in practice? \n4. The experiment setting is not self-contained. What is the exact protocol from Hwang et al. (2021) in Line 250? It should be given in the appendix.\n5. What is the deep network $f$ used for DCCA?\n6. What if the features in X_k are the representations from foundational pre-trained encoders but possibly of different vector dimension across different views?\n7. The NESum recovers after epoch 600 in Figure 3 for DCCA_PRIVATE. Why is it that? But based on (d), the model seems to collapse.", "limitations": "Please refer to 'Weaknesses'", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. Clear and consistent performance improvement by the proposed method in the both synthetic and practical settings. \n2. The proposed method sound reasonable; by the proposed regularization, DCCA mimics linear CCA in terms of collapsing. Since the latter is resilient to collapse by definition, NR-DCCA becomes robust to collapse by mimicking the linear CCA.", "weaknesses": "1. Readability. It is quite hard to find the exact configuration used for real-world experiments such as what encoders are used for DCCA. The content in the appendix is huge, but they are not organized well.\n2. The degree of model collapse will depend on the model complexity (e.g., the number of parameters in the MLP). But there is no clear analysis on this aspect. Would the proposed noise regularization can prevent the collapse issue under any degree of MLP complexity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n>Readability. weak1\n\nWe apologize for not specifying the structure of the MLP in the paper. All our MLPs use the Leaky ReLu activation function. The first linear layer is feature\\_dim \\* hidden\\_dim and the final linear layer is hidden\\_dim \\* embedding\\_dim. \n\nFor the synthetic dataset, which is simple, we have only one hidden layer with a dimension of 256\\. For the real-world dataset, we have used MLPs with three hidden layers, and the dimension of the middle hidden layer is 1024\\.\n\nTo enhance reproducibility, we will release all the experiment settings and source codes after the blind review process.\n\n> weak2\n\nThanks a lot for your suggestion, which is very reasonable. It is essential to test the robustness of NR to model complexity, and we supplemented our synthetic datasets with the following experiment  \n(hidden\\_dim \\= 128):\n\n|  Epoch/R2 |    1 hidden layer  |  |        2 hidden layer  |  |    3  hidden layer  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n|  | DCCA | NR-DCCA  | DCCA | NR-DCCA  | DCCA | NR-DCCA  |\n| 100 | 0.284 \\+/- 0.012 | **0.295 \\+/- 0.005** | 0.161 \\+/- 0.013 | **0.304 \\+/- 0.006** | 0.071 \\+/- 0.084 | **0.299 \\+/- 0.010** |\n| 800 | 0.137 \\+/- 0.028 | **0.313 \\+/- 0.004** | \\-0.072 \\+/- 0.071 | **0.307 \\+/- 0.005** | \\-0.975 \\+/- 0.442 | **0.309 \\+/- 0.005** |\n| 1200 | 0.106 \\+/- 0.027 | **0.312 \\+/- 0.005** | \\-0.154 \\+/- 0.127 | **0.303 \\+/- 0.006** | \\-1.412 \\+/- 0.545 | **0.308 \\+/- 0.006** |\n\nSince synthetic data is simple, increasing the depth of the network makes both DCCA and NR-DCCA less effective, but the nature of NR for preventing model collapse is still maintained. We can clearly see that by increasing the network depth, the DCCA collapses more severely, while NR remains effective.\n\n\n>Q1 In Line 137\n\nThe questions you mentioned are very helpful in understanding CCA. The CCA family is not strong enough to deal with missing views (missing data for a view at some sample points), which requires that the data for each view of the sample is complete. The MVRL domain has a specialized approach to address the missing view scenario, which is called partial multi-view learning, and it may not be the focus of this paper.\n\n>Q2 What are the practical applications of the CCA methods? \n\nYour suggestions are very reasonable and I will add various practical application scenarios for the CCA methods in the appendix:\n\nThe application of CCA mainly focuses on multi-view data. Multi-view data may come from multiple domains, such as computer vision, natural language, speech, and so on \\[1\\].  \nThe features of different views are utilized to obtain a unified form of representation that captures the correlation between different views \\[2\\], which can be used in various downstream tasks such as classification, retrieval, clustering, and dimension reduction \\[2\\].  \n\n\\[1\\] Yan, Xiaoqiang, et al. \"Deep multi-view learning methods: A review.\" *Neurocomputing* 448 (2021): 106-129.\n\\[2\\] Hardoon, David R., Sandor Szedmak, and John Shawe-Taylor. \"Canonical correlation analysis: An overview with application to learning methods.\" *Neural computation* 16.12 (2004): 2639-2664.\n\n(No room for more references)\n\n>Q3 In line 137 \n\nI apologize for the potential misunderstanding caused. Let's take the Caltech101 dataset as an example, the training set has 6400 images, so the same image has been fed to three different feature extractors producing three features a 1, 984-d HOG feature, a 512-d GIST feature, and a 928-d 681 SIFT feature. Then for this dataset. X\\_1: 1984\\*6400 , X\\_2: 512\\*6400 , X\\_3: 928\\*6400\n\n> Q4 The experiment setting \n\nWe apologize for the misunderstanding and will add Hwang et al. (2021) in the appendix. Our experiment setting is the same, in the training set, we only utilize the multi-view features for multi-view learning and train the encoder to capture the correlation between multiple views. Subsequently, using the trained encoder, the test set features are projected into a new space, and the newly obtained features are used as a multi-view representation. This representation is then used in the downstream classification task to report the average classification metric (F1\\_score) through 5-fold cross-validation using the SVC linear classifier.\n\n>Q5 What is the deep network used for DCCA?\n\nAgain, we apologize for not specifying the structure of the MLP in the paper.   \nThe first linear layer is feature\\_dim \\* hidden\\_dim. All our MLPs use the Leaky ReLu activation function.  \nFor the manually generated dataset, which is simple enough, we have only one hidden layer with a dimension of 256\\. For the real-world dataset, we have used MLPs with three hidden layers, and the dimension of the middle hidden layer is 1024\\.\n\n>Q6 What if \n\nThe questions you mentioned are very helpful in understanding CCA, and it's exactly what CCA methods are trying to solve: how to utilize features generated by different pre-trained foundational encoders. DCCA uses an MLP for each view, where the first linear layer dimension is feature\\_dim \\* hidden\\_dim, and the structure of the MLP behind the first layer is the same.   \nThis way our MLP will project different features onto the same size of feature space.\n\n>Q7 The NESum \n\nYour question is very helpful in helping us understand model collapse. We believe that the addition of DropOut to DCCA\\_private, which is a common regularization technique, does, to some extent, prevent redundancy among network weights (NESum rises), but it is clear that its effect is not stable for DCCA, especially as seen in (a), where the performance variance of different datasets is very large, which means DCCA\\_private  is highly dependent on the dataset and is not generalizable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbhdjBY66t", "reviewer": "Reviewer_ForS", "summary": "The authors propose NR-DCCA, a novel approach of Deep Canonical Correlation Analysis(DCCA) equipping with noise regularization, in order to prevent DCCA-based methods from model collapse in the multi-view representation learning(MVRL) task. First, the authors analyze the difference between Linear CCA and DCCA, and then draw a conclusion on the cause of model collapse of DCCA. After that they propose the NR-DCCA method, which integrates noise-regularization into DCCA. Theoretical analysis is also provided to further illustrate the effect of noise regularization. Experimental results show the effectiveness of NR-DCCA.", "questions": "1. why can noise regularization enforce DCCA to mimic the behavior of Linear CCA and therefore prevent DCCA from model collapse?\n2. The authors need to polish their expressions. According to the math formulat of NESum, a full-rank matrix with considerably different eigenvalues can still have low NESum score. For example, diag(1000,1,1) has lower NESum than diag(2,1,0), both of them are full-rank.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: Although noise regularization is not a novel approach, the authors point out the key difference between Linear CCA and DCCA, which may be the cause of model collapse of DCCA, and succesfully use noise regularization to alleviate this issue. Therefore, I think this paper is innovative enough.\n2. Quality: The paper is highly qualified because of several reasons. First, it is meaningful to find the key difference between Linear CCA and DCCA and its effect, which can help researchers get deeper understanding of CCA and DNN-based methods; Second, sufficient theoretical analyses are provided. According to my knowledge, these proofs are right. Third, the experiments are well organized and explained.\n3. Clarity: The writing style is good and the core ideas are easy to understand.\n4. Significance: The significance of the paper is not that considerable compared to its originality and quality. However, it is still good.", "weaknesses": "1. The authors hypothesize the root cause of model collapse of DCCA is that the weight matrices of network is not guaranteed to be full-rank, which possibly leads to overfitting. The experimental results are good and I think this hypothesis is probably right to some extent. However, as I know the relation between weight matrices’full-rank property and the overfitting phenomenon is still underexplored. The authors do not provide enough proofs to support this opinion. Besides, ‘full-rank’ is not a precise(even misleading) enough expression if the authors determine to use NESum as metric to evaluate the redundacy of weight matrices.\n2. The theoretical proof of Theorem 2 only shows that noise regularization do not cause a serious degradation to DCCA’s performance. However, why noise regularization works in DCCA still remains unclear. The authors just write they try to enforce DCCA to mimic the behavior of Linear CCA but no more illustrations are provided.\n3. Some texts in the figures are too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>However, as I know the relation between weight matrices’full-rank property and the overfitting phenomenon is still underexplored. The authors do not provide enough proofs to support this opinion.\n\nThanks for your comment. There is theoretical work devoted to the phenomenon that weight matrices tend to be low-rank in overfitting networks \\[1,2\\].  \nIn the field of representation learning, \\[1\\] mentions that due to the interaction between network weight layers, the weight matrix collapses (be low-rank), which in turn affects the quality of the representation.  \nWe can observe very clearly that the performance of DCCA decreases with training while its NESum decreases rapidly, so we hypothesize that the weight matrix collapse (low-rank) also occurs in DCCA.    \nMore theoretical work, such as analyzing how the layers in DCCA interact with each other and why they become low-rank, is also very important and we leave it for future work.\n\nIn this paper, we mainly provide practical solutions to model collapse, and the theoretical analysis is for justification purposes. \n\n\\[1\\] Jing, Li, et al. \"Understanding dimensional collapse in contrastive self-supervised learning.\" *arXiv preprint arXiv:2110.09348* (2021).\n\n\\[2\\] Barrett, David GT, and Benoit Dherin. \"Implicit gradient regularization.\" arXiv preprint arXiv:2009.11162 (2020).\n\n> Besides, ‘full-rank’ is not a precise(even misleading) enough expression if the authors determine to use NESum as metric to evaluate the redundacy of weight matrices.\n\nIndeed, there is a difference between the notion of the rank of a matrix and that of NESum. NESum was originally proposed as a measure of whether the eigenspace is dominated by a small number of very large eigenvalues. We simply follow the previous research and use this metric to measure the eigenvalue distribution and redundancy of the matrix.   \nIn some cases where the eigenvalues are suddenly decreasing, e.g., NESum (2,1,0.1) \\= 1 \\+ 0.5 \\+ 0.05 \\= 1.55 (1-\\>0.1), NESum (2,2,0) \\= 1 \\+ 1 \\+ 0 \\= 2 (2-\\>0), the full-rank matrix will have a smaller NESum.   \nIn order to better analyze the weight matrices in DNNs, we add the average of the cosine similarity between the dimensions in the weight matrices as another measure of redundancy, and we use DCCA and NR-DCCA in synthetic datasets as an example: \n\n|  Epoch |    NESum |  | Cosine Similarity |  |\n| :---- | :---- | :---- | :---- | :---- |\n|  | DCCA | NR-DCCA  | DCCA | NR-DCCA  |\n| 100 | 0.260 | 0.227 | 0.074 | 0.070 |\n| 800 | 0.149 | 0.330  | 0.093 | 0.052 |\n| 1200 | 0.141 | 0.354 | 0.098 | 0.050 |\n\nIt can be seen that the metrics of cosine similarity and NESum are synchronized, with weight redundancy in DCCA increasing (cosine similarity increases, NESum decreases) and NR-DCCA redundancy decreasing (cosine similarity decreases, NESum increases).  \nIn particular, we visualize the correlation matrix and the eigenvalues of the first Linear layer in DCCA and NR-DCCA. Again, it can be found that the redundancy of the weight matrices in DCCA is rising (**please see the pdf material**).\n\n>The theoretical proof of Theorem 2 only shows that noise regularization do not cause a serious degradation to DCCA’s performance. However, why noise regularization works in DCCA still remains unclear. The authors just write they try to enforce DCCA to mimic the behavior of Linear CCA but no more illustrations are provided.\n\nYour suggestions are very helpful for the understanding of NR.   \nFirst of all, the key difference between DCCA and Linear CCA is that the weight matrix in DCCA cannot be guaranteed to be full-rank during the optimization process.  Combined with the previous research, we believe that this difference leads to model collapse in DCCA, and therefore we need to design a method to constrain the behavior of DCCA.  \nAs for the full-rank property of Linear CCA, we prove through Theorem 1 that it is equivalent to CIP.   \nSo we design NR, through which DCCA has the property of CIP, and its behavior is constrained to be consistent with that of Linear CCA.   \nTheorem 2 shows that NR can guarantee that the generated features will not be degenerated (reconstruction and denoising) from the feature level.\n\n>Some texts in the figures are too small.  \n\nWe apologize for the small size of the text in the images. Due to space constraints, we put too many images under the same figure, which will be improved in our subsequent releases.\n\n\n>why can noise regularization enforce DCCA to mimic the behavior of Linear CCA and therefore prevent DCCA from model collapse?\n\nThanks for your question. The key difference between DCCA and Linear CCA is that the weight matrix in DCCA cannot be guaranteed to be full-rank during the optimization process.  Combined with the previous studies, we believe that this difference leads to model collapse in DCCA, and therefore we need to design a method to constrain the behavior of DCCA.  \nAs for the full-rank property of Linear CCA, we prove through Theorem 1 that it is equivalent to CIP.   \nSo we design NR, through which DCCA has the property of CIP, and its behavior is constrained to be consistent with that of Linear CCA. \n\n>The authors need to polish their expressions. According to the math formulat of NESum, a full-rank matrix with considerably different eigenvalues can still have low NESum score. For example, diag(1000,1,1) has lower NESum than diag(2,1,0), both of them are full-rank.\n\nYour suggestion is valuable, and we have added the cosine similarity of the elements of the weight matrix as an indicator of the redundancy of the weights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ac1EbduJ42", "reviewer": "Reviewer_BL52", "summary": "This paper reveals a new observation in multi-view representation learning, that is, the performance of DCCA-based methods will gradually decrease as training progresses. The authors explore the possible reasons from the rank of weights and conclude that the Correlation Invariant Property is the key to preventing this problem, and propose NR-DCCA. Experiments also show the effectiveness of the proposed NR-DCCA.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The perspective of observation is novel.\n2. The method proposed by the author is simple and effective.\n3. The theoretical analysis seems sound.", "weaknesses": "1. In fact, the authors call this degradation phenomenon model collapse, which is not very accurate. Collapse should be very extreme. The performance degradation is a bit like an overfitting problem.\n2. The method is effective, but it cannot be denied that it is too simple.\n3. The legend may be better if it is a vector diagram.\n4. The authors' theoretical analysis is based on Gaussian white noise. Is it the same for other types of noise?\n5. Is NR loss also universal and effective on other DCCA methods? I think this is very important because it avoids the risk that DCCA is a special case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>In fact, the authors call this degradation phenomenon model collapse, which is not very accurate. Collapse should be very extreme. The performance degradation is a bit like an overfitting problem.\n\nYour question is very helpful in understanding the model collapse of DCCA.  \nPrevious studies have referred to weights being redundant and low-rank as an over-parametrization issue\\[1,3\\], and then the impaired feature quality and downstream task performance caused by this issue is called collapse \\[2\\]. Similarly, in this paper, we call the degradation of DCCA in the downstream task, model collapse.   \nIt is worth noting that this is not the same as overfitting in traditional supervised learning, the collapse of DCCA is not due to to it is memorizing training data, leading to a reduction in generalizability.\n\n\\[1\\] Wang, Zhennan, et al. \"Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles.\" *Advances in Neural Information Processing Systems* 33 (2020): 19099-19110.  \n\\[2\\] Jing, Li, et al. \"Understanding dimensional collapse in contrastive self-supervised learning.\" *arXiv preprint arXiv:2110.09348* (2021).\n\n\\[3\\] Barrett, David GT, and Benoit Dherin. \"Implicit gradient regularization.\" arXiv preprint arXiv:2009.11162 (2020).\n\n>The method is effective, but it cannot be denied that it is too simple.\n\nOur NR method is intuitively efficient, and it rests on a number of theoretical guarantees, including why NR guarantees CIP properties that constrain the behavior of the network (Theorem 1), and why NR guarantees that features do not degrade (Theorem 2).  \nOur NR method can be generalized to other methods such as DGCCA.\n\nWe believe a simple method is a plus, rather than a minus, as it can be used in different types of DCCA very easily, and it can be widely used as a plug-and-play module.\n\n>The legend may be better if it is a vector diagram.  \n\n\nWe are grateful for your suggestion, we will improve the quality of the pictures in the next version.\n\n>The authors' theoretical analysis is based on Gaussian white noise. Is it the same for other types of noise?  \n\nYour question is very helpful in understanding NR. From our theoretical analysis, the most important feature of noise is that the sampled noise matrix is a full-rank matrix. Therefore, continuous distributions such as the uniform distribution can also be applied to NR, which demonstrates the robustness of the NR method. Here are our experiments with synthetic datasets (mean and variance are from multiple datasets (different common rates)):\n\nAs you can see, their effects on preventing model collapse and improving the performance of DCCA are almost identical: \n\n| Epoch | DCCA | NR-DCCA (Guassion Noise) | NR-DCCA (Uniform Noise) |\n| :---- | :---- | :---- | :---- |\n| 100 | 0.284 \\+/- 0.012 | 0.295 \\+/- 0.005 | 0.291 \\+/- 0.004 |\n| 800 | 0.137 \\+/- 0.028 | 0.313 \\+/- 0.004 | 0.313 \\+/- 0.005 |\n| 1200 | 0.106 \\+/- 0.027 | 0.312 \\+/- 0.005 | 0.316 \\+/- 0.005 |\n\n>Is NR loss also universal and effective on other DCCA methods? I think this is very important because it avoids the risk that DCCA is a special case.  \n\nWe strongly agree with you that it is important that the NR method works for other DCCA methods. In particular, we test the results of NR-DGCCA in this paper and compare them with other methods such as DGCCA. Please see Appendix A.8 DGCCA and NR-DGCCA in our paper.    \nMoreover, we have added two new DCCA methods, DCCA\\_EY and DCCA\\_GHA, which replace the matrix factorization in CCA with matrix operations, and in the follow-up work, we can check whether NR can be used in them \\[1\\].\n\n\\[1\\] Chapman, James William Harvey, Ana Lawry Aguila, and Lennie Wells. \"A Generalized EigenGame With Extensions to Deep Multiview Representation Learning.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their questions and constructive feedback. In the general response, we respond to the five core issues :   
**More DCCA methods**, **Larger MVRL experiments**, **Effects of MLP structures**, and **New metric of weight redundancy and Contributions.**   The image quality mentioned by the reviewers will be fixed in the next version.

**More DCCA methods**  
In this paper, we focus on the family of DCCA methods, and we add DCCA\_EY DCCA\_GHA, the latest two DCCA-based methods \[1\].  
Due to time constraints, we conducted experiments on synthetic datasets (mean and variance are from multiple datasets (different common rates)):

| Epoch/R2 | DCCA | NR-DCCA  | DDCA\_EY | DDCA\_GHA |
| :---- | :---- | :---- | :---- | :---- |
| 100 | 0.284 \+/- 0.012 | **0.295 \+/- 0.005** | 0.187 \+/- 0.005 | 0.209 \+/- 0.005 |
| 400 | 0.206 \+/- 0.025 | **0.310 \+/- 0.005** | 0.266 \+/- 0.008 | 0.273 \+/- 0.010 |
| 800 | 0.137 \+/- 0.028 | **0.313 \+/- 0.004** | 0.248 \+/- 0.009 | 0.272 \+/- 0.010 |
| 1200 | 0.106 \+/- 0.027 | **0.312 \+/- 0.005** | 0.214 \+/- 0.012 | 0.256 \+/- 0.011 |

DCCA\_EY and DCCA\_GHA utilize an efficient algorithm for matrix eigenvalue decomposition in CCA loss that can be replaced by inter-matrix operations. This algorithm is fast and does not depend on a large batchsize (no gradient bias).   
There is still model collapse, but they collapse slower than DCCA.

\[1\] Chapman, James William Harvey, Ana Lawry Aguila, and Lennie Wells. "A Generalized EigenGame With Extensions to Deep Multiview Representation Learning."

**Larger MVRL experiments**  
The largest dataset used for the current MVRL method is the PolyMnist dataset with 5 views, 60,000 images per view and 10 categories. However, it is really necessary to test MVRL in larger and more complex scenarios. To this end, we constructed 2 views of data in Cifar100 (100 categories, 60,000 images) by CLIP and BLIP pre-training image feature encoders respectively. Using the same PolyMnist experimental settings as in the paper, except that the embedding dimension was changed to 300, the average 5-fold F1 score results are as follows:

| Epoch/F1\_score | DCCA | NR-DCCA  | Concat |
| :---- | :---- | :---- | :---- |
| 50 | 0.749 | 0.752 | 0.733 |
| 500 | 0.672 | 0.753 |  |

We can see that although DCCA starts out with better results than concat, it quickly collapses, while NR-DCCA shows stable performance.  


**Effects of MLP structures**  
It is essential to test the robustness of NR to model complexity, and we supplemented our synthetic datasets with the following experiment (hidden\_dim \= 128):

|  Epoch/R2 |    1 hidden layer  |  |        2 hidden layer  |  |    3  hidden layer  |  |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
|  | DCCA | NR-DCCA  | DCCA | NR-DCCA  | DCCA | NR-DCCA  |
| 100 | 0.284 \+/- 0.012 | **0.295 \+/- 0.005** | 0.161 \+/- 0.013 | **0.304 \+/- 0.006** | 0.071 \+/- 0.084 | **0.299 \+/- 0.010** |
| 800 | 0.137 \+/- 0.028 | **0.313 \+/- 0.004** | \-0.072 \+/- 0.071 | **0.307 \+/- 0.005** | \-0.975 \+/- 0.442 | **0.309 \+/- 0.005** |
| 1200 | 0.106 \+/- 0.027 | **0.312 \+/- 0.005** | \-0.154 \+/- 0.127 | **0.303 \+/- 0.006** | \-1.412 \+/- 0.545 | **0.308 \+/- 0.006** |

Since synthetic data is simple, increasing the depth of the network makes both DCCA and NR-DCCA less effective, but the nature of NR for preventing model collapse is still maintained. We can clearly see that by increasing the network depth, the DCCA collapses more severely, while NR remains effective.

**A new metric of weight redundancy**  
There is a difference between the notion of the rank of a matrix and that of NESum.   
NESum was originally proposed as a measure of whether the eigenspace is dominated by a small number of very large eigenvalues. In order to better analyze the weight matrices in DNNs, we add the average of the cosine similarity between the dimensions in the weight matrices as another measure of redundancy, and we use DCCA and NR-DCCA in synthetic datasets as an example: 

|  Epoch |    NESum |  | Cosine Similarity |  |
| :---- | :---- | :---- | :---- | :---- |
|  | DCCA | NR-DCCA  | DCCA | NR-DCCA  |
| 100 | 0.260 | 0.227 | 0.074 | 0.070 |
| 800 | 0.149 | 0.330  | 0.093 | 0.052 |
| 1200 | 0.141 | 0.354 | 0.098 | 0.050 |

It can be seen that the metrics of cosine similarity and NESum are synchronized, with weight redundancy in DCCA increasing (cosine similarity increases, NESum decreases) and NR-DCCA redundancy decreasing (cosine similarity decreases, NESum increases).  
In particular, we visualize the correlation matrix and the eigenvalues of the first Linear layer in DCCA and NR-DCCA. Again, it can be found that the redundancy of the weight matrices in DCCA is rising (**please see the pdf material**).

**Contributions**  
Our NR method is intuitively efficient, and it rests on a number of theoretical guarantees, including why NR guarantees CIP properties that constrain the behavior of the network (Theorem 1), and why NR guarantees that features do not degrade (Theorem 2).  

Moreover, our motivation is new, as for the first time, we explicitly demonstrate the difference in behavior between DCCA and Linear CCA in an attempt to explain and alleviate the problem of model collapse in DCCA.  
Secondly, our NR method is new. While noise regularization previously had to rely on the Autoencoder architecture to implicitly regularize the network, we now rely on the (G)CCA loss to regularize the network from another perspective using noise, which opens up the possibility of wider use of noise regularization. 


Overall, we would like to thank the reviewers once again for your detailed and effective suggestions. Improving and supplementing the experiments based on your suggestions will undoubtedly make our paper more convincing. We hope that our response has adequately addressed your concerns.

Meta-review: This submission proposed a method for Deep CCA in preventing model collapse by adding regularization in learning. 3/4 reviewers rates the submission as borderline accept, and the other one rates it as borderline reject with major concern of limited novelty. After reading the submission, comments, and rebuttal，the AC recognize the merits of this submission and suggest accepting this paper.
Decision: Accept (poster)