Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:10:11

Abstract: Multi-View Representation Learning (MVRL) aims to learn a unified representation of an object from multi-view data.
Deep Canonical Correlation Analysis (DCCA) and its variants share simple formulations and demonstrate state-of-the-art performance. However, with extensive experiments, we observe the issue of model collapse, i.e., the performance of DCCA-based methods will drop drastically when training proceeds. The model collapse issue could significantly hinder the wide adoption of DCCA-based methods because it is challenging to decide when to early stop. To this end, we develop NR-DCCA, which is equipped with a novel noise regularization approach to prevent model collapse. Theoretical analysis shows that the Correlation Invariant Property is the key to preventing model collapse, and our noise regularization forces the neural network to possess such a property. A framework to construct synthetic data with different common and complementary information is also developed to compare MVRL methods comprehensively. The developed NR-DCCA outperforms baselines stably and consistently in both synthetic and real-world datasets, and the proposed noise regularization approach can also be generalized to other DCCA-based methods such as DGCCA.

Main paper: 
 introduction: 
 In recent years, multi-view representation learning (MVRL) has emerged as a core technology for learning from multi-source data and providing readily useful representations to downstream tasks (Sun et al., 2023; Yan et al., 2021) , and it has achieved tremendous success in various applications, such as video surveillance (Guo et al., 2015; Feichtenhofer et al., 2016; Deepak, K. et al., 2021) , medical diagnosis (Wei et al., 2019; Xu et al., 2020) and social media (Srivastava & Salakhutdinov, 2012; Karpathy & Fei-Fei, 2015; Mao et al., 2014; Fan et al., 2020) . Specifically, we collect multi-source data of the same object, and each data source can be regarded as one view of the object. For instance, an object can be described simultaneously through texts, videos, and audio, which contain both common and complementary information of the object (Yan et al., 2021; Zhang et al., 2019b; Hwang et al., 2021; Geng et al., 2021) , and the MVRL aims to learn a unified representation of the object from the multi-view data.The key challenge of MVRL is to learn the intricate relationships of different views. The Canonical Correlation Analysis (CCA), which is one of the early and representative methods for MVRL, transforms all the views into a unified space by maximizing their correlations (Hotelling, 1992; Horst, 1961; Hardoon et al., 2004; Lahat et al., 2015; Yan et al., 2023; Sun et al., 2023) . Through correlation maximization, CCA can identify the common information between different views and extract them to form the representation of the object. On top of CCA, DCCA further utilizes the powerful deep neural networks(DNNs) to transform each view and then adopts CCA to maximize the correlation of the transformed views (Andrew et al., 2013) . Indeed, there are quite a few variants of DCCA, such as DGCCA (Benton et al., 2017) , DCCAE (Wang et al., 2015) , DVCCA (Wang et al., 2016) , DTCCA (Wong et al., 2021) and DCCA GHA (Chapman et al., 2022) . However, with extensive experiments, we observe that the DCCA-based methods generally perform well at the early training stage, while their performance drops drastically when the training proceeds. This issue is referred to as model collapse. Essentially, the model collapse is mainly attributed to the overly powerful transformation abilities of DNNs, and the maximized view correlation may not come from views, but it may come from the model correlation among neural networks, as shown in Figure 1 . Although representation produced by DCCA-based methods is naturally fullrank, the feature space may still become degenerated due to unregulated neural networks. In contrast, CCA-based methods utilize linear transformations, which are forced to be full-rank matrices, and hence there is no such model collapse issue.Though early stopping could be adopted to prevent model collapse (Prechelt, 1998; Yao et al., 2007) , it remains challenging when to stop. The model collapse issue of DCCA-based methods prevents the adoption in large models, and currently, many applications still use simple concatenation to combine different views (Yan et al., 2021; Zheng et al., 2020; Nie et al., 2017) . Therefore, how to develop a DCCA-based MVRL method without model collapse remains an interesting and open question.We prove that the main reason for CCA not having the model collapse issue is that the full-rank property holds in its transformation matrix, while the DNNs in DCCA do not possess such property. To this end, we develop a novel idea that employs noise regularization to enforce the DNNs to be "full-rank". Note that the proposed noise regularization approach is novel and particularly tailored for DCCA-based methods, which is different from the existing approaches that directly inject noise into the neural networks (Poole et al., 2014; He et al., 2019; Gong et al., 2020) .Overall, this paper develops NR-DCCA, a DCCA method equipped with a generalized noise regularization approach. The proposed noise regularization approach constrains the DNNs to be "full-rank", and it can also be applied to other DCCA-based methods. To justify the approach, the formulation of CCA is analyzed and rigorous proofs are provided. Comprehensive experiments using both synthetic datasets and real-world datasets demonstrate the consistent outperformance and stability of the developed NR-DCCA method. Our contributions are four-fold:• The model collapse issue in DCCA-based methods for MVRL is identified, demonstrated, and explained.• A simple yet effective noise regularization approach is proposed and the NR-DCCA method is developed to prevent model collapse.• Rigorous proofs are provided to demonstrate that the full-rank property of the transformation in the CCA method is the key to preventing model collapse, which justifies the developed noise regularization approach from a theoretical perspective.• A novel framework is proposed to construct synthetic data with different common and complementary information for comprehensively evaluating MVRL methods. 
multi-view representation learning: 
 MVRL aims to uncover relationships among multi-view data in an unsupervised manner, thereby obtaining semantically rich representations that can be utilized for various downstream tasks (Sun et al., 2023; Yan et al., 2021) . A number of works have been proposed to deal with MVRL from different aspects. DMF-MVC (Zhao et al., 2017) utilizes deep matrix factorization to extract a shared representation from multiple views. MDcR (Zhang et al., 2016a) maps each view to a lowerdimensional space and applies kernel matching to enforce dependencies across the views. CPM-Nets (Zhang et al., 2019a) formalizes the concept of partial MVRL and many works have been proposed for such issue (Zhang et al., 2020; Tao et al., 2019; Li et al., 2022; Yin & Sun, 2021) . AE 2 -Nets (Zhang et al., 2019b) utilizes a two-level autoencoder framework to obtain a comprehensive representation of multi-view data. DUA-Nets (Geng et al., 2021) takes a generative modeling perspective and dynamically estimates the weights for different views. MVTCAE (Hwang et al., 2021) explores MVRL from an information-theoretic perspective, which can capture the shared and view-specific factors of variation by maximizing or minimizing specific total correlation. Our work focuses on CCA as a simple, classic, and theoretically sound approach as it can still achieve stateof-the-art performance consistently.Canonical Correlation Analysis (CCA) projects the multi-view data into a unified space by maximizing their correlations (Hotelling, 1992; Horst, 1961; Hardoon et al., 2004; Lahat et al., 2015; Yan et al., 2023; Sun et al., 2023) . It has been widely applied in various scenarios that involve multiview data, including dimension reduction (Zhang et al., 2016b; Sun et al., 2010a; Avron et al., 2013) , classification (Kim et al., 2007; Sun et al., 2010b), and clustering (Fern et al., 2005; Chang & Lin, 2011) . To further enhance the nonlinear transformability of CCA, Kernel CCA (KCCA) uses kernel methods, while Deep CCA (DCCA) employs DNNs. Since DNNs is parametric and can take advantage of large amounts of data for training, numerous DCCA-based methods have been proposed. Benton et al. (2017) utilizes DNNs to optimize the objective of Generalized CCA, with the aim of revealing connections between multiple views more effectively. To better preserve view-specific information, Wang et al. (2015) introduces the reconstruction errors of autoencoders to DCCA. Going a step further, Wang et al. (2016) proposes Variational CCA and utilizes dropout and private autoencoders to project common and view-specific information into two distinct spaces. Furthermore, there are many studies exploring efficient methods for computing the correlations between multiview data when dealing with more than two views such as MCCA, GCCA and TCCA (Horst, 1961; Nielsen, 2002; Kettenring, 1971; Hwang et al., 2021) . Some research focuses on improving the efficiency of computing CCA by avoiding the need for singular value decomposition (SVD) (Chang et al., 2018; Chapman et al., 2022) . However, the model collapse issue of DCCA-based methods has not been explored and addressed.Noise regularization is a pluggable approach to regularize the neural networks during training (Bishop, 1995; An, 1996; Sietsma & Dow, 1991; Gong et al., 2020) . In supervised tasks, Sietsma & Dow (1991) might be the first to propose that, by adding noise to the train data, the model will generalize well on new unseen data. Moreover, Bishop (1995); Gong et al. (2020) analyze the mechanism of the noise regularization and He et al. (2019) ; Gong et al. (2020) indicate that noise regularization can also be used for adversarial training to improve the generalization of the network. In unsupervised tasks, Poole et al. (2014) systematically explores the role of noise injection at different layers in autoencoders, and distinct positions of noise perform specific regularization tasks. However, how to make use of the noise regularization for DCCA-based methods, especially for preventing model collapse, has not been studied. 
preliminaries: 
 In this section, we will explain the objectives of the MVRL and then introduce CCA and DCCA as representatives of the CCA-based methods and DCCA-based methods, respectively. Lastly, the model collapse issue in DCCA-based methods is demonstrated.Suppose the set of datasets from K different sources that describe the same object is represented by X, and we defineX = {X 1 , • • • , X k , • • • , X K }, X k ∈ R d k ×n, where x k represents the k-th view (k-th data source), n is the sample size, and d k represents the feature dimension for the k-th view. And we use X ′ k to denote the transpose of X k . The objective of MVRL is to learn a transformation function Ψ that projects the multi-view data X to a unified representation Z ∈ R m×n , where m represents the dimension of the representation space, as shown below:Z = Ψ(X) = Ψ(X 1 , • • • , X k , • • • , X K ).(1) After applying Ψ for representation learning, we expect that the performance of using Z would be better than directly using X for various downstream tasks.Among various MVRL methods, CCA projects the multi-view data into a common space by maximizing their correlations. We first define the correlation between two views as follows:Corr(W 1 X 1 , W 2 X 2 ) = tr((Σ -1/2 11 Σ 12 Σ -1/2 22 ) ′ Σ -1/2 11 Σ 12 Σ -1/2 22 ) 1/2(2)where tr denotes the matrix trace, Σ 11 , Σ 22 represent the self-covariance matrices of the projected views, and Σ 12 is the cross-covariance matrix between projected views (D'Agostini, 1994; Andrew et al., 2013) . The correlation between the two projected views can be regarded as the sum of all singular values of the normalized cross-covariance (Hotelling, 1992; Anderson et al., 1958) .For multiple views, their correlation is defined as the summation of all the pairwise correlations (Nielsen, 2002; Kettenring, 1971) , which is shown as follows:EQUATIONEssentially, CCA searches for the linear transformation matrices {W k } k that maximize correlation among all the views. Mathematically, it can be represented as:EQUATIONOnce W * k is obtained, the multi-view data are projected into a unified space. Lastly, all projected data are concatenated to obtain Z= [W * 1 X 1 ; • • • ; W * k X k ; • • • ; W * K X K ] for downstream tasks.On top of the CCA, DCCA integrates neural networks and CCA to capture the nonlinear relationship among multi-view data. The major difference of DCCA is that the transformation matrix W k is replaced with multi-layer perceptrons (MLP), and the parameters in MLP are updated through backpropagation (Andrew et al., 2013) . Specifically, each W k is replaced with a neural network f k , which can be viewed as a nonlinear transformation. Similar to CCA, the goal of DCCA is to solve the following optimization problem:EQUATIONAgain, the unified representation is obtained byZ = [f * 1 (X 1 ); • • • ; f * k (X k ); • • • ; f * K (X K )] for downstream tasks.Despite exhibiting promising performance, DCCA shows a significant decline in performance as the training proceeds. We attribute this decline-in-performance phenomenon as model collapse. Model collapse could make the performance of DCCA-based methods even worse than classic CCA methods and simple feature concatenation. As illustrated in (a) of Figure 4 , one can see that the performance of DCCA drops drastically during training, though its best performance at early iterations could exceed that of CCA. The correlation between unrelated data increases as the collapsed model transforms any data to a degenerated feature space. In (b) of Figure 4 , we can see that except for the proposed method (NR-DCCA), the correlation between unrelated data increases significantly. We could also analyze the model collapse in real-world data by feature visualization as shown in Figure 9 in Appendix A.9. However, it should be noted that the full-rank property of representation always holds for DCCA (verified in Appendix A.6.1), while this cannot prevent the model collapse. Indeed, the model collapse of DCCA is mainly due to the difference between W k and f k , which requires further investigation. 
dcca with noise regularization(nr-dcca) 4.1 method: 
 In this section, we present NR-DCCA, which makes use of the noise regularization approach to prevent model collapse in DCCA. Indeed, the developed noise regularization approach can be applied to variants of DCCA methods, such as Deep Generalized CCA (DGCCA) (Benton et al., 2017) . An overview of the NR-DCCA framework is presented in Figure 2 .Figure 2 : Illustration of NR-DCCA. We take the CUB dataset as an example: similar to DCCA, the k-th view X k is transformed using f k to obtain new representation f k (X k ) and then maximize the correlation between new representations. Additionally, for the k-th view, we incorporate the proposed NR loss to regularize f k .The key idea in NR-DCCA is to generate a set of i.i.d Gaussian white noise, denoted asA = {A 1 , • • • , A k , • • • , A K }, A k ∈ R d k ×n, with the same shape as the multi-view data X k . In CCA, the correlation with noise is invariant to the full-rank linear transformation Wk : Corr(X k , A k ) = Corr(W k X k , W k A k ) (rigorous proof provided in Proposition 3). However, for DCCA, Corr(X k , A k ) might not equal Corr(f k (X k ), f k (A k ))because the powerful neural networks f k have overfitted to the maximization problem in DCCA and "created" correlation itself . Therefore, we enforce the DCCA to mimic the behavior of CCA by adding a NR lossζ k = |Corr(f k (X k ), f k (A k )) -Corr(X k , A k )|,and hence the formulation of NR-DCCA is:EQUATION)where α is the hyper-parameter weighing the NR loss. NR-DCCA can be trained through backpropagation with the randomly generated A in each epoch, and the unified representation is obtained directly using {f * k } k in the same manner as DCCA.In this section, we provide the rationale for why the developed noise regularization can help to prevent model collapse. Without loss of generality, we assume that all the datasets X k are zerocentered with respect to row (Hotelling, 1992) . Then we make use of the following proposition to simplify our analysis: Proposition 1 Given a specific matrix B and a zero-centered C with respect to rows, the product BC is also zero-centered with respect to rows.This implies that W k A k and W k X k are both zero-centered matrices. When computing the covariance matrix, there is no need for an additional subtraction of the mean of row, which simplifies our subsequent derivations.We first analyze CCA in detail, and the Moore-Penrose Inverse (MPI) (Petersen et al., 2008) will be used for analysis, and the MPI is defines as follows:Definition 1 Given a specific matrix Y , its Moore-Penrose Inverse (MPI) is denoted asY + . Y + satisfies: Y Y + Y = Y , Y + Y Y + = Y + , Y Y + is symmetric, and Y + Y is symmetric.The MPI Y + is unique and always exists for any Y . Furthermore, when matrix Y is invertible, its inverse matrix Y -is exactly Y + . Using the definition of MPI, we can rewrite the formulation of CCA. In particular, Corr(•, •) can be derived by replacing the inverse with MPI. Using Corr(X k , A k ) as an example, the following proposition holds: Proposition 2 (MPI-based CCA) For the k-th view data X k and the Gaussian white noiseEQUATIONUtilizing the new form of Corr, we discover that the transformation matrix W k obtained from multi-view data X = {X 1 , • • • , X k , • • • , X K }, X k ∈ R d k ×n (d k < n) through CCA(X k , A k ) = Corr(W k X k , W k A k ). Proposition 4 For any k, if Corr(W k X k , W k A k ) = Corr(X k , A k )and W k is a square matrix, then W k must be a full-rank matrix.Proofs of all the above propositions are presented in the Appendix. Combining both Proposition 3 and 4, we have Theorem 1 holds.Theorem 1 (Full-rank of CCA) If W k is a square matrix for any k, thenη k = 0 ⇐⇒ W k is full-rank, where η k = |Corr(W k X k , W k A k ) -Corr(X k , A k )|.It is widely acknowledged that forced by the loss function, CCA searches for full-rank representation W k X K and thereby obtains a full-rank matrix W k (refer to Lemma 2 in Appendix A.3). However, Theorem 1 is profound, as it first time provides the equivalent condition of the full-rank property of W k by connecting it to the correlation with noise, and then we can transplant this condition to DCCA. Essentially, CCA searches for W k as a full-rank matrix, and it is robust to random noise A k , given the fact that the correlations between X k and A k will be invariant to transformation W k . This indicates the W k will not learn to "create" correlation during the maximization process. However, the correlation increases in the DCCA training, which is closely related to the model collapse. We would like the DCCA to hold the same property as CCA, therefore, we define the "full-rank" property for the neural networkf k : Definition 2 ("Full-rank" of f k ) Given k, the neural network f k is called "full-rank" if ζ k = 0.Based on the Definition 2, the proposed NR-DCCA maximizes the correlation among views and also enforces the neural network to be "full-rank". As the neural network f k shares the same property of the full-rank matrix W k , the model collapse issue can be eliminated. It should be noted that the "full-rank" property of f k is different from the full-rank of the representation f k (X k ), as the latter is defined by the matrix rank, while the former one is the new concept in this paper defined by ζ k = 0. 
numerical experiments: 
 We conduct extensive experiments on both synthetic and real-world datasets to answer the following research questions:• RQ1: How can we construct synthetic datasets to evaluate the MVRL methods in a comprehensive manner? • RQ2: Does NR-DCCA avoid model collapse across all synthetic MVRL datasets? • RQ3: Does NR-DCCA perform consistently in real-world datasets?We follow the protocol described in Hwang et al. (2021) for evaluating the MVRL methods. For each dataset, we construct a train dataset and a test dataset. All methods are trained in an unsupervised manner on the train dataset. Subsequently, the test dataset is used to obtain the representation, which will be evaluated in downstream tasks. For the regression task, we employ Ridge Regression (Hoerl & Kennard, 1970) and use R2 as the evaluation metric. For the classification task, we use Support Vector Classifier (SVC) (Chang & Lin, 2011) and report the average F1 scores. All tasks are evaluated using 5-fold cross-validation, and the reported results correspond to the average values of the respective metrics. Baseline methods include CONCAT, CCA (Hotelling, 1992) , PRCCA (Tuzhilina et al., 2023) , KCCA (Akaho, 2006) , DCCA (Andrew et al., 2013) , DGCCA (Benton et al., 2017) , DCCAE/DGCCAE (Wang et al., 2015) , DCCA PRIVATE/DGCCA PRIVATE (Wang et al., 2016) , and MVTCAE (Hwang et al., 2021) . Details of the experiment settings including datasets and baselines are presented in Appendix A.5. Hyper-parameter settings, including ridge regularization and α of NR, are discussed in Appendix A.6. We also analyze the computational complexity of different DCCA-based methods in Appendix A.8 and the learned representations are visualized in Appendix A.9. In the main paper, we mainly compare DCCA and NR-DCCA, while the results related to DGCCA are similar and presented in Appendix A.10.We construct synthetic datasets to assess the performance of MVRL methods, and the framework is illustrated in Figure 3 . We believe that the multi-view data describes the same object, which is represented by a high-dimensional embedding G d×n , where d is the feature dimension and n is the size of the data, and we call it God Embedding. Each view of data is regarded as a nonlinear transformation of part (or all) of G. For example, we choose K = 2, d = 100, and then X 1 = ϕ 1 (G[0 : 50 + CR/2, :]), X 2 = ϕ x (G[50 -CR/2 : 100], :), where ϕ 1 and ϕ 2 are non-linear transformations, and CR is referred to as common rate. The common rate is defined as follows: Definition 3 (Common Rate) For two view data X = {X 1 , X 2 }, common rate is defined as the percentage overlap of the features in X 1 and X 2 that originate from G.One can see that the common rate ranges from 0% to 100%. The larger the value, the greater the correlation between the two views, and a value of 0 indicates that the two views do not share any common dimensions in G. Additionally, we construct the downstream tasks by directly transforming the God Embedding G. Each task T j = ψ j (G), where ψ j is a transformation, and T j represents the j-th task. By setting different G, common rates, ϕ k , and ψ j , we can create various synthetic datasets to evaluate the MVRL methods. Finally, X k are observable to the MVRL methods for learning the representation, and the learned representation will be used to classify/regress T j to examine the performance of each method. Detailed implementation is given in Appendix A.7.We generate the synthetic datasets with different common rates, and the proposed NR-DCCA and other baseline methods are compared, as presented in (a) of An interesting observation is that, most CCA-based methods perform better on data with high common rates, consistent with CCA's assumption of maximizing the correlation between data from different views. However, this also indicates that their performance may degrade with low correlations among views.We  
conclusions: 
 We propose a novel noise regularization approach for DCCA in the context of MVRL, and it can prevent model collapse during the training, which is an issue observed and analyzed in this paper for the first time. Specifically, we theoretically analyze the full-rank property in CCA and demonstrate that it is the key to preventing model collapse. To this end, an analogy of the "full-rank" property is defined for DCCA, and the noise regularization can be considered as an implicit constraint to make the neural networks "full-rank". Additionally, synthetic datasets with different common rates are generated and tested on, which provide a benchmark for fair and comprehensive comparisons of different MVRL methods. The NR-DCCA developed in the paper inherits the merits of both CCA and DCCA to achieve stable and consistent outperformance in both synthetic and real-world datasets. More importantly, the proposed noise regularization approach can also be generalized to other DCCA-based methods (e.g., DGCCA).In future studies, we wish to explore the potential of noise regularization in other representation learning tasks, such as contrastive learning and generative models. It is also interesting to further investigate the "full-rank" definition of neural networks, and how it is different from other neural network regularization approaches, such as orthogonalization (Bansal et al., 2018; Huang et al., 2020) and weight decay (Loshchilov & Hutter, 2017; Zhang et al., 2018; Krogh & Hertz, 1991) . Additionally, it is also interesting but challenging to investigate what (e.g., parameters, gradients, features) and how the neural network has been regularized through the noise. Our ultimate goal is to make the developed noise regularization a pluggable and useful module for neural network regularization. 
 
 Appendix: All of our experiments are conducted with fixed random seeds and all the performance of downstream tasks is the average value of a 5-fold cross-validation. The CCA-zoo package is adopted as the implementation of various CCA/GCCA-based methods, and the original implementation of MVTCAE is employed. Both baselines and our developed NR-DCCA/NR-DGCCA are implemented in the same PyTorch environment (see requirements.txt in the source codes). All the datasets used in the paper are either provided or open datasets. Detailed proofs of all the propositions in the main paper can be found in the Appendix. Both source codes and appendix can be downloaded from the supplementary material.A.1 PROOF OF PROPOSITION 1Let B i,j and C i,j denote the (i, j)-th entry of B and C, respectively. Then we have:(BC) i,j = r=1 B i,r C r,jSince each row of C has a mean of 0, we have n j=1 C r,j = 0, ∀r. For the mean value of i-th row of BC, we can write:1 n n j=1 (BC) i,j = 1 n n j=1 r=1 B i,r C r,j = 1 n r=1 n j=1 B i,r C r,j = 1 n r=1 B i,r   n j=1 C r,j   = 1 n r=1 B i,r • 0 = 0 (8) A.2 PROOF OF PROPOSITION 2 Corr(X k , A k ) = tr((Σ -1/2 11 Σ 12 Σ -1/2 22 ) ′ Σ -1/2 11 Σ 12 Σ -1/2 22 ) 1/2 = tr(Σ -1/2 22 Σ ′ 12 Σ -1/2 11 Σ -1/2 11 Σ 12 Σ -1/2 22 ) 1/2 = tr(Σ -1/2 22 Σ -1/2 22 Σ ′ 12 Σ -1/2 11 Σ -1/2 11 Σ 12 ) 1/2 = tr(Σ -1 22 Σ ′ 12 Σ -1 11 Σ 12 ) 1/2 = 1 (n -1) 2 tr((A k A ′ k ) -1 (X k A ′ k ) ′ (X k X ′ k ) -1 (X k A ′ k )) 1/2 = 1 (n -1) 2 tr((A k X ′ k ) -1 (A k X ′ k )(X k X ′ k ) -1 (X k A ′ k )) 1/2 = 1 (n -1) 2 tr((A k A ′ k ) + (A k X ′ k )(X k X ′ k ) + (X k A ′ k )) 1/2 = 1 (n -1) 2 tr(A ′ k (A k A ′ k ) + A k X ′ k (X k X ′ k ) + X k ) 1/2 = 1 (n -1) 2 tr(A + k A k X + k X k ) 1/2 (9)The first row is based on the definition of Corr, the second row is because trace is invariant under cyclic permutation, the fifth row is to replace matrix inverse by MPI and the ninth row is due to et al., 2008) .Y + = Y ′ (Y Y + ) (PetersenFirstly, we have the k-th view data X k to be full-rank, as we can always delete the redundant data, and the random noise A k is full-rank as each column is generated independently. Then by utilizing Proposition 2, we derive that the correlation between X k and A k remains unchanged before and after the transformation:Corr(W k X k , W k A k ) = 1 (n -1) 2 tr((W k A k ) + W k A k (W k X k ) + W k X k ) 1/2 = 1 (n -1) 2 tr((W + k W k A k ) + (W k A k A + k ) + W k A k (W + k W k X k ) + (W k X k X + k ) + W k X k ) 1/2 = 1 (n -1) 2 tr((W + k W k A k ) + W + k W k A k (W + k W k X k ) + W + k W k X k ) 1/2 = 1 (n -1) 2 tr(A + k A k X + k X k ) 1/2 = Corr(X k , A k )(10) The first row is based on Proposition 2, the second row is because given two matrices B and C, (BC) + = (B + BC) + (BC + C) + always holds (Petersen et al., 2008) , and the third row utilizes the properties of full-rank and square matrix et al., 2008) .W k : W + k = W - k , which means W + k W k = W k W + k = I d k (PetersenThis proposition is equivalent to its contra-positive proposition: if W k is not a full-rank matrix, there exists random noise dataA k such that η k = |Corr(W k X k , W k (A k )) -Corr(X k , A k )| is not 0.And we find that when W k is not full-rank, there exists A k = X k such that η k ̸ = 0. We have the following derivation:η k = |Corr(W k X k , W k A k ) -Corr(X k , A k )| = 1 (n -1) 2 tr((W k A k ) + (W k A k )(W k X k ) + (W X k )) 1/2 - 1 (n -1) 2 tr(A + AX + X) 1/2 = 1 (n -1) 2 tr((W + k W k A k ) + (W k A k A + k ) + W k A k (W + k W k X k ) + (W k X k X + k ) + W k X k ) 1/2 - 1 (n -1) 2 tr(A + k A k X + k X k ) 1/2 = 1 (n -1) 2 tr((W + k W k A k ) + W + k W k A k (W + k W k X k ) + W + k W k X k ) 1/2 - 1 (n -1) 2 tr(A + k AX + k X k ) 1/2(11) The first row is the definition of NR loss with respect to W k , the second row is based on the new form of CCA, the third row is because given two specific matrices B and C, it holds the equality (BC) + = (B + BC) + (BC + C) + (Petersen et al., 2008), and the fourth row utilizes the properties of full-rank matrix: for full-rank matrices X k and A k , whose sample size is larger than dimension size, they fulfill: et al., 2008) . Petersen et al., 2008) .X k X k + = I d k , A k A k + = I d k (given a specific full-rank matrix Y , if its number of rows is smaller than that of cols, it holds that Y + = Y ′ (Y Y ′ ) -, which means that Y Y + = I) (PetersenA k = X k : η k = 1 (n -1) 2 tr((W + k W k X k ) + W + k W k X k (W + k W k X k ) + W + k W k X k ) 1/2 - 1 (n -1) 2 tr(X + k X k X + k X k ) 1/2 = 1 (n -1) 2 tr((W + k W k X k ) + W + k W k X k ) 1/2 - 1 (n -1) 2 tr(X + k X k ) 1/2 . (12) The first row is to replace A k with X k , the second row is because X + k X k X + k = X + k and (W + k W k X k ) + W + k W k X k (W + k W k X k ) + = W + k W k X k , which are based on the definition of MPI that given a specific matrix Y , Y + Y Y + = Y + (Next, we need to prove the following two lemmas:Lemma 1 Given a specific matrix Y and its MPI Y + , let Rank(Y ) and Rank(Y + Y ) be the ranks of Y and Y + Y , respectively. It is true that:Rank(Y ) = Rank(Y + Y ) Rank(Y + Y ) = tr(Y + Y )Proof Firstly, the column space of Y + Y is a subspace of the column space of Y . Therefore, Rank(Y + Y ) ≤ Rank(Y ). On the other hand, according to the definition of MPI (Petersen et al., 2008) , we know that Y = Y (Y + Y ). Since the rank of a product of matrices is at most the minimum of the ranks of the individual matrices, we have Rank(Y ) ≤ Rank(Y + Y ). Combining the two inequalities, we have Rank MPI (Petersen et al., 2008) ), Y + Y is an idempotent and symmetric matrix, and thus its eigenvalues must be 0 or 1. So the sum of its eigenvalues is exactly its rank. Considering matrix trace is the sum of eigenvalues of matrices, we have Rank((Y ) = Rank(Y + Y ). Furthermore, since (Y + Y )(Y + Y ) = Y + Y (it holds that Y + = Y + Y Y + according to the defi- nition ofY + Y ) = tr(Y + Y ). Lemma 2 Rank(W k X k ) < Rank(X k ), when W k is not a full-rank matrix and X k is a full-rank matrix.Proof Since the rank of a product of matrices is at most the minimum of the ranks of the individual matrices, we have Rank(W k X k ) ≤ min(Rank(W k ), Rank(X k )). Considering X k is full-rank, Rank(X k ) = min(d k , n) and then Rank(W k X k ) ≤ min(Rank(W k ), Rank(X k )) = min(Rank(W k ), min(d k , n)). Since W k is not full-rank, we have Rank(W k ) < d k . In conclu- sion, Rank(W k X k ) < min(d k , min(d k , n)) and then Rank(W k X k ) < d k ≤ Rank(X k ).As a result, we can know that when the random noise data A k is exactly X k and W k is not full-rank, η k can not be zero:η k = 1 (n -1) 2 tr((W + k W k X k ) + W + k W k X k ) 1/2 - 1 (n -1) 2 tr(X + k X k ) 1/2 = 1 (n -1) 2 Rank(W + k W k X k ) 1/2 - 1 (n -1) 2 Rank(X) 1/2 ̸ = 1 (n -1) 2 Rank(X k ) 1/2 - 1 (n -1) 2 Rank(X k ) 1/2 ̸ = 0 (13)The first row is due to Equation 12, the second row is based on Lemma 1 that tr((W+ k W k X k ) + W + k W k X k ) = Rank(W + k W k X k ) and tr(X + k X k ) = Rank(X), and the third row is because of Lemma 2.Finally, we have if η k is always constrained to 0 for any A k , then W k must be a full-rank matrix.We make 6 groups of multi-view data originating from the same G ∈ R d×n (we set n = 4000, d = 100). Each group consists of tuples with 2 views (2000 tuples for training and 2000 tuples for testing) and a distinct common rate. Common rates of these sets are from {0%, 20%, 40%, 60%, 80%, 100%} and there are 50 downstream regression tasks. We report the mean and standard deviation of R2 score across all the tasks.PolyMnist (Sutter et al., 2021) : A dataset consists of tuples with 5 different MNIST images (60, 000 tuples for training and 10, 000 tuples for testing). Each image within a tuple possesses distinct backgrounds and writing styles, yet they share the same digit label. The background of each view is randomly cropped from an image and is not used in other views. Thus, the digit identity represents the common information, while the background and writing style serve as view-specific factors. The downstream task is the digit classification task. CUB (Wah et al., 2011) : A dataset consists of tuples with deep visual features (1024-d) extracted by GOOGLENET and text features (300-d) obtained through DOC2VEC (Le & Mikolov, 2014) (480 tuples for training and 600 tuples for testing). This MVRL task utilizes the first 10 categories of birds in the original dataset and the downstream task is the bird classification task. Caltech (Deng et al., 2018) : A dataset consists of tuples with traditional visual features extracted from images that belong to 101 object categories, including an additional background category (6400 tuples for training and 9144 tuples for testing). Following Yang et al. (2021) , three features are used as views: a 1, 984-d HOG feature, a 512-d GIST feature, and a 928-d SIFT feature.In this section, we discuss the effects of hyper-parameter r in ridge regularization. Ridge regularization is commonly used across almost all (D)CCA methods, whichimproves numerical stability. It works by adding an identity matrix I to the estimated covariance matrix. However, ridge regularization mainly regularizes the features, rather than the transformation (i.e., W k in CCA and f k in DCCA) and it cannot prevent the neural networks from degenerating (i.e., model collapse). To further support our arguments, we provide the experimental results with different ridge parameters on a real-world dataset CUB as shown in Figure 7 . One can see that the ridge regularization even damages the performance of DCCA. In our NR-DCCA, we actually set the ridge parameter to zero. We conjecture the reason is that the large ridge parameter could make the neural network even "lazier" to actively project the data into a better feature space, as the full-rank property of features and covariance matrix are already guaranteed, and this is also evidenced by the "Square sum of feature covariance" shown in the figure. The choice of the hyper-parameter α is essential in NR-DCCA. Different from the conventional hyper-parameter tuning procedures, the determination of α is simpler, as we can search for the smallest α that can prevent the model collapse, and the model collapse can be directly observed on the validation data. Specifically, we increase the α adaptively until the model collapse issue is tackled, i.e., the correlation with noise will not increase or the performance of DCCA will not drop with increasing training epochs, then the optimal α is found. To further illustrate the influence of α in NR-DCCA, we present performance curves of NR-DCCA in CUB under different α. As shown in Figure 8 , if α is too large, the convergence of the training becomes slow; if α is too small, model collapse still remains. Additionally, one can see the NR-DCCA outperforms DCCA robustly with a wide range of α. We draw n random samples with dim d from a Gaussian distribution as G ∈ R d×n to represent complete representations of n objects. We define the non-linear transformation ϕ k as the addition of noise to the data, followed by passing it through a randomly generated MLP. To generate the data for the k-th view, we select specific feature dimensions from G based on a given common rate 3 and then apply ϕ k to those selected dimensions. And we define ψ j as a linear layer, and task T j is generated by directly passing G through ψ j .A.8 COMPLEXITY ANALYSIS In this section, we compare the computational complexity of different DCCA-based methods. Assuming that we have data from K views, with each view containing N samples and D feature dimensions, then we have the computational complexity of each method in Table 1 . - - O(K * N * D) MLP Encoder O(K * N * L * H 2 ) O(K * N * L * H 2 ) O(2 * K * N * L * H 2 ) O(2 * K * N * L * H 2 ) MLP Decoder - O(K * N * L * H 2 ) O(K * N * L * H 2 ) - Reconstruction Loss - O(K * N * D) O(K * N * D) - Correlation Maximization O((M * K) 3 ) O((M * K) 3 ) O((M * K) 3 ) O((M * K) 3 ) Noise Regularization - - - O(2 * K * (M * K) 3 )• Complexity of MLP: We will use DNNs with the same MLP structure, consisting of L hidden layers, each with H neurons. Therefore, the computational complexity of one pass of the data through the DNNs can be expressed asO(N * (D * H + D * M + L * H 2 )).To simplify, we use O(N * L * H 2 ).• Complexity of Corr: During the process of calculating Cor among K views, three main computations are involved. The calculation complexity of the covariance is O(N * (M * K) 2 . Second, the complexity of the inverse matrix and the eigenvalues are O((M * K) 3 . As a result, the computational complexity of calculating Cor can be considered as O((M * K) 3 ).• Complexity of reconstruction loss: The reconstruction loss, also known as the mean squared error (MSE) loss, has a complexity of O(N * D).To further demonstrate the effectiveness of our method, we employ 2D-tSNE visualization to depict the learned representations of the CUB dataset (test set) under different methods. Each data point is colored based on its corresponding class, as illustrated in Figure 9 . There are a total of 10 categories, with 60 data points in each category. A reasonable distribution of learned representations entails that data points belonging to the same class are grouped together in the same cluster, which is distinguishable from clusters representing other classes. Additionally, within each cluster, the data points should exhibit an appropriate level of dispersion, indicating that the data points within the same class can be differentiated rather than collapsing into a single point. This dispersion is indicative of the preservation of as many distinctive features of the data as possible.From Figure. 9, we can observe that CCA, DCCA / DGCCA have all confused the data from different categories. Specifically, CCA completely scatter the data points as it cannot handle nonlinear relationships. By incorporating autoencoders, DCCAE / DGCCAE and DCCA PRIVATE / DGCCA PRIVATE have partially separated the data; however, they have not fully separated the green and orange categories. NR-DCCA / NR-DGCCA is the only method that successfully separates all categories.It is worth noting that our approach not only separates the data into different clusters but also maintains dispersion within each cluster. Unlike DCCA PRIVATE / DGCCA PRIVATE, where the data points within a cluster form a strip-like distribution, our method ensures that the data points within each cluster remain appropriately scattered.  


Decision: This submission proposed a method for Deep CCA in preventing model collapse by adding regularization in learning. 3/4 reviewers rates the submission as borderline accept, and the other one rates it as borderline reject with major concern of limited novelty. After reading the submission, comments, and rebuttal，the AC recognize the merits of this submission and suggest accepting this paper.

Reviews: review: paper_summary: paper_summary: This work focuses on multi-view learning. Specifically, it studies the deep canonical correlation analysis and its variants. This study observes the issue of model collapse and proposes a regularization learning strategy to release the problem, then solve the early stop challenging.
strength_weakness: strength_weakness: 1. Adding regularization has been fully explored in different machine learning scenarios. To this end, the proposed method is lack of research novelty, which may diminish the paper contribution.
2. The compared methods are relatively old, adding more recent publication for comparison helps to support the draft.
3. Numerical results only contain some small datasets, using more large-scale ones will be helpful, especially in current large-scale learning era.
questions: questions: Please refer to the weaknesses section for reference.
limitations: limitations: None
paper_summary: paper_summary: The paper proposes noise regularization term to prevent collapse issues found in deep canonical correlation analysis (DCCA) methods. The term makes DCCA to behave in a way similar to linear CCA, which is robust to collapse by definition, thereby making DCC with the regularization robust against collapse issue.
strength_weakness: strength_weakness: 1. Readability. It is quite hard to find the exact configuration used for real-world experiments such as what encoders are used for DCCA. The content in the appendix is huge, but they are not organized well.
2. The degree of model collapse will depend on the model complexity (e.g., the number of parameters in the MLP). But there is no clear analysis on this aspect. Would the proposed noise regularization can prevent the collapse issue under any degree of MLP complexity?
questions: questions: 1. In Line 137, the setting assumes the same sample size for all the datasets X_k. Is it necessary, and how about other CCA methods?
2. What are the practical application of the CCA methods? Despite the highly theoretical nature of the work, I think it would be better if there is at least one line of comment that explains a practical use case of the CCA methods in the introduction or appendix for the readers who are not familiar with the field.
3. In line 137, what are the actual examples of X_k in practice? 
4. The experiment setting is not self-contained. What is the exact protocol from Hwang et al. (2021) in Line 250? It should be given in the appendix.
5. What is the deep network $f$ used for DCCA?
6. What if the features in X_k are the representations from foundational pre-trained encoders but possibly of different vector dimension across different views?
7. The NESum recovers after epoch 600 in Figure 3 for DCCA_PRIVATE. Why is it that? But based on (d), the model seems to collapse.
limitations: limitations: Please refer to 'Weaknesses'
paper_summary: paper_summary: The authors propose NR-DCCA, a novel approach of Deep Canonical Correlation Analysis(DCCA) equipping with noise regularization, in order to prevent DCCA-based methods from model collapse in the multi-view representation learning(MVRL) task. First, the authors analyze the difference between Linear CCA and DCCA, and then draw a conclusion on the cause of model collapse of DCCA. After that they propose the NR-DCCA method, which integrates noise-regularization into DCCA. Theoretical analysis is also provided to further illustrate the effect of noise regularization. Experimental results show the effectiveness of NR-DCCA.
strength_weakness: strength_weakness: 1. The authors hypothesize the root cause of model collapse of DCCA is that the weight matrices of network is not guaranteed to be full-rank, which possibly leads to overfitting. The experimental results are good and I think this hypothesis is probably right to some extent. However, as I know the relation between weight matrices’full-rank property and the overfitting phenomenon is still underexplored. The authors do not provide enough proofs to support this opinion. Besides, ‘full-rank’ is not a precise(even misleading) enough expression if the authors determine to use NESum as metric to evaluate the redundacy of weight matrices.
2. The theoretical proof of Theorem 2 only shows that noise regularization do not cause a serious degradation to DCCA’s performance. However, why noise regularization works in DCCA still remains unclear. The authors just write they try to enforce DCCA to mimic the behavior of Linear CCA but no more illustrations are provided.
3. Some texts in the figures are too small.
questions: questions: 1. why can noise regularization enforce DCCA to mimic the behavior of Linear CCA and therefore prevent DCCA from model collapse?
2. The authors need to polish their expressions. According to the math formulat of NESum, a full-rank matrix with considerably different eigenvalues can still have low NESum score. For example, diag(1000,1,1) has lower NESum than diag(2,1,0), both of them are full-rank.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper reveals a new observation in multi-view representation learning, that is, the performance of DCCA-based methods will gradually decrease as training progresses. The authors explore the possible reasons from the rank of weights and conclude that the Correlation Invariant Property is the key to preventing this problem, and propose NR-DCCA. Experiments also show the effectiveness of the proposed NR-DCCA.
strength_weakness: strength_weakness: 1. In fact, the authors call this degradation phenomenon model collapse, which is not very accurate. Collapse should be very extreme. The performance degradation is a bit like an overfitting problem.
2. The method is effective, but it cannot be denied that it is too simple.
3. The legend may be better if it is a vector diagram.
4. The authors' theoretical analysis is based on Gaussian white noise. Is it the same for other types of noise?
5. Is NR loss also universal and effective on other DCCA methods? I think this is very important because it avoids the risk that DCCA is a special case.
questions: questions: See Weaknesses.
limitations: limitations: Yes.

Comments: title: Looking forward to response and discussion
comment: Thanks very much for your recognition of our work!
I appreciate the effort put into the rebuttal, which addressed some of my concerns. After reading the other reviews and replies, I have decided to raise my score accordingly.
Dear reviewers,

Thanks very much for your kind suggestions and raised concerns. 

We put a lot of effort into the replies and additional experiments, and we'd like to know if we've resolved your concerns. Considering that there's not much time left in the discussion phase, please take the time to reply.