Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective

1 INTRODUCTION . One pillar of unsupervised representation learning is multiview learning . Extracting shared information from multiple “ views ” ( e.g. , image and audio ) of data entities has been considered a major means to fend against noise and data scarcity . A key computational tool for multiview learning is canonical correlation analysis ( CCA ) ( Hotelling , 1936 ) . The classic CCA seeks linear transformation matrices such that transformed views are maximally correlated . A number of works studied nonlinear extensions of CCA ; see kernel CCA in ( Lai & Fyfe , 2000 ) and deep learning-based CCA ( DCCA ) in ( Andrew et al. , 2013 ; Wang et al. , 2015 ) . DCCA and its variants were shown to largely outperform the classical linear CCA in many tasks . In recent years , a series of self-supervised learning ( SSL ) paradigms were proposed . These SSL approaches exhibit a lot of similarities with DCCA approaches , except that the “ views ” are noisy data “ augmenented ” from the original clean data . To be specific , different views are generated by distorting data—e.g. , using rotating , cropping , and/or adding noise to data samples ( Dosovitskiy et al. , 2015 ; Gidaris et al. , 2018 ; Chen et al. , 2020 ; Grill et al. , 2020 ) . Then , neural encoders are employed to map these artificial views to embeddings that are highly correlated across views . This genre—which will be referred to as artificial multiview SSL ( AM-SSL ) —includes some empirically successful frameworks , e.g. , BYOL ( Grill et al. , 2020 ) and Barlow Twins ( Zbontar et al. , 2021 ) . Notably , many DCCA and AM-SSL approaches involve ( explicitly or implicitly ) searching for highly correlated representations from multiple views , using neural feature extractors ( encoders ) . The empirical success of DCCA and AM-SSL bears an important research question : How to under- stand the role of cross-view correlation in deep multiview learning ? Furthermore , how to use such understanding to design theory-backed learning criteria to serve various purposes ? Intuitively , it makes sense that many DCCA and AM-SSL paradigms involve latent correlation maximization in their loss functions , as such loss functions lead to similar/identical representations from different views—which identifies view-invariant essential information that is often identityrevealing . However , beyond intuition , theoretical support of latent correlation-based deep multiview learning had been less studied , until recent works started exploring this direction in both nonlinear CCA and AM-SSL ( see , e.g. , ( Lyu & Fu , 2020 ; von Kügelgen et al. , 2021 ; Zimmermann et al. , 2021 ; Tian et al. , 2021 ; Saunshi et al. , 2019 ; Tosh et al. , 2021 ) ) , but more insights and theoretical underpinnings remain to be discovered under more realistic and challenging settings . In this work , we offer an understanding to the role of latent correlation maximization that is seen in a number of DCCA and AM-SSL systems from a nonlinear mixture learning viewpoint—and use such understanding to assist various learning tasks , e.g. , clustering , cross-view translation , and cross-sample generation . Our detailed contributions are : ( i ) Understanding Latent Correlation Maximization - Shared Component Identification . We start with a concept that has been advocated in many multiview learning works . In particular , the views are nonlinear mixtures of shared and private latent components ; see , e.g. , ( Huang et al. , 2018 ; Lee et al. , 2018 ; Wang et al. , 2016 ) . The shared components are distortion/view invariant and identity-revealing . The private components and view-specific nonlinear mixing processes determine the different appearances of the views . By assuming independence between the shared and private components and invertibility of the data generating process , we show that maximizing the correlation of latent representations extracted from different views leads to identification of the ground-truth shared components up to invertible transformations . ( ii ) Imposing Additional Constraints - Private Component Identification . Using the understanding to latent correlation maximization-type loss functions in DCCA and AM-SSL , we take a step further . We show that with carefully imposed constraints , the private components in the views can also be identified , under reasonable assumptions . Learning private components can facilitate tasks such as cross-view and cross-sample data generation ( Huang et al. , 2018 ; Lee et al. , 2018 ) . ( iii ) Finite-Sample Analysis . Most existing unsupervised nonlinear mixture identification works , e.g. , those from the nonlinear independent component analysis ( ICA ) literature ( Hyvarinen & Morioka , 2016 ; 2017 ; Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020 ; Locatello et al. , 2020 ; Gresele et al. , 2020 ) , are based on infinite data . This is perhaps because finite sample analysis for unsupervised learning is generally much more challenging relative to supervised cases—and there is no existing “ universal ” analytical tools . In this work , we provide sample complexity analysis for the proposed unsupervised multiview learning criterion . We come up with a success metric for characterizing the performance of latent component extraction , and integrate generalization analysis and numerical differentiation to quantify this metric . To our best knowledge , this is the first finite-sample analysis of nonlinear mixture model-based multiview unsupervised learning . ( iv ) Practical Implementation . Based on the theoretical understanding , we propose a latent correlation-maximization based multiview learning criterion for extracting both the shared components and private components . To realize the criterion , a notable innovation is a minimax neural regularizer that serves for extracting the private components . The regularizer shares the same purpose of some known independence promoters ( e.g. , Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2005a ) ) but is arguably easier to implement using stochastic gradient algorithms . Notation . The notations used in this work are summarized in the supplementary material . 2 BACKGROUND : LATENT CORRELATION IN DCCA AND AM-SSL . In this section , we briefly review some deep multiview learning paradigms that use latent correlation maximization and its close relatives . 2.1 LATENT CORRELATION MAXIMIZATION IN DCCA . DCCA methods aim at extracting common information from multiple views of data samples . Such information is expected to be informative and essential in representing the data . • DCCA . The objective of DCCA can be summarized as follows ( Andrew et al. , 2013 ) : maximize f ( 1 ) , f ( 2 ) Tr ( E [ f ( 1 ) ( x ( 1 ) ) f ( 2 ) ( x ( 2 ) ) ⊤ ] ) , s.t . E [ f ( q ) ( x ( q ) ) f ( q ) ( x ( q ) ) ⊤ ] = I , ( 1 ) where x ( q ) ∈ RMq ∼ Dq is a data sample from view q for q = 1 , 2 , Dq is the underlying distribution of the qth view , f ( 1 ) : RM1 → RD and f ( 2 ) : RM2 → RD are two neural networks . CCA was found particularly useful in fending against unknown and strong view-specific ( private ) interference ( see theoretical supports in ( Bach & Jordan , 2005 ; Ibrahim & Sidiropoulos , 2020 ) ) . Such properties were also observed in DCCA research ( Wang et al. , 2015 ) , while theoretical analysis is mostly elusive . An equivalent representation of ( 1 ) is as follows minimize f ( 1 ) , f ( 2 ) E [ ∥∥∥f ( 1 ) ( x ( 1 ) ) − f ( 2 ) ( x ( 2 ) ) ∥∥∥2 2 ] , s.t . E [ f ( q ) ( x ( q ) ) f ( q ) ( x ( q ) ) ⊤ ] = I , ( 2 ) which is expressed from latent component matching perspective . Both the correlation maximization form in ( 1 ) and the component matching form in ( 2 ) are widely used in the literature . As we will see in our proofs , although the former is popular in the literature ( Andrew et al. , 2013 ; Wang et al. , 2015 ; Chen et al. , 2020 ) , the latter is handier for theoretical analysis . • Slack Variable-Based DCCA . In ( Benton et al. , 2017 ) and ( Lyu & Fu , 2020 ) , a deep multiview learning criterion is used : minimize f ( q ) 2∑ q=1 E [ ∥∥∥f ( q ) ( x ( q ) ) − g∥∥∥2 ] , s.t . E [ |gi|2 ] = 1 , E [ gigj ] = 0 . ( 3 ) The slack variable g represents the common latent embedding learned from the two views . Conceptually , this is also latent correlation maximization ( or latent component matching ) . To see this , assume that there exists f ( q ) ( x ( q ) ) = g for all x ( q ) . The criterion amounts to learning [ f ( 1 ) ( x ( 1 ) ) ] k = [ f ( 2 ) ( x ( 2 ) ) ] k—which has the maximally attainable correlation . 2.2 LATENT CORRELATION MAXIMIZATION/COMPONENT MATCHING IN AM-SSL . Similar to DCCA , the goal of AM-SSL is also to find identity-revealing embeddings of data samples without using labels . The idea is often realized via intentionally distorting the data to create multiple artificial views . Then , the encoders are require to produce highly correlated ( or closely matched ) embeddings from such views . In AM-SSL , the views x ( 1 ) and x ( 2 ) are different augmentations ( e.g. , by adding noise , cropping , and rotation ) of the sample x . • Barlow Twins . The most recent development , namely , the Barlow Twins network ( Zbontar et al. , 2021 ) is appealing since it entails a succinct implementation . Specifically , the Barlow Twins network aims to learn a single encoder f : RM → RD for two distorted views . The cost function is as follows : minimize f D∑ i=1 ( 1− Cii ) 2 + λ D∑ i=1 D∑ j ̸=i C2ij , where Cij = E [ [ f ( x ( 1 ) ) ] i [ f ( x ( 2 ) ) ] j ] √ E [ [ f ( x ( 1 ) ) ] 2i ] √ E [ [ f ( x ( 2 ) ) ] 2j ] . When the learned embeddings are constrained to have zero mean , i.e. , E [ f ( x ( q ) ) ] = 0 , Cij is the cross-correlation between f ( x ( 1 ) ) and f ( x ( 2 ) ) . Note that the normalized representation of crosscorrelation in Cij is equivalent to the objective in ( 1 ) with the orthogonality constraints . • BYOL . The BYOL method ( Grill et al. , 2020 ) uses a cross-view matching criterion that can be distilled as follows : minimize f ( 1 ) , f ( 2 ) E [ ∥∥∥f ( 1 ) ( x ( 1 ) ) − f ( 2 ) ( x ( 2 ) ) ∥∥∥2 2 ] ( 4 ) where f ( q ) ( · ) means that the output of the network is normalized . In BYOL , the networks are constructed in a special way ( e.g. , part of f ( 2 ) ’ s weights are moving averages of the correspond part of f ( 1 ) ’ s weights ) . Nonetheless , the cross-view matching perspective is still very similar to that in latent component matching in ( 2 ) . • SimSiam . The loss function of SimSiam ( Chen & He , 2021 ) has a similar structure as that of BYOL , but with a Siamese network , which , essentially , is also latent component matching .

Summary: The paper offers a theoretical analysis of neural canonical correlation analysis (CCA) type methods. This reveals conditions under which these methods may work. The paper then proposes regularizers that approximately realize these conditions, which result in a new CCA-type algorithm. Empirical results indicate that the approximate algorithm behaves as dictated by the theory.