Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective

1 INTRODUCTION . One pillar of unsupervised representation learning is multiview learning . Extracting shared information from multiple “ views ” ( e.g. , image and audio ) of data entities has been considered a major means to fend against noise and data scarcity . A key computational tool for multiview learning is canonical correlation analysis ( CCA ) ( Hotelling , 1936 ) . The classic CCA seeks linear transformation matrices such that transformed views are maximally correlated . A number of works studied nonlinear extensions of CCA ; see kernel CCA in ( Lai & Fyfe , 2000 ) and deep learning-based CCA ( DCCA ) in ( Andrew et al. , 2013 ; Wang et al. , 2015 ) . DCCA and its variants were shown to largely outperform the classical linear CCA in many tasks . In recent years , a series of self-supervised learning ( SSL ) paradigms were proposed . These SSL approaches exhibit a lot of similarities with DCCA approaches , except that the “ views ” are noisy data “ augmenented ” from the original clean data . To be specific , different views are generated by distorting data—e.g. , using rotating , cropping , and/or adding noise to data samples ( Dosovitskiy et al. , 2015 ; Gidaris et al. , 2018 ; Chen et al. , 2020 ; Grill et al. , 2020 ) . Then , neural encoders are employed to map these artificial views to embeddings that are highly correlated across views . This genre—which will be referred to as artificial multiview SSL ( AM-SSL ) —includes some empirically successful frameworks , e.g. , BYOL ( Grill et al. , 2020 ) and Barlow Twins ( Zbontar et al. , 2021 ) . Notably , many DCCA and AM-SSL approaches involve ( explicitly or implicitly ) searching for highly correlated representations from multiple views , using neural feature extractors ( encoders ) . The empirical success of DCCA and AM-SSL bears an important research question : How to under- stand the role of cross-view correlation in deep multiview learning ? Furthermore , how to use such understanding to design theory-backed learning criteria to serve various purposes ? Intuitively , it makes sense that many DCCA and AM-SSL paradigms involve latent correlation maximization in their loss functions , as such loss functions lead to similar/identical representations from different views—which identifies view-invariant essential information that is often identityrevealing . However , beyond intuition , theoretical support of latent correlation-based deep multiview learning had been less studied , until recent works started exploring this direction in both nonlinear CCA and AM-SSL ( see , e.g. , ( Lyu & Fu , 2020 ; von Kügelgen et al. , 2021 ; Zimmermann et al. , 2021 ; Tian et al. , 2021 ; Saunshi et al. , 2019 ; Tosh et al. , 2021 ) ) , but more insights and theoretical underpinnings remain to be discovered under more realistic and challenging settings . In this work , we offer an understanding to the role of latent correlation maximization that is seen in a number of DCCA and AM-SSL systems from a nonlinear mixture learning viewpoint—and use such understanding to assist various learning tasks , e.g. , clustering , cross-view translation , and cross-sample generation . Our detailed contributions are : ( i ) Understanding Latent Correlation Maximization - Shared Component Identification . We start with a concept that has been advocated in many multiview learning works . In particular , the views are nonlinear mixtures of shared and private latent components ; see , e.g. , ( Huang et al. , 2018 ; Lee et al. , 2018 ; Wang et al. , 2016 ) . The shared components are distortion/view invariant and identity-revealing . The private components and view-specific nonlinear mixing processes determine the different appearances of the views . By assuming independence between the shared and private components and invertibility of the data generating process , we show that maximizing the correlation of latent representations extracted from different views leads to identification of the ground-truth shared components up to invertible transformations . ( ii ) Imposing Additional Constraints - Private Component Identification . Using the understanding to latent correlation maximization-type loss functions in DCCA and AM-SSL , we take a step further . We show that with carefully imposed constraints , the private components in the views can also be identified , under reasonable assumptions . Learning private components can facilitate tasks such as cross-view and cross-sample data generation ( Huang et al. , 2018 ; Lee et al. , 2018 ) . ( iii ) Finite-Sample Analysis . Most existing unsupervised nonlinear mixture identification works , e.g. , those from the nonlinear independent component analysis ( ICA ) literature ( Hyvarinen & Morioka , 2016 ; 2017 ; Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020 ; Locatello et al. , 2020 ; Gresele et al. , 2020 ) , are based on infinite data . This is perhaps because finite sample analysis for unsupervised learning is generally much more challenging relative to supervised cases—and there is no existing “ universal ” analytical tools . In this work , we provide sample complexity analysis for the proposed unsupervised multiview learning criterion . We come up with a success metric for characterizing the performance of latent component extraction , and integrate generalization analysis and numerical differentiation to quantify this metric . To our best knowledge , this is the first finite-sample analysis of nonlinear mixture model-based multiview unsupervised learning . ( iv ) Practical Implementation . Based on the theoretical understanding , we propose a latent correlation-maximization based multiview learning criterion for extracting both the shared components and private components . To realize the criterion , a notable innovation is a minimax neural regularizer that serves for extracting the private components . The regularizer shares the same purpose of some known independence promoters ( e.g. , Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2005a ) ) but is arguably easier to implement using stochastic gradient algorithms . Notation . The notations used in this work are summarized in the supplementary material . 2 BACKGROUND : LATENT CORRELATION IN DCCA AND AM-SSL . In this section , we briefly review some deep multiview learning paradigms that use latent correlation maximization and its close relatives . 2.1 LATENT CORRELATION MAXIMIZATION IN DCCA . DCCA methods aim at extracting common information from multiple views of data samples . Such information is expected to be informative and essential in representing the data . • DCCA . The objective of DCCA can be summarized as follows ( Andrew et al. , 2013 ) : maximize f ( 1 ) , f ( 2 ) Tr ( E [ f ( 1 ) ( x ( 1 ) ) f ( 2 ) ( x ( 2 ) ) ⊤ ] ) , s.t . E [ f ( q ) ( x ( q ) ) f ( q ) ( x ( q ) ) ⊤ ] = I , ( 1 ) where x ( q ) ∈ RMq ∼ Dq is a data sample from view q for q = 1 , 2 , Dq is the underlying distribution of the qth view , f ( 1 ) : RM1 → RD and f ( 2 ) : RM2 → RD are two neural networks . CCA was found particularly useful in fending against unknown and strong view-specific ( private ) interference ( see theoretical supports in ( Bach & Jordan , 2005 ; Ibrahim & Sidiropoulos , 2020 ) ) . Such properties were also observed in DCCA research ( Wang et al. , 2015 ) , while theoretical analysis is mostly elusive . An equivalent representation of ( 1 ) is as follows minimize f ( 1 ) , f ( 2 ) E [ ∥∥∥f ( 1 ) ( x ( 1 ) ) − f ( 2 ) ( x ( 2 ) ) ∥∥∥2 2 ] , s.t . E [ f ( q ) ( x ( q ) ) f ( q ) ( x ( q ) ) ⊤ ] = I , ( 2 ) which is expressed from latent component matching perspective . Both the correlation maximization form in ( 1 ) and the component matching form in ( 2 ) are widely used in the literature . As we will see in our proofs , although the former is popular in the literature ( Andrew et al. , 2013 ; Wang et al. , 2015 ; Chen et al. , 2020 ) , the latter is handier for theoretical analysis . • Slack Variable-Based DCCA . In ( Benton et al. , 2017 ) and ( Lyu & Fu , 2020 ) , a deep multiview learning criterion is used : minimize f ( q ) 2∑ q=1 E [ ∥∥∥f ( q ) ( x ( q ) ) − g∥∥∥2 ] , s.t . E [ |gi|2 ] = 1 , E [ gigj ] = 0 . ( 3 ) The slack variable g represents the common latent embedding learned from the two views . Conceptually , this is also latent correlation maximization ( or latent component matching ) . To see this , assume that there exists f ( q ) ( x ( q ) ) = g for all x ( q ) . The criterion amounts to learning [ f ( 1 ) ( x ( 1 ) ) ] k = [ f ( 2 ) ( x ( 2 ) ) ] k—which has the maximally attainable correlation . 2.2 LATENT CORRELATION MAXIMIZATION/COMPONENT MATCHING IN AM-SSL . Similar to DCCA , the goal of AM-SSL is also to find identity-revealing embeddings of data samples without using labels . The idea is often realized via intentionally distorting the data to create multiple artificial views . Then , the encoders are require to produce highly correlated ( or closely matched ) embeddings from such views . In AM-SSL , the views x ( 1 ) and x ( 2 ) are different augmentations ( e.g. , by adding noise , cropping , and rotation ) of the sample x . • Barlow Twins . The most recent development , namely , the Barlow Twins network ( Zbontar et al. , 2021 ) is appealing since it entails a succinct implementation . Specifically , the Barlow Twins network aims to learn a single encoder f : RM → RD for two distorted views . The cost function is as follows : minimize f D∑ i=1 ( 1− Cii ) 2 + λ D∑ i=1 D∑ j ̸=i C2ij , where Cij = E [ [ f ( x ( 1 ) ) ] i [ f ( x ( 2 ) ) ] j ] √ E [ [ f ( x ( 1 ) ) ] 2i ] √ E [ [ f ( x ( 2 ) ) ] 2j ] . When the learned embeddings are constrained to have zero mean , i.e. , E [ f ( x ( q ) ) ] = 0 , Cij is the cross-correlation between f ( x ( 1 ) ) and f ( x ( 2 ) ) . Note that the normalized representation of crosscorrelation in Cij is equivalent to the objective in ( 1 ) with the orthogonality constraints . • BYOL . The BYOL method ( Grill et al. , 2020 ) uses a cross-view matching criterion that can be distilled as follows : minimize f ( 1 ) , f ( 2 ) E [ ∥∥∥f ( 1 ) ( x ( 1 ) ) − f ( 2 ) ( x ( 2 ) ) ∥∥∥2 2 ] ( 4 ) where f ( q ) ( · ) means that the output of the network is normalized . In BYOL , the networks are constructed in a special way ( e.g. , part of f ( 2 ) ’ s weights are moving averages of the correspond part of f ( 1 ) ’ s weights ) . Nonetheless , the cross-view matching perspective is still very similar to that in latent component matching in ( 2 ) . • SimSiam . The loss function of SimSiam ( Chen & He , 2021 ) has a similar structure as that of BYOL , but with a Siamese network , which , essentially , is also latent component matching .

Summary: ### Scope & Problem setting The paper studies the problem of representation learning from multi-view data from an identifiability perspective. In particular, the focus is on latent correlation maximisation approaches as commonly used in (nonlinear) canonical correlation analysis (CCA) and artificial multi-view self-supervised learning (AM-SSL). The paper postulates a generative model by which views $(x^1,x^2)$ are generated by invertible functions $f_i$ applied to a mixture of independent shared $z$ and private $c^i$ components, $x^i=f^i(z,c^i)$, for $i=1,2$.    ### Theory  It is shown that latent correlation maximisation (with a suitable objective and constraints, as well as with invertible encoders) identifies, or separates, both (i) the shared component $z$ and (ii) the private components $c^1,c^2$ up to arbitrary invertible functions. Further, (iii) a finite sample analysis is provided.   ### Algorithm & Experiments The paper proposes a learning objective that combines latent correlation maximisation with a regulariser and a reconstruction objective to encourage independence among $(z,c^1,c^2)$ and invertibility of the encoders. Several experiments on synthetic data and image benchmarks are used to validate the theory and compare the proposed method against DCCA, BarlowTwins, and BYOL.