Neural network architectures for disentangling the multimodal structure of data ensembles

1 INTRODUCTION . Neural networks are being employed increasingly in high-stakes application areas , such as face recognition [ Taigman et al . ( 2014 ) ; Huang ( 2012 ) ; Sun et al . ( 2013 ) ; Chen et al . ( 2015 ) ; Xiong et al . ( 2016 ) ] , and medical technologies [ Kermany et al . ( 2018 ) ; Madani et al . ( 2018 ) ; Topol ( 2019 ) ] . Developing a set of neural network architectures that are causally explainable is important in developing a trustworthy machine learning , where `` A causes B '' means `` the effect of A is B '' , a measurable and experimentally repeatable quantity [ Holland ( 1986 ) ] . Forward causal inference models the mechanism of data formation , and estimates the effects of interventions [ Pearl ( 2000 ) ; Imbens & Rubin ( 2015 ) ; Spirtes et al . ( 2000 ) ; Vasilescu et al . ( 2021 ) ; Vasilescu & Terzopoulos ( 2002a ; 2005 ; 2004 ) ] . Unlike , conventional statistics and conventional machine learning that model the observed data distribution , and make predictions about a variable that has been co-observed with another . Inverse causal inference estimates the causes of effects given an estimated forward causal model that is inverted subject to a set of observations that constrain the solution set [ Vasilescu ( 2011 ) ; Vasilescu & Terzopoulos ( 2007 ) ] . There are two conceptual frameworks for causal inference : DAGs or path analysis and potentialoutcome . Donald Rubin and his collaborators have advocated the potential outcome approach which framed causal inference as a missing data problem [ Imbens ( 2020 ) ] . Judea Pearl has been advocating do-calculus – a directed acyclic graph approach as a mathematical language that he has unified with structural equations and counterfactuals [ Bollen & Pearl ( 2013 ) ] . Judea Pearl ’ s causation ladder [ Pearl ( 2000 ) ] provides a way of thinking about causal discovery , causal reasoning , and decision making . Pearl & Bareinboim ( 2014 ) ; Bareinboim & Pearl ( 2016 ) have parameterized the differences between experimental and observational studies based on possible sources of error . Tensor data analysis is a type of structural equation modeling that has been employed to perform dimensionality reduction , to develop regression models , and to model cause-and-effect , Fig . 1 . Tensor factor analysis has been employed in psychometrics [ Tucker ( 1966 ) ; Harshman ( 1970 ) ; Carroll & Chang ( 1970 ) ; Bentler & Lee ( 1979 ) ; Kroonenberg & de Leeuw ( 1980 ) ] , econometrics [ Kapteyn et al . ( 1986 ) ; Magnus & Neudecker ( 1988 ) ] , chemometrics [ Bro ( 1997 ) ; Acar et al . ( 2014 ) ] , signal processing [ de Lathauwer ( 1997 ; 2008 ) ; Cichocki et al . ( 2009 ) ] , computer vision [ Vasilescu & Terzopoulos ( 2002b ) ; Wang & Ahuja ( 2003 ) ] , computer graphics [ Vasilescu ( 2002 ) ; Davis & Gao ( 2003 ) ; Vasilescu & Terzopoulos ( 2004 ) ; Vlasic et al . ( 2005 ) ; Hsu et al . ( 2005 ) ] , and machine learning [ Vasilescu ( 2009 ) ; Vasilescu & Terzopoulos ( 2005 ) ] . In machine learning , tensor methods have been effectively employed to reparameterize neural networks . Neural network weights have been organized into “ data tensors ” and dimensionally reduced in order to achieve greater computational efficiency [ Lebedev et al . ( 2014 ) ; Novikov et al . ( 2015 ) ; Kim et al . ( 2015 ) ; Khrulkov ( 2020 ) ; Onu et al . ( 2020 ) ; Iwen et al . ( 2021 ) ] . Tensor methods have also been applied in regression analysis [ Kolda et al . ( 2005 ) ; Chu & Ghahramani ( 2009 ) ; Tang et al . ( 2013 ) ; Anandkumar et al . ( 2014 ) ; Kossaifi et al . ( 2017 ) ; Wang et al . ( 2017 ) ; Benesty et al . ( 2021 ) ; Vendrow et al . ( 2021 ) ] . This paper introduces a set of causal capsule architectures for forward and inverse causal inference that implement tensor factor analysis operations . These architectures are consistent with capsule theory proposed by Geoffrey Hinton and his collaborators [ Hinton et al . ( 2011 ) ; Sabour et al . ( 2017 ) ; Hinton ( 2021 ) ] . Forward causal inference , the estimation of effects of causes , is performed with a causal autoencoder architecture that consists of several causal capsules that compute the causal factor representations , and a tensor-autoencoder that governs the causal factor interaction , Fig 2 . A causal capsule is formed from a set of constrained “ cluster ” -based autoencoders1that transform the basis vectors spanning the “ cluster ” subspace , such that a causal factor representation is invariant of the “ cluster ” membership , i.e. , invariant to all the other causal factors [ Vasilescu ( 2009 ) ] . A tensor autoencoder is an autoencoder with a vectorized tensor code formed from the multilinear ( tensor ) product of factor representations . A recurrent non-linear causal capsule chain that employs kernel activations computes the optimal linearized subspace for every causal factor , and implements the kernel multilinear principal component analysis or the kernel multilinear independent component analysis , Fig 3 . For distributed computation , we break the chain links andeach causal representation is computed separately , shuttling causal information between capsules . For a scalable architecture , causal representations for an object whole can be computed efficiently by parts , Fig 4 . As Hinton ( 2021 ) has also indicated , a part-based causal capsule architectures may also be “ glommed ” together to analyze a hierarchy of data columns [ Vasilescu et al . ( 2021 ) ; Vasilescu & Kim ( 2019 ) ; de Lathauwer ( 2008 ) ] . The hierarchical neural network architecture is a compositional 1In the context of multifactor data analysis , a cluster is a set of observations for which all factors are fixed except one . Data belonging to the same cluster may not form a cluster in Euclidean space and not easily identifiable by an EM algorithm [ Dempster et al . ( 1977 ) ] . hierarchical computation of causal factor representation and implements the Incremental M -mode Block SVD algorithm . 2 Algorithm 1 M -mode SVD algorithm . Input the data tensor D ∈ CI0×···×IM . 1 . For m : = 0 , . . . , M , Let Um be the left orthonormal matrix of [ UmSmVTm ] : = svd ( D [ m ] ) a 2 . Set Z : = D ×0 U0T ×1 U1T · · · ×m Um T · · · ×M UM T. Output mode matrices U0 , U1 . . . , UM and the core tensor Z . aThe computation of Um in the SVD D [ m ] = UmΣVm T can be performed efficiently , depending on which dimension of D [ m ] is smaller , by decomposing either D [ m ] D [ m ] T = UmΣ2UmT ( note that VmT = Σ+UmTD [ m ] ) or by decomposing D [ m ] TD [ m ] = VmΣ 2Vm T and then computing Um = D [ m ] VmΣ + . Inverse causal inference is performed with a multilinear projection architecture [ Vasilescu & Terzopoulos ( 2007 ) ; Vasilescu ( 2009 ) ] that is performed by inverting an estimated forward model subject to data constraints . Fig . 5 . The architectures are derived based on two mathematical principles : ( i ) linear autoencodersdecoders weights are the principal component analysis basis vectors , sec . 2 , ( ii ) the object-whole representation can be derived bottom-up in closed form from a part-based hierarchical causal factor representation , sec 3.2 . After reviewing the mathematical foundations of our work in the next section , we discuss forward causal models and depict their neural network arcitectures in Section 3 and discuss inverse causal inference and depict the multilinear projection neural network architectures in Section 4 . Section 5 concludes the paper . 2 LINEAR AUTOENCODER AND LINEAR PCA . An autoencoder-decoder that minimizes the reconstruction loss function , l = I∑ i=1 ‖di −Bci‖+ λ‖BTB− I‖ ( 1 ) and has a linear decoder learns a set of weights , br that are identical to the PCA basis vectors when the weights of each neuron , cr are computed sequentially . An autoencoder is implemented with a cascade of Hebb neurons [ Hebb ( 1949 ) ] . The contribution of each neuron , c1 , . . . , cr , are the PCA sequentially computed and subtracted from a centered training data set , and the difference is driven through the next Hebb neuron , cr+1 [ Sejnowski et al . ( 1989 ) ; Sanger ( 1989 ) ; Rumelhart et al . ( 1986 ) ; Ackley et al . ( 1985 ) ; Oja ( 1982 ) ] . The weights of a Hebb neuron , cr , are updated by ∆br ( t+ 1 ) = η ( d− r∑ ir=1 bir ( t ) cir ( t ) ) cr ( t ) = η ( d− r∑ ir=1 bir ( t ) b T ir ( t ) d ) dTbr ( t ) , br ( t+ 1 ) = ( br ( t ) + ∆br ( t+ 1 ) ) ‖br ( t ) + ∆br ( t+ 1 ) ‖ where d ∈ CI0 is a vectorized centered observation with I0 measurements , η is the learning rate , br are the autoencoder weights of the r neuron , cr is the activation , and t is the time iteration . Backpropagation [ LeCun et al . ( 1988 ; 2012 ) ] is equivalent to performing PCA gradient descent [ Jolliffe ( 1986 ) ] . 3 CAUSAL INFERENCE . Throughout this article , we will denote scalars by lower case italic letters ( a , b , ... ) , vectors by bold lower case letters ( a , b , ... ) , matrices by bold uppercase letters ( A , B , ... ) , and higher-order tensors by bold uppercase calligraphic letters ( A , B , ... ) . Index upper bounds are denoted by italic uppercase letters ( i.e. , 1 ≤ a ≤ A or 1 ≤ i ≤ I ) . The zero matrix is denoted by 0 , and the identity matrix is denoted by I . The TensorFaces paper [ Vasilescu & Terzopoulos ( 2002a ) is a gentle introduction to tensor factor analysis , Kolda and Bader [ Kolda & Bader ( 2009 ) is a nice survey of tensor methods and references [ Vasilescu ( 2009 ) ; de Lathauwer ( 1997 ) ; Bro ( 1997 ) provide an in depth treatment of tensor factor analysis . 2By comparison , a hierarchical Tucker is a resource efficient hierarchical computational scheme that employs a hierarchical re-balancing of the modes trick in which one flattens a data tensor in multiple modes at the same time to avoid computing SVDs of skinny matrices [ Hackbusch & Kühn ( 2009 ) ; Grasedyck ( 2010 ) ; Perros et al . ( 2015 ) ] . Algorithm 2 Kernel Multilinear PCA/ICA ( K-MPCA/MICA ) algorithm . Input the data tensor D ∈ CI0×···×IM , where mode m = 0 is the measurement mode , and the desired ranks R̃1 , . . . , R̃M . 1 . For m : = 1 , . . . , M , Compute the elements of the mode-m covariance matrix , for j , k : = 1 , . . . , Im , as follows : [ D [ m ] D [ m ] T ] jk : = I1∑ i1=1 ... Im−1∑ im-1=1 Im+1∑ im+1=1 ... IM∑ iM=1 K ( di1 ... im-1 j im+1 ... iM , di1 ... im-1 k im+1 ... iM ) . For K-MPCA : Set Um to the left matrix of the SVD of D [ m ] D [ m ] T = UmΣ2UmT Truncate to R̃m columns Um ∈ CIm×R̃m . For K-MICA : Compute Um : = Cm ∈ CIm×R̃m based on [ Vasilescu & Terzopoulos ( 2005 ) ] . The initial SVD truncates to R̃m . 2 . Set T : = D ×1 U+1 · · · ×m U+m · · · ×M U + M . 3 . Local optimization via alternating least squares : Iterate for n : = 1 , . . . , N For m : = 1 , . . . , M , Set Xm : = D ×1 U+1 · · · ×m−1 U + m−1 ×m+1 U + m+1 · · · ×M U + M . Set Um to the R̃m leading left-singular vectors of the SVD of Xm , [ m ] a . Set T : = XM ×M U+M . until convergence . Output the converged extended core tensor T ∈ CI0×R̃1×···×R̃M and causal factor mode matrices U1 , . . . , UM .

Summary: This paper is an attempt to connect tensor factor analysis with DNN learning. The paper explores the forward causal inference and inverse causal inference problems. The forward causal inference is performed using causal capsules architecture that uses M-mode SVD for subspace learning (autoencoder). The authors show that the M-mode SVD rotates the basis vectors of various clusters such that the causal representation is the same regardless of the data cluster (Fig.1). They introduce causal capsule hierarchy architecture that uses tensor decompositions for forward and inverse causal inference.