Neural network architectures for disentangling the multimodal structure of data ensembles
Authors: M. Alex O. Vasilescu
Venue: ICLR 2022 Submitted (2022)
Keywords: autoencoders
SEMs

Abstract: We introduce neural network architectures that model the mechanism that generates data and address the difficult problem of disentangling the multimodal structure of data ensembles. We provide (i) an autoencoder-decoder architecture that implements the $M$-mode SVD and (ii) a generalized autoencoder that employs a kernel activation and implements the doubly nonlinear Kernel-MPCA.  The neural network projection architecture decomposes an unlabeled data given an estimated forward model and a set of observations that constrain the solution set.

Under review as a conference paper at ICLR 2022
Causal Capsules and Tensor Autoencoders
Anonymous authors Paper under double-bline review
ABSTRACT
This paper introduces a set of neural network architectures for forward and inverse causal inference that are consistent with capsule theory and implement multilinear (tensor) factor analysis methods. Forward causal inference addressed with a causal autoencoder- decoder architecture composed of a set of causal capsules that estimate the latent variables representing the constituent factor of data formation, and jensor-autoencoder that governs the latent variable interaction. A recurrent non non-linear causal capsule chain that employs kernel activations computes the optimal linearized subspace for every causal factor, and implements the kernel multilinear principal component analysis or the kemnel multilinear independent component analysis. For distributed computation, we break the chain links and each causal representation is computed separately, shuttling causal information between capsules. The causal factor representations may be computed efficiently by restructuring the input into hierarchy of parts with a set of part-based causal capsules that are "glommed" together to create part-based hierarchy of causal capsules. Inverse causal inference, the estimation of causes of effects, is addressed with a multilinear projection architecture that inverts the estimated forward causal model and employs set of observations constrain the solution set rendering the problem well-posed.
INTRODUCTION
Neural networks are being employed increasingly in high-stakes application areas, such face recognition [Taigman et al. (2014); Huang (2012); Sun et al. (2013); Chen et al. (2015); Xiong et al. (2016)], and medical technologies [Kermany et al. (2018); Madani et al. (2018); Topol (2019)). De- veloping a set of neural network architectures that are causally explainable is important in developing a trustworthy machine learning, where "A causes B" means "the effect of A is B", a measurable and experimentally repeatable quantity [Holland (1986)].
Forward causal inference models the mechanism of data formation, and estimates the effects of interventions [Pearl (2000); Imbens & Rubin (2015); Spirtes et al. (2000); Vasilescu et al. (2021); Vasilescu & Terzopoulos (2002a; 2005; 2004)]. Unlike, conventional statistics and conventional machine learning that model the observed data distribution, and make predictions about variable that has been co-observed with another. Inverse causal inference estimates the causes of effects given an estimated forward causal model that is inverted subject to a set of observations that constrain the solution set [Vasilescu (2011); Vasilescu & Terzopoulos (2007)].
There are two conceptual frameworks for causal inference: DAGS or path analysis and potential- outcome. Donald Rubin and his collaborators have advocated the potential outcome approach which framed causal inference as a missing data problem [Imbens (2020)]. Judea Pearl has been advocating do-calculus directed acyclic graph approach structural equations and counterfactuals [Bollen & Pearl (2013)). Judea Pearl's causation ladder [Pearl (2000)) provides a way of thinking about causal discovery, causal reasoning, and decision making. Pearl & Bareinboim (2014); Bareinboim & Pearl (2016) have parameterized the differences between experimental and observational studies based on possible sources of error.
mathematical language that he has unified with
Tensor data analysis is a type of structural equation modeling that has been employed to perform dimensionality reduction, develop regression models, and to model -auue-and-effect, Fig. 1. Tensor factor analysis has been employed in psychometrics [Tucker (1966); Harshman (1970); Carroll & Chang (1970); Bentler & Lee (1979); Kroonenberg & de Leeuw (1980)), econometrics [Kapteyn al. (1986); Magnus Neudecker (1988)). chemometrics [Bro (1997); Acar et al. (2014)), signal processing [de Lathauwer (1997; 2008); Cichocki et al. (2009)), computer vision [Vasilescu & Terzopoulos (2002b); Wang & Ahuja (2003)], computer graphics [Vasilescu (2002); Davis & Gao (2003); Vasilescu & Terzopoulos (2004); Vlasic al. (2005); Hsu al. (2005)), and machine learning [Vasilescu (2009); Vasilescu & Terzopoulos (2005)]. In machine learning, tensor methods have been effectively employed to reparameterize neural networks. Neural network weights have been organized

---

Under review as a conference paper at ICLR 2022
Caral Countex
Madca ManncCorapateinnn
Pock
DOOW 4o
Ag- Lanourmcet
Al- CreoArce Docoders (N
Astoessoeers
a
Sormalbed PCA Ceetfiiint Marix
U,-
M
bnitatration
(Cor: Tamer Corpuration T
(Ress
Z
Jow
Spuce
(b)
Figure 1: (a) M-mode SVD estimates the param- eters of tensor factor model from a collection of vectorized images that have been acquired com- binatorially. (b) M- -ode SVD computes re- gression model and computes the column and row space from a collection of images where each image is grid of numbers, "data matrix" 2-way array. (All images in this paper have been vectorized, except in this sub-figure.)
Figure 2: Naive neural network implementation of the M-mode SVD. alg. Depiction of the TensorFaces model estimation, Fig.1(a) Comput- ing each mode matrix, Um. naively with a single autoencoder-decooee
The core tensor, T. com- puted by an autoencoder that is initialized with the vectorized multilinear (tensor) codes formed from the product of factor representations.
into "data tensors" and dimensionally reduced in order to achieve greater computational efficiency [Lebedev et al. (2014); Novikov et al. (2015); Kim et al. (2015); Khrulkov (2020); Onu al. (2020); Iwen et al. ().21)). Tensor methods have also been applied in regression analysis [Kolda et al. (2005); Chu & Ghahramani (2009); Tang et al. (2013); Anandkumar et al. (2014); Kossaifi et al. (2017); Wang et al. (2017); Benesty et al. (2021); Vendrow et al. (2021)).
This paper introduces a set of causal capsule architectures for forward and inverse causal inference that implement tensor factor analysis operations. These architectures are consistent with capsule theory proposed by Geoffrey Hinton and his collaborators [Hinton et al. (2011) 1); Sabour et al. (2017); Hinton (2021))
Forward causal inference, the estimation of effects of causes, is performed with a causal autoencoder architecture that consists of several causal capsules that compute the causal factor representations, and fensor-autoencoder that governs the causal factor interaction, Fig 2. A causal capsule formed from a set of constrained "cluster" -based autoencoders 'tanderrrshhhhtee that transform the basis vectors spanning the "cluster" subspace, such that a causal factor representation invariant of the "cluster" membership. ie., invariant to all the other causal factors [Vasilescu (2009)]. A tensor autoencoder is an autoencoder with vectorized tensor code formed from the multilinear (tensor) product of factor representations.
recurrent non-linear causal capsule chain that employs kernel activations computes the optimal linearized subspace for every causal factor, and implements the kernel multilinear principal component analysis the kernel multilinear independent component analysis, Fig 3. For distributed computation, we break the chain links andeach causal representation is computed separately, shuttling causal information between capsules.
For a scalable architecture, causal representations for an object whole can be computed efficiently by parts, Fig 4. As Hinton (2021) has also indicated, a bart-based causal capsule architectures may also be "glommed" together analyze a hierarchy of data columns [Vasilescu et al. (2021); Vasilescu & Kim (2019); de Lathauwer (2008)]. The hierarchical neural network architecture is a compositional
II the context of multifactor data analysis, cluster is set of observations for which all factors are fixed except one. Data belonging the same cluster may not form cluster in Euclidean space and not easily identifiable by an EM algorithm Dempster et al. (1977)).
2

---

Under review as a conference paper at ICLR 2022
hierarchical computation of causal factor representation and implements the Incremental M-mode Block SVD algorithm.
Inverse causal inference is performed with mul- tilinear projection architecture [Vasilescu Ter-
Algorithm M-mode SVD algorithm.
zopoulos (2007); Vasilescu (2009)] that is per- formed by inverting estimated forward model subject to data constraints. Fig.
Input the data tensor DD Clax
For m := ...... .M,
Let U be the left orthonormal matrix
The architectures are derived based on two mathematical principles: (i)linear autoencoders- decoders weights are the principal component analysis basis vectors, sec. 2, (ii) the object- object-whole representation can be derived bottom-up in closed form from a part-baue bart-based hierarchical causal factor representation, sec 3.2.
of [ m V == (Dd mm))
2. Set Z := D Xo Uo" U Xm
Um T M UM
Output mode matrices Uo, U...., Um and the tensor Z.
"The computation of Um in the SVD [m] EV.. m can be performed efficiently. depend- ing on which dimension of [(m) is smaller, by de- composing either Dim]] [mn]" = Um Un, (note that V = E+U (mm) or by decomposing D (mm "D [m] V V and then computing U D [m] Et
After reviewing the mathematical foundations of our work in the next section, we discuss forward causal models and depict their neural network arci- tectures in Section 3 and discuss inverse causal inference and depict the multilinear projection neural network architectures in Section 4. Section 5 concludes the paper.
LINEAR AUTOENCODER AND LINEAR PCA
An autoencoder-decoderr that minimizes the reconstruction loss function,
Σ ld. -B,,l + AJJB'B - I]]
(I)
and has linear decoder learns set of weights, b, that identical to the PCA basis vectors when the weights of each neuron, C- are computed sequentially. An autoencoder is implemented with a cascade of Hebb neurons [Hebb (1949)]. The contribution of each neuron, .I...., G. are the PCA sequentially computed and subtracted from a centered training data set, and the difference is driven through the next Hebb neuron, Ga1 [Sejnowski et al. (1989); Sanger (1989); Rumelhart et al. (1986); Ackley al. (1985); Oja (1982)]. The weights of Hebb neuron, G. are updated by
Ab,(t 1) = d d- 1.-Eb,,t (b,(()c(tt) c(t) =n (n b. (ebb ( (t),
4=1 b(((1) (b,(1) (t, [b, + (b,(
where d E Coo is a vectorized centered observation with 4o measurements, is the learning rate, b, the autoencoder weights of the neuron, c, is the activation, and t is the time iteration. Back- propagation [LeCun et al. (1988; 2012)] is equivalent to performing PCA gradient descent [Jolliffe (1986)].
3 CAUSAL INFERENCE
Throughout this article, we will denote scalars by lower case italic letters (a, b, b..), vectors by bold lower case letters (a, b...... matrices by bold uppercase letters (A, B.... and higher-order tensors by bold uppercase calligraphic letters (AA, B, ...). Index upper bounds are denoted by italic uppercase letters (i.e., 1 < a < A or 1 I i & I). The zero matrix denoted by O. and the identity matrix is denoted by I. The TensorFaces paper [Vasilescu & Terzopoulos (2002a) is gentle introduction to tensor factor analysis, Kolda and Bader [Kolda & Bader (2009) is nice survey of tensor methods and references [Vasilescu (2009); de Lathauwer (1997); Bro (1997) provide an in depth treatment of tensor factor analysis.
By comparison, hicrarchical Tucker is resource efficient hicrarchical computational scheme that employs hierarchical re- balancing of the modes trick in which flattens a data tensor in multiple modes the same time avoid computing SVDs of skinny matrices [Haacbuschh & Kühn (2009); Grasedyck (2010); Perros al. (2015)).
3

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Kernel Multilinear PCA/ICA (K-MPCA/MICA) algorithm.
Input the data tensor D E Cloxmxlarr where mode m 0 is the measurement mode, and the desired ranks Rí Ri...., RM-
1. For == l...., M,
Compute the elements of the mode-m covariance matrix, for j, k: == ,.... Im as
follows:
h
Im
1m+1
JM
D [m
((d fm-1...Mm din...mm fmmt
[m]
fm-1=1
$ms1=1
FM-1
For K-MPCA:
Set Um to the left matrix of the SVD of of D(m Truncate Rm columns Um C. Rim.
D [m]
U U
For K-MICA: Compute U, = C. E C A. based on
[Vasilescu & Terzopoulos (2005)]. The initial SVD truncates to Rm
2. Set T = 1 U
Xm U XM U.-
3. Local optimization via alternating least squares:
Iterate for = N
For m == 1,..., M,
Set X == D U m U - x U
m++1 XM U
Set Um to the Rm leading left-singular vectors of the SVD of Xm,mm Set T := As X U
until convergence.
Output the converged extended core tensor T Ro x AM and causal factor mode matrices Ul...., UM.
"See Alg. footnote
Linear kernel:
[(u, v) = u'y = u
Polynomial kemel of degree d:
K(n, v) = (u'v)d
Polynomial kernel up to degree
[(u, v) = =(uvv 1)d
Sigmoidal kernel:
K(u, v) ttanh(aa ( V 3)
Gaussian (radial basis function (RBF)) kernl::
((u, v) = exp
uu-v
2
Table 1: Common kernel functions. Kernel functions are symmetric, positive semi-definite func- tions(coreeponding to symmetric, positive semi-defiiite Gram matrices). The linear kernel does not modify or warp the feature space.
3.1 FORWARD CAUSAL INFERENCE
Forward causal inference frames questions terms of interventions. What if causal factor e were changed by one unit, how much would the observed measurements, d, be expected to change?
For modeling individual level-effects rather than the average effects of causes, observations are acquired by systematically varying each causal factor while holding the rest of the causal factors fixed.
Within the tensor mathematical framework, a M-way array or "data-tensor", D E C1 Xmm IM contains a collection of vectorized and centered observations," dip...mm..in E R-o that are the result of M causal factors. The m causal factor (1 < < M) takes one of Im values that indexed by im. & im S Jm. An observation that result the confluence M causal factors is modeled by multilinear tensor equation with multimode latent variables.
dip...iim
T x; (r.,, + + XM (rT + + + Eig...,
where T Z x Up is the extended core which modulates the interaction between the latent variables, Fi Fi Fi . that represent the causal factors, and c,., E (O,, Σ,,) ). is an IID Gaussian noise.
'Reference [(Vasilescu, 2009, Appendix A)] evaluates some of the arguments found in highly eited publica- tions in favor of treating image matrix (tensor) rather than vector. While technically speaking, is not incorreet to treat image matrix in linear/tensor algebra, most arguments do not stand up to analytical scrutiny, and it is preferable vectorize image and treat it single observation rather than collection of independent column/row observations.
4

---

Under review as a conference paper at ICLR 2022
The M-mode SVD, Alg. 1 and Alg. 2, and their neural network architecture counterparts. Fig. 2, Fig. may be employed to represent data in terms of their causal factors by minimizing the reconstruction loss function,
M Σ m=
1 - T X1 U, x U XM Uni
UU
I|I,
where Ti' is the extended core and the mode matrices, U.. spans the causal factor representation. Each mode matrix, Um computed using alternating least squares where a set of M least squares
are computed by moving the mode matrices U, U,,..., m Umit.... Uv to the knowns side of the
equation, setting Xm D
Xm U X U m- XM Uh for distributed computation or setting Xm, = (Kmm11 X 1 Xm Un) for sequential computation, and optimizing the loss function
1 T Xm Um! | U I||
| U [mq
\| U II||=
(3)
where Xm D X1 X mm
U
m-]
X
U
m-
XM U
distributed computation (4)
= (nm X U X Um = T Xm U
sequential computation (5)
The mode matrix Um is set to the subspace of the matrixized Xm, S V == svd(Xa (mm)
Figure 3()) displays recurrent causal chain that unrolls the for-loop from step 3, Alg 2 and sequentially computes the mode matrices by performing an SVD on the matrixized X,, computed from eq.( 5). For a distributed computation, M different Xm are computed according to eq. 4), where mode matrices are shuttled between the different threads.
3.2 DERIVATION: INVARIANCE AND HIERARCHY OF CAUSAL CAPSULES
In this section, takes advantages of the principle that an SVD can be computed from its parts. On that basis, we derive a causal factor representation that is statistical invariant to "cluster" membership (i.e., all other causal factors). On that basis, we provide a scalable architecture by deriving a compositional bottom-up computation of an object whole representation. Thus, the naive and direct implementation of the M-mode SVD is replaced with compositional hierachical part-based distributed architecture.
Computing the mode matrices. U may be viewed as equivalent to computing a set of mutually constrained, cluster-based PCAs. When dealing with data that can be separated into clusters, the standard machine learning approach is to compute separate PCA. When data from different clusters are generated by the same underlying process (e.g., facial images of the same people under different viewing conditions), the underlying data can be concatenated in the measurement mode and the common causal factor can be modeled by one PCA.5
Thus, we define a constrained, cluster-based PCA as the computation of a set of PCA basis vectors that are rotated such that the latent representation is constrained be the invariant of the cluster.
MPCA performs M constrained, bassed PCAs, since the computation of the mode-m matrix Um. which involves a mode-m data tensor flattening and subsequent SVD, is equivalent to performing constrained, cluster-based PCA; i.e., data cluster concatenation followed by SVD. This is self evident when employing our modified datum-centris flattening operator, Fig. 7.
In the context of our multifactor data analysis, we define cluster as set of observations for which all factors fixed except one, the factor. Note that there are N,, 11/2... Im- Im+1 ... Ins possible clusters and the data in each cluster varies with the same causal mode. Thus, the data across different clusters share one of the underlying causal factors. The constrained, auster-based PCA concatenates the clusters in the measurement mode and analyzes the data with a linear model, such as PCA or ICA [Bartlett et al. (2002); Common (1994); Lathauwer et al. (1995); De Lathauwer et al. (1996); Anandkumar al. (2014).
To see this, let Di.....- 1im.fm--mm--i-. E xlxxxx=-xxxxxx x x1--x1 denote a subtensor of D that is obtained by fixing all modes except causal factor mode m and mode (the measurement mode). Matrixizing this subtensor in the measurement mode O, we obtain D Sa..fm--mmmm-. to E CloXIm, This data matrix comprises cluster of data obtained by varying the m causal factor, to which one can traditionally
*Sec Alg. footnotc
'The active appearance model concatenated two different measurements, facial feature locations and texture, to compute person representation for particular viewpoint invariant of measurement. More generally, one can concatenate the measurements of a person from different viewpoint clusters to compute person representation that is invariant of the measurement and the viewpoint [Cootes et al. (2001); Si ct al. (2013)).
'Ohservations in the cluster may not be in Euclidean proximity in the measurement space. Consequently, cluster may he easily identified through a standard EM algorithm.
5

---

Under review as a conference paper at ICLR 2022
apply PCA. Since there are N,, -m-1mm+1mmmm----- Ia possible clusters that share the same underlying space associated with the emth factor, the data can be concatenated and PCA performed in order to extract the same representation for the m factor regardless of the cluster. Now, consider the MPCA computation of mode matrix U Fig. 3(a), which can be written in terms of matrixized subtensors as
D 1...
D=
D.....
Vm Em Um
(6)
D D Im- Im41... [m]]
Clearly, this is equivalent to computing a set of Nm = 12 7m-1mm+1..- Ta cluster-based PCAs concurrently by combining them into single statistical model and representing the underlying causal factor m common to the clusters. Thus, rather than computing a separate linear PCA model for each cluster, MPCA concatenates the clusters into a single statistical model and computes representation (coefficient vector) for mode that is invariant relative the other causal factor modes 1,... (m - 1), (m + .), M. Thus, MPCA is a multilinear, constrained, cluster-based PCA.
To clarify the relationship, let us number each of the matrices Di Pmp-:-5-1immtt 18m 1im+1 [ms] parenthetical superscript
D (x) m with
k-1 = 1 Ee Ek=Lk (fn 1 Ti $ Nn-
Let each of the linear SVDs be
D (n)
U(n) (n) U((rr m
(7)
(8)
D [m]
U (1) (1)
U
(Nw) (()) 2m
diag([ U0
(N D),
(9)
SVD
U Em W diag(( U
u(Nm) D)
(10)
Un Emi
U ((!)
Wo l
(11)
where diag(-) denotes a diagonal matrix whose elements are each of the elements of its vector argument. The mode matrix U("m) is the measurement matrix that contains the eigenvectors that span the observed data in cluster Am. 1 S nh S N... MPCA can be thought as computing a rotation matrix, W.. that contains a set of blocks W(") along the diagonal that transform the PCA cluster eigenvectors, (mm)) such that the mode matrix Um is the same regardless of cluster membership. (--1( The constrained "cluster"-based PCAs may also be implemented with set of concurrent "cluster"-baas" PCAs.
Object wholes appear to have have hierarchy of perceptual parts. Part-based causal capsule architectures may be "glommed" together to create hierarchy of part-based causal capsules that analyze a hierarchy of data columns [Vasilescu et al. (2021); Vasilescu & Kim (2019); de Lathauwer (2008)). This hierarchical architecture implements the incremental hierarchical multilinear (tensor) block decomposition algorithm.
Causal factors of object wholes may be computed efficiently from their parts, Fig 4. The matrixized data tensor may be organized into part "clusters" by applying permutation
D Tx P P PD (m.)
(12)
where P is permutation matrix. The resulting hierarchical architecture implements the Incremental M-mode Block SVD algorithm. The Incremental M-mode Block SVD is a generalized hierarchical part-based decomposition that computes an exact global decomposition and is suitable for streaming data [Vasilescu et al. (2021).
3.3 NONLINEAR CAUSAL CAPSULES AND KERNEL MPCA/KERNEL MICA
An autoencoder with a non-linear activation function represents an observation with
d = fa f. [B.d. d + a +a))
(13)

---

Under review as a conference paper at ICLR 2022
Seynesti SVIN
Di
STD
500
Rebakcwes
Fi
Si 0%
My EpPFi
US Welrr
Eigo Mpp
UN
St 1o
US WHTT
Upi Sm Um
U
W
PM
PS
09
W. 1
W:
W.
W
W.
(c)
(d)
g.
distributed compatation
Inzttalization
(I..
U. T x U. saparial computation
stenialoom
TKUU-
15
diard
W'kd"
WE'Kd
WC
U. 1
U,
0.
UM
(f)
Figure 3: Face recognition example. (a) An ensemble of vectorized images is organized into is matrixized into a data matrix, Dp from which one can compute the mode
D E Iy x matrix, Up. that spans the person representation. This depicts how a single SVD(D can be written in terms of (i) constrained cluster-based autoencoder (PCA) and (ii) concurrent autoencoder. This is depicted as a neural network architecture (b), (c) and (d), respectively. (b) Mode matrix computation using a single autoencoder-decoodr: (c) Mode matrix computation using constrained cluster-
woweennodde-decodde based on the derivation in part (a). (d) Concurrent- inncurrent-autoencoddee (e) The neural network architecture consists of a chain constrained autoencoders-decoolee
where the weights of one constrained autoencoder-deee -ecoodr are the inputs of the next one. This constrained recurrent causal chain is the unrolled for-loop that computes the mode matrices by employing alternating least squares. When the autoencoders employ kernels then the architecture implements K-MPCA K-MICA, Alg. 2.
where fe, fa are the encoder, decoder activation functions, and B, B, are the encoder, decoder weights respectively. Kernel PCA (KPCA) is often given an example of a "true" nonlinear model. KPCA first applics a nonlinear transformation to the data and then performs a linear decomposition. Thus, KPCA derives its nonlinearity from its preprocessing step. Other nonlinear methods include nonlinear PCA (NLPCA) [Kramer (1991), well kernel PCA (KPCA) [Schölkoph et al. (1998) and kernel LDA (KLDA) [Yang (2002) methods in which kernel functions that satisfy Mercer's theorem correspond to inner products in dnfinite-dimensionn
space. An alternative approach is to apply linear models nonlinear problems through the "kernel trick", specifically the kernel PCA [Schölkoph et al. (1998) and kernel ICA [Yang et al. (2005) techniques. Kernel PCA/ICA are
"The so-called "kernel trick" maps the original non-linear measurements into higher- digees-aimeniional space, where a lincar classifier is subsequently used; this makes a lincar classification in the space equivalent non-linear classification in the original space. This is done using Mercer's theorem, which states that any continuous, symmetric, positive semi-definite kemel K ((u, v) can be expressed an inner product in high- dimensional space. Wherever inner product hetween vectors used, it is replaced with kemel of the
7

---

Under review as a conference paper at ICLR 2022
Ui Wr
W. Us
Wa
People
W
U
6-1
a
Figure 4: (a) Learning levels of abstractions bottom-up with a hierarchy causal part-based capsules. Each causal capsule in Fig 3 (f) can be replaced with a hierarchy of parts and wholes. The constrained cuuser-bbased PCA (b), may be computed with hierarchy of autoencoders, as derived in eq. (6)-(10), and depicted in Fig 3.
nonlinear versions of their conventional linear counterparts, but of course they multimodal factor models.
The kernel trick can also be applied our multilinear, multifactor PCA/ICA models to further nonlincarize them, thus enabling them to deal with arbitrarily nonlinear data, Alg. 2.
To accomplish this, recall that the computation of the mode-m covariance matrix D [mm] (m))imm involves inner products d! f... fou diz...mm. m- & im between pairs of images in the image data tensor D associated with causal factor mode m, for = 1...., M. We replace the inner products with generalized distance measure between images, ((d;,...mm11 1 fm a1...ar>> dig...m-11 km 1 ) where K(.,)) ) is suitable kernel function (Table 1), which corresponds to an inner product in some expanded feature space. This generalization naturally leads to Kernel Multilinear PCA (K-MPCA) Algorithm, where the covariance step computation in in Algorithm is replaced by
I
Im+11
[ m
Σ
Σ Y===
K(d foms di (mmtandas
m
jk
Ém =1
im+1==1
Similarly, a Kernel Multilinear ICA (K-MICA) Algorithm results from making the same generalization in the MICA algorithm [Vasilescu & Terzopoulos (2005). Algorithm specifies both K-MPCA and K-MICA. Figure 3(d) unrolls the for-loop in step 3 of Alg. 2.
INVERSE CAUSAL INFERENCEMULTILINN PROJECTIONS
AND MULTIPLE LINEAR
Inverse causal inference estimates the causes of effects, and addresses the why question. Inverse problems often violate one of the conditions of a well- problem, and Donald Rubin has referred the "why" question "cocktail party chatter" [Gelman & Imbens (2013). For problem to be well-posed posed solution must exit, it must be unique and the solution's 's behaviour ought to change continuously with the initial conditions [Hadamard (1952). Often, there are multiple combinations of same causal factors that have the same potential outcome. In imaging, these types of outcomes are known as visual illusions.
Therefore, inverse causal inference is the estimation of causes of effects given an estimated forward causal model that inverted subject to a set of observations that constrain the solution set and render the problem well-posed [Vasilescu et al. Simiilr to reverse causal inference [Gelman & Imbens (2013)), inverse causal inference may be employed as a model checking mechanism and motivation for forward inference question.
vectors. Thus, a linear algorithm easily transformed into nonlinear algorithm. This trick has been applied numerous algorithms in machine learning and statistics.
8

---

Under review as a conference paper at ICLR 2022
Multilinear projection simultaneously projects more unlabeled test images that are not part of the train- ing data set into multiple constituent causal factor spaces associated with data formation, in order to infer the mode labels:
Compule the representation, (ev
plul
x;
For
Faz
Xx
Tiy
xx1
CP or M-mode SVD(T ts x d ry... or...... OT OTF.
Nie
Nic
Topologically the multilinear projection architecture, Fig. 5, is inverted M-mode SVD architecture. When the dimensionality of vec(R) is larger than the number of measurements in d, then the system of equations is under determined. There are three possible solutions dimensionality reduction of the mode matrices, and mod- cling the mechanism of data formation by multiple linear or tensor models. Instead of performing a multilinear projection. [Vasilescu& & Terzopoulos (2002b) perform a set of linear projections.
Factorize ot into Iatent variables:
Tinswrr coda cac (R.)
CPa.Munkk
RIV
R
R
R/y
Figure 5: Neural network architecture of the multilinear projection algorithm [Vasilescu & Terzopoulos (2007) given estimated interaction causal model, T (i.e., Tn)-
5 CONCLUSION
This paper introduces deep causal learning architectures that implement tensor factor analysis operations and model the mechanism of data formation. The tensor factor analysis methods, the M M-mode SVD, the Kernel MPCA/MICA, and the associated causal capsules architectures transform the "cluster" eigenvectors such that the constituent causal representations are invariant of the cluster membership, ie.. invariant of other causal factors of data formation. Causal representation may be computed efficiently by "glomming" together hierarchy of part-based causal capsules. The hierarchical part-based causal architecture implements the compositional hierarchical tensor factorization, the Incremental M-mode Block SVD. Each part-based capsule analyzes a data column from a hierarchy of data columns [Vasilescu al. (2021). Inverse causal inference, the estimation of causes of effects, is accomplished with a multilinear projection algorithm. The neural architecture that implements the multilinear projection is inverted M-mode SVD architecture. Tensor causal factor analysis and their associates neural networks have properties consistent with the capsule theory. Tensor causal factor analysis has been applied on real and synthetic data in many domains, including face recognition where the approach is known as TensorFaces and computer graphics where set of Tensor are synthesized for arbitrary geometries.
REFERENCES
Evrim Acar, Evangelos E Papalexakis, Gözde Gürdeniz, Morten A Rasmussen, Anders Lawaetz, Mathias Nilsson, and Rasmus Bro. Structure-revcaling data fusion. BMC bioinformatics, 15(1):239, 2014. 1
David H. Ackley. Geoffrey A. Hinton, and Terrnce J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147-169 1985. ISSN 0364-0213. doi: http:://doi.orr//00000066
003544002330558800022 URL https://///// http///www../// www.://iiieeeieee..............e htt.://aceeee..ccc/////////////s wwtp///ieee///tttiteeeee//e/e/ee 553640213858001222 3,
Animashrce Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decompositions
for learning latent variable models. 15(1):27732832 2014. 2, 5
Elias Bareinhoim and Judea Pearl. Causal inference and the data-fusion problem. Proc. of the National Academy of Sciences, 1 113(2)):735-52, 2016. ISSN 0027-8424, doi: 10.1073/pna..51100011 URL https: : / www. pnas. org www.entttttttocoooooooooooooct 111/22/73322
M.S. Bartlett, J.R. Movellan, and TJJ. Sejnowski. Face recognition by independent component analysis. IEEE
Transactions on Neural Nenworks, 13(6))1450-64, 2002. 5
Jacob Benesty, Constantin Paleologu, Marura Dogariu, and Silviu Ciochin. Identification of linear and bilin- ear systems: A unified study. Electronics, 10(15), 2021. ISSN 2079-9292. doi: 10.3390Velectronics001.. 10151790. URL https:///// hwwp:wwnn.. mdpi. com/ 20/ -9- /
Peter M. Bentler and Sik-Yum Lee. A statistical development of three-mode factor analysis. British J. of Math. and Stat. Psych., 32(1))87-104, 1979. doi: // o/online aanineelibrre
URL. https:
wiwey
wtt.199........... .x..
V. Blanz and T. A. Vetter. Morphable model for the synthesis of 3D faces. In Proc. ACM SIGGRAPH 99 Conf., pp. 187-194, 1999. 16

---

Under review as a conference paper at ICLR 2022
Kenneth A. Bollen and Judea Pearl. Eight Myths About Causality and Structural Equation Models, pp. 301-328.
Springer Netherlands, Dordrecht, 2013. ISBN 978-94-00076609--3. doi: 00110779978-944077--0994 1_1.. URL https: /tt/ org/ 1007 978
Rasmus Bro. Parafac: Tutorial and applications. In Chemom, Intell. Lab Syst., Special Issue 2nd Internet Cont.
in Chemometrics (INCINC'99) 96), volume 38, pp. 149-171, 1997. 1, 15
J. D. Carroll and J. J. Chang. Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart- Young' decomposition. Psychomerrika, 35:283-319, 1970.
Carroll, S. Pruzansky, and J. B. Kruskal. CANDELINC: A general approach multidimensional analysis of many-way arrays with linear constraints on parameters. Psychometrika, 43:3-24, 1980. 16
Chen, Ranjan, Kumar, C. H. Chen, unconstrained face verification with deep convolutional neural networks. IEEE International Conf. on Computer Vision Workshop (ICCVW), pp. 660-368, Dec 2015. doi: 1091CC/....00.5555
Patel, and Chellappa. An end-to-end system for
Wei Chu and Zoubin Ghahramani. Probabilistic models for incomplete multi-dimensional arryy.. volume of Proceedings of Machine Learning Research, pp. 89-96 Hilton Clearwater Beach Resort, Clearwa- ter Beach, Florida USA, 16-18 Apr 2009. PMLR. URL http:://pooceeeeeeeeee ngs pre..pre///v/ chu09a h
Andrzej Cichocki, Rafal Zdunck, Anh Huy Phan, and Shun-ichi Amari. Nonnegative matrix and tensor factorizations: applications exploratory multi-way data analysis and blind source separation. John Wiley & Sons, 2009.
1
Pierre Common. Independent component analysis, a new concept? Signal Processing, 36:287-314, 1994, URL
https: /tallha.... archives ouvertes.............. fr/hall 0041110000. 5
T.F. Cootes, G.J. Edwards, and C.J. Taylor. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, (566):61-6855 2001.
Davis and H. Gao. Recognizing human action efforts: adaptive three-mode PCA framework. Proc. IEFE Inter. Conf. on Computer Vision, (ICCV), pp. 1463-69, Nice, France, Oct 13-16 2003.
L. de Lathauwer. Decompositions of higher-order tensor block termspart ii: Definitions and unigueness. SIAM J. Matrix Analysis and Applications, 303)):033-1066, 2008. doi: 10.1 137/17770709977229 URL
https: ////s///cc1/ crg/ 131111777667722, 2,
L. De Lathauwer, B. De Moor, and J. Vandewalle. Independent component analysis based higher-order statistics only. In Proceedings of 8th Workshop Statistical Signal and Array Processing, pp. 356 359, 1996. doi: 10. N0.110//SAP..196.55444
Lieven de Lathauwer. Signal Processing Based on Multilinear Algebra. PhD dissertation, Katholicke Univ. Leuven, Belgium, 1997. 1, 15, 16
A. P. Dempster. N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series (Methodologicall) 39(1):1-22, 1977. doi: https://doi.org// 10.111111111111 11115117-6111.197..bb com/doi/ abs / 10. 11111/1.211111 6161 1977..1977..... tb01600...... 1977.111100000 x.
URL https:/// /tts///rs.../ on ieleinnelee wwwrrrrrrrrrr...e. warrrrbbrrleley.y.............ee
Andrew Gelman and Guido Imbens. Why ask why? forward causal inference and reverse causal questions. Tech.report, But.Bureau of Econ Research, 2013.
Lars Grasedyck. Hicrarchical singular value decomposition of tensors. SIAM on Matrix Analysis and Applications, 31(():2019 2010.
Wolfgang Hackbusch and Stefan Kühnn. A new scheme for the tensor representation. Journal of Fourier Analysis and Applications, (315):706-777 2009.
J. Hadamard. Lectures Caucky's Problem in Linear Partial Differential Equations. Dover Publisher, 1952. 8
R. Harshman. Foundations of the PARAFAC procedure: Model and conditions for an explanatory factor analysis. Tech. Report Working Papers in Phonetics 16, UCLA, CA, Dec 1970.
Hatamizadeh, Demetri Terzopoulos, and Audriy Myronenko. End-to-end boundary aware networks for medical image segmentation. Inter. Workshop on Machine Learning Medical Imaging, pp. 187-194. Springer. 2019. 16
D. O. Hebb. The organization behavior; A neuropsychologica theory. John Wiley And Sons, Inc., New York, 1949. 3,
10

---

Under review as a conference paper at ICLR 2022
Geoffrey Hinton. How represent part-whole hierarchies in neural network, 2021. 2
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders, 2011. 2
Paul W. Holland. Statistics and causal inference: Rejoinder. J. of the American Statixticcal Association, 81(396):
968970, 1986. URL https: / /usersses./// nber. wwgg/rdege///o/ooo holland JASA 1986. pdf.
Eugene Hsu, Kari Pulli, and Jovan Popovic. Style translation for human motion, ACM Transactions on Graph- ics, 24(3) 2005, URL http: httpp://ppppp...ooo csail mit edu/ jowan/ assets paperss
2suu2205-sst. pdf.
G. B. Huang. Learning hierarchical representations for face verification with convolutional deep belief networks. In IEEE Conf. Computer Vision and Pottern Recognition (CVPR), pp. 2518 25, Jun 2012. doi: 10.1109/ CVPR.2011.6244966
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Mas- sachusetts, Amherst, Oct 2007. 17, 18
Guido W. Imbens. Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics. Journal of Economic Literature, 129-79, December 2020, doi: 10. 12./ell 20191597. URL https: / /www. aeaweb. org/ hatp/artaaa articles 2eeeeeslldll..... 12w111277/// 2et........11.......e.... 20997597.
Guido W. Imbens and Donald Rubin. Causal Inference for Statistics, Social, and Biomedical Sciencex: An Introduction. Cambridge Univ. Press, 2015. ISBN 052 1885884
Mark A Iwen, Deanna Needell, Elizaveta Rebrova, and Ali Zare. Lower memory oblivious (tensor) subspace embeddings with fewer random bits: modewise methods for least squares. SIAM Journal on Matrix Analysis and Applications, 42(1))377-411 2021.
LTT Jolliffe. Principal Component Analysis. Springer-Verlat New York, 1986. 3, 15
A. Kapteyn, H. Neudecker, and T. Wansbeek, An approach to n-mode component analysis. Psychometrika, 51 (2):269-275, Jun 1986.
Vahid Kazemi and Josephine Sullivan. One millisccond face alignment with ensemble of regression trees. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR '14. pp. 1867-74, Washington, DC, USA, 2014. IEEE Computer Society. ISBN 978- 1778----79--1118- doi: 10.1 (0.1109/CVPR22014.222 URL
http:/ /wt. www.rr//1/... /ttp////////./ 4. 241. 16
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang. Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei. Magdalene Y.L. Ting, Jie Zhu, Christina Li. Sierra Hewett, Jason Dong, Ian Ziyar. Alexander Shi, Runze Zhang. Lianghong Zheng, Rui Hou. William Shi, Fu, Yaou Duan, Viet A.N. Huu. Cindy Wen. Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable discases by image-based deep learning. Cell, 172(5): 1122 1121122111111 1131,c9, 2018, ISSN 0092-8674, doi: 2018. 22.0002.110........1222111000000
https://doi. 1116 c01 article/ /plie//ppiii http://pp/////////// 74183016661115518
URL https: / www. www.eccciieccccccooocooocc rect..rrr....ooooooooo......e..e com/ sc scienc/
Valentin Khrulkov. Geometrical Methods Machine Learning and Tensor Analysis. PhD dissertation, Skolkovo
Institute of Science and Technology, 2020. 2
Yong-Deok Kim, Eunhycok Park, Sungjoo Yoo, Taclim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. CoRR, 00///5511.065300 2015. URL htt.t:ttt///iiiiii org/ abaaabbs5555 1tt//11111110001
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Muchine Learning Research, 10: 10:1755-1758, 2009. 16
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(33)455-500
2009. 3, 15
Tamara G. Kolda, Brett W. Bader, and Joseph Kenny. Higher-order web link analysis using multilinaar algebra. In ICDM 2005: Proceedings of the 5th IEEE International Conference on Data Mining, pp. 202.242-22 249, 2005. doi: 10. 109/CCD..200.
Jean Kossaiti, Aran Khanna, Zachary C. Lipton, Tomasso Furlanello, and Anima Anandkumar. Tensor contraction layers for parsimonious deep nets. Computer Vision and Pattern Recognition (CVPR), Tensor Methods Computer Vision Workshop, pp. 1940-46, Jul 2017. 2
11

---

Under review as a conference paper at ICLR 2022
M. A. Kramer. Nonlinear principal components analysis using autoassociative neural networks. AIChE J., 32(2): 233-243, Feb 1991.
P. M. Kroonenberg and J. de Lecuw. Principal component analysis of threc-mode data by means of alternating least squares algorithms. Psychometrika, 45:69-99 97, 1980.
Lieven Lathauwer. Comon, Bart De Moor, and Joos Vandewalle. Higher- power method application in independent component analysis. Proc. of the International Symposium on Nonlinear Theory and its Applications (NOLTA 95), pp. 91-96, 1995.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding- up convolutional neural networks using fine-tuned cp-decomposition. CoRR, abs/1412.6553, 2014. URL
http:// aariivi org/ abs/ aww/a/a///22/// 111.. 6553.
Yann LeCun, D Touresky, G Hinton, and T Sejnowski. A theoretical framework for back- packkpropagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pp. 21-28, 1988. 3, 15
Yann A. LeCun, Léon Bottou, Genevieve Orr, Klaus-Robert Müller. Efficient BackProp, pp. 9- Springer doi: (10007998--3-42--335 35289-888- URL https: doi wtw.ddii...rrroooooo wtt.cr/g//ooooo 100.097000700000 900000797788100 3- 642 35289-228899-- 8_3. 3, 15
Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 778-3-642-352899
I. Macedo, F. V. Brazil, and I.. Velho. Expression transfer hetween photographs through multilinear aam's, s. pp. 239-246 246, Oct 2006. doi: 10.11099/BBGRRA122006..88 16
Ali Madani, Mehdi Moradi, Alexandros Karargyris, and Tanveer Syeda- Syeda-Mahmood, Semi- Semi-supervised learning with generative adversarial networks for chest x-ray classification with ability of data domain adaptation. In 2018 IEEE 1,5th International Symposium on Biomedical Imaging (ISBI 2018). pp. 1038-1042, 2018. doi: 10. 1090091/..22 211.B1...18.8888 8.8363749.
R. Magnus and Neudecker. Matrix Diffferentall Calculus with Applications Statistics and Econometrics. John Wiley Sons, 1988.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gamett (eds.), Advances in Neural Information Processing Systems 28, pp. 442-450. Curran Associates, Inc., 2015. URL http:// nips cc/ paper/ 5777-777777ttrtt zing neural newworkss pdf.
Erkki Oja. A simplified neuron model as a principal component analyzer. 15:267-2735, 1982. 3, 15
Charles C. Onu, Jacob E. Miller, Doina Precup. fully tensorized recurrent neural network. CoRR, abs/2010.04196, 2020. URL ht pttttt/:://////e org/ a/////////// 2ass/2110 02211....1............
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge Univ. Press, 2000. ISBN 9780521773621.
URL https::tp//. h /tcck//cckkscccooc/c//..../..... google. www/book...eooooooooooooooooooo. d=wnGU d
Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across populations. Statistical Science, 29(4)557995, 2014.
I. Perros, Chen, R. Vuduc, and Sun. Sparse hicrarchical tucker factorization and its application healthcare. In Proc. IEEE Inter. Conf. on Data Mining, pp. 943-948, Nov 2015. doi: 10. (09/ICDM.2015.2.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. 1986. 3, 15
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Guyon, U.
Luxbburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httppoooodddnnggoooo wwbbbbbbeeeeee 2bt.eeeeaabbbbdbbbebbee 08877774844-444 33b94ee-5334494---aaeeee .pdf. 2
Terry Sanger. Optimal unsupervised learnig in single layer linear feedforward neural network. 12:459-473, 1989. 3,
B. Schölkoph, A. Smola, and K.-R. Muller, Nonlinear component analysis as a kerel eigenvalue problem. Neural Computation, 1299)1129911319 1998. 7
Terrence Sejnowski, Sumantra Chattarji, Patric Sfanton. Induction of Synaptic Plasticity by Hebbian Covariance in the Hippocampus, pp. 105-124. Addison- 1989. 3, 15
Weiguang Si, Kota Yamaguchi, and M. Alex O. Vasilescu. ing with multilinear (tensor) active appearance models. Scbbaanorooof//Sffcccaaallee1e10 b2h44 Jun 2013. URL https: /ttpss///pfffttt fs.....
Face track-
http://pdfs..e http://pdfi.semantic
semanti cacha a
3etp5///7ccccaaaeeeee33110 1e33110 67del 8 F2
12

---

Under review as a conference paper at ICLR 2022
Peter Spirtes, Clark N Glymour. Richard Scheines, and David Heckerman, Causation, prediction, and search. MIT press, 2000.
Yi Sun, Xiangang Wang, and Xiaoou Tang. Hybrid deep learning for face verification. In Proc. IEFE International Conf. on Computer Vision (ICCV), pp. 1489-96, Dec 2013. doi: 10.1109/1CC222013..888
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1701-08, 2014. doi: 10. 109///PP.2201..20..
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. Tensor analyzers. volume 28 of Proceedings of Machine Learning Research, pp. 163-171, Atlanta, Georgia, USA, 17-19 Jun 2013. URL http: /ptppeeeiiggss...c//ee.p.....sse mlr. prwss/ v28 / tang13 html. 2
Eric J Topol. pirg-peefoemance medicine: the convergence of human and artificial intelligence. Nature medicine, 25(1):44-56 56, 2019.
L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31:279-3 -1199931111101111 1966.
M. A. O. Vasilescu, Human motion signatures: Analysis, synthesis, recognition. In Proc. Int. Conf. on Pattern Recognition, volume 3, pp. 456-44466-- 460, Quebee City, Aug 2002. 1
M. A. O. Vasilescu. Multilinear projection for face recognition via canonical decomposition. In Proc. IEEE Inter. Conf. on Automatic Face Gesture Recognition (FG 201 pp. 476-483, Mar 2011. doi: 10.1 109/FG. 2011 1771144445 17
M. A. O. Vasilescu and D. Terzopoulos. Multilinear analysis of image ensembles: TensorFaces. In Proc. European Conf. on Computer Vision (ECCV 2002), pp. 447-460, Copenhagen, Denmark, May 2002a. 1, 3, 15, 16
M. A. O. Vasilescu and D. Terzopoulos. Multilinear analysis for facial image recognition. In Proc. Int. Conf. on Pattern Recognition, volume 2. pp. 511-514, Quehec City, Aug 2002b. 1,
M. A. O. Vasilescu and D. Terzopoulos. TensorTextures: Multilinear amage-based rendering. ACM Transactions on Graphics, 23(3):33--34,, Aug 2004. Prac. ACM SIGGRAPH 2004 Conf., Los Angeles, CA.
M. A. O. Vasilescu and D. Terzopoulos. Multilinear independent components analysis. Proc. IEFE Conf. on Computer Vision and Pattern Recognition, volume I, pp. 547-553, San Diego, CA, 2005. 1, 1,4,8 8
M. A. O. Vasilescu D. Terzopoulos. Multilinear projection for appearance-based recognition in the tensor framework. In Proc. 1h IEFE Inter. Conf. on Computer Vision (ICCV'07), pp. 1-8, 2007. 1,3,9,1
M. Alex O. Vasilescu. A Multilinear (Tensor) Algebraic Framework for Computer Graphics, Computer Vision and Machine Learning. PhD dissertation, University of Toronto, 2009. 1, 3, 4, 15
M. Alex O. Vasilescu and Fric Kim. Compositional hierarchical tensor factorization: Representing hierarchical intrinsic and extrinsic causal factors. In The 25th ACM SIGKDD Conf. on Knowledge Discovery and Data Mining (KDDI9): Tensor Methods for Emerging Data Science Challenges Workshop. Aug. 2019. 2,
M. Alex O. Vasilescu, Eric Kim, and Xiao Zeng. Causalx: Causal explanations and block multilinear factor analysis. In 2020 25th International Conference of Pattern Recognition (ICPR 2020), pp. 10736-10743, Jan 2021. 2, 6, 8,
Joshua Vendrow, Jamie Haddock, and Deanna Needell. generalized hierarchical nonnegative tensor decompo- sition, 2021. 2
D. Vlasic, M. Brand, H. Plister, and J. Popovic. Face transfer with multilinear models. ACM Transactions on Graphics (TOG), 24(3) Jul 2005. URL http:/ http://peoplo.. csail mi edu/ Jovan/ assets assets/ pttppppppppsssseeepppppp 2005-- ftm. paf.
Hongcheng Wang and Narendra Ahuja. Facial expression decomposition. In Proc; 9th IEEE Inter. Conf. on Computer Vision (ICCV), pp. 958-6 v5,, 2003. doi: 10.1 10.1109/....2..1223888888 1238452.
M. Wang. Y. Panagakis, P. Snape, and S. Zafeiriou. Learning the multilinear structure of visual data. In 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp- 6053-6061, Jul 2017. doi: 10. 109 2017.641. URL https: : / /oo..
org/ 11t...119911000 /wt/////////// 2017 6.1.2
C. Xiong, L. Liu, X. Zhao, S. Yan, and T. K. Kim. Convolutional fusion network for face verification in the wild. IEFE Transactiony Circuits and Systems for Video Technology, 26(3):517-528, Mar 2016. ISSN 1051-8215. doi: 10. 00.1199/TCSVT22255220611
13

---

Under review as a conference paper at ICLR 2022
J. Yang. X. Gao, D. Zhang, and J. Yang. Kernel ICA: An alternative formulation and its application face recognition. Pattern Recognition, 38 18(10):1844877 2005.
M. Yang. Kernel eigenfaces vs. kernel fisherfaces: Face recognition using kernel methods. In Proc. 5th IEEE Inter. Conf. on Automatic Face Gesture Recognition, pp. 215-220, Washington, DC, May 2002. doi: 2011009/AFFR.20002.5.222222 7
14

---

Under review as a conference paper at ICLR 2022
A MATHEMATICAL BACKGROUND
Throughout this article, we will denote scalars by lower case italic letters (a, b, ...). vectors by bold lower case letters (a, b...,, matrices by bold uppercase letters (A, B...), and higher-order tensors by bold uppercase calligraphic letters (AA, B, ..... Index upper bounds are denoted by italic uppercase letters (i.e., 1 Σ a ك A or 1 I i S I). The zero matrix is denoted by 0, and the identity matrix is denoted by I. The TensorFacee paper [Vasilescu & Terzopoulos (2002a) is gentle introduction to tensor factor analysis, [Kolda Bader (2009) is a great survey of tensor methods and references [Vasilescu (2009); de Lathauwer (1997); Bro (1997) provide an in depth treatment of tensor factor analysis.
A.I PCA COMPUTATION WITH LINEAR AUTOENCODER
An autoencoderr-eccoder that minimizes the reconstruction loss function for a set of observations, d E C
1 = Σ d. -cBll + AB -|I,
(14)
and has linear decoder learns of weights, h that identical the elements of the PCA basis matrix, B Cl when the weights of each neuron are computed sequentially, Fig. 6. An autoencoder is implemented with a cascade of Hebb neurons [Hebb (1949). The contribution each neuron, C..... G. is sequentially computed and subtracted from a centered training data set, and the difference is driven through the next Hebb neuron, G,1 [Sejnowski et (1989); Sanger (1989); Rumelhart et al. (1986); Ackley et al. (1985); Oja (1982).
The weights of Hebb neuron, C., are updated by
(b, + = d
ba (t)
c(t) (15)
=1 d -
bi, (O) d' dtb,(t),
1=1
b,(t + (f) (b 1))
(t) + Ab,(t + +1]l
b
by
where E Coo vectorized centered observation with I, measurements, 1) is the learning rate, b, are the autoencoder weights of the neuron, is the activation, and is the time iteration. pack-propagation [LeCun et al. (1988; 2012) performs PCA gradient descent [Jol- liffe (1986).
Figure 6: Autoencoder-decodes architec- ture and Principal Component Analysis. (All images have been vectorized, but they are displayed as a grid of numbers. The eigenvector b, is the mean and activation ei is set to 1.)
A.2 RELEVANT TENSOR ALGEBRA
Briefly, the natural generalization of matrices (i.e., lin- ear operators defined over a vector space), tensors define multilinear operators over a set of vector spaces. A "data lensor" denotes an M-way data array.
Definition (Tensor) Tensors are multilinear mappings over a set of vector spaces, C.m, 1 S m S M, to a range vector space Co
A: (ch x C22 x ... x C. Cl.
(16)
The order of tensor A E Cloxhx IM is M + 1. An element of A is denoted Am AM or Oig.h.........ppp where S im Im.
The mode-m vectors of an Mordeer tensor A E Cloxix-XXx are the Im dimeensional vectors obtained from A by varying index im while keeping the other indices fixed. In tensor terminology. column vectors are the mode-0 vectors and row vectors as mode-1 vectors. The mode-m vectors of a tensor are also known as fibers. The mode-m vectors are the column vectors of matrix A Amo that results from matrixizing (a.k.a. flattening) the tensor A.
Definition (Mode-m Matrixizing) The mode-m m matrixizing of tensor A E C1ox is de- fined as the matrix A mn) (lo 1x) As the parenthetical ordering indicates, the
15

---

Under review as a conference paper at ICLR 2022
Algorithm 3 M-mode SVD algorithm.
Input the data tensor D E C Clax-
1. For := 0, ..... M,,
Let Um m be the left orthonormal matrix of S V == svd(D(m)) (m])
2. Set 2 D x Un R X Si Xm Um M UM TM
Output mode matrices Uo. UI.... MM and the core tensor 2.
"The computation of Um in the SVD [im] Um EVm T can be performed efficiently, depending on
which dimension D [m]] is smaller, by decomposing either D [mn [mniimm U, U'' (note that Vm E Um ' (m)) by decomposing D [l D [m] V V²V,,. m
T and then computing U.. Dim) V -
mode-m column vectors are arranged by sweeping all the other mode indices through their ranges, with smaller mode indexes varying more rapidly than larger ones; thus,
[Amil)n
ai aiponamooo
where
(17)
M
j = im and
==1+
Σ (1n
4
A generalization of the product of two matrices is the product of tensor and matrix [de Lathauwer (1997); Carroll et al. (1980).
Definition (Mode-m Product, x The mode-m product of tensor Chxlxx...xla x1m x-- and matrix B E CJmXIm, denoted by A Xm. B, is lensor of dimensionality Cl xxx Im X x X.xIMM whose entries are computed by
[A Xm B) Blii..em
Σ tm
Oig..o Embbett 1 jmmmm:
C=Ax
mutriatiee Cmm] = BA [m]
tensorize
The M-mode SVD, Alg. 1 [Vasilescu Terzopoulos (2002a) is "generalization" of the conventional
matrix (i.e., 2-mode) SVD which may be written in tensor notation as
D = U
D S X Up X1 U1
The M-mode SVD orthogonalizes the M spaces and decomposes tensor the mode-m product, denoted X of -orthonnormal mode matrices, and a core tensor Z
D=Z X Uo Xm Um XM UM. D [m) U [m)
(18)
(U M U Um-i 8m.1
U... U,)
(19)
vec( (U
U Un Un) vec(2))
(20)
The latter two equations express the decomposition matrix form and terms of vee operators.
A.3 COMPOSITIONAL HIERARCHICAL BLOCK TENSORFACES
Training Data: In our experiments, we employed gray-level facial training images rendered from 3D scans 100 subjects. The seans were recorded using a CyberwareTMM 3030PS laser scanner and are part of the 3D morphable faces database created at the University of Freiburg [Blanz & Vetter (1999). Each subject was combinatoriall y imaged in Maya from different viewpoints
60 to +-60 in 10a steps on the horizontal plane. Φ = 09) with 15 different illuminations ( -35 to +35 in 5" increments a plane inclined at 45%).
Data Preprocessing: Facial images were warped to an average face template by piecewise affine transformation given a set facial landmarks obtained by employing Dlib software [King (2009); Kazemi & Sullivan (2014); Si et al. (2013); Macedo et al. (2006); Hatamizadeh et al. (2019). Illumi- nation was normalized with an adaptive contrast histogram equalization algorithm, but rather than performing contrast correction on the entire image, subtiles of the image were contrast normalized, and tiling artifacts were eliminated through interpolation. Histogram clipping was employed to avoid over-saturated regions.
16

---

Under review as a conference paper at ICLR 2022
Experiments: Each image, R f was convolved with five filters banks (H. 1...S). The filtered images, d Xo H., resulted five facial part hierarchies composed of (i) independent pixel parts (ii) parts seg- mented from different layers of a Gaussian pyramid that were equally or (iii) unequally weighed, (iv) parts were segmented from Laplacian pyramid that were equally or (v) unequally weighed. We ran five experiments with five facial part hierarchies from which a person repre- sentation computed, Fig. 8. The composite person signature was computed for every test image by em- ploying the multilinear projection algorithm |Vasilescu (2011); Vasilescu & Terzopoulos (2007), and signatures were compared with a nearest neighbor classifier.
Aaa
To validate the effectiveness our system on real-world images, we report results on "LFW" dataset (LFW) [Huang et al. (2007). This dataset contains 13,233 facial images of 5,749 people. The photos are unconstrained (i.e., "in the wild"), and include variation due to pose, illumination, expression, and occlusion. The dataset consists of train/test splits of the data. We report the mean accuracy and standard deviation across all splits in Table 2. Fig. 8(b-c) depicts the experimental ROC curves. We follow the supervised "Unrestricted, labeled outside data" paradigm.
Figure 7: Matrixizing 3rd order tensor, A. The tensor can be matrixized in 3 ways.
While we cannot celebrate closing the gap on human performance, our results are promising. DeepFace, CNN model, improved the prior verification rates LFW from 70% 97 .7.% by training on 4.4M images 200 x 200 pixels from 4, 030 people, the same order of magnitude as the number of people in the LFW database.
Results;
We trained on less than one percent (1%) of the 4.4M total images used to train DeepFace. Images were rendered from 3D scans of 100 subjects with an the intraocular distance of approximately 20 pixels and with facial region captured by 10, 414 pixels (image size 100 100 pixels). We have currently achieved verification rates just shy of 80% on LFW. When data is limited, CNN models do not convergence generalize.
Roo
Sevrd
I
U2
Figure 8: Compositional hierarchical Block Tensor learns a hierarchy of features, and reesents each person part-based compositional representation. Figure depicts the training data factorization, D Tx H x, U, Xv Uy Xp Up. where an observation is represented d(p, v, = Tx x, Xpp and Tx spans the hierarchical causal factor variance.
17

---

Under review as a conference paper at ICLR 2022
Test Dataset
PCA
TensorFaces
compositional hierarchical Block TensorFaces
Weighted Gaussian Pyramid
Weighted
Gaussian Pyramid
Laplacian Pyramid
Pixels
Laplacian Pyramid
Freiburg
65.23%
71.64%
90.50%
88. 17%
94.17%
90.96%
93.98%
LFW
69.23%
66.25%
72.72%
76.72%
77.85%
77.58%
78.93%
11.51
11.60
12.14
#1.65
11.8
11.45
11.7
Table 2: Empirical results reported for LFW PCA, TensorFaces and compositional hierarchical Block TensorFaces. Pixels denotes independent facial part analysis Gaussian/ use a multi resolution pyramid analyze facial features different scales. Weighted denotes weighted composite signature.
Freiburg Experiment:
); 6 illuminations (1600, ,4.00,,.30*, ,.:5), 45 people Test on Freiburg: views (+50 =40" =20", =10", 00)) ) illums (=50", =40", =20 =10 0%), 45 different people
Train on Freiburg: views (1600, ,,,30 ,1:5%;;;
LFW Experiment: Models were trained on approximately half of one percent (0.5% 1%) of the
4.4M images used to train DeepFace.
Train Freiburg:
15 views (=600 ,=500, =4000" -400, =200, ==0 09), 15 illuminations 5500 +440% ,+300
200%, = =100 =5 0²), 100 people
Test on LFW: We report the mean accuracy and standard deviation across standard literature partitions [Huang et al. (2007), following the
Unrestricted, labeled outside data supervised protocol.
Summary: This paper contributes the tensor algebraic paradigm and models cause-and- cause-and-effect as hierarchical block tensor interaction between intrinsic and extrinsic hierarchical causal factors of data formation.
data tensor expressed function hierarchical data tensor is unified tensor model wholes and parts from which new compositional hierarchical block tensor factorization was derived. The interpretable, hierarchical, and statistically invariant to
resulting causal factor representations demonstrated the context of facial images by training all other causal factors. Our approach was on very small set of synthetic images. While we have not closed the gap on human performance, we report encouraging face verification results Faces in the Wild datasets. CNN verification rates improved the 70% prior art to 97.35% when they employed 4.4M images from 4, 030 people, the same order of magnitude as the number of people in the LFW database. We have currently achieved verification rates just shy of 80% LFW by employing synthetic images from 100 people for a total of less than one percent (1%) of the total images employed by DeepFace. By comparison, when data is limited, CNN models do not convergence, or generalize.
test data sets-the Freiburg, and the Labeled
REFERENCES
Evrim Acar, Evangelos E Papalexakis, Gözde Gürdeniz, Morten A Rasmussen, Anders J Lawaetz, Mathias Nilsson, and Rasmus Bro. Structure-revealing data fusion. BMC hioinformatics, 15(1):239, 2014.
David H. Ackley, Geoffrey A. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147-16, 1985. ISSN 0364-0213. doi: http:///doi.org10..100 1016/ 50364-0213(858800 2-4. URL https::///// sttpp://wert wwwp///ww....///////////////w//. htt.:sss/www.eeeneddeeeeee www.ncediieett......c.......ee.. cww/sciinneeeeeeeeeeeecccccec..e comm//cience//rticel//bb.... 5036402138580012 3, 15
Animashree Anandkumar, Rong Gc, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. 15(1):2733283, 2014. 2,
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proc. the National
Academy Sciences, 13(27):7345-52, 2016. ISSN 0027-8424. doi: 10.10773pmas.151000 URL https: : / www. pnas. wttg///ot//eooooo www.oottttttcooooooooooooooo /ttpen/////////// 111111/27332
M.S. Bartlett, J.R. Movellan, and TJJ. Sejnowski. Face recognition by independent component analysis. IEEE Transactions Neural Networks, 13(6):1450-644 2002.
18

---

Under review as a conference paper at ICLR 2022
Jacob Benesty, Constantin Paleologu, Laura-Maria Dogariu, and Silviu Ciochin. Identification of linear and bilin- ear systems: A unified study. Electromics, 10(15), 2021. ISSN 2079-9292. doi: 10.3390/electroniess0011 10 790.
URL https:// www. mdpi. com/ 20 19- 01292////15/1/////
Peter M. Bentler and Sik- Yum Lee. A statistical development of thrre-mode factor analysis. British J. of
Math. and Stat. Psych., 3270144, 1979. doi: 0111111j22244-811111999999 wwwey. com /tw//bs///i///eeee/e wttpp//....11/..... 1tt.::1111....9-111111112
URL. https:
(ww.:/44--31..9991 tw00754 .x..
Blanz and T. A. Vetter. Morphable model for the synthesis of 3D faces. In Proc. ACM SIGGRAPH 99 Conf. pp. 187-194, 1999. 16
Kenneth A. Bollen and Judea Pearl. Eight Myths About Causality and Structural Equation Models, pp. 301-328. Springer Netherlands, Dordrecht, 2013. ISBN 778-94-007-660443 doi: 1110079778-9400077-711 15. URL https: /dc1. crg/ 10. 000000778000 04-007-6094-333 000----04----5 3_15.
Rasmus Bro. Parafac: Tutorial and applications. In Chemom, Intell. Lab Syst., Special Issue 2nd Internet Cont.
in Chemometrics (INCINC'96) 96), volume 38, pp. 149-171, 1997. 1, 15
J. D. Carroll and J. Chang. Analysis of individual differencess in multidimensional scaling via an N-way generalization of 'Eckart- 'Eckart-Young' Young' decomposition. Psychometrika, 35:283-319, 1970.
J. D. Carroli, S. Pruzansky, and J. B. Kruskal, CANDELINC: A general approach to multidimensional analysis of many-way arrays with linear constraints on parameters. Psychometrika, 45:3-24, 1980. 16
J. C. Chen, R. Ranjan, A. Kumar, C. Н. Chen, V. M. Patel, and R. Chellappa. An end-to-end system for unconstrained face verification with deep convolutional neural networks. In IEEE International Conf. on Computer Vision Workshop (ICCVW), pp. 360-368, Dec 2015. doi: 10. 109/CCCW.2015.555
Wei Chu and Zoubin Ghahramani. Probabilistic models for incomplete multi- linetidiimensoonl arrys. volume 5 of Proceedings of Machine Learning Research, pp. 89-96, Hilton Clearwater Beach Resort, Clearwa- ter Beach, Florida USA, 16-18 Apr 2009. PMLR. URL http::/ppocceeiiii www.smmr..ppp mlrrpress//v/
chu09a 9a. html. 2
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative matrix and lensor factorizations: applications exploratory multi-way data analysis and blind source separation. John Wiley & Sons, 2009.
Pierre Common. Independent component analysis, a new concept? Signal Processing, 36:287-333 314, 1994, URL https: / /hall.aaaaaa/ archi ves ouvertes. fr/hal 0004 004417833 7283. 5
T.F. Cootes, G.J. Edwards, and C.J. Taylor. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 63(6)6681 685, 2001.
J. Davis and H. Gao. Recognizing human action efforts: An adaptive three-mode PCA framework. In Proc. IEEE Inter. Conf. on Computer Vision, (ICCV), pp. 1463 69, Nice, France, Oct 13-16 2003. 1
L. de Lathauwer. Decompositions of higher-order tensor in block termspart ii: Definitions and uniqueness. 00107/0070999991.. SIAM J. Matrix Analysis and Applications, 30(3):1033-100 1066, 2008. doi: 10.1137/070690722. URL
https: /cci orgg 10. 1 070690729. 2,
De I lathauwer, B. De Moor, and Vandewalle. Independent component analysis based on higher-order statistics only. In Proceedings of 8th Workshop on Statistical Signal and Array Prucessing, pp. 356-359, 1996. doi: 10. 1t..109/SAP..99.5555500. 534.38990
Lieven de Lathauwer. Signal Processing Based on Multilinear Algebra. PhD dissertation, Katholieke Univ. Leuven, Belgium, 1997. 1, 3, 15, 16
A. P. Dempster. N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1 -22 1977. doi: htips://doi.org/10.. 10. 11 111/11 11..//17-6161111.. 1661177666111977.h66 1977.h0060000.. URL https:///// www.....oo on in iibrar.rr... wi wil.y. com/dci/abs/ 10. httpa/as......11111111111---1...
600 x.
Andrew Gelman and Guido Imbens. Why ask why? forward causal inference and reverse causal questions. Tech.report, Nat.Burean of Econ Research, 2013. 8
Lars Grasedyck. Hierarchical singular value decomposition tensors. SIAM J. on Matrix Analysis and Applications, 244999419, 2010.
Wolfgang Hackbusch and Stefan Külh.. A new scheme for the tensor representation. Journal of Fourier Analysis and Applications, 1565):006-722, 2009. 3
J. Hadamard. Lectures Caucky's Problem in Linear Partial Differential Equations. Dover Publisher, 1952. 8
R. Harshman. Foundations of PARAFAC procedure: Model and conditions for an explanatory factor analysis.
Tech. Report Working Papers Phonetics 16, UCLA, CA, Dee 1970.
19

---

Under review as a conference paper at ICLR 2022
Ali Hatamizadeh, Demetri Terzopoulos, and Andriy Myronenko. End-to-end boundary aware networks for medical image segmentation. In Inter. Workshop Machine Learning in Medical Imaging, pp. 187-194. Springer, 2019, 16
D. Hebb. The organization of behavior: A neurupsychological theory. John Wiley And Sons, Inc., New York, 1949. 3,
Geoffrey Hinton. How to represent part-whole hicrarchies in neural network, 2021. 2
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders, 2011.
Paul W. Holland. Statistics and causal inference: Rejoinder. J. of the American Statistical Association, 81(39)); 968970, 1986. URL https: / /userss.... aneerr.nber..................... holland JASA 11c__9990 1986. pdf.
.bber wwwer.or/eee..... wogg//rdehee j BBAAMM/ causal /
Eugene Hsu, Kari Pulli, and Jovan Popovic. Style translation for human motion. ACM Transactions on Graph- ics, 24(3):1082-85 89, 2005. URL ptttp://peoplaee csall mit edu/ jovan http://stttssssssseeeeseeeeees
hsu- 2005-000..220
G.B. B. Huang. Learning hierarchical representations for face verification with convolutional deep belief networks. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2518 25, Jun 2012. doi: 10.1
CVPR.2011..247968.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned- Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Mas- sachusetts, Amherst, Oct 2007. 17, 18
Guido W. Imbens. Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics. Journal of Economic Literature, 58( (8844):1199977)) December 2020. doi: 12.72e/ 20191597. https:
/ww.. aeaweb org/
Guido W. Imbens and Donald Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge Univ. Press, 2015. ISBN 052 0521188884,
Mark A Iwen, Deanna Needell, Elizaveta Rebrova, and Ali Zare. Lower memory oblivious (tensor) subspace embeddings with fewer random bits: modewise methods for least squares. SIAM Journal on Matrix Analysis and Applications, 42(1):376 416, 2021. 2
LT. Jolliffe. Principal Component Analysis. Springer-Vette New York, 1986. 3, 15
A. Kapteyn, H. Neudecker, and T. Wansbeek. An approach to n-mode component analysis. Psychometrika, 51 (2):269 275, Jun 1986.
Vahid Kazemi and Josephine Sullivan. One millisecond face alignment with ensemble of regression trees. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition, CVPR '14. pp. 1867-74, Washington, DC, USA, 2014. IEEE Computer Society. ISBN 978- 1-4799-51188 doi: 10.1 11110//VP..2014.000 URL
http:// //tp///x.////
dci. oro.
2014 241. 16
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang. Xiaokang Wu. Fangbing Yan, Justin Dong. Made K. Prasadha, Jacqueline Pei, Magdalene Y.I.. Ting, Jie 7hu, Christina I.I, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang. Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A.N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable diseases by image- based deep learning. Cell, 172(5):112 1122-1121111111 1.3,, 2018. ISSN 0092-8674. doi: https://doi. /trg//ooooo////o .rg/110. /016/ cell 2cw..2211.......cem. 2018.02.010. article/ p1 /00920002202000 7067774130000 41880153455...
URL https: / www. sci encedi rect. com/ sc ence/
Valentin Khrulkov. Geometrical Methods in Machine Leurning and Tensor Analysis. PhD dissertation, Skolkovo Institute of Science and Technology, 2020. 2
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taclim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. CoRR, abs/1511..06500, 2015. URL http: /arxiv org/ abs /1111 11111114. 06530.
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755-111 -158, 2009. 16
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51((3):555-- -50000 2009. 3, 1.5
Tamara G. Kolda, Brett W. Bader, Joseph Kenny. Higher-order web link analysis using multilinear algebra. In ICDM 2005: Proceedings of the 5th IEEE International Conference on Data Mining, pp. 242 249, 2005. doi: 10. 109/ICDM.MMM.. 009//CD..0005.777
20

---

Under review as a conference paper at ICLR 2022
Jean Kossaifi, Aran Khanna, Zachary C. Liptonn Tomasso Furlanello, and Anima Anandkumar. Tensor contraction layers for parsimonious deep nets. In Computer Vision and Pattern Recognition (CVPR), Tensor Methods in Computer Vision Workshop, pp. 1940 46000-6,600 Jul 2017. 2
M. A. Kramer. Nonlinear principal components analysis using autoassociative neural networks. AIChE J., 32(2):
233 243, Feb 1991.
M. Kroonenberg and J. de Leeuw. Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika, 45:69-97, 1980. 1
Lieven De Lathauwer, Comon, Bart De Moor, and Jons Vandewalle. Higher-order power method application in independent component analysis. Proc. of the International Sympoxium on Nonlinear Theory and its Applications (NOLTA 195,, pp. 91-96, 1995. 5
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky, Speeding- up convolutional neural networks using fine-tuned cp-decomposition. CoRR, abs/1412.6553, 2014. URL
http:/ /arxiv..vvv...///ii...a./.aiitv. org/abagae..... 1412. 6553.
Yann LeCun, D Touresky, G Hinton, and T Sejnowski. A theoretical framework for back prock--aggattonn. In Proceedings of the 1988 connectionist models summer school, volume 1. pp. 21-28, 1988. 3, 15
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Rohert Müller. Efficient BackProp, pp. 9-48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 778-3-642-3528999 doi: 10.1007/978-3-66 3-1---3--4-2--8--- URL ht doi wtt.c//g///oo.o. 100709700077700 3- 642 35289-22888888 3. 3, 15
I. Macedo, E. V. Brazil, and L. Velho. Expression transfer between photographs through multilincar aam's ss pp- 239-246, Oct 2006. doi: 10.1 1091099BRRAAA..2000000 18. 16
Ali Madani, Mchdi Moradi, Alexandros Karargyris, and Tanveer Syeda- Madda-Maaoood.... supervipe lcarning with generative adversarial networks for chest x-ray classification with ability of data domain adaptation. In 2018 IEEE 1.5th International Symposium on Biomedical Imaging (ISBI 2018). pp. 1038-1042, 2018. doi: 10.1 109/1B1I...00 20T.B......11.13.33 8.8363749.
J. R. Magnus and H. Neudecker. Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wilcy & Sons, 1988.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In C. Cortes, N. D. Lawrence, D. D. Lec, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. paper / 5787-7777777 tensort z ng- neural -net works pdf.
Curran Associates, Inc., 2015. URL http:
nips ce/
Erkki Oja. A simplified neuron model principal component analyzer. 15:267-270 1982. 3, 15
Charles C. Onu, Jacob E. Miller, Doina Precup. fully tensorized recurrent neural network. CoRR, abs/2010.04196, 2020. URL https:// /ttss/////vvv../ org/ wwws/201111 0. 04 96. 2
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge Univ. Press, 2000. ISBN 9780521773621. URL
google. com/ books 2id=waGU 1 TsW3BQC.
Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across populations. Statistical Science, 7994), 2014.
I. Perros, R. Chen, R. Vuduc, and J. Sun. Sparse hierarchical tucker factorization and its application to healthcare. In Proc. IEEE Inter. Conf. on Data Mining, pp. 943-948, Nov 2015. doi: 10.1 10.1109/CDM.20155.2
David F. Rumelhart, Geoffrey F. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. 1986. 3, 15
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 30. Curran Associates, Inc., 2017. URL ht http:p////////ooopooooooooo neuri ps. cc/ httpp/apprr//// /201 17 Fi e 2aad a 7bbe f2 8
.dff
Terry Sanger. Optimal unsupervised learnig in single layer linear feedforward neural network. 12:459 473. 1989. 3, 15
B. Schälkoph, A. Smola, and K.-R. Muller. Nonlinear component analysis kemee eigenvalue problem. Neural Computation, 10(5):1299-1319, 1998.
Terrence Sejnowski. Sumantra Chattarji. and Patric Sfanton. Induction of Synaptic Plasticity by Hebbian Covariance in Hippocampus, pp. 105-124. Addison- Wesley, 1989. 3, 15
Weiguang Si, Kota Yamaguchi, and M. Alex O. ing with multilinear (tensor) active appearance models. 0e331
Vasilescu.
Face track- http://pdls..emmnticc URL https: / ppfss
0oobaaoogooo44//000aa22120011110 semant cacho a 6
Jun 2013.
67dcl 8 a2..2244bbbbt www.ff.ppfffffcco
21

---

Under review as a conference paper at ICLR 2022
Peter Spirtes, Clark N Glymour. Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Hybrid deep learning for face verification. In Proc. IEEE International Conf. on Computer Vision (ICCV), pp. 1489-96, 96, Dee 2013. doi: 10.1109/ICCV:2013.1
Y. Taigman, M. Yang. M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1701 2014. doi: 10.1 109/CVPR.2014.222
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. Tensor analyzers. volume 28 of Proceedings
of Machine Learning Research, pp. 163-1711 Atlanta, Georgia, USA, 17-19 Jun 2013. URL http: // //proceedings mlr. prw.s/ressseee tangl3 html.
Eric J Topol. Higgh-eeformanee medicine: the convergence of human and artificial intelligence. Nature medicine, 25(1))44-56, 2019. 1
L. R. Tucker. Some mathematical notes three-mode factor analysis. Psychometrika, 31:279-311, 1966.
M. A. Vasilescu. Human motion signatures: Analysis, synthesis, recognition. In Proc. Int. Conf. on Pattern Recognition, volume pp. 456-460, Quehee City, Aug 2002. 1
M. A. Vasilescu. Multilinear projection for face recognition via canonical decomposition. In Proc. IEEE Inter. Conf. on Automatic Face Gesture Recognition (FG 201 I), pp. 476-483, Mar 2011. doi: 10.1 10.1109/FG.
2011.5771445. 1,17 17
M. A. O. Vasilescu and D. Terzopoulos. Multilincar analysis of image ensembles: TensorFaces. In Proc. European Conf. on Computer Vision (ECCV 2002). pp- 447-460, Copenhagen, Denmark, May 2002a. 1, 15, 16
M. A. O. Vasilescu and D. Terzopoulos. Multilinear analysis for facial image recognition. In Proc. Int. Conf, on Pattern Recognition, volume 2. pp. 511-514, Quebec City, Aug 2002b. 1,
M. A. O. Vasilescu and D. Terzopoulos. TensorTextures: Multilinear image- based rendering. ACM Transactions on Graphics, 23(3):33--342, Aug 2004. Proc. ACM SIGGRAPH 2004 Conf., Los Angeles, CA.
M. A. O. Vasilescu and D. Terzopoulos. Multilinear independent components analysis. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, volume pp. 447-553, San Diego, CA, 2005, 1, 1,4,8
M. A. Vasilescu and D. Terzopoulos. Multilinear projection for appearance- recognition in the lensor framework. In Proc. 11th IEEE Inter. Conf. on Computer Vision (ICCV'07), pp. 1-8, 2007. 1, 9, 17
M. Alex O. Vasilescu. A Multilinear (Tensor) Algebraic Framework for Computer Graphics, Computer Vision, and Machine Learning. PhD dissertation, University of Toronto, 2009. 1,2, 3, 15
M. Alex O. Vasilescu and Eric Kim. Compositional hierarchical tensor factorization: Representing hicrarchical intrinsic and extrinsic causal factors. In The 25th ACM SIGKDD Conf. on Knowledge Discovery and Data Mining (KDDI9): Tensor Methods for Emerging Data Science Challenges Workshop, Aug. 2019. 2,
M. Alex O. Vasilescu, Eric Kim, and Xiao S. Zeng. Causalx: Causal explanations and block multilincar factor analysis. In 2020 25th International Conference of Pattern Recognition (ICPR 2020), pp. 10736-10743, Jan 2021. 1, 2, 6, 8,
Joshua Vendrow, Jamie Haddock, and Deanna Needell, generalized hierarchical nonnegative tensor decompo-
sition, 2021. 2
D. Vlasic, M. Brand, H. Pfister, and Popovic. Face transfer with multilinear models. ACM Transactions on Graphics (TOG), 24(3):426444 Jul 2005. URL http:://peop... csail 1 má t. edu/ /jvan// assets/ assets/paperss httppaapprssseeeee wwweerrsllllssee 2005- ftm pd...
Hongcheng Wang and Narendra Ahuja. Facial expression decomposition. Proc; IEEE Inter: Conf. on Computer Vision (ICCV), pp. 958 65, 2003. doi: 10. 119///1C.20331 1238452.
M. Wang, Y. Panagakis, P. Snape, and S. Zafciriou. Leamiing the multilinear structure of visual data. In 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 6053 6061, Jul 2017. doi: 10. 10. 2017.641. URL https: /ooi. weecompttersocetee
org/ 10. 39///VPR 2017 641.2
C. Xiong, I. Liu, X. Zhao, S. Yan, and K. Kim. Convolutional fusion network for face verification in the wild. IEEE Transactions on Circuits and Systems for Video Technology. 26(3)5117-528. Mar 2016. ISSN 1051-8215. doi: 10. N0.11099TCSVT.20155244666
Yang, X. Gao, D. Zhang, and Yang. Kernel ICA: An alternative formulation and its application face recognition. Pattern Recognition, 38 38(10)1784 87, 2005.
M. Yang. Kernel eigenfaces vs. kernel fisherfaces: Face recognition using kernel methods. In Proc. IEEÉ Inter. Conf. on Automatic Face Gesture Recognition, pp. 215 220, Washington, DC, May 2002. doi: 10.1 109///...20000 45222277
22


Reviews:
[{"review_id": "zgIie0kIY57", "reviewer": "Reviewer_oC53", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* the paper adresses an important and timely problem: that of causal discovery\n* the paper proposes an innovative approach to combine Tensor Factor Analysis with capsule network architecture that uses auto-encoding architectures\n\nWeaknesses:\n* This paper suffers from a major lack of clarity in writing. Here are some important points\n- The introduction essentially lists a number of papers that use Tensor-Decomposition for various applications but does not clearly identify a gap in knowledge or a problem that will be addressed by the contribution. It states that the paper seeks to \"connect tensor causal factor analysis with deep learning\" but does not adequately articulate why this is needed or a good idea.\n- There is a lack of clarity in definitions: (i) several acronyms are not defineds (e.g. MPCA/MICA). (ii) in section 2, equation (1), there is no desciption of the variables (what is d, B, c ?) (iii) Throughout all figures appear to show image recognition/classification tasks but nowhere is the task or computation described. (iv) formatting is not at the level of publications (Figures are not referred in the text in the order they appear, a number of citations are compiled with an error showing \"?\", punctuation and sentence construction is as mishandled in a number of areas).\n\n*Most crucial is the lack of experimental evidence. The figures appear to show some image recognition task that is never described and there is no comparison with other methods, nor any description of the advantages of the proposed method.\n\n*Conclusion is quasi non-existant and there is no discussion about future directions for this work and how it may be improved in the future.\n", "summary_of_the_paper": "The authors present a framework in which Tensor Factor Analysis is incorporated in a causal graph structure implemented by a series of \"capsule\" auto-encoders.  The rationale is that these capsules can be combined in some way to create meaningful hierarchical architecture useful for causal inference.", "summary_of_the_review": "In my opinion, this paper is not clearly written, and does not contain compelling and convincing evidence that the proposed method adresses a gap in knowledge or improves on current method. It suffers from major structural problems as outlined in detail above. It is quite possible that the contribution is, in fact, significant but I could not evaluate it because of these issues. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qe0o9yR1-Qk", "reviewer": "Reviewer_Bzxf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. As far as I understand, the authors provide a causal invariant representation of the data in a subspace. The representation can be used for streaming tasks. \n2. The cascade idea depicted in Fig. 5 is neat!\n3. The extension to non-linear Causal capsules using kernels is intuitive and nicely introduced.\n\nConcerns & questions:\n1. The paper is confusing to read. The motivation and the flow of the paper is tedious to follow. This might be due to the lack of knowledge on my end. Although, I am fairly familiar with causality theory (by Pearl etc.), capsule networks and tensor decompositions. The authors should provide more background in their paper.\n2. It will be great if the authors can include some set of experiments to demonstrate the benefits of their approach in practice. \n3. I can foresee some issues with the scalability of this work. I request the authors to highlight them in the paper. This will be a good indicator of how useful this technique can be for real world applications.\n4. Missing citations on Pg. 2 and some minor typos. \n", "summary_of_the_paper": "This paper is an attempt to connect tensor factor analysis with DNN learning. The paper explores the forward causal inference and inverse causal inference problems. The forward causal inference is performed using causal capsules architecture that uses M-mode SVD for subspace learning (autoencoder). The authors show that the M-mode SVD rotates the basis vectors of various clusters such that the causal representation is the same regardless of the data cluster (Fig.1). They introduce causal capsule hierarchy architecture that uses tensor decompositions for forward and inverse causal inference. ", "summary_of_the_review": "I would be more confident about my review of the work, if the authors can include some set of experiments in the paper. It will be very helpful if the authors can point out the benefits over the existing SOTA techniques on why to use their proposed causal representation. Some comments on the scalability of their approach will also be appreciated. \n\nI am willing to reconsider my ratings if the authors address some of my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "lTbtJSawLAg", "reviewer": "Reviewer_4HKq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern with this paper is the effectiveness of the proposed method. Since there is no experimental evaluation, I don’t know how well the proposed method will perform in real problems. Also, if this paper claims a theoretical contribution, there should be strong theoretical justifications about the advances of the proposed method. However, I only see the method descriptions instead of in-depth theoretical analyzes. Besides, this paper is very hard to follow. It is not well structured and has many typos. \n", "summary_of_the_paper": "This paper tried to build the connection between tensor factorization and capsule network, and a forward causal inference with causal capsule method was proposed. However, since there is no theoretical justification or experimental evaluation, I cannot judge the effectiveness of the proposed method.", "summary_of_the_review": "I think this paper's contributions are not well justified. Given its current status, I cannot vote for its acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "0wu42WF85fF", "reviewer": "Reviewer_GQkp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is hard to read. Even though I read multiple times, I'm not sure if I fully understand this paper. The idea to connect tensor analysis with casual inferences is good but this paper doesn't have enough contexts. \n\nSome issues:\n\n- There are a few typos and incorrect latex references that impacts the reading experiences. For example:\n  - In the second paragraph in page 2, \"by proposing a causal capsule hierarchy of parts and wholes that is consistent the capsule theory\" should be \"consistent with the capsule thoery\".\n  - In the second last paragraph in page 2, there are two \"?\" in the citations after \"which may result in t\napproximate subspace\".\n  - Right above section 3.2 in page 8, there is a \"Fig ??\" after \"not just a complete observation,\".\n- It would be better to provide some empricial results to show the effectness of the proposed models. The face recognitions task is used as an example in many places in the paper, but there are no experiments to evaluate how the proposed model performs.\n- It's worth to discuss what kind of tensors are suitable for the proposed model. It's also worth to show the proposed model can run in a reasonable speed for modern datasets. ", "summary_of_the_paper": "The authors present some analysis and describes a new methodology on how to do forward/inverse causal inferences with tensor analysis. There are no empricial results nor theoratic analysis to back the authors' claims.  ", "summary_of_the_review": " I don't think this paper is ready to be read. I think there are many places that can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four experts reviewed this paper and rated the paper below the acceptance threshold. The reviewers raised many concerns regarding the paper, mainly the lack of empirical studies and clarity. Some reviewers also suggested the authors better positioning the paper in the literature by discussing more related works. The rebuttal did not address all concerns. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject