ACML 2020 TUTORIAL

Tensor Networks in Machine Learning:
Recent Advances and Frontiers
Qibin Zhao
Tensor Learning TeamRIKEN AIP
https://qibinzhao.github.io1

Success of Machine Learning
Big Data

Computation

Model & Algorithm

Tensor Networks

Mathematics

Quantum Physics

2

Role of Tensor and Tensor Networks(Liu et al. IJCAI Workshop 2020)3

AgendaThis tutorial aims to introduce tensor networks fundamental, andto overview recent progress of tensor networks applied tomachine learning.

Part I

Part II

Part III

Tensor Methods
for Data
Representation

Tensor Networks
in Deep Learning
Modeling

Frontiers andFuture Trends

4Part 1:
Tensor Networks for Data
Representation

5

Multiway Structured Data

Feature Maps in CNN

Video Data

fMRI Data(Spatial x Spatial x Filter)
WIKIMEDIA COMMONS(Spatial x Spatial x Time)(Spatial x Spatial x Spatial)

EEG(Spatial x Time x Frequency)

Hyperspectral Image(Spatial x Spatial x Spectral)
6

Medical Knowledge Graph(Entity x Entity x Relation)(Wang et al., 2017)

Mod
i=1,2

Tensor: Generalization of Vector andMatrixx

otations and Graphical Representations

X(i,:,:)

7

Lateral Slices

Horizontal Slices

k= Mo
1, de2, 3
...
,K

X

X(i,:,:)

Frontal Slices

X (:, j,:)X(:,:,k)Column (Mode-1)
FibersMode-1i=1,2,...,I

ontal Slices

j =1,2,..., J
Mode-2

6,5,1

Row (Mode-2)
FibersTube (Mode-3)
FibersX(1,3,:)

X(1,:,3)

j =1,2,..., J
Mode-2Lateral Slices
X (:, j,:)x6,5,1

X(:,3,1)
Frontal Slices

Figure 1.2: A 3rd-order tensor X R I J K , with entries xi,j,k = X(i, j, k ) and its
tensors: slices (middle) and fibers (bottom). All fibers are treated as column vectX(:,:,k)7

zed version of (1) isJ⇤Imm ⇤···⇤IM
decomposition
holds
and is U
definedhe m-mode product of a tensor X ⇧Tucker
RI1 ⇤···⇤I
withexactly
a matrix⇧ Ras:

Tensor Decomposition

(m)
(M )
(m+1)
(m
T (X(2) ), . . . , rank(X( N ) ) ,rank
X) = rank(X(1(1)
ML (1)) ), rank
=
U
G
(U
⇤
·
·
·
⇤
U
⇤
U
⇤
·
·
·
⇤
U
)
(2)
is a tensor Y of size I1 ⇥ · · · ⇥ Im 1 ⇥ J ⇥ Im+1 ⇥, · · · ⇥ IM definedm U and (m)with X(n)3.3. The Tucker Tensor Format

R In

I1

In

1 In+175

IN for n = 1, 2, . . . , N.

Im=
xi1 i2operator
For morematrices.
detailed descriptionsof multilinear
algebra,..iM...iM ujim .between
iproductm =1
roneckerSuchadecomposition
becomes
The independent Tucker format has the following important
where
the multi-indicesareorderedin a reverse
lexicographic
order
dome
Bader
[2].
The
two
populartensordecompositions
aretheTuckermodel
andertiesif the equalityin (3.10)
holds
exactly(seee.g., (Jiang
specific constraints are imposed. For example, when all factor matrices et al.

(little-endian).
and references therein):
The both
Tuckerofdecomposition
is regarded
said to be as
in an
independent generalizations
TuckerRAFAC(CP)
model,
whichcanbehigher-order(1) , B (2) , . . . , B
mnwise orthonormal
and
the
core
tensorGisall-orthogonal(i.e.,
any
subtensors(
n)
1.
Tensor
(CP)
rank
of
any
tensor,X=
JG;
B
format if the all factor matrices, B , are full column rank, while a
I1 I2
IN , and the rank of its core tensor, G
R1 R2
R
R
lar value decomposition
(SVD)
[1].
Tuckerformatis termed
anorthonormal
format,higher-order
if in addition,singular
all the value
efinition in [13])andordered,
this
model
is
calledare thesame, i.e., The standard Tucker(n) = U(n) , arefactormatrices,B
orthogonal.
⇤···⇤IM denote an M th-order tensor, then Tucker model, illustrated in Fig. 1, isrank(G). tensor
model
usuallyus
hasmanyorthogonal
matrices.For
CP ( X ) = rankVD) [13], which
providesusefulfactor
properties.
instance,
theCPcore

Tucker Decomposition(HOSVD):I I
I(1) , B (2) , . . . , B ( N
(1)T
(2)T
(M
)T
2.
If
a
tensor,X
R=
JG;
B
ed by G = X ⇥⇥rank.
· ·⇥N Usize N-tuple. A more
caseof Tucker
Multilinear( R1 , Rrestrictedcalled1U
2 U The·minimal
2 , . . . , R N ) is
mits an(Mindependent
Tuckerformat with multilinear
I
I
I
(1)
(2)
)
2
N
1
tensorX
R2, if theXthe= multilinear-rank
G ⇥1 U ⇥2 of
U an ·Nth-order· · ⇥RM
U
,
(1)
,
R
,
.
.
.
,
R
,
then
2
N
1
e number of components
in all factor
is isthesameas:(i.e., R1 = · · · = RM )Tucker decomposition
holdsmatrices
exactly anddefined
1

2

N

N

⇤R
(m) if⇧r
Im⇤Rm=denotesn.
=M rank(X(X, RCP
is Msuperdiagonal(i.e.,
⌃=
only
rRM
model
canin
n)
p (3.14)is called therankcore
and0(XU
factormatrixMLg(tensor(1) ), rank
) ), ·. ·. .·, rank
( N[2].
)) arX1)...r1R(2=R In I1

with X(n)In

1 In+1

IN for n = 1, 2, . . . , N.

p num of version
rank-one
icizedoftensors:(1) Polyadic
isMoreover,(CPD)
without loss
Canonical
Decomposition: of generality, under the assumption

R2
R N , we haveR⇥
The independent Tucker format has the following important prop(m)(M ) (1)
(m
(1)
T R R
(2)(m+1)(M) 1) Rrank(
X
R N . (3)
=
U
G
(U
⇤
·
·
·
⇤
U
⇤
U
⇤
·
·
·
⇤
U
)
, 2016)
(2)
X
=
u
⌅
u
⌅
·
·
·
⌅
u
,
2 3
CP et al.,erties if theinr equalitym)(m)rr (3.10) holdsrexactly (see1 e.g., (Jiang

andr=1
references therein):

2 Thisfollows

from the fact that mode-n matricizations, X(
IN , are not independent of one another, which makes it imp
R In I1 In 1 In+1
(1) , B (2) , . . . , B ( N ) K
chose
the multilinear
rankBarbitrarily.
1. Tensor (CP) rank of toany
tensor,
X = JG;
8
I I
I
R R
R

Kronecker product operator between matrices. Such a decomposition becomes
R1

2

N

, and the rank of its core tensor, G

R 1

2

N

,

Tensor Methods‣ HOSVD: A method for tensor denosing and dimension reduction‣ Tensor completion‣ Tensor imaging and sensing‣ Multi-linear blind source separation (BSS)
Reversing a forward problem

arw

Compression

se

Fo
r

er

v
In

d

Low rank tensor

Identification

fig: (Malcolm Sambridge, 2016)

9

10

2,
-

(ii) ordering:
σ1 ! σ2 ! . . . ! σmin(I1 ,I2 ) ! 0.

Multilinear Dimension Reduction
The σ are singular values of F and the vectors U
and U
are, resp., an
tructuredleft anddata
an ith right singular
vector.
The
decompositionis visualized in Figure 3.
Multi-linear
extension
of
SVD,
PCA
and
ICA
‣
(4)

3
g

i

)

e
e
e
e

op
le

Pe

rs

)

ns

n

sio
es

rn.
s.
=
d
n
3

pr
Ex

(5)
Views

Illumination

s
”.
e
al
al
f
s
g

(2)
i

(b) we state the following theorem.
Now
Theorem 2 (N th-order SVD). Every complex (I1 × I2 × · · · × IN )-tensor A
be written as the productHigher-order singular value decomposition

ms

3
y
w

(1)
i

A = S ×1 U(1) ×2 U(2) · · · ×N U(N ) ,in which
"
!
(n)
(n)
(n)
VALUE DECOMPOSITION
1261
isSINGULAR
a unitary(In × In )-matrix,. . . UIn Orthonormal
1. U(n) = U1 U2 A MULTILINEAR
2. S is a complex (I1 × I2 × · · · × IN )-tensor of which the subtensors Sin
obtained by fixing the nth index to α, have theproperties ofProjections(i) all-orthogonality: two subtensors Sin =α and Sin =β are orthogonal forpossible(c) values of n, α and β subject to α "= β:Figure 2: The facial image database (28 subjects, 60 images per

(6)

#S

,S

$=0

when

α "= β,subject). (a) The 28 subjects shown in expression 2 (smile), viewin =α
in =βpoint 3 (frontal), and illumination 2 (frontal). (b) Part of the image
set for subject 1. Left to right, the three panels show images captured in illuminations 1, 2, and 3. Within each panel, images of
expressions 1, 2, and 3 (neural, smile, yawn) are shown horizontally while images from viewpoints 1, 2, 3, 4, and 5 are shown
vertically. The image of subject 1 in (a) is the image situated at
the center of (b). (c) The 5th -order data tensor D for the imagei(Vasilescu
i =I
n =14. Visualizationn =2of the HOSVDnMultilinear Analysis of Image Ensembles: iTensorFaceset al., ECCV
2002)
Fig.
for anthird-ordertensor.
ensemble; only images in expression 1 (neutral) are shown.

(ii) ordering:

(7)

%S

% ! %S% ! . . . ! %S%!0

A Multi-linear Singular Value Decomposition (Lathauwer et al , SIAM J. Matrix Anal. Appl., 2000)

for all possible values of n.
10
...
(n)
of matrices (i.e., theproducts of by
the σcorresponding
entries vanishes);%, thesymbolized, are n-mode
singularat va
The Frobenius-norms%Ssum ofTensor Blind Source Separation‣ Blind source separation (BSS) is applied to multiple latent factors
G. Zhou and A. Cichocki

Fig. 3. Illustration of FSTD and MBSS based on fiber sampling of a 3D tensor. Note that on one hand, the columns of Y(n) are sampledfrom the fibers of the original data tensor Y; on the other hand, these columns/fibers are the linear combinations of the columns of factors
A(n) . Thus we can run various BSS algorithms (optionally, with PCA) on the sampled fibers directly to extract A(n)G. Zhou and A. Cichocki, “Fast and unique Tucker decompositions
via multiway blind source separation”, 201211

Tensor Decomposition
for
Efficient
Learning
Algorithm
What are Tensors Useful For?
[A. Anandkumar, JMLR 2014]
WhatareTensors
UsefulFor?
Unsupervised learning

Method of Moments for unsupervised learning

Estimate
moments E[x ], E[x ⊗ x ], E[x ⊗ x ⊗ x ] . . . from data.
moments‣ Estimate
Find best model in class that fits moments.
‣ Learning through higher order moments‣ Tensor decomposition via power methodijMultimodal data

T = E[x1 ⊗ x2 ⊗ x3 ].

x2

x3

ICA

Time

h1 h2

hk
xd

x1 x2

Multiview and Topic Models

12

T (i, j, k) = E[x1 (i)x2 (j)x3 (k)]T (i, j, k) = E[x1 (i)x2 (j)x3 (k)]
x3
x3
x2
x2
x1

x1

cts
u
od
h
Pr3

Users

Users

ct
u
od
Prh1
h2

k

Higher order moments
T = E[x1 ⊗ x2 ⊗ x3 ].

Shared latent representation.
HMM
s TimejHigher order moments

Multimodal data
Different data types.
Different for
dataLVMs
types.
Tractable
Learning
Shared latent representation.

GMM

i

x1

i

Article

Missing Values Problems in ML

Perturbational Gene-Expression Signatures for
system (userx item x time), Knowledge graph‣ Recommender Drug
Combinatorial
Discovery

ational prediction
data

‣

Image
inpainting/denoising,drug repositioningnships among
multiple
objects.

Fig. 1. Experimental resultand 20 before and after rec
Chen-Tsung
Huang, Chiao-Hui
Hsieh, Yun-Hsien
Chung, Yen-Jen
Oyang, HsuanCheng Huang,
Hsueh-Fen Juanhsuancheng@ym.edu.tw (H.C.H.)
yukijuan@ntu.edu.tw (H.-F.J.)

Fig. 2. Comparison of clo
used
for the color composit
HIGHLIGHTS

air ⇔matrixuple ⇔ 3 dim. array

Compound signatures
targeting non-oncogene
addiction
for area. TRALS c
inthe most
combinatorial drug
image.
However, it also r
discovery

covered. To sum all, the pr
results
from theThese signaturesare visual pers

3 / 29reproducible and linked tocancer hallmarks and drug
4.
sensitivity

13

Two synergistic drugCOTensor Completion
Problem formulation (low-rank approximation)

min Rank(X ), s.t. Y⌦ = X⌦<latexit sha1_base64="D/7PrqjGN4C9VkngxHKgOYK/4yc=">AAACV3icbZBPa9tAEMXXapO4Sf847bGXpaaQQhBSGkigFEx6ya1pqRMHy4jReuws3l2J3VGpEfp4+RA+9txrem5XjkmapAPLPn4zjxleVijpKIoWreDR47X1jfaTza2nz56/6Gy/PHV5aQX2Ra5yO8jAoZIG+yRJ4aCwCDpTeJbNPjX9s+9onczNN5oXONIwNXIiBZBHaSdNtDRplWigCwGKD2qeEP6g6iuYWb1zy9/t8uQDdyGFzX/Dz733s8Yp1PzjLR3c0LTTjcJoWfyhiFeiy1Z1knZ+JuNclBoNCQXODeOooFEFlqRQWG8mpcMCxAymOPTSgEY3qpZB1PytJ2M+ya1/hviS/uuoQDs315mfbE5193sN/F9vWNLkcFRJU5SERlwvmpSKU86bVPlYWhSk5l6AsNLfysUFWBDks7+zJdNNJvH9BB6K070wfh/ufdnv9o5W6bTZa/aG7bCYHbAeO2YnrM8Eu2S/2BX73Vq0/gTrQft6NGitPK/YnQq2/wLlO7WW</latexit>

X

Main Approaches:‣ Low-rank approximation via convex optimization (high computation cost)‣ Decomposition based approach (rank selection problem)‣ Combining convex optimization with decomposition approach‣ Incorporating priori knowledge, such as auxiliary information from other data,smoothness, sparsity, non-negativity and etc.
14

traditional
methods
thata priorplacedistribution
the ARD overprior overMorespecifically,
we place
Fig. 1. Proba
latent governedvariables byorR-dimensional
weight parameters,
the either
latent factors,
hyper- suchofas Bayesiancomponent analysis [33], factorizationourparameters= [principle
1 , . . . , R ], where each r controls
considersmodel
as latent varirth method
componentin A(n) .allThus,
the parameters
prior of the mode-n
ables
overis whicha sparsity-inducing prior is placed
factormatrixgiven byFor simplicwith shared Ihyperparameters.
ncluding bothY
Probabilistic modelingMoreoftensordecomposition
(n)
place
1 a prior distribution overp A(n) specifically,=
N aiwe0,⇤, 8n 2 [1, N ], (7) ters, are
colle
n
Fig.
1.
the latent factors,
governedbyR-dimensional
hyper(1)
(N
in =1
{A
,
.
.
.
,
A
factorizGroup sparsity priorparameters
imposed on= factor[ 1 , . . . ,matrices
R ], where each r controls
the joint distri
(n)
where
⇤
=
diag(
)
denotestheinversecovariance
rth component in A . Thus, the prior of the mode-nvariables as
matrix,
also
known
as
the
precision
matrix,andis
Bayesian inference factor
for posteriors
of modelparametersmatrix is given
byFor⇣
shared by latent factor matrices in all modes. We can=pInfurther define a hyperpriorover hyperparameters , p(Y ⌦ , ⇥)cludinY(n)1(n)
which pis A
factorized
over Nlatentdimensionalityto], (7) ters, a=a0,⇤, 8n 2due
[1, N
in
c
d
the independent assumption
in =1
By combining{A(1) ,
parametersinthe joiR
where ⇤ = diag(Y) denotes
the inverse covariancer r
the logarithm
variab
p(
)
=
Ga(
|c
,
d
),
(8)
r
0
0
is
a matrix,b also known as the precision matrix, andAppendix for
r=1

Learning Optimal Tensor Rank‣‣‣

shared by latent factor matrices in all modes. We canb) denotesa Gammadistribution
furtherdefinea hyperpriorover
hyperparameters , p(Y ⌦⌧ Ga(x|a,A(1) · · · A(n) · · · A(N ) where
⌧,⇥
`(⇥)
=
which is factorized overa latentdimensionality
due
to2b xa 1 e bx
the independentX
Ga(x|a,assumptionb) =(9)
By com
1
(a)
Tr
⇤
param
Y
2
R
Y
and (a) is the Gamma function.
Xp( ) =Ga( r |cr0 , dr0 ),(8) the log
Since the sparsity is enforced in the latent dimen+
Appen
r=1
sions, the initialization point of the dimensionalityof 2015)(Zhao et al, TPAMI
where
Ga(x|a,b) isdenotesGamma
latentspace(i.e., R)
usuallya set
to itsdistribution
maximum
possible value, while
15 the effective dimensionality canba xa 1 e bx

r

`(⇥)

Applications to Hyperspectral Image Denoising

Low-rank Tensor
Approximation

Non-Local Meets Global: An Integrated Paradigm for Hyperspectral Denoising (He et al., CVPR 2019)

16

ted
sor
en-

recovery.
In therest of this Undersection, we
first establish
an upperTensorCompletion
Multiple
Transformation
bound of MCMT under a single linear transformation, i.e.
alwaysthat,
globally
low-rank
K =is not1. After
we extend
the results to the case of‣ Imagemultiplesimilartransformations.
patches are often low-rank‣ Non-local

Lackofexample
theoreticalanalysis‣An–
Non-local
in Image Restoration(4)
3.1. Single linearTricktransformation

Assume thatM0 2 Rdenotesthe“true”
maGroup matching
Concatenating
Unfolding
trix
that
we
want
to
recover,
and
its
rank
equalsR.
The-thnoisedvariantofMisgenerated
by
Y
=
M
+
H
where
0
0
ons
the
entries
of
H
obey
the
i.i.d.
Gaussiandistribution,i.e.
A
lineartransformationfromhigh-ranktolow-rank
structure
hen
2
H(i,
j)
⇠
N
(0,
)
for
all
i
2
[m
],
j
2
[m
].
With
the
1
2
sor
Summary
singlelineartransformation,we
simplify(2)
as
Low-rankness
underlineartransformation
A significant
low-rank
structure appears
under sometransformations.
es,

redig

sed

m1 ⇥m2

ProblemminkQ(X)k⇤s.t. kP⌦ (X)P⌦ (Y)kF  ,The conventional
analysis for guarantee is no longer suitable.
m1 ⇥mtheoretical
2

X2R

Zhun Sun (RIKEN-AIP)CVPR2019

Linear transformation

June 20

3/7(5)where the subscript of Q 2 Rm1 ⇥m2 ⇥n1 ⇥n2 is removed
Guaranteed Matrix Completion under Multiple Linear Transformations(Li et al, CVPR 2019)>
for brevity. Let Q(M0 ) = UDV be the truncated
17

28
28
29
29

29
29
29
29
29
29

29
29
30
30
30
30

30
30
30
30

Experiment
Observation

FBCP

Illustrative Experiment
SPC

TRALS

FaLRTC

MCMT

NNM

MCMT
(Sketching)

U

R

C

RC

URC

(RIKEN-AIP)18
CVPR2019

JuPrior Arts

Latent Convex Tensor Decomposition‣ Most tensor decompositions are non-convex
In tensor
decomposition,
the most
related work is
totoourstudy is low-rank
the latent
Latentconvextensordecomposition
optimize‣
convex tensor decomposition (LCTD)2 ,3 .
components

I
J

K
I
JK

{ {

3 components

unfoldings

unfoldings

J

K

IK

IJ

Low-rank approximation‣

LatentHayashi,
convexdecomposition
cannotensureexactoptirecovery
Tomioka,andtensor
Kashima,“Estimation of low-rank
tensors
via convex
mization”.
3(Tomioka,
Hayashi,“Convex
Kashima,Tensor
Low-rank
tensor via convex
Tomiokaand Suzuki,
Decomposition
via optimization)
Structured Schatten Norm
Regularization”.
(Tomioka, Suzuki, Convex tensor decomposition via structured schatten norm regularization)
2

Chao Li (RIKEN-AIP)

Mathematical Seminar19

May 30, 201911 / 33

The difference
from
our
model
Reshuffled Tensor Decomposition
unfolding v.s. reshuffling.
(a) Tensor Decomposition

3 components{(c) How reshasping and reordering findthe potential low-rank structure of the data.

I
J

K
I

unfoldings

unfoldings

J

K

JK

reshaping

IK(b) Our method

full rank

IJ

N components{

full rankIJ

reorderingK

reshapingreshaping

reordering

reordering

&

&

m2

m1
n1

rank-1

mNn2
m1 n1= m2 n2=nN= mN nN=IJK

Figure Illustration of the difference with the ”latent” model.
Beyond Unfolding: Exact Recovery of Latent Convex Tensor Decomposition under Reshuffling(LiChao
et al,LiAAAI
2020)(RIKEN-AIP)
Mathematical Seminar
May 30, 201912 / 33
20

problem suchsensing, K
matrix/tensor
comAssume that theLinear-inverse
observation
X asœcompressed
RI1 ◊I2 ◊···◊I
is a mixtureof N compopletion and deconvolution.
Reshuffled
Decomposition
nents, thenthe recovery ofTensor
the originalcomponents can be formulated as
Note
Main
Results(Li et al, AAAI 2020)
tensor decomposition,i.e., of the solution is eliminated by imposing additional
In general, the ambiguity‣ Formulation

constraints, e.g. sparsity, low-rankness and smoothness.

X = R (A ) + R (A ) + · · · + R (A ).

(1)

Low-rank tensor
1 decomposition
12such as21 CP, Tucker, Tensor-TrainN
N(TR)and Tensor-Ring (TR) decomposition .

Additional Assumption

each component
where Ai , i œWe[N]further
denoteassume
latent thatcomponents(original A
images)andlow-rank
Ri de- si has the+ w.r.t.
+ operation+
notes the corresponding =reshufflingAi .
The latent components can be exactly recovered by solving t
NOTE
Andrzej Cichocki et al. Nonnegative matrix and tensor factorizations: applications toproblem:
exploratory multi-way data analysis and blind source separation. John Wiley & Sons,
NNÿÿ
Recovering A2009.
only
from
X
is
a
ill-posed
problem.
i
Optimization objective: minÎAi Îú , s.t., X =Ri (Ai ),observation

component 1

component 2

component N

1‣

Chao Li (RIKEN-AIP)

Mathematical Seminar

Ai , iœ[N]i=1

PurposeConditions for Exact Recovery

May 30, 201910 / 33

i=1

where
we employ the matrix nuclear norm Î · Îú in the model as
guarantee‣ Theoretical
The aim of our work is to study what the condition is for exact recoveryof the matrix rank.
Theorem
Condition)of thelatent(Exact-Recovery
components, andto develop an effective algorithm for thisdecomposition.
The estimated Âi , obtained by Reshuffled-TD, are equal to the true Aú forall i, when

Chao Li (RIKEN-AIP)i

1
max µi (Ai ) <,
i=1,...,N3N ≠ 2Mathematicalı Seminar

where N denotes the number of the components.
21

May 30, 2019(7)

3 / 33

Low-rank image steganography

Reshuffled Tensor Decomposition

In our study, we’ like tohide a full-size RGB images (e.g. 2K ◊ 2K ) into a grayscale image;choose the images from different fields, e.g. natural, cartoon;
try to give an invisible change on the cover images.
covercontainer

strength para.
secretour method
reconstruction

Random permutation

Fig. 14 Block diagram of the system scheme used in the experiment.
Chao Li (RIKEN-AIP)

Tensor Learning Unit

February 23, 2019

Application: Image steganography

22

21 / 30

Convex Formulation of TT Rank Minimization

On tensor train rank minimization: Statistical efficiency and scalable algorithm (Imaizumi, et al.,NeurIPS 2017)
23

,
iN

i1

T

i1

· · r· = rN= r.
r. According
1 =According
all I1 , I21, · ·2· , IN isNequal to I, and r1 = 1r2==r2· ·=·=equa-equaN 1
20 and15,wecomputational
complexityof threeour threealgorithms
tiontion
10, 10,20 and15,welist list
the thecomputational
complexityof ouralgorithms

x

(1)
(2)
(N )
10: 10:
Output:G·is· ·, equal,· G· · , G .(NallOutput:
I , I G, · ·G·, (1), I, G, (2)to) . I, and r···

I ⇥I ⇥···⇥I

x

in

2
N
For
tensorR 21⇥···⇥I
numberof observed
entries
M assume, we assume
9: 9:
End
while
N
End
while
For
tensor
X 2XR2I1 ⇥I
withwith
numberof observed
entries
M , we

!

n

···

i

3.4. Computational (n)Complexity

Computational
Complexity
8:3.4.
Updatecorresponding
GimG(n)bydescent
method.
8:
Updatecorrespondingby gradientdescent
method.
m gradient

,8

G
ra
or
4
du
W
L
S
at
ch
rit
3 ea
eS
eh
oo Sc rn
in
er
l
h
@
of oo ge:
@G f
Co l o Un
1, in(n) =ifm,8 ···
t
2
A
(
p
x
Te
x ,N
1
A
ut
u
i1
i1
B
t
G
n
o
···
···
O
e
.
ST
s
r
r
i
i
or
adS
N
N
O (N
4 R
,
cai
n
W
u
L
A
S
y
· · (I N r N
eech CT 3 ea
=
i1
rit
t
·
·
2
)
1, e h ··ioo Sc rn
)
N
··
in
er
)
yi
l
h
@
·
of oo g
e:
@G f
1·
,
N
Q
· ·i
Co l o Un
1, in(n) = N .
k=
N
fA
n+
m
,8 ···(
pu u
x
1 GA (k
x ,N
i1
i1
)
ik B
te t
···
···
O
.
S
n
rS
iN
iN
TR Q 1
O (N
,n
ci
k= A
(
r
y
· · I Nx N
=
G( !
i1
1 C
·

7: 7:
End
End

N N
1 1

2

for forevery
iteration
in table
1. 1.
Thoughthethetime
complexity
willwill
exponentiallyevery
iteration
in tableThoughtime
complexity
exponentially

)

,8

Table
1: Computational
complexityof TTWOPT,STTOPT,TTSGD
for every
iteration
Table
1: Computational
complexityof TTWOPT,STTOPT,TTSGD
for every
iteration（Bengua, et al., ICSPCS 2016)
1)

O(I
O(I N
+ rN2 I+Nr 21I)N

1)

O(MO(MIr) Ir)N 1N +2 I N2 1
NI N1 I 1
Nr+r I N) 1 )i2 =2i==34O(ri21i=2TTWOPT
1=iTTWOPT
2 3i
i2 2=O(ri2 =4 N) 1 ) O(IO(I
2 =
2

O(M
Ir) Ir)
O(M
O(Ir)
O(Ir)

11 11

Higher-order tensor)

Experiments
4. 4.
Experiments···

OneOneadvantageof gradient-based
optimizationis that
we do
too tooadvantageof gradient-based
optimizationis that
we notdo neednot need
itune
i
=
2
i
=
3
i
=
4
i2 =
=
2
i
=
3
i
=
4
2 1=i21222
2
2
so many
hyper
parameters,
we caneasilyget any
wantedaccuracy
withintune
so many
hyper
parameters,
we caneasilyget any
wantedaccuracy
within(2

N 1N) 1 )
O(rO(r

)4. 4.
Experiments
ExperimentsO(Mr N r1N) 1 )
O(M

k=
n+

TTSGD
TTSGD

)

STTOPTSTTOPT

Approximated entries

1

QN

Algorithm
complexity
complexity
AlgorithmTime
Time
complexitySpaceSpace
complexity

ik

TTSGD
O(r1 N
O(Ir)
TTSGD
O(rN
) 1)
O(Ir)
Table
1: Computational
complexityof TTWOPT,STTOPT,
TTSGD
for forevery
iteration
Table
1: Computational
complexityof TTWOPT,STTOPT,
TTSGD
every
iteration

(1

1)

)

N

STTOPT O(MO(MSTTOPTr N 1r)NG (kN
1 NN 1 I
1)TTWOPTO(r I
TTWOPT
complexity
andandspace
complexity.
complexity
space
complexity. O(riNadvantageof gradient-based
optimizationis thatneed
OneOneadvantageof gradient-based
optimizationis thatwe we
do donotnotneedtootoo,N.

tune
so many
hyper
parameters,
easilywantedaccuracy
withintune
so many
hyper
parameters,
we wecancaneasilygetgetanyanywantedaccuracy
within

P

n=

1

)
N
= YN

Recovered data

=

(x

f y
= i1 1···
2 kyiN

i1

···

iN

)i1 ·

TT/TR decomposition

· ·i

2

Low-rank TT/TR approximation

ion
t
c
i
red

G (n
in )
.

1,
·

··

or

11 11

G
x
m
ns 1 i m(n)i1···or n ,
y
x
iN
m
G
a
i
k2
<
1·
)
er s a(f·n
G > ·iN F .
at sp inm
=
ed a
1
=
n
, a . rse @ f nQ 1 2 k inm G = YN<
nd W @ Gte n
y
nn
i
m
e
= G
i1
n
(
n
a
n
) T=1 G (n···or QNrr ar i m s)o =1
(n
iN
n= e
inm )
an ran r,
in )
,
n
t
(
.
o
.
nt c n+en Gge ge nlxm If
xentio 1 so (nth a y t
w
i1
son rinsm ) e a ll he y e
···
t
,
r
i
c
iN
(3
o
c
h
o
m af n G c e o m

2Algorithm
Timecomplexity
Space
complexity
Algorithm
Timecomplexity
Space
complexity
so they
cancandealdeal
withlarge-scale
data.
Besides,
TTSGD
uses
thetheleast
time
so they
withlarge-scale
data.
Besides,
TTSGD
uses
least
time

High-ordertensorization

Observed data1

increase
by bydata
dimensions,
STTOPT
andandTTSGD
is freefromdimensionality
increasedata
dimensions,
STTOPT
TTSGD
is freefromdimensionality

Te
ns

Observed entries

ik

1 2

G (k

1

1

N N

k=

2

nQ

2
1

k)
iT
k

1

1

and
space
complexity.
and
space
complexity.
tiontion
10,complexity
20complexity
and
15,we
list
the
computational
complexityof ourthree
algorithms
10,
20 and
15,we
list
the
computational
complexityof ourthree
algorithms

T

Tensor Ring with Low-rank Cores
for
every
iteration
in table
1. Thoughcomplexity
exponentially
for
every
iteration
in table
1. Thoughthe thetimetime
complexity
will will
exponentially3.4.3.4.
Computational
Complexity
Computational
Complexity
increasebydata
dimensions,
STTOPT
and
TTSGD
is
free
from
dimensionality⇥I⇥···⇥IN N STTOPT and TTSGD is free from dimensionality
I12⇥I
2 ⇥···⇥I
dimensions,
Forincrease
tensor
Xby2
R2I1R
with
numberof observed
entriesMM, we, weassumeFortensorXdata
withnumberof observed
entriesassume
so
they
can
deal
with
large-scale
data.
Besides,
TTSGD
uses
the
least
so
they
can
deal
with
large-scale
data.
Besides,
TTSGD
uses
the
least
timetime
all all
I , I , I· · ,· ·,·I· , Iis equal
to I,
r =
· · ·=
Accordingequais equal
to andI, andr r= =r ·=· · r= r = r.
= r.
According
equa-

inm )

(n

@G

@f

‣ High computational efficiency, rank-robustness, theoretical support
n+

24

QN

L. Yuan et al. , “Tensor ring decomposition with rank minimization on latent space: An efficient approach fortensor completion ”, (AAAI 2019)

What Is Tensor Network?

powerful tool to describe strongly entangled‣ Aquantum
many-body systems in physics
of N-order tensor as contractions of‣ Approximation
O(N) smaller tensors

25Tensor Network DiagramJ<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>II(1 ⇥ 1)(I ⇥ 1)(I ⇥ J)

ScalarVectorMatrix<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="EZEmgoPYGnSmLv9tAdgJRCcw6xM=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjcsK9gHToWTSTBuaTIbkjlCGrvwGt7p2J279Epf+iWk7C9t6IHA4517OzQkTwQ247reztr6xubVd2Cnu7u0fHJaOjltGpZqyJlVC6U5IDBM8Zk3gIFgn0YzIULB2OLqb+u0npg1X8SOMExZIMoh5xCkBK/kVrwtcMoO9i16p7FbdGfAq8XJSRjkavdJPt69oKlkMVBBjfM9NIMiIBk4FmxS7qWEJoSMyYL6lMbE5QTY7eYLPrdLHkdL2xYBn6t+NjEhjxjK0k5LA0Cx7U/E/z08hugkyHicpsJjOg6JUYFB4+n/c55pREGNLCNXc3orpkGhCwba0kBLKie3EW25glbRqVe+yWnu4Ktdv83YK6BSdoQry0DWqo3vUQE1EkUIv6BW9Oc/Ou/PhfM5H15x85wQtwPn6BYuultM=</latexit><latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="CdDEFOmD07BOoEI/ujiP3GVwNE8=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosutFdBfuA6VAyaaYNTTJDkhFK6cpvcKtrd+LWL3Hpn5i2s7CtBwKHc+7l3Jww4Uwb1/12cmvrG5tb+e3Czu7e/kHx8Kip41QR2iAxj1U7xJpyJmnDMMNpO1EUi5DTVji8nfqtJ6o0i+WjGSU0ELgvWcQINlbyy/cdwwTVyDvvFktuxZ0BrRIvIyXIUO8Wfzq9mKSCSkM41tr33MQEY6wMI5xOCp1U0wSTIe5T31KJbU4wnp08QWdW6aEoVvZJg2bq340xFlqPRGgnBTYDvexNxf88PzXRdTBmMkkNlWQeFKUcmRhN/496TFFi+MgSTBSztyIywAoTY1taSAnFxHbiLTewSprVindRqT5clmo3WTt5OIFTKIMHV1CDO6hDAwjE8AKv8OY8O+/Oh/M5H8052c4xLMD5+gWyTpbr</latexit><latexit sha1_base64="iPTNFxUONcbVk7ikbpNWDPrDMUc=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjbqqYB8wHUomTdvQZDIkd4QydOU3uNW1O3Hrl7j0T0zbWdjWA4HDOfdybk4YC27Adb+dldW19Y3N3FZ+e2d3b79wcNgwKtGU1akSSrdCYpjgEasDB8FasWZEhoI1w+HNxG8+MW24ih5hFLNAkn7Ee5wSsJJfumsDl8zg+7NOoeiW3SnwMvEyUkQZap3CT7uraCJZBFQQY3zPjSFIiQZOBRvn24lhMaFD0me+pRGxOUE6PXmMT63SxT2l7YsAT9W/GymRxoxkaCclgYFZ9Cbif56fQO8qSHkUJ8AiOgvqJQKDwpP/4y7XjIIYWUKo5vZWTAdEEwq2pbmUUI5tJ95iA8ukUSl75+XKw0Wxep21k0PH6ASVkIcuURXdohqqI4oUekGv6M15dt6dD+dzNrriZDtHaA7O1y/ZwpcE</latexit>IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

K

J<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

Ik<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>

I<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="dnHEp4lgdA4li6dUWQoii/vavnY=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e98f9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX8uAlEU=</latexit>

I1

I1

IN

<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit><latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>N<latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>(I ⇥ J ⇥ K)

(I1 ⇥ I2 ⇥ · · · ⇥ IN )3-order Tensor

N-order Tensor<latexit sha1_base64="2dY5dknUY7GVUPyF4XCBF6lv4hM=">AAACC3icbVDLSgMxFM34rPU12qWbYBHqpsxUQZdFNz42FewD2qFk0kwbmskMyR1hKP0Ev8Gtrt2JWz/CpX9i2s7Cth4IOZxzL+dy/FhwDY7zba2srq1vbOa28ts7u3v79sFhQ0eJoqxOIxGplk80E1yyOnAQrBUrRkJfsKY/vJ74zSemNI/kI6Qx80LSlzzglICRunahdNsBHjKN77L//rRrF52yMwVeJm5GiihDrWv/dHoRTUImgQqiddt1YvBGRAGngo3znUSzmNAh6bO2oZKYHG80PX6MT4zSw0GkzJOAp+rfjREJtU5D30yGBAZ60ZuI/3ntBIJLb8RlnACTdBYUJAJDhCdN4B5XjIJIDSFUcXMrpgOiCAXT11yKH45NJ+5iA8ukUSm7Z+XKw3mxepW1k0NH6BiVkIsuUBXdoBqqI4pS9IJe0Zv1bL1bH9bnbHTFynYKaA7W1y86S5pu</latexit><latexit sha1_base64="RlY4qj8aJU3bHXR0cym/vC92qLw=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQN2WmCrosurEbqWAf0JYhk8m0oZnMkNwRSukX+BF+g1tduxO34tI/MX0g2nogcDjnXs7N8RPBNTjOp5VZWV1b38hu5ra2d3b37P2Dho5TRVmdxiJWLZ9oJrhkdeAgWCtRjES+YE1/cDXxm/dMaR7LOxgmrBuRnuQhpwSM5NmFYtVzO8AjpnHVK89YhwYx6B/15sSz807JmQIvE3dO8miOmmd/dYKYphGTQAXRuu06CXRHRAGngo1znVSzhNAB6bG2oZKYpO5o+p0xLhglwGGszJOAp+rvjRGJtB5GvpmMCPT1ojcR//PaKYQX3RGXSQpM0llQmAoMMZ50gwOuGAUxNIRQxc2tmPaJIhRMg39S/GhsOnEXG1gmjXLJPS2Vb8/ylct5O1l0hI5REbnoHFXQNaqhOqLoAT2hZ/RiPVqv1pv1PhvNWPOdQ/QH1sc3LISh5A==</latexit>

26

Tensor Network OperationsabA<latexit sha1_base64="94tL1q9O9cS77J172E010NqOwHQ=">AAAB/3icbVC7TsMwFL3hWcqrwMhiUSExVUlBgrHAwlgk+kBtVDmu01q1nch2kKqoA9/ACjMbYuVTGPkTnDYDbTmSpaNz7tU9PkHMmTau++2srK6tb2wWtorbO7t7+6WDw6aOEkVog0Q8Uu0Aa8qZpA3DDKftWFEsAk5bweg281tPVGkWyQczjqkv8ECykBFsrPTYFdgMgxBd90plt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPEGnVumjMFL2SYOm6t+NFAutxyKwk1lAvehl4n9eJzHhlZ8yGSeGSjI7FCYcmQhlv0d9pigxfGwJJorZrIgMscLE2I7mrgRiYjvxFhtYJs1qxTuvVO8vyrWbvJ0CHMMJnIEHl1CDO6hDAwgIeIFXeHOenXfnw/mcja44+c4RzMH5+gUI7Zad</latexit>

T

a b

I
<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>A<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="PPCkjOkrnSVQ1Z4QUirmGsykYQg=">AAACC3icbVDLSsNAFL3xWesr2qWbwSK4KkkVdFnrxmUF+4A2lMl00g6dScLMRAiln+A3uNW1O3HrR7j0T5y0EWzrgQuHc+7lHo4fc6a043xZa+sbm1vbhZ3i7t7+waF9dNxSUSIJbZKIR7LjY0U5C2lTM81pJ5YUC5/Ttj++zfz2I5WKReGDTmPqCTwMWcAI1kbq26WewHrkB+gG/bJ63y47FWcGtErcnJQhR6Nvf/cGEUkEDTXhWKmu68Tam2CpGeF0WuwlisaYjPGQdg0NsaDKm8zCT9GZUQYoiKSZUKOZ+vdigoVSqfDNZhZQLXuZ+J/XTXRw7U1YGCeahmT+KEg40hHKmkADJinRPDUEE8lMVkRGWGKiTV8LX3wxNZ24yw2skla14l5UqveX5Vo9b6cAJ3AK5+DCFdTgDhrQBAIpPMMLvFpP1pv1bn3MV9es/KYEC7A+fwBbzZqC</latexit>

X<latexit sha1_base64="KRj5Yh3W8hsYBioRpujyXxO60FU=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkJE+uWKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE+c5a/</latexit>

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>a⌦b<latexit sha1_base64="M//2C6WLxtNTfHKeYcKIzygckwU=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjeKmgn3AdCiZNG1Dk8mQ3BHK0JXf4FbX7sStX+LSPzFtZ2FbDwQO59zLuTlhLLgB1/12VlbX1jc2c1v57Z3dvf3CwWHDqERTVqdKKN0KiWGCR6wOHARrxZoRGQrWDIc3E7/5xLThKnqEUcwCSfoR73FKwEp+6a4NXDKD7886haJbdqfAy8TLSBFlqHUKP+2uoolkEVBBjPE9N4YgJRo4FWycbyeGxYQOSZ/5lkbE5gTp9OQxPrVKF/eUti8CPFX/bqREGjOSoZ2UBAZm0ZuI/3l+Ar2rIOVRnACL6CyolwgMCk/+j7tcMwpiZAmhmttbMR0QTSjYluZSQjm2nXiLDSyTRqXsnZcrDxfF6nXWTg4doxNUQh66RFV0i2qojihS6AW9ojfn2Xl3PpzP2eiKk+0coTk4X7/bVpcF</latexit><latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>J

X ⇥1 a ⇥ 2 b ⇥ 3 c<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit><latexit sha1_base64="Mdj9E+YBNnyEN8zwAEnvpiuL0nQ=">AAACN3icbZDLSsNAFIYn9VbrLerSzWARXJWkFXRn0Y3LCvYCbQiT6aQdOpOEmYlQQh7Hh/AZ3OrClbgRt76BkzQF23pg4Oc75/Cf+b2IUaks690ora1vbG6Vtys7u3v7B+bhUUeGscCkjUMWip6HJGE0IG1FFSO9SBDEPUa63uQ263cfiZA0DB7UNCIOR6OA+hQjpZFrXg84UmOMWNJLB4pyIl07R54PUQHqcE68gjTmALtm1apZecFVYReiCopquebnYBjimJNAYYak7NtWpJwECUUxI2llEEsSITxBI9LXMkDaz0nyj6bwTJMh9EOhX6BgTv9uJIhLOeWenswOlMu9DP7X68fKv3ISGkSxIgGeGfkxgyqEWWpwSAXBik21QFhQfSvEYyQQVjrbBRePpzoTezmBVdGp1+xGrX5/UW3eFOmUwQk4BefABpegCe5AC7QBBk/gBbyCN+PZ+DC+jO/ZaMkodo7BQhk/v61NrX4=</latexit><latexit sha1_base64="4tiu7aGizBnvfz6YaX5e2UsfHTU=">AAACE3icbVDLSgMxFM3UV62vUZfdBIvgqsxUQZdFNy4r2Ae0Q0nSTBuaZIYkI5ShCz/Cb3Cra3fi1g9w6Z+YaUewrQcCh3Pu5dwcHHOmjed9OYW19Y3NreJ2aWd3b//APTxq6ShRhDZJxCPVwUhTziRtGmY47cSKIoE5bePxTea3H6jSLJL3ZhLTQKChZCEjyFip75Z7ApkRDiGCvcgwQTX8VXDfrXhVbwa4SvycVECORt/97g0ikggqDeFI667vxSZIkTKMcDot9RJNY0TGaEi7lkpk44J09okpPLXKAIaRsk8aOFP/bqRIaD0R2E5mB+plLxP/87qJCa+ClMk4MVSSeVCYcGgimDUCB0xRYvjEEkQUs7dCMkIKEWN7W0jBYmo78ZcbWCWtWtU/r9buLir167ydIiiDE3AGfHAJ6uAWNEATEPAInsELeHWenDfn3fmYjxacfOcYLMD5/AH7/Z4f</latexit>(I ⇥ J)

J<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>cX<latexit sha1_base64="KRj5Yh3W8hsYBioRpujyXxO60FU=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkJE+uWKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE+c5a/</latexit><latexit sha1_base64="iPTNFxUONcbVk7ikbpNWDPrDMUc=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjbqqYB8wHUomTdvQZDIkd4QydOU3uNW1O3Hrl7j0T0zbWdjWA4HDOfdybk4YC27Adb+dldW19Y3N3FZ+e2d3b79wcNgwKtGU1akSSrdCYpjgEasDB8FasWZEhoI1w+HNxG8+MW24ih5hFLNAkn7Ee5wSsJJfumsDl8zg+7NOoeiW3SnwMvEyUkQZap3CT7uraCJZBFQQY3zPjSFIiQZOBRvn24lhMaFD0me+pRGxOUE6PXmMT63SxT2l7YsAT9W/GymRxoxkaCclgYFZ9Cbif56fQO8qSHkUJ8AiOgvqJQKDwpP/4y7XjIIYWUKo5vZWTAdEEwq2pbmUUI5tJ95iA8ukUSl75+XKw0Wxep21k0PH6ASVkIcuURXdohqqI4oUekGv6M15dt6dD+dzNrriZDtHaA7O1y/ZwpcE</latexit><latexit sha1_base64="gJU+xKS8gcd+TGzfn1VAZmamC10=">AAACAXicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmUF+4DpUDJppg1NMkOSEcrQld/gVtfuxK1f4tI/MdPOwrYeCBzOuZd7csKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMnvCWxGBPOsO+1Xqm7NnQGtEq8gVSjQ7Fd+eoOYpIJKQzjW2vfcxAQZVoYRTqflXqppgskYD6lvqcSC6iCbRZ6ic6sMUBQr+6RBM/XvRoaF1hMR2sk8ol72cvE/z09NdBNkTCapoZLMD0UpRyZG+f/RgClKDJ9YgoliNisiI6wwMbalhSuhyDvxlhtYJe16zbus1R+uqo3bop0SnMIZXIAH19CAe2hCCwjE8AKv8OY8O+/Oh/M5H11zip0TWIDz9Qt3B5gI</latexit>

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

BI(1 ⇥ 1 ⇥ 1)<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>a<latexit sha1_base64="xPJ872JvyatRpZQAUPNv3i1Wxnc=">AAACC3icbVDLSsNAFJ3UV62vaJduBotQNyWpgi6LblxWsA9oQ5lMJ+3QmSTM3Agh9BP8Bre6didu/QiX/onTNoi2HhjmcM69nMvxY8E1OM6nVVhb39jcKm6Xdnb39g/sw6O2jhJFWYtGIlJdn2gmeMhawEGwbqwYkb5gHX9yM/M7D0xpHoX3kMbMk2QU8oBTAkYa2OWq2wcumcY//9nArjg1Zw68StycVFCO5sD+6g8jmkgWAhVE657rxOBlRAGngk1L/USzmNAJGbGeoSExOV42P36KT40yxEGkzAsBz9XfGxmRWqfSN5OSwFgvezPxP6+XQHDlZTyME2AhXQQFicAQ4VkTeMgVoyBSQwhV3NyK6ZgoQsH09SfFl1PTibvcwCpp12vuea1+d1FpXOftFNExOkFV5KJL1EC3qIlaiKIUPaFn9GI9Wq/Wm/W+GC1Y+U4Z/YH18Q3BmJoj</latexit><latexit sha1_base64="lARkly2hBl77atLgnGdkxY/CRV8=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKE++WKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE7TZa9</latexit>A<latexit sha1_base64="epGFPGJ1Wl8nOqnfv7WyY0wMIF8=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdVNy4r2AdMh5JJ0zY0mQxJRihDV36DW127E7d+iUv/xEw7C9t6IHA4517uyQljzrRx3W+nsLa+sblV3C7t7O7tH5QPj1paJorQJpFcqk6INeUsok3DDKedWFEsQk7b4fgu89tPVGkmo0cziWkg8DBiA0awsZLfFdiMCObpzbRXrrhVdwa0SrycVCBHo1f+6fYlSQSNDOFYa99zYxOkWBlGOJ2WuommMSZjPKS+pREWVAfpLPIUnVmljwZS2RcZNFP/bqRYaD0RoZ3MIuplLxP/8/zEDK6DlEVxYmhE5ocGCUdGouz/qM8UJYZPLMFEMZsVkRFWmBjb0sKVUGSdeMsNrJJWrepdVGsPl5X6bd5OEU7gFM7Bgyuowz00oAkEJLzAK7w5z8678+F8zkcLTr5zDAtwvn4BUruX8Q==</latexit><latexit sha1_base64="futOwDfv0EATwUk9NtBVM1D4uvk=">AAAB/3icbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLUjcsK9iHtUDJppg1NMkOSEcrQhd/gVtfuxK2f4tI/MdPOwrYeCBzOuZd7coKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Ft5refqNIskg9mElNf4KFkISPYWOmxJ7AZBSGq90tlt+LOgFaJl5My5Gj0Sz+9QUQSQaUhHGvd9dzY+ClWhhFOp8VeommMyRgPaddSiQXVfjoLPEXnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhjZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1qxbusVO+vyrV63k4BTuEMLsCDa6jBHTSgCQQEvMArvDnPzrvz4XzOR9ecfOcEFuB8/QIKgJae</latexit>

J<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>N<latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>

X ⇥31 A ⇥2 B ⇥3 c

A ⇥ 1 b 1 ⇥ · · · ⇥N b N

N-order<latexit sha1_base64="BkSzgrl85rludy/ChhGSAiJ4l1A=">AAACNXicbVDLSsNAFJ3UV62vqks3g0VwVZIq6EaounFVKthaaEKZTCft0MmDmRuhhH6NH+E3uNWVCxeCuPUXnKQBbeuBgcM593LuHDcSXIFpvhmFpeWV1bXiemljc2t7p7y711ZhLClr0VCEsuMSxQQPWAs4CNaJJCO+K9i9O7pO/fsHJhUPgzsYR8zxySDgHqcEtNQrX9g+gSElIrmc2MB9pnpWJrkedjXNJJv2Q1C53fi1G71yxayaGfAisXJSQTmavfKH3Q9p7LMAqCBKdS0zAichEjgVbFKyY8UiQkdkwLqaBkQnOkn2zQk+0kofe6HULwCcqX83EuIrNfZdPZmeqOa9VPzP68bgnTsJD6IYWECnQV4sMIQ47Qz3uWQUxFgTQiXXt2I6JJJQ0M3OpLj+RHdizTewSNq1qnVSrd2eVupXeTtFdIAO0TGy0BmqoxvURC1E0SN6Ri/o1Xgy3o1P42s6WjDynX00A+P7B8nIrSM=</latexit><latexit sha1_base64="Xeo+HX1fxYI8BSjcjkm/MnhKzUs=">AAACPnicbZDLSsNAFIYn9VbrLerSzWARXJWkFXRZK4jLCvYCTQyT6aQdOrkwMxFKyBv5ED6DO9GNG3fi1qWTNhXbemDg5zvn8J/53YhRIQ3jRSusrK6tbxQ3S1vbO7t7+v5BW4Qxx6SFQxbyrosEYTQgLUklI92IE+S7jHTc0VXW7zwQLmgY3MlxRGwfDQLqUYykQo5+bflIDjFiSTe1JPWJcMz7GpxRmFzOcHUKXQ82clL7JdjRy0bFmBRcFmYuyiCvpqO/W/0Qxz4JJGZIiJ5pRNJOEJcUM5KWrFiQCOERGpCekgFShnYy+W8KTxTpQy/k6gUSTujfjQT5Qox9V01mB4rFXgb/6/Vi6V3YCQ2iWJIAT428mEEZwiw82KecYMnGSiDMqboV4iHiCEsV8ZyL66cqE3MxgWXRrlbMWqV6e1auN/J0iuAIHINTYIJzUAc3oAlaAINH8AxewZv2pH1on9rXdLSg5TuHYK607x9ebq+1</latexit>(M ⇥ L ⇥ N )I<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>

··· I

I1

<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit><latexit sha1_base64="ivNZUtdnuq0ySbBM5e8K0dntoiE=">AAACC3icbVDLSgMxFM34rPU12qWbYBHqpsxUQZdFNy5UKtgHtEPJpJk2NJMZkjvCUPoJfoNbXbsTt36ES//EtJ2FbT0QcjjnXs7l+LHgGhzn21pZXVvf2Mxt5bd3dvf27YPDho4SRVmdRiJSLZ9oJrhkdeAgWCtWjIS+YE1/eD3xm09MaR7JR0hj5oWkL3nAKQEjde1C6a4DPGQa32b//WnXLjplZwq8TNyMFFGGWtf+6fQimoRMAhVE67brxOCNiAJOBRvnO4lmMaFD0mdtQyUxOd5oevwYnxilh4NImScBT9W/GyMSap2GvpkMCQz0ojcR//PaCQSX3ojLOAEm6SwoSASGCE+awD2uGAWRGkKo4uZWTAdEEQqmr7kUPxybTtzFBpZJo1J2z8qVh/Ni9SprJ4eO0DEqIRddoCq6QTVURxSl6AW9ojfr2Xq3PqzP2eiKle0U0Bysr19Iz5p3</latexit>

A

···

N

<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>L<latexit sha1_base64="vSd1wp8j1e0s+mXgeoiWMaOjNFc=">AAACAHicbVC7TsMwFL3hWcqrwMhiUSExVUlBgrHAwlgk+hBtVDmu01q1nch2kKqoC9/ACjMbYuVPGPkTnDYDbTmSpaNz7tU9PkHMmTau++2srK6tb2wWtorbO7t7+6WDw6aOEkVog0Q8Uu0Aa8qZpA3DDKftWFEsAk5bweg281tPVGkWyQczjqkv8ECykBFsrPTYFdgMCeboulcquxV3CrRMvJyUIUe9V/rp9iOSCCoN4VjrjufGxk+xMoxwOil2E01jTEZ4QDuWSiyo9tNp4gk6tUofhZGyTxo0Vf9upFhoPRaBncwS6kUvE//zOokJr/yUyTgxVJLZoTDhyEQo+z7qM0WJ4WNLMFHMZkVkiBUmxpY0dyUQE9uJt9jAMmlWK955pXp/Ua7d5O0U4BhO4Aw8uIQa3EEdGkBAwgu8wpvz7Lw7H87nbHTFyXeOYA7O1y/VQ5cP</latexit>(I ⇥ K)<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit><latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>b<latexit sha1_base64="gJU+xKS8gcd+TGzfn1VAZmamC10=">AAACAXicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmUF+4DpUDJppg1NMkOSEcrQld/gVtfuxK1f4tI/MdPOwrYeCBzOuZd7csKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMnvCWxGBPOsO+1Xqm7NnQGtEq8gVSjQ7Fd+eoOYpIJKQzjW2vfcxAQZVoYRTqflXqppgskYD6lvqcSC6iCbRZ6ic6sMUBQr+6RBM/XvRoaF1hMR2sk8ol72cvE/z09NdBNkTCapoZLMD0UpRyZG+f/RgClKDJ9YgoliNisiI6wwMbalhSuhyDvxlhtYJe16zbus1R+uqo3bop0SnMIZXIAH19CAe2hCCwjE8AKv8OY8O+/Oh/M5H11zip0TWIDz9Qt3B5gI</latexit><latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>

I

K

I

<latexit sha1_base64="CdDEFOmD07BOoEI/ujiP3GVwNE8=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosutFdBfuA6VAyaaYNTTJDkhFK6cpvcKtrd+LWL3Hpn5i2s7CtBwKHc+7l3Jww4Uwb1/12cmvrG5tb+e3Czu7e/kHx8Kip41QR2iAxj1U7xJpyJmnDMMNpO1EUi5DTVji8nfqtJ6o0i+WjGSU0ELgvWcQINlbyy/cdwwTVyDvvFktuxZ0BrRIvIyXIUO8Wfzq9mKSCSkM41tr33MQEY6wMI5xOCp1U0wSTIe5T31KJbU4wnp08QWdW6aEoVvZJg2bq340xFlqPRGgnBTYDvexNxf88PzXRdTBmMkkNlWQeFKUcmRhN/496TFFi+MgSTBSztyIywAoTY1taSAnFxHbiLTewSprVindRqT5clmo3WTt5OIFTKIMHV1CDO6hDAwjE8AKv8OY8O+/Oh/M5H8052c4xLMD5+gWyTpbr</latexit>cb<latexit sha1_base64="JtahwqYLiJdwEJa47CBn25f/Th8=">AAACAHicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ae2Q8mkmTY0yQxJRihDN36DW127E7f+iUv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKEUb9ccavuDGiVeDmpQI5Gv/zTG0QkEVQawrHWXc+NjZ9iZRjhdFrqJZrGmIzxkHYtlVhQ7aezxFN0ZpUBCiNlnzRopv7dSLHQeiICO5kl1MteJv7ndRMTXvspk3FiqCTzQ2HCkYlQ9n00YIoSwyeWYKKYzYrICCtMjC1p4UogprYTb7mBVdKqVb2Lau3+slK/ydspwgmcwjl4cAV1uIMGNIGAhBd4hTfn2Xl3PpzP+WjByXeOYQHO1y+V/5bn</latexit>

AB<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>(I ⇥ 1)I<latexit sha1_base64="EZEmgoPYGnSmLv9tAdgJRCcw6xM=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjcsK9gHToWTSTBuaTIbkjlCGrvwGt7p2J279Epf+iWk7C9t6IHA4517OzQkTwQ247reztr6xubVd2Cnu7u0fHJaOjltGpZqyJlVC6U5IDBM8Zk3gIFgn0YzIULB2OLqb+u0npg1X8SOMExZIMoh5xCkBK/kVrwtcMoO9i16p7FbdGfAq8XJSRjkavdJPt69oKlkMVBBjfM9NIMiIBk4FmxS7qWEJoSMyYL6lMbE5QTY7eYLPrdLHkdL2xYBn6t+NjEhjxjK0k5LA0Cx7U/E/z08hugkyHicpsJjOg6JUYFB4+n/c55pREGNLCNXc3orpkGhCwba0kBLKie3EW25glbRqVe+yWnu4Ktdv83YK6BSdoQry0DWqo3vUQE1EkUIv6BW9Oc/Ou/PhfM5H15x85wQtwPn6BYuultM=</latexit>aJ<latexit sha1_base64="ZrFty2l/Zyqi7tkI+zQEBL5il24=">AAACC3icbVDLSgMxFM3UV62v0S7dBIvgqsxUQZdVNy4r2Ae0Q0nSTBuaZIYkIwyln+A3uNW1O3HrR7j0T8y0I9jWAxcO59zLPRwcc6aN5305hbX1jc2t4nZpZ3dv/8A9PGrpKFGENknEI9XBSFPOJG0aZjjtxIoigTlt4/Ft5rcfqdIskg8mjWkg0FCykBFkrNR3yz2BzAiH8Br+Mtx3K17VmwGuEj8nFZCj0Xe/e4OIJIJKQzjSuut7sQkmSBlGOJ2WeommMSJjNKRdSyUSVAeTWfgpPLXKAIaRsiMNnKl/LyZIaJ0KbDezgHrZy8T/vG5iwqtgwmScGCrJ/FGYcGgimDUBB0xRYnhqCSKK2ayQjJBCxNi+Fr5gMbWd+MsNrJJWreqfV2v3F5X6Td5OERyDE3AGfHAJ6uAONEATEJCCZ/ACXp0n5815dz7mqwUnvymDBTifP44tmqI=</latexit>(1 ⇥ 1)<latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit><latexit sha1_base64="futOwDfv0EATwUk9NtBVM1D4uvk=">AAAB/3icbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLUjcsK9iHtUDJppg1NMkOSEcrQhd/gVtfuxK2f4tI/MdPOwrYeCBzOuZd7coKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Ft5refqNIskg9mElNf4KFkISPYWOmxJ7AZBSGq90tlt+LOgFaJl5My5Gj0Sz+9QUQSQaUhHGvd9dzY+ClWhhFOp8VeommMyRgPaddSiQXVfjoLPEXnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhjZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1qxbusVO+vyrV63k4BTuEMLsCDa6jBHTSgCQQEvMArvDnPzrvz4XzOR9ecfOcEFuB8/QIKgJae</latexit>

Ab<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit><latexit sha1_base64="ZTPE9JepZqhbQcCQ/sGsbjQq6lY=">AAACDnicbVDLSgMxFM3UV62vUcGNm2ARXJWZKuiy6MZlhb6gHUuSZtrQJDMkGaGM/Qe/wa2u3Ylbf8Glf2LajmBbDwROzrmXezg45kwbz/tyciura+sb+c3C1vbO7p67f9DQUaIIrZOIR6qFkaacSVo3zHDaihVFAnPaxMObid98oEqzSNbMKKaBQH3JQkaQsVLXPeoIZAY4hAje1+DvB3fdolfypoDLxM9IEWSodt3vTi8iiaDSEI60bvtebIIUKcMIp+NCJ9E0RmSI+rRtqUSC6iCd5h/DU6v0YBgp+6SBU/XvRoqE1iOB7eQkoF70JuJ/Xjsx4VWQMhknhkoyOxQmHJoITsqAPaYoMXxkCSKK2ayQDJBCxNjK5q5gMbad+IsNLJNGueSfl8p3F8XKddZOHhyDE3AGfHAJKuAWVEEdEPAInsELeHWenDfn3fmYjeacbOcQzMH5/AGO6Juy</latexit>bB<latexit sha1_base64="94tL1q9O9cS77J172E010NqOwHQ=">AAAB/3icbVC7TsMwFL3hWcqrwMhiUSExVUlBgrHAwlgk+kBtVDmu01q1nch2kKqoA9/ACjMbYuVTGPkTnDYDbTmSpaNz7tU9PkHMmTau++2srK6tb2wWtorbO7t7+6WDw6aOEkVog0Q8Uu0Aa8qZpA3DDKftWFEsAk5bweg281tPVGkWyQczjqkv8ECykBFsrPTYFdgMgxBd90plt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPEGnVumjMFL2SYOm6t+NFAutxyKwk1lAvehl4n9eJzHhlZ8yGSeGSjI7FCYcmQhlv0d9pigxfGwJJorZrIgMscLE2I7mrgRiYjvxFhtYJs1qxTuvVO8vyrWbvJ0CHMMJnIEHl1CDO6hDAwgIeIFXeHOenXfnw/mcja44+c4RzMH5+gUI7Zad</latexit><latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>

J<latexit sha1_base64="lARkly2hBl77atLgnGdkxY/CRV8=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKE++WKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE7TZa9</latexit><latexit sha1_base64="kQZKAm+Je6wlXWdcxpet4z5qQgM=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxsIiAfMByRH2NnPJkt29Y3dPOI78Alut7cTWn2PpP3GTXGESHww83pthZl4Qc6aN6347hY3Nre2d4m5pb//g8Kh8fNLWUaIotGjEI9UNiAbOJLQMMxy6sQIiAg6dYHI/8zvPoDSL5JNJY/AFGUkWMkqMlZqPg3LFrbpz4HXi5aSCcjQG5Z/+MKKJAGkoJ1r3PDc2fkaUYZTDtNRPNMSETsgIepZKIkD72fzQKb6wyhCHkbIlDZ6rfycyIrRORWA7BTFjverNxP+8XmLCWz9jMk4MSLpYFCYcmwjPvsZDpoAanlpCqGL2VkzHRBFqbDZLWwIxtZl4qwmsk3at6l1Va83rSv0uT6eIztA5ukQeukF19IAaqIUoAvSCXtGbkzrvzofzuWgtOPnMKVqC8/ULR1OTag==</latexit>

b1 b2

M

<latexit sha1_base64="hGnHz8MLv8rMHMwQ+qxmuoezNic=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDOZTYbMzi4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cEgiuDau++2srW9sbm2Xdsq7e/sHh5Wj47aOU0VZi8YiVl2CNRNcspbhRrBuohiOiGAdMr7L/c4TU5rH8tFMEhZEeCh5yCk2VvL9CJsRCTMy7df7lapbc2dAq8QrSBUKNPuVH38Q0zRi0lCBte55bmKCDCvDqWDTsp9qlmA6xkPWs1TiiOkgm2WeonOrDFAYKzvSoJn69yLDkdaTiNjNPKNe9nLxP6+XmvAmyLhMUsMknT8KU4FMjPIC0IArRo2YWIKp4jYroiOsMDW2poUvJJraTrzlBlZJu17zLmv1h6tq47ZopwSncAYX4ME1NOAemtACCgm8wCu8Oc/Ou/PhfM5X15zi5gQW4Hz9AvENmEU=</latexit><latexit sha1_base64="1Q4yvgXLVYzclbvyhhESC1z1g5A=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMQhWYTcKWgZtLCOYB2SXMDOZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cHDCmTae9+2srW9sbm2Xdsq7e/sHh5Wj47aOU0Voi8Q8Vl2MNOVM0pZhhtNuoigSmNMOHt/lfueJKs1i+WgmCQ0FGkoWMYKMlYJAIDPCUYanfb9fqXo1bwZ3lfgFqUKBZr/yEwxikgoqDeFI657vJSbMkDKMcDotB6mmCSJjNKQ9SyUSVIfZLPPUPbfKwI1iZUcad6b+vciQ0HoisN3MM+plLxf/83qpiW7CjMkkNVSS+aMo5a6J3bwAd8AUJYZPLEFEMZvVJSOkEDG2poUvWExtJ/5yA6ukXa/5l7X6w1W1cVu0U4JTOIML8OEaGnAPTWgBgQRe4BXenGfn3flwPuera05xcwILcL5+Ae96mEQ=</latexit><latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>‣ Basic linear and multilinear algebra operations‣ Convenient to denote complex operations
27

bN<latexit sha1_base64="zmlt1uF7jlS9gDkPqDQ/iydNTJA=">AAACAnicbVDLSgMxFM3UV62vqks3wSK4KjNV0GXRjSupYB/QGUqSZtrQJDMkGaEMs/Mb3Oranbj1R1z6J2baWdjWAxcO59zLPRwcc6aN6347pbX1jc2t8nZlZ3dv/6B6eNTRUaIIbZOIR6qHkaacSdo2zHDaixVFAnPaxZPb3O8+UaVZJB/NNKaBQCPJQkaQsZLvC2TGOExxNrgfVGtu3Z0BrhKvIDVQoDWo/vjDiCSCSkM40rrvubEJUqQMI5xmFT/RNEZkgka0b6lEguognWXO4JlVhjCMlB1p4Ez9e5EiofVUYLuZZ9TLXi7+5/UTE14HKZNxYqgk80dhwqGJYF4AHDJFieFTSxBRzGaFZIwUIsbWtPAFi8x24i03sEo6jbp3UW88XNaaN0U7ZXACTsE58MAVaII70AJtQEAMXsAreHOenXfnw/mcr5ac4uYYLMD5+gUdMJhh</latexit>(1 ⇥ · · · ⇥ 1)<latexit sha1_base64="u0qsZ+VivbajxBS4ja+M1rFHIB4=">AAACD3icbVDLSsNAFJ3UV62vqAsXbgaLUDclqYIui25cVrAPaEKZTCbt0MmDmRuhhHyE3+BW1+7ErZ/g0j9x2mZhWw8MHM65l3PneIngCizr2yitrW9sbpW3Kzu7e/sH5uFRR8WppKxNYxHLnkcUEzxibeAgWC+RjISeYF1vfDf1u09MKh5HjzBJmBuSYcQDTgloaWCe1GwHeMiUQ/0Y1Jxj+2JgVq26NQNeJXZBqqhAa2D+OH5M05BFQAVRqm9bCbgZkcCpYHnFSRVLCB2TIetrGhGd42azD+T4XCs+DmKpXwR4pv7dyEio1CT09GRIYKSWvan4n9dPIbhxMx4lKbCIzoOCVGCI8bQN7HPJKIiJJoRKrm/FdEQkoaA7W0jxwlx3Yi83sEo6jbp9WW88XFWbt0U7ZXSKzlAN2egaNdE9aqE2oihHL+gVvRnPxrvxYXzOR0tGsXOMFmB8/QLgKpxz</latexit>

ScalarTensor Network Models

28Tensor Train Decomposition(Oseledets, SIAM J. Sci. Comput. 2011)Χ ∈ ℝ-.×-0×⋯×-2" ($)

N-order
…

=

" (()
&(

&$

'$

Χ ∈ ℝ$%×$'×⋯×$)

0 (4)

82

×⋯ ×

82
94 -th!"# ,"% ,⋯,"'

,*

7;

85:2

8;:2

95 -th96 -th

85

,1

,/
×

0 (6)

×⋯ ×

84

,-% ,-' ,⋯,-/'575

92 -th

" (5)

0 (5)

74
×

&54$

&3

'3

0 (2)

=

=

&34$

'(

72

" (3)

×⋯ ×

,*

×⋯ ×
(1)

,-.*

(*)

("%

("0

("'

)#(+)
$

)#(.)
&

)#(0)
/

)#(-)
(

("#

(/)

,1.*

(-)

=
"#$ ,#& ,⋯,#(

29

Tensor Ring Decomposition(Zhao et al., arXiv 2016, ICASSP 2019),1 (-)

Χ ∈ ℝ&'×&)×⋯×&+

N-order
……

=

,.

4-

Χ ∈ ℝ$%×$'×⋯×$)

1 (/)

4
1 (0) 06-

4/6-

40

×

!"# ,"% ,⋯,"'

Trace

,.

:, -th

8.

(-)

,-

* (0)

70

×⋯×

8/;,

80;,

:/ -th:. -th

,*
×

("#

×⋯×

8,

,=

("(.)
-

* (/)

7/

7.

8,

12% ,2' ,⋯,24!"# ,"% ,⋯,"'

* (.)

89

("(/)
'

=

,0

* (,)
7,

=

("(,)
%

1 (.) 4
.

45

,/

("(*)
#

×⋯×
(*)

8/

89 :0 -th

,0

,.

,01-

×⋯×
(0)

("%

("/

30

,21(2)("'

Tensor Ring DecompositionIn(I1 ⇥ I2 ⇥ · · · ⇥ IN ).. . ...

<latexit sha1_base64="RlY4qj8aJU3bHXR0cym/vC92qLw=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQN2WmCrosurEbqWAf0JYhk8m0oZnMkNwRSukX+BF+g1tduxO34tI/MX0g2nogcDjnXs7N8RPBNTjOp5VZWV1b38hu5ra2d3b37P2Dho5TRVmdxiJWLZ9oJrhkdeAgWCtRjES+YE1/cDXxm/dMaR7LOxgmrBuRnuQhpwSM5NmFYtVzO8AjpnHVK89YhwYx6B/15sSz807JmQIvE3dO8miOmmd/dYKYphGTQAXRuu06CXRHRAGngo1znVSzhNAB6bG2oZKYpO5o+p0xLhglwGGszJOAp+rvjRGJtB5GvpmMCPT1ojcR//PaKYQX3RGXSQpM0llQmAoMMZ50gwOuGAUxNIRQxc2tmPaJIhRMg39S/GhsOnEXG1gmjXLJPS2Vb8/ylct5O1l0hI5REbnoHFXQNaqhOqLoAT2hZ/RiPVqv1pv1PhvNWPOdQ/QH1sc3LISh5A==</latexit><latexit sha1_base64="B3e7+lpSwHTmeIEAyPDu/F7BNWU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92W/XHGr7gxolXg5qUCORr/80xtEJBFUGsKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LZVYUO2ns1On6MwqAxRGypY0aKb+nUix0HoiAtspsBnpZS8T//O6iQmv/ZTJODFUkvmiMOHIRCj7Gw2YosTwiSWYKGZvRWSEFSbGprOwJRBTm4m3nMAqadWq3kW19nBZqd/k6RThBE7hHDy4gjrcQQOaQGAIL/AKb07qvDsfzue8teDkM8ewAOfrF9A5lEg=</latexit>r Gn r<latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit>...

<latexit

A<latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit sha1_base64="33X/Ozbrrw3EbiPWwTgMGrKHiD8=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GXVjcsK9gFNKJPppB06k4SZiVBCdn6DW127E7f+iEv/xEmbhW09cOFwzr3cw/FjzpS27W+rtLa+sblV3q7s7O7tH1QPjzoqSiShbRLxSPZ8rChnIW1rpjntxZJi4XPa9Sd3ud99olKxKHzU05h6Ao9CFjCCtZFcV2A9JpinNxkaVGt23Z4BrRKnIDUo0BpUf9xhRBJBQ004Vqrv2LH2Uiw1I5xmFTdRNMZkgke0b2iIBVVeOsucoTOjDFEQSTOhRjP170WKhVJT4ZvNPKNa9nLxP6+f6ODaS1kYJ5qGZP4oSDjSEcoLQEMmKdF8aggmkpmsiIyxxESbmha++CIznTjLDaySTqPuXNQbD5e15m3RThlO4BTOwYEraMI9tKANBGJ4gVd4s56td+vD+pyvlqzi5hgWYH39Aq6RmBs=</latexit>IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

I1

I2<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>=<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="1CH/E/k7vw4iob5ofCdhFnqOU70=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aQne6WyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXyUpjY</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

GN<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

G1IN<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>

G2 I2<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit><latexit sha1_base64="1h/TU896uDFwcMUPY8XDmR6Idps=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRha6kgn1AG8pkOmmHTiZxZiKUkKXf4FbX7sStH+LSP3HSZmFbD1w4nHMv93C8iDOlbfvbKqysrq1vFDdLW9s7u3vl/YOWCmNJaJOEPJQdDyvKmaBNzTSnnUhSHHictr3xdea3n6hULBQPehJRN8BDwXxGsDaS2wuwHhHMk5u0f9cvV+yqPQVaJk5OKpCj0S//9AYhiQMqNOFYqa5jR9pNsNSMcJqWerGiESZjPKRdQwUOqHKTaegUnRhlgPxQmhEaTdW/FwkOlJoEntnMQqpFLxP/87qx9i/dhIko1lSQ2SM/5kiHKGsADZikRPOJIZhIZrIiMsISE216mvviBanpxFlsYJm0alXnrFq7P6/Ur/J2inAEx3AKDlxAHW6hAU0g8Agv8Apv1rP1bn1Yn7PVgpXfHMIcrK9fv/KYuA==</latexit>

Ai1 ,...,iN = Tr<latexit sha1_base64="WfasgN5FCkhV4WO3X5Z4+iBhRg8=">AAACS3icbVBBaxNBGJ2NttbU1qhHL4OhEKGE3Vqol0JbD3oqEZs2kN0Os7PfJkNnZ5aZb6Vh2R/WH+EP8CS91bM38eAkzcG2Phh4vPc9vm9eWirpMAy/B61Hj1dWn6w9ba8/29h83nnx8tSZygoYCqOMHaXcgZIahihRwai0wItUwVl68WHun30F66TRJzgrISn4RMtcCo5eYp0vccFxKriqDxtWSxZtxyoz6LapZMcN3acxwiXWJ7aJFeTYi0trMlbr/ag5P15k05x+ZHosmU5iKydTfMs63bAfLkAfkmhJumSJAetcx5kRVQEaheLOjaOwxKTmFqVQ0LTjykHJxQWfwNhTzQtwSb34fEO3vJLR3Fj/NNKF+m+i5oVzsyL1k/Nz3X1vLv7PG1eYv09qqcsKQYvbRXmlKBo6b5Jm0oJANfOECyv9rVRMueUCfd93tqRF4zuJ7jfwkJzu9KN3/Z3Pu92Do2U7a+Q1eUN6JCJ75IB8IgMyJIJckR/khvwMvgW/gt/Bn9vRVrDMvCJ30Fr5C80LtGE=</latexit>NY

Gn [in ]n=1<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>‣ Highly compact representation
O(I ) ) O(N Ir )2N<latexit sha1_base64="i5PyY18Of5H/pIFlQi0z1Sm9HWc=">AAACAXicbVA9SwNBEJ3zM8avqKXNYhCswl0UtAzaWEYxH3A5wt5mL1mye3vs7inhSOVvsNXaTmz9JZb+EzfJFSbxwcDjvRlm5oUJZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODptaporQBpFcqnaINeUspg3DDKftRFEsQk5b4fBm4rceqdJMxg9mlNBA4H7MIkawsZLfuWf9gcFKyaduqexW3CnQMvFyUoYc9W7pp9OTJBU0NoRjrX3PTUyQYWUY4XRc7KSaJpgMcZ/6lsZYUB1k05PH6NQqPRRJZSs2aKr+nciw0HokQtspsBnoRW8i/uf5qYmugozFSWpoTGaLopQjI9Hkf9RjihLDR5Zgopi9FZEBVpgYm9LcllCMbSbeYgLLpFmteOeV6t1FuXadp1OAYziBM/DgEmpwC3VoAAEJL/AKb86z8+58OJ+z1hUnnzmCOThfv3TSmAc=</latexit><latexit sha1_base64="WgdJwn+nCPtIqydBVm+TOF18Oew=">AAACCnicbVDLSgMxFM34rPU11qWbYBHqpsxUQZdFN7rRCvYB7VgyadqGJpkhyYhlmD/wG9zq2p249Sdc+idm2lnY1gMXDufcyz0cP2RUacf5tpaWV1bX1nMb+c2t7Z1de6/QUEEkManjgAWy5SNFGBWkrqlmpBVKgrjPSNMfXaZ+85FIRQNxr8ch8TgaCNqnGGkjde1ChyM9xIjFt0np5lo+VI67dtEpOxPAReJmpAgy1Lr2T6cX4IgToTFDSrVdJ9RejKSmmJEk34kUCREeoQFpGyoQJ8qLJ9kTeGSUHuwH0ozQcKL+vYgRV2rMfbOZJlXzXir+57Uj3T/3YirCSBOBp4/6EYM6gGkRsEclwZqNDUFYUpMV4iGSCGtT18wXnyemE3e+gUXSqJTdk3Ll7rRYvcjayYEDcAhKwAVnoAquQA3UAQZP4AW8gjfr2Xq3PqzP6eqSld3sgxlYX78PA5pg</latexit><latexit sha1_base64="QQRPCsunN2kfSfGz5ClIGJRvbDg=">AAACCHicbVDLSgMxFM3UV62vUZdugkWomzJTBV0W3ehGK9gHtGPJpGkbmmSGJFMow/yA3+BW1+7ErX/h0j8x087Cth64cDjnXu7h+CGjSjvOt5VbWV1b38hvFra2d3b37P2DhgoiiUkdByyQLR8pwqggdU01I61QEsR9Rpr+6Dr1m2MiFQ3Eo56ExONoIGifYqSN1LXtDkd6iBGL75PS7dPdadcuOmVnCrhM3IwUQYZa1/7p9AIccSI0ZkiptuuE2ouR1BQzkhQ6kSIhwiM0IG1DBeJEefE0eQJPjNKD/UCaERpO1b8XMeJKTbhvNtOcatFLxf+8dqT7l15MRRhpIvDsUT9iUAcwrQH2qCRYs4khCEtqskI8RBJhbcqa++LzxHTiLjawTBqVsntWrjycF6tXWTt5cASOQQm44AJUwQ2ogTrAYAxewCt4s56td+vD+pyt5qzs5hDMwfr6BbYEmag=</latexit>‣ Circular permutation invariancee n ⇥ · · · ⇥ IN ⇥ I1 ⇥ · · · In 1 ) ) {Gn , . . . , GN , G1 , . . . , Gn 1 }A(I<latexit sha1_base64="0y2IbDyLwbHRSb3hrUI3aLBxCVA=">AAACSnicbVDLSsNAFJ3U97vq0s1gEXRhSVTQpY+NbkTFqtCUMJnc6tDJJMzcKCXkv/wIf8CN4ErX7sSN0zaIrwsD555zL2fuCVMpDLruo1MZGh4ZHRufmJyanpmdq84vXJgk0xwaPJGJvgqZASkUNFCghKtUA4tDCZdh56CnX96CNiJR59hNoRWzayXagjO0VFA98+9EBChkBLkfM7zhTOZ7RUFXjwLlo4jBUJ9HCZqyOQqOv5A3QAPd9rla94q1oFpz626/6F/glaBGyjoJqs9+lPAsBoVcMmOanptiK2caBZdQTPqZgZTxDruGpoWKWc9W3r+9oCuWiWg70fYppH32+0bOYmO6cWgne+eZ31qP/E9rZtjeaeVCpRmC4gOjdiYpJrQXJI2EBo6yawHjWti/Un7DNONo4/7hEsaFzcT7ncBfcLFR9zbrG6dbtd39Mp1xskSWySrxyDbZJYfkhDQIJ/fkibyQV+fBeXPenY/BaMUpdxbJj6oMfwJjCrQk</latexit><latexit sha1_base64="LBy+jxL/gy/TREWMf/c8LzdW3Js=">AAACVHicbZBNSwMxEIbTrR+1flU9egkWwUMtuyroUfSgJ6liVeiWkk3TNjSbLMmsUpb9c/4IwaM3r3r2YLbdQ1t9IfDmmRlmeINIcAOu+15wiguLS8ullfLq2vrGZmVr+8GoWFPWpEoo/RQQwwSXrAkcBHuKNCNhINhjMLzM6o/PTBuu5D2MItYOSV/yHqcELOpUfP+O9wdAtFYv2E/8kMCAEpFcpR1Zw77oKjC1aXpj6dTXq+U9MzSRh17qp51K1a27Y+G/xstNFeVqdCofflfROGQSqCDGtDw3gnZCNHAqWFr2Y8MiQoekz1rWShIy007GKaR435Iu7iltnwQ8ptMTCQmNGYWB7cxONfO1DP5Xa8XQO2snXEYxMEkni3qxwKBwFinucs0oiJE1hGpub8V0QDShYIOf2RKEWSbefAJ/zcNR3TuuH92eVM8v8nRKaBftoQPkoVN0jq5RAzURRa/oE32h78Jb4ccpOouTVqeQz+ygGTkbv/m0tnY=</latexit>‣ If any r =1, TR becomes TT.

31!

Efficient Computation... ...

r Gn r<latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

I1

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>

GNIN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

G2 I2

IN

<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

G1<latexit sha1_base64="QOOglJyQgF9kzwOVxe+hlrTGwZI=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZelblxWsA9oh5JJM21okhmSjFCGrvwGt7p2J279Epf+iZl2Frb1QOBwzr3ckxPEnGnjut9OYWNza3unuFva2z84PCofn7R1lChCWyTikeoGWFPOJG0ZZjjtxopiEXDaCSZ3md95okqzSD6aaUx9gUeShYxgY6VeX2AzJpinjdmgXHGr7hxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n88gzdGGVIQojZZ80aK7+3Uix0HoqAjuZRdSrXib+5/USE976KZNxYqgki0NhwpGJUPZ/NGSKEsOnlmCimM2KyBgrTIxtaelKILJOvNUG1km7VvWuqrWH60q9kbdThDM4h0vw4AbqcA9NaAGBCF7gFd6cZ+fd+XA+F6MFJ985hSU4X79UT5fy</latexit><latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit><latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit><latexit sha1_base64="1h/TU896uDFwcMUPY8XDmR6Idps=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRha6kgn1AG8pkOmmHTiZxZiKUkKXf4FbX7sStH+LSP3HSZmFbD1w4nHMv93C8iDOlbfvbKqysrq1vFDdLW9s7u3vl/YOWCmNJaJOEPJQdDyvKmaBNzTSnnUhSHHictr3xdea3n6hULBQPehJRN8BDwXxGsDaS2wuwHhHMk5u0f9cvV+yqPQVaJk5OKpCj0S//9AYhiQMqNOFYqa5jR9pNsNSMcJqWerGiESZjPKRdQwUOqHKTaegUnRhlgPxQmhEaTdW/FwkOlJoEntnMQqpFLxP/87qx9i/dhIko1lSQ2SM/5kiHKGsADZikRPOJIZhIZrIiMsISE216mvviBanpxFlsYJm0alXnrFq7P6/Ur/J2inAEx3AKDlxAHW6hAU0g8Agv8Apv1rP1bn1Yn7PVgpXfHMIcrK9fv/KYuA==</latexit><latexit sha1_base64="B3e7+lpSwHTmeIEAyPDu/F7BNWU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92W/XHGr7gxolXg5qUCORr/80xtEJBFUGsKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LZVYUO2ns1On6MwqAxRGypY0aKb+nUix0HoiAtspsBnpZS8T//O6iQmv/ZTJODFUkvmiMOHIRCj7Gw2YosTwiSWYKGZvRWSEFSbGprOwJRBTm4m3nMAqadWq3kW19nBZqd/k6RThBE7hHDy4gjrcQQOaQGAIL/AKb07qvDsfzue8teDkM8ewAOfrF9A5lEg=</latexit>r H r

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit>

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit>

...

I2

<latexit sha1_base64="33X/Ozbrrw3EbiPWwTgMGrKHiD8=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GXVjcsK9gFNKJPppB06k4SZiVBCdn6DW127E7f+iEv/xEmbhW09cOFwzr3cw/FjzpS27W+rtLa+sblV3q7s7O7tH1QPjzoqSiShbRLxSPZ8rChnIW1rpjntxZJi4XPa9Sd3ud99olKxKHzU05h6Ao9CFjCCtZFcV2A9JpinNxkaVGt23Z4BrRKnIDUo0BpUf9xhRBJBQ004Vqrv2LH2Uiw1I5xmFTdRNMZkgke0b2iIBVVeOsucoTOjDFEQSTOhRjP170WKhVJT4ZvNPKNa9nLxP6+f6ODaS1kYJ5qGZP4oSDjSEcoLQEMmKdF8aggmkpmsiIyxxESbmha++CIznTjLDaySTqPuXNQbD5e15m3RThlO4BTOwYEraMI9tKANBGJ4gVd4s56td+vD+pyvlqzi5hgWYH39Aq6RmBs=</latexit>IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

A

In<latexit<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="1CH/E/k7vw4iob5ofCdhFnqOU70=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aQne6WyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXyUpjY</latexit><latexit=

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit... ...
B I2<latexit sha1_base64="RlY4qj8aJU3bHXR0cym/vC92qLw=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQN2WmCrosurEbqWAf0JYhk8m0oZnMkNwRSukX+BF+g1tduxO34tI/MX0g2nogcDjnXs7N8RPBNTjOp5VZWV1b38hu5ra2d3b37P2Dho5TRVmdxiJWLZ9oJrhkdeAgWCtRjES+YE1/cDXxm/dMaR7LOxgmrBuRnuQhpwSM5NmFYtVzO8AjpnHVK89YhwYx6B/15sSz807JmQIvE3dO8miOmmd/dYKYphGTQAXRuu06CXRHRAGngo1znVSzhNAB6bG2oZKYpO5o+p0xLhglwGGszJOAp+rvjRGJtB5GvpmMCPT1ojcR//PaKYQX3RGXSQpM0llQmAoMMZ50gwOuGAUxNIRQxc2tmPaJIhRMg39S/GhsOnEXG1gmjXLJPS2Vb8/ylct5O1l0hI5REbnoHFXQNaqhOqLoAT2hZ/RiPVqv1pv1PhvNWPOdQ/QH1sc3LISh5A==</latexit><latexit sha1_base64="B3e7+lpSwHTmeIEAyPDu/F7BNWU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92W/XHGr7gxolXg5qUCORr/80xtEJBFUGsKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LZVYUO2ns1On6MwqAxRGypY0aKb+nUix0HoiAtspsBnpZS8T//O6iQmv/ZTJODFUkvmiMOHIRCj7Gw2YosTwiSWYKGZvRWSEFSbGprOwJRBTm4m3nMAqadWq3kW19nBZqd/k6RThBE7hHDy4gjrcQQOaQGAIL/AKb07qvDsfzue8teDkM8ewAOfrF9A5lEg=</latexit>...

<latexit sha1_base64="RlY4qj8aJU3bHXR0cym/vC92qLw=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQN2WmCrosurEbqWAf0JYhk8m0oZnMkNwRSukX+BF+g1tduxO34tI/MX0g2nogcDjnXs7N8RPBNTjOp5VZWV1b38hu5ra2d3b37P2Dho5TRVmdxiJWLZ9oJrhkdeAgWCtRjES+YE1/cDXxm/dMaR7LOxgmrBuRnuQhpwSM5NmFYtVzO8AjpnHVK89YhwYx6B/15sSz807JmQIvE3dO8miOmmd/dYKYphGTQAXRuu06CXRHRAGngo1znVSzhNAB6bG2oZKYpO5o+p0xLhglwGGszJOAp+rvjRGJtB5GvpmMCPT1ojcR//PaKYQX3RGXSQpM0llQmAoMMZ50gwOuGAUxNIRQxc2tmPaJIhRMg39S/GhsOnEXG1gmjXLJPS2Vb8/ylct5O1l0hI5REbnoHFXQNaqhOqLoAT2hZ/RiPVqv1pv1PhvNWPOdQ/QH1sc3LISh5A==</latexit>(I1 ⇥ I2 ⇥ · · · ⇥ IN )In

(I1 ⇥ I2 ⇥ · · · ⇥ IN )I1

<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>n<latexit sha1_base64="zuUMLfouRsfgdF3AvuQVA1PsBDk=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu3LfqnsVtwZ0CrxclKGHI1+6ac3iEgiqDSEY627nhsbP8XKMMLptNhLNI0xGeMh7VoqsaDaT2ehp+jcKgMURsqONGim/r1IsdB6IgK7mYXUy14m/ud1ExPe+CmTcWKoJPNHYcKRiVDWABowRYnhE0swUcxmRWSEFSbG9rTwJRBT24m33MAqaVUr3mWlen9Vrt3m7RTgFM7gAjy4hhrUoQFNIPAIL/AKb86z8+58OJ/z1TUnvzmBBThfv/PomNk=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>=<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="mn9J7LFSCc3YsmGI1D0wKY8HO7s=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRTVdSwT6gDWUynbRDZ5I4MxFKyNJvcKtrd+LWD3Hpnzhps7CtBy4czrmXezhexJnStv1tFdbWNza3itulnd29/YPy4VFbhbEktEVCHsquhxXlLKAtzTSn3UhSLDxOO97kNvM7T1QqFgYPehpRV+BRwHxGsDaS2xdYjwnmSSMd3A3KFbtqz4BWiZOTCuRoDso//WFIYkEDTThWqufYkXYTLDUjnKalfqxohMkEj2jP0AALqtxkFjpFZ0YZIj+UZgKNZurfiwQLpabCM5tZSLXsZeJ/Xi/W/rWbsCCKNQ3I/JEfc6RDlDWAhkxSovnUEEwkM1kRGWOJiTY9LXzxRGo6cZYbWCXtWtW5qNbuLyv1m7ydIpzAKZyDA1dQhwY0oQUEHuEFXuHNerberQ/rc75asPKbY1iA9fULwYiYuQ==</latexit>IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

I1

H2

HN

<latexit sha1_base64="Qgkh4DhDZb/epcqT/LTJbI/qBbk=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu1X+6WyW3FnQKvEy0kZcjT6pZ/eICKJoNIQjrXuem5s/BQrwwin02Iv0TTGZIyHtGupxIJqP52FnqJzqwxQGCk70qCZ+vcixULriQjsZhZSL3uZ+J/XTUx446dMxomhkswfhQlHJkJZA2jAFCWGTyzBRDGbFZERVpgY29PCl0BMbSfecgOrpFWteJeV6v1VuXabt1OAUziDC/DgGmpQhwY0gcAjvMArvDnPzrvz4XzOV9ec/OYEFuB8/QKVdJid</latexit>

I2<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

H1<latexit sha1_base64="nKqATXhVI8HwzzcJt8za2drnZZg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu17/VLZrbgzoFXi5aQMORr90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtK9f6qXLvN2ynAKZzBBXhwDTWoQwOaQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGT4Zic</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit><latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>Hnr<latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit>

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

I2<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit><latexit sha1_base64="xhn3btZ/dx2t068ny9g8D1vsomE=">AAAB/XicbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWEYwH5AcYW8zlyzZ3Tt294RwBH+DrdZ2YutvsfSfuEmuMIkPBh7vzTAzL0w408bzvp219Y3Nre3CTnF3b//gsHR03NRxqig2aMxj1Q6JRs4kNgwzHNuJQiJCjq1wdDf1W0+oNIvloxknGAgykCxilBgrtbqcyAHHXqnsVbwZ3FXi56QMOeq90k+3H9NUoDSUE607vpeYICPKMMpxUuymGhNCR2SAHUslEaiDbHbuxD23St+NYmVLGnem/p3IiNB6LELbKYgZ6mVvKv7ndVIT3QQZk0lqUNL5oijlrond6e9unymkho8tIVQxe6tLh0QRamxCC1tCMbGZ+MsJrJJmteJfVqoPV+XabZ5OAU7hDC7Ah2uowT3UoQEURvACr/DmPDvvzofzOW9dc/KZE1iA8/ULIfOWKQ==</latexit>,

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit sha1_base64="33X/Ozbrrw3EbiPWwTgMGrKHiD8=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GXVjcsK9gFNKJPppB06k4SZiVBCdn6DW127E7f+iEv/xEmbhW09cOFwzr3cw/FjzpS27W+rtLa+sblV3q7s7O7tH1QPjzoqSiShbRLxSPZ8rChnIW1rpjntxZJi4XPa9Sd3ud99olKxKHzU05h6Ao9CFjCCtZFcV2A9JpinNxkaVGt23Z4BrRKnIDUo0BpUf9xhRBJBQ004Vqrv2LH2Uiw1I5xmFTdRNMZkgke0b2iIBVVeOsucoTOjDFEQSTOhRjP170WKhVJT4ZvNPKNa9nLxP6+f6ODaS1kYJ5qGZP4oSDjSEcoLQEMmKdF8aggmkpmsiIyxxESbmha++CIznTjLDaySTqPuXNQbD5e15m3RThlO4BTOwYEraMI9tKANBGJ4gVd4s56td+vD+pyvlqzi5hgWYH39Aq6RmBs=</latexit><latexit

sha1_base64="FfmTf9AMYIsDK0iwW9AWSBdIOuI=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgddzFgJZBG8uI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/N/PYzVZrF8slMEhoIPJQsYgQbKz26rtsvVzzXmwOtEz8nFcjR6Jd/eoOYpIJKQzjWuut7iQkyrAwjnE5LvVTTBJMxHtKupRILqoNsfuoUXVhlgKJY2ZIGzdW/ExkWWk9EaDsFNiO96s3E/7xuaqKbIGMySQ2VZLEoSjkyMZr9jQZMUWL4xBJMFLO3IjLCChNj01naEoqpzcRfTWCdtKquf+VWH2qV+m2eThHO4BwuwYdrqMM9NKAJBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfNfk7w=</latexit><latexit

IN

A

...

<latexit...

h... ...
B I2... ...

<latexit sha1_base64="QOOglJyQgF9kzwOVxe+hlrTGwZI=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZelblxWsA9oh5JJM21okhmSjFCGrvwGt7p2J279Epf+iZl2Frb1QOBwzr3ckxPEnGnjut9OYWNza3unuFva2z84PCofn7R1lChCWyTikeoGWFPOJG0ZZjjtxopiEXDaCSZ3md95okqzSD6aaUx9gUeShYxgY6VeX2AzJpinjdmgXHGr7hxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n88gzdGGVIQojZZ80aK7+3Uix0HoqAjuZRdSrXib+5/USE976KZNxYqgki0NhwpGJUPZ/NGSKEsOnlmCimM2KyBgrTIxtaelKILJOvNUG1km7VvWuqrWH60q9kbdThDM4h0vw4AbqcA9NaAGBCF7gFd6cZ+fd+XA+F6MFJ985hSU4X79UT5fy</latexit><latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>IN<latexit sha1_base64="MRqN8fAcIo2T3cPqnG7aiRbtrdg=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgIeEuCloGbSwTMB+QHGFvs5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYrQFol4pLoB1pQzSVuGGU67saJYBJx2gsnDzO88U6VZJJ9MGlNf4JFkISPYWKl5NShX3Ko7B1onXk4qkKMxKP/0hxFJBJWGcKx1z3Nj42dYGUY4nZb6iaYxJhM8oj1LJRZU+9n80Cm6sMoQhZGyJQ2aq38nMiy0TkVgOwU2Y73qzcT/vF5iwjs/YzJODJVksShMODIRmn2NhkxRYnhqCSaK2VsRGWOFibHZLG0JxNRm4q0msE7atap3Xa01byr1+zydIpzBOVyCB7dQh0doQAsIUHiBV3hzUufd+XA+F60FJ585hSU4X78U85NK</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>

Computationi=r<latexit sha1_base64="zuUMLfouRsfgdF3AvuQVA1PsBDk=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu3LfqnsVtwZ0CrxclKGHI1+6ac3iEgiqDSEY627nhsbP8XKMMLptNhLNI0xGeMh7VoqsaDaT2ehp+jcKgMURsqONGim/r1IsdB6IgK7mYXUy14m/ud1ExPe+CmTcWKoJPNHYcKRiVDWABowRYnhE0swUcxmRWSEFSbG9rTwJRBT24m33MAqaVUr3mWlen9Vrt3m7RTgFM7gAjy4hhrUoQFNIPAIL/AKb86z8+58OJ/z1TUnvzmBBThfv/PomNk=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>In

r Gn r<latexit sha1_base64="B3e7+lpSwHTmeIEAyPDu/F7BNWU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92W/XHGr7gxolXg5qUCORr/80xtEJBFUGsKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LZVYUO2ns1On6MwqAxRGypY0aKb+nUix0HoiAtspsBnpZS8T//O6iQmv/ZTJODFUkvmiMOHIRCj7Gw2YosTwiSWYKGZvRWSEFSbGprOwJRBTm4m3nMAqadWq3kW19nBZqd/k6RThBE7hHDy4gjrcQQOaQGAIL/AKb07qvDsfzue8teDkM8ewAOfrF9A5lEg=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="1CH/E/k7vw4iob5ofCdhFnqOU70=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aQne6WyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXyUpjY</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>=<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

GN<latexit sha1_base64="o6Mqnn7mVJCjNqHzNwHAi/DZiXE=">AAAB/XicbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWEYwH5AcYW8zlyzZ3Tt294RwBH+DrdZ2YutvsfSfuEmuMIkPBh7vzTAzL0w408bzvp219Y3Nre3CTnF3b//gsHR03NRxqig2aMxj1Q6JRs4kNgwzHNuJQiJCjq1wdDf1W0+oNIvloxknGAgykCxilBgrtbqKyAHHXqnsVbwZ3FXi56QMOeq90k+3H9NUoDSUE607vpeYICPKMMpxUuymGhNCR2SAHUslEaiDbHbuxD23St+NYmVLGnem/p3IiNB6LELbKYgZ6mVvKv7ndVIT3QQZk0lqUNL5oijlrond6e9unymkho8tIVQxe6tLh0QRamxCC1tCMbGZ+MsJrJJmteJfVqoPV+XabZ5OAU7hDC7Ah2uowT3UoQEURvACr/DmPDvvzofzOW9dc/KZE1iA8/ULK4OWLw==</latexit><latexit sha1_base64="mn9J7LFSCc3YsmGI1D0wKY8HO7s=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRTVdSwT6gDWUynbRDZ5I4MxFKyNJvcKtrd+LWD3Hpnzhps7CtBy4czrmXezhexJnStv1tFdbWNza3itulnd29/YPy4VFbhbEktEVCHsquhxXlLKAtzTSn3UhSLDxOO97kNvM7T1QqFgYPehpRV+BRwHxGsDaS2xdYjwnmSSMd3A3KFbtqz4BWiZOTCuRoDso//WFIYkEDTThWqufYkXYTLDUjnKalfqxohMkEj2jP0AALqtxkFjpFZ0YZIj+UZgKNZurfiwQLpabCM5tZSLXsZeJ/Xi/W/rWbsCCKNQ3I/JEfc6RDlDWAhkxSovnUEEwkM1kRGWOJiTY9LXzxRGo6cZYbWCXtWtW5qNbuLyv1m7ydIpzAKZyDA1dQhwY0oQUEHuEFXuHNerberQ/rc75asPKbY1iA9fULwYiYuQ==</latexit>

HN<latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

G1IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>

H1<latexit sha1_base64="nKqATXhVI8HwzzcJt8za2drnZZg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu17/VLZrbgzoFXi5aQMORr90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtK9f6qXLvN2ynAKZzBBXhwDTWoQwOaQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGT4Zic</latexit>

O(I N )
<latexit sha1_base64="QQRPCsunN2kfSfGz5ClIGJRvbDg=">AAACCHicbVDLSgMxFM3UV62vUZdugkWomzJTBV0W3ehGK9gHtGPJpGkbmmSGJFMow/yA3+BW1+7ErX/h0j8x087Cth64cDjnXu7h+CGjSjvOt5VbWV1b38hvFra2d3b37P2DhgoiiUkdByyQLR8pwqggdU01I61QEsR9Rpr+6Dr1m2MiFQ3Eo56ExONoIGifYqSN1LXtDkd6iBGL75PS7dPdadcuOmVnCrhM3IwUQYZa1/7p9AIccSI0ZkiptuuE2ouR1BQzkhQ6kSIhwiM0IG1DBeJEefE0eQJPjNKD/UCaERpO1b8XMeJKTbhvNtOcatFLxf+8dqT7l15MRRhpIvDsUT9iUAcwrQH2qCRYs4khCEtqskI8RBJhbcqa++LzxHTiLjawTBqVsntWrjycF6tXWTt5cASOQQm44AJUwQ2ogTrAYAxewCt4s56td+vD+pyt5qzs5hDMwfr6BbYEmag=</latexit>

G2 I2<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit><latexit sha1_base64="1h/TU896uDFwcMUPY8XDmR6Idps=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRha6kgn1AG8pkOmmHTiZxZiKUkKXf4FbX7sStH+LSP3HSZmFbD1w4nHMv93C8iDOlbfvbKqysrq1vFDdLW9s7u3vl/YOWCmNJaJOEPJQdDyvKmaBNzTSnnUhSHHictr3xdea3n6hULBQPehJRN8BDwXxGsDaS2wuwHhHMk5u0f9cvV+yqPQVaJk5OKpCj0S//9AYhiQMqNOFYqa5jR9pNsNSMcJqWerGiESZjPKRdQwUOqHKTaegUnRhlgPxQmhEaTdW/FwkOlJoEntnMQqpFLxP/87qx9i/dhIko1lSQ2SM/5kiHKGsADZikRPOJIZhIZrIiMsISE216mvviBanpxFlsYJm0alXnrFq7P6/Ur/J2inAEx3AKDlxAHW6hAU0g8Agv8Apv1rP1bn1Yn7PVgpXfHMIcrK9fv/KYuA==</latexit>

H2<latexit sha1_base64="Qgkh4DhDZb/epcqT/LTJbI/qBbk=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu1X+6WyW3FnQKvEy0kZcjT6pZ/eICKJoNIQjrXuem5s/BQrwwin02Iv0TTGZIyHtGupxIJqP52FnqJzqwxQGCk70qCZ+vcixULriQjsZhZSL3uZ+J/XTUx446dMxomhkswfhQlHJkJZA2jAFCWGTyzBRDGbFZERVpgY29PCl0BMbSfecgOrpFWteJeV6v1VuXabt1OAUziDC/DgGmpQhwY0gcAjvMArvDnPzrvz4XzOV9ec/OYEFuB8/QKVdJid</latexit>

Computation
O(N Ir4 )<latexit sha1_base64="UwWmRHXLLvv7R1PDq+G7mjr/3kw=">AAACCnicbVDLSsNAFJ3UV62vWJduBotQNyWpBV0W3ehGK9gHtLFMppN26GQSZiZiCfkDv8Gtrt2JW3/CpX/ipM3Cth64cDjnXu7huCGjUlnWt5FbWV1b38hvFra2d3b3zP1iSwaRwKSJAxaIjoskYZSTpqKKkU4oCPJdRtru+DL1249ESBrwezUJieOjIacexUhpqW8Wez5SI4xYfJuUb67FQ+2kb5asijUFXCZ2RkogQ6Nv/vQGAY58whVmSMqubYXKiZFQFDOSFHqRJCHCYzQkXU058ol04mn2BB5rZQC9QOjhCk7Vvxcx8qWc+K7eTJPKRS8V//O6kfLOnZjyMFKE49kjL2JQBTAtAg6oIFixiSYIC6qzQjxCAmGl65r74vqJ7sRebGCZtKoV+7RSvauV6hdZO3lwCI5AGdjgDNTBFWiAJsDgCbyAV/BmPBvvxofxOVvNGdnNAZiD8fULEiuaYg==</latexit>

32Compact Representations

33

TT Networks for Optimization
Problems
Tensor166Networks for Large-Scale
Optimization
Problems
166
TT Networks for Optimization Problems
R n‣ TT format of a large vector

2.2. Graphical Representation of Fundamental Tensor Networks

}

xT

a

I = I1 I2 … INAJ = J1 J2 … JN

I1
J1

X(1)
I2

}

J2

I1

I3
J3

Rn -1

JN

(n-1)
A
A(1)
2.2. Graphical Representation of Fundamental
Tensor Networks A 301

I = I1 I2 …IN

I1

I2

I3

1 2

A

N

1

2

I1
I2
I = I1 I2 … IN
J2J
I1 J1 x I2I
J = J1 J2 … JN
J1
J2

}

3
3

J3
(1)
IX
3

}

P n-1

N

R1

J3

IN

JN (n-1)

IN X
JN

XR(1)n X

R1

In

AA(n)

IN

I1
KI n-1
K …K
matrixK
K of Ka large‣ TTK= K format
I
a

}

(n+1)

(n)

T
xX

IN
I n-1

P1

n

301

X(n-1)

R1

T
max
x
Ax
R

Optimization
problem:
Ln

Ln

}
X

R n-1

I n+1 I n-1

I1

Pn

P1

A A(n+1)
(1)

I1

In

Pn+1

R1

n+2
RnR-1

Rn

X(n+1)

P n-1
P n+2

Pn

) (N)
A(nA

(n+1)

A

R n+1 (n-1) R n-1
R n+2

XX(n+2)X(n+2)

In

)
X(nX
(N)

Rn

X(n+1)

R n+2

Pn+1

(n+2)

A

P n+2

X(n+2)

A(N)
IN

I n+2

R n+1

X(N)
IN

I n+2

I n+1

IN

I n+2

R n+1

I n+1

In

n+2)
(n(-1)

AA

)
X(nX

IN

I n+2

I n+1 I n-1

XR(1)n X(n+1)

x (n)

(N)

(n+2)

X
(n-1)
R n+1X

R n+2

X(N)

Figure A2.20: Forms of tensor train decompositions for a vector, a RI , matrix,
A RI J , and 3rd-order tensor, A RI J K (by applying a suitable tensorization).
I3
I1
I2
IN
I = I1 I2 …IN

TT-cores need
be storedKand processed,
whichof
K= KtoK3 makes theKnumber
K2
N
1K2 …KN
1
parameters
to scale linearly in the tensor order, N , of a data tensor and
A
J2
J
JN
all mathematical
J operations
1 thenI1 JareI3 3 only on Ithe low-orderI
I2 performedN
and relatively small size core tensors.
The2.20:
TT rank
as andecompositions
N 1 -tuple
the formn)I , matrix,
FigureFormsisofdefinedtensor train
for ofa vector,a (RA‣ Fast ALS/DMRG algorithm‣ Large-scale SVD/PCA/CCA‣ Parallel computing

RI J , and 3rd-order tensor, A

rankTT X

rT T

X(n)

X

RI J K (by applying a suitable tensorization).

R1 , . . . , R N 1 ,Rnrank X n,(2.21)

I1 stored
In In and
1 IN processed,
where X needis an nthwhichcanonical
ofTT-corestoRbemakesmatricization
the number ofnn)n
the tensor to
X.scaleSince
the TTrank
determines
requirements
parameterslinearlyin thetensorNmemory, of a dataandA(tensor
Lorder,of mathematicala tensor train,operations
it has a strong
impacton theonlycomplexity,
i.e., theall
are thenperformed
on the low-ordersuitabilityof tensor
traincore
representation
for a given raw data tensor.
andrelatively
small size
tensors.
The TT
numbersamples
be stored
scalesin the tensor
Therankofisdata
definedas anto N
1 -tupleof linearly
the form
order, N , and the size, I, and quadratically in the maximum
X(n)TT rank
rankXrR1 , . . . , RN 1 , Rn rank X n , (2.21)
bound,TTR, that is T T

Rn -1

Ln

A(n)

X(n)

X(n)
In

x (n) R

n -1

Rn

R n

R n

Rn -1

X(n)

In

X(n)

Rn

In

x (n)
Rn

_

In

_

In
A( n )A( n ) Rn -1

In

In

X(n)

Rn

In
x (n)

x (n)

where
RI1 In In 1 IN is an nth canonical matricization of
N X n
Rn 1 RO the
N R2TTI , rank
R : determines
max Rn , memory
I : max
In . (2.22)
the tensor
X.
requirements
n InSincennFigure 6.7: The conceptual TT network for the computation of a single extremeof na 1tensor train, it has a strong impact on the complexity, i.e., theeigenvalue, l, and the corresponding eigenvector, x
R I , for a symmetric matrix
suitabilityof ittensor
train representation
for a low-rank
given rawTT
data
tensor.
This is why
is crucially
important to haveapproximaI I . The frame matrix maps the TT core into a large vector. The tensor net7. A
A RforFigureThe
conceptual
TT network
the computation of a single extremeThenumberof datasamples
to be storedlinearly
the tensor
tions
drawbackof theTTformatis 6.7:
thatscales
the
ranks
of aintensor
train
order,
N , and the size, I, andquadratically inmaximum
TT rank
7
N 2the corresponding
work
corresponds
quotient, where
eigenvalue,
l,the
and
eigenvector,
x to theR I ,Rayleighfor a symmetricmatrixthe matrix A and vectors x R I
In the worst case scenario the TT ranks can grow up to I

for an N th-order(Cichocki et al., 2016)34

Part 2:
Tensor Networks in Deep Learning

35

Challenges in DNNs
Large
Model

of input increases for‣ Dimension
more complex data and task

Data

Computation

storage (memory, disk
‣ Hugerequirement)

layer improves model‣ Wider
capacity but parameters increaseinference and energy‣ Slow
costlydramatically
number of training samples,‣ Largelarge number of model parameters,
large computation cost36

to deploy models on‣ Hardsmall devices (e.g. smart
phone)

Model Compression
Goal: Make a lightweight model that is fast, memory-efficient andenergy-efficient

p
m
o
C

n
o
i
s
res

Duringtraining
Or
After
training

Weight parameter‣ To compress model parameters with comparable performance‣ To increase capacity without increasing model parameters‣ To improve computation efficiency
37

Taxonomy of Model Compression

Knowledge Distillation(Hinton et al., NIPS 2014 Workshop)

Weight Pruning(Han et al., NIPS 2015)

Layer

Layer

W
<latexit sha1_base64="jsCvViEpXziPr7GYTUbXuMCt0Dk=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FKgzqNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvX/k7lOg=</latexit>

W ⇡ AB<latexit sha1_base64="fEm9CghZIsgvGhmvBPgCPVYqIuQ=">AAACEXicbVC7TsMwFHXKq5RXgAmxWFRITFVSkGAsZWEsEn1ITVQ5rtNatePIdhBVVfERfAMrzGyIlS9g5E9w0gy05Ui2js+5V/f6BDGjSjvOt1VYWV1b3yhulra2d3b37P2DlhKJxKSJBROyEyBFGI1IU1PNSCeWBPGAkXYwukn99gORioroXo9j4nM0iGhIMdJG6tlHXsBhG3oojqV4hOnrOrvrPbvsVJwMcJm4OSmDHI2e/eP1BU44iTRmSKmu68TanyCpKWZkWvISRWKER2hAuoZGiBPlT7IvTOGpUfowFNKcSMNM/dsxQVypMQ9MJUd6qBa9VPzP6yY6vPInNIoTTSI8GxQmDGoB0zxgn0qCNRsbgrCkZleIh0girE1qc1MCPjWZuIsJLJNWteKeV6p3F+VaPU+nCI7BCTgDLrgENXALGqAJMHgCL+AVvFnP1rv1YX3OSgtW3nMI5mB9/QLcBpxQ</latexit>(M ⇥ N ) (M ⇥ r) (r ⇥ N )<latexit sha1_base64="PknIA7FAUfjZnID1MOJitrlc4f8=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GITdiNgpZBGyuJYB6QXcLsZDYZMjO7zMwKIaTzG2y1thNbf8TSP3GSbGESD1w4nHMv53LChDNtXPfbya2tb2xu5bcLO7t7+wfFw6OmjlNFaIPEPFbtEGvKmaQNwwyn7URRLEJOW+Hwduq3nqjSLJaPZpTQQOC+ZBEj2FjJLyvfMEE1uj9H3WLJrbgzoFXiZaQEGerd4o/fi0kqqDSEY607npuYYIyVYYTTScFPNU0wGeI+7VgqsQ0KxrOfJ+jMKj0UxcqONGim/r0YY6H1SIR2U2Az0MveVPzP66Qmug7GTCapoZLMg6KUIxOjaQGoxxQlho8swUQx+ysiA6wwMbamhZRQTGwn3nIDq6RZrXgXlerDZal2k7WThxM4hTJ4cAU1uIM6NIBAAi/wCm/Os/PufDif89Wck90cwwKcr199M5db</latexit><latexit sha1_base64="6p50TwTpSQ8J1Mjj1uQFA2ufsWg=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAixCbtR0DJoY6NEMA/ILmF2MkmGzMwuM7NCWNL5DbZa24mtP2LpnzhJtjCJBy4czrmXczlhzJk2rvvtrKyurW9s5rby2zu7e/uFg8OGjhJFaJ1EPFKtEGvKmaR1wwynrVhRLEJOm+HwZuI3n6jSLJKPZhTTQOC+ZD1GsLGSX7rzDRNUo/sz1CkU3bI7BVomXkaKkKHWKfz43YgkgkpDONa67bmxCVKsDCOcjvN+ommMyRD3adtSiW1QkE5/HqNTq3RRL1J2pEFT9e9FioXWIxHaTYHNQC96E/E/r52Y3lWQMhknhkoyC+olHJkITQpAXaYoMXxkCSaK2V8RGWCFibE1zaWEYmw78RYbWCaNStk7L1ceLorV66ydHBzDCZTAg0uowi3UoA4EYniBV3hznp1358P5nK2uONnNEczB+foFQYKXNg==</latexit>A<latexit sha1_base64="wgzcIFwT9VG+mjDxxajZt60GmGA=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXUxjKC+YDkCHubvWTJ7t6xuyeEI/gbbLW2E1t/i6X/xM3lCpP4YODx3gwz84KYM21c99sprK1vbG4Vt0s7u3v7B+XDo5aOEkVok0Q8Up0Aa8qZpE3DDKedWFEsAk7bwfhu5refqNIsko9mElNf4KFkISPYWKmNeoFAN6hfrrhVNwNaJV5OKpCj0S//9AYRSQSVhnCsdddzY+OnWBlGOJ2WeommMSZjPKRdSyUWVPtpdu4UnVllgMJI2ZIGZerfiRQLrScisJ0Cm5Fe9mbif143MeG1nzIZJ4ZKMl8UJhyZCM1+RwOmKDF8YgkmitlbERlhhYmxCS1sCcTUZuItJ7BKWrWqd1GtPVxW6rd5OkU4gVM4Bw+uoA730IAmEBjDC7zCm/PsvDsfzue8teDkM8ewAOfrF4eslSY=</latexit><latexit sha1_base64="BaKXy+/5N7Yjeak1nKO2MFMrwWU=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GITdiNgpZBGxshgnlAdgmzk9lkyMzsMjMrhJDOb7DV2k5s/RFL/8RJsoVJPHDhcM69nMsJE860cd1vJ7e2vrG5ld8u7Ozu7R8UD4+aOk4VoQ0S81i1Q6wpZ5I2DDOcthNFsQg5bYXD26nfeqJKs1g+mlFCA4H7kkWMYGMlv3zvGyaoRuocdYslt+LOgFaJl5ESZKh3iz9+LyapoNIQjrXueG5igjFWhhFOJwU/1TTBZIj7tGOpxDYoGM9+nqAzq/RQFCs70qCZ+vdijIXWIxHaTYHNQC97U/E/r5Oa6DoYM5mkhkoyD4pSjkyMpgWgHlOUGD6yBBPF7K+IDLDCxNiaFlJCMbGdeMsNrJJmteJdVKoPl6XaTdZOHk7gFMrgwRXU4A7q0AACCbzAK7w5z8678+F8zldzTnZzDAtwvn4BenaXWg==</latexit>

B<latexit sha1_base64="aQddn4q9tiQ+8L6Ze0UCeiWo4+M=">AAAB/HicbVA9TwJBEJ3DL8Qv1NJmIzGxIndIoiXBxhIT+UjgQvaWPVjZ3bvs7pmQC/4GW63tjK3/xdJ/4gJXCPiSSV7em8nMvCDmTBvX/XZyG5tb2zv53cLe/sHhUfH4pKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML6d+e0nqjSL5IOZxNQXeChZyAg2VmqhXiBQvV8suWV3DrROvIyUIEOjX/zpDSKSCCoN4VjrrufGxk+xMoxwOi30Ek1jTMZ4SLuWSiyo9tP5tVN0YZUBCiNlSxo0V/9OpFhoPRGB7RTYjPSqNxP/87qJCW/8lMk4MVSSxaIw4chEaPY6GjBFieETSzBRzN6KyAgrTIwNaGlLIKY2E281gXXSqpS9q3Llvlqq1bN08nAG53AJHlxDDe6gAU0g8Agv8ApvzrPz7nw4n4vWnJPNnMISnK9fMDaU/Q==</latexit>

Linear
Low-rank Matrix/Tensor Factorization(Novikov et al., NIPS 2015)

Quantization and Sharing of Weight(Han et al., ICLR 2016)
38

Compression
Low-rank matrix factorization of weights in fully connected layer

W ⇡ AB<latexit sha1_base64="fEm9CghZIsgvGhmvBPgCPVYqIuQ=">AAACEXicbVC7TsMwFHXKq5RXgAmxWFRITFVSkGAsZWEsEn1ITVQ5rtNatePIdhBVVfERfAMrzGyIlS9g5E9w0gy05Ui2js+5V/f6BDGjSjvOt1VYWV1b3yhulra2d3b37P2DlhKJxKSJBROyEyBFGI1IU1PNSCeWBPGAkXYwukn99gORioroXo9j4nM0iGhIMdJG6tlHXsBhG3oojqV4hOnrOrvrPbvsVJwMcJm4OSmDHI2e/eP1BU44iTRmSKmu68TanyCpKWZkWvISRWKER2hAuoZGiBPlT7IvTOGpUfowFNKcSMNM/dsxQVypMQ9MJUd6qBa9VPzP6yY6vPInNIoTTSI8GxQmDGoB0zxgn0qCNRsbgrCkZleIh0girE1qc1MCPjWZuIsJLJNWteKeV6p3F+VaPU+nCI7BCTgDLrgENXALGqAJMHgCL+AVvFnP1rv1YX3OSgtW3nMI5mB9/QLcBpxQ</latexit>(M ⇥ N ) (M ⇥ r) (r ⇥ N )<latexit sha1_base64="PknIA7FAUfjZnID1MOJitrlc4f8=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GITdiNgpZBGyuJYB6QXcLsZDYZMjO7zMwKIaTzG2y1thNbf8TSP3GSbGESD1w4nHMv53LChDNtXPfbya2tb2xu5bcLO7t7+wfFw6OmjlNFaIPEPFbtEGvKmaQNwwyn7URRLEJOW+Hwduq3nqjSLJaPZpTQQOC+ZBEj2FjJLyvfMEE1uj9H3WLJrbgzoFXiZaQEGerd4o/fi0kqqDSEY607npuYYIyVYYTTScFPNU0wGeI+7VgqsQ0KxrOfJ+jMKj0UxcqONGim/r0YY6H1SIR2U2Az0MveVPzP66Qmug7GTCapoZLMg6KUIxOjaQGoxxQlho8swUQx+ysiA6wwMbamhZRQTGwn3nIDq6RZrXgXlerDZal2k7WThxM4hTJ4cAU1uIM6NIBAAi/wCm/Os/PufDif89Wck90cwwKcr199M5db</latexit><latexit sha1_base64="6p50TwTpSQ8J1Mjj1uQFA2ufsWg=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAixCbtR0DJoY6NEMA/ILmF2MkmGzMwuM7NCWNL5DbZa24mtP2LpnzhJtjCJBy4czrmXczlhzJk2rvvtrKyurW9s5rby2zu7e/uFg8OGjhJFaJ1EPFKtEGvKmaR1wwynrVhRLEJOm+HwZuI3n6jSLJKPZhTTQOC+ZD1GsLGSX7rzDRNUo/sz1CkU3bI7BVomXkaKkKHWKfz43YgkgkpDONa67bmxCVKsDCOcjvN+ommMyRD3adtSiW1QkE5/HqNTq3RRL1J2pEFT9e9FioXWIxHaTYHNQC96E/E/r52Y3lWQMhknhkoyC+olHJkITQpAXaYoMXxkCSaK2V8RGWCFibE1zaWEYmw78RYbWCaNStk7L1ceLorV66ydHBzDCZTAg0uowi3UoA4EYniBV3hznp1358P5nK2uONnNEczB+foFQYKXNg==</latexit><latexit sha1_base64="BaKXy+/5N7Yjeak1nKO2MFMrwWU=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GITdiNgpZBGxshgnlAdgmzk9lkyMzsMjMrhJDOb7DV2k5s/RFL/8RJsoVJPHDhcM69nMsJE860cd1vJ7e2vrG5ld8u7Ozu7R8UD4+aOk4VoQ0S81i1Q6wpZ5I2DDOcthNFsQg5bYXD26nfeqJKs1g+mlFCA4H7kkWMYGMlv3zvGyaoRuocdYslt+LOgFaJl5ESZKh3iz9+LyapoNIQjrXueG5igjFWhhFOJwU/1TTBZIj7tGOpxDYoGM9+nqAzq/RQFCs70qCZ+vdijIXWIxHaTYHNQC97U/E/r5Oa6DoYM5mkhkoyD4pSjkyMpgWgHlOUGD6yBBPF7K+IDLDCxNiaFlJCMbGdeMsNrJJmteJdVKoPl6XaTdZOHk7gFMrgwRXU4A7q0AACCbzAK7w5z8678+F8zldzTnZzDAtwvn4BenaXWg==</latexit>

Compression: O(M N ) ! O(r(M + N ))<latexit sha1_base64="xgFPYmra3/9MA6JfYrnHQL9v2rk=">AAACJ3icbVDLSsNAFJ3UV62vqEs3g0VoEWpSBV0W3bhprWAf0IQymU7boZMHMxOlhHyGH+E3uNW1O9Fl/8RJm0UfHrhwOOde7r3HCRgV0jB+tcza+sbmVnY7t7O7t3+gHx41hR9yTBrYZz5vO0gQRj3SkFQy0g44Qa7DSMsZ3SV+65lwQX3vSY4DYrto4NE+xUgqqatfWC6SQ4xY9BAXqrUitDgdDCXi3H+B8x4vVM9rxWJXzxslYwq4SsyU5EGKelefWD0fhy7xJGZIiI5pBNKOEJcUMxLnrFCQAOERGpCOoh5yibCj6WMxPFNKD/Z9rsqTcKrOT0TIFWLsOqozOVUse4n4n9cJZf/GjqgXhJJ4eLaoHzIofZikBHuUEyzZWBGEOVW3QjxEHGGpslzY4rixysRcTmCVNMsl87JUfrzKV27TdLLgBJyCAjDBNaiAe1AHDYDBK3gHH+BTe9O+tG/tZ9aa0dKZY7AAbfIHv8alwA==</latexit>

Low-rank tensor network factorization of weights‣ Step 1: W ! W (Matrix to d-order tensor)
Step 2: W ⇡ TT(G · · · G ) (Tensor network representation)‣<latexit sha1_base64="2sTKVCKnks+X5IzQYzhPljB+N/k=">AAACFXicbVC7SgNBFJ2Nrxhfq5YiDAbBKuxGQcugjWUE84DsEmYnk2TIPJaZWSUsqfwIv8FWazuxtbb0T5xNtjCJBy4czrmXe++JYka18bxvp7Cyura+UdwsbW3v7O65+wdNLROFSQNLJlU7QpowKkjDUMNIO1YE8YiRVjS6yfzWA1GaSnFvxjEJORoI2qcYGSt13eMg4rAFA0UHQ4OUko8w4MgMMWJW7rplr+JNAZeJn5MyyFHvuj9BT+KEE2EwQ1p3fC82YYqUoZiRSSlINIkRHqEB6VgqECc6TKdvTOCpVXqwL5UtYeBU/TuRIq71mEe2MztRL3qZ+J/XSUz/KkypiBNDBJ4t6icMGgmzTGCPKoING1uCsKL2VoiHSCFsbHJzWyI+sZn4iwksk2a14p9XqncX5dp1nk4RHIETcAZ8cAlq4BbUQQNg8ARewCt4c56dd+fD+Zy1Fpx85hDMwfn6BS6Vnrs=</latexit>

1<latexit sha1_base64="cSY0IJ21pwMVZNsfQmMNAa2Konw=">AAACN3icbVDJSgNBEO1xjXGLevTSGIR4CTNR0JtBD3qMkA0yIfT0dJImPdNDd40kDPkcP8Jv8KoHT+JFvPoHdhYkiw8KHu9VUVXPiwTXYNvv1srq2vrGZmorvb2zu7efOTisahkryipUCqnqHtFM8JBVgINg9UgxEniC1bze7civPTKluQzLMIhYMyCdkLc5JWCkVuYauwGBLiUC17BLokjJPnaB9SEpl4e5P/Ou5WCX+hI0ntH8s1Yma+ftMfAycaYki6YotTKfri9pHLAQqCBaNxw7gmZCFHAq2DDtxppFhPZIhzUMDUnAdDMZPzrEp0bxcVsqUyHgsTo7kZBA60Hgmc7RkXrRG4n/eY0Y2lfNhIdRDCykk0XtWGCQeJQa9rliFMTAEEIVN7di2iWKUDDZzm3xgqHJxFlMYJlUC3nnPF94uMgWb6bppNAxOkE55KBLVET3qIQqiKIn9IJe0Zv1bH1YX9b3pHXFms4coTlYP79CuawW</latexit>dLoss function:
L(W , x, y) ! L({G1 , · · · , Gd }, x, y)<latexit sha1_base64="IMztCH48GuNAMfTMPqWMUISd7DI=">AAACaXicbVHBahsxENVuE8dN03bTXEpyGWoCDhizmxbSY2gOzSEHF+LY4DVGq5VtEWm1lWbTmsXfGXLsud9QiNY2JXYyIObpzRvN8JTkUlgMwwfPf7W1Xdupv959s/f23ftg/8ON1YVhvMu01KafUMulyHgXBUrezw2nKpG8l9xeVPXeHTdW6OwaZzkfKjrJxFgwio4aBT8hVhSnjEq4gmacKOi1oEq/l2l2ArERkylSY/QvWFNDXP6/fh9FroGlGm0LnrBpPN94bxQ0wna4CHgOohVokFV0RsGfONWsUDxDJqm1gyjMcVhSg4JJPt+NC8tzym7phA8czKjidlgurJnDsWNSGGvjToawYJ92lFRZO1OJU1ZL281aRb5UGxQ4/josRZYXyDO2HDQuJKCGymdIheEM5cwByoxwuwKbUkMZut9Ym5KoufMk2nTgObg5bUef26c/vjTOv63cqZMj8ok0SUTOyDm5JB3SJYzck39ezdvx/vr7/kf/cCn1vVXPAVkLv/EIkGm2Tg==</latexit>

Compression:

O(M N ) ! O(dr

p
2 d

M

p
d

N)

<latexit sha1_base64="GusYkOpTliikInbTcJajOdvg7Kk=">AAACO3icbZDLSgMxFIYzXmu9VV26CRah3ZSZKuiy2I2bVgXbCp2xZDJpG5q5mJxRyjAP5EP4DG51Ke7ErXvTC9KqBwIf/38O5+R3I8EVmOarsbC4tLyymlnLrm9sbm3ndnabKowlZQ0ailDeuEQxwQPWAA6C3USSEd8VrOUOqiO/dc+k4mFwDcOIOT7pBbzLKQEtdXJV2yfQp0QkF2mhVi9iW/JeH4iU4QOe9Tx5W7bVnYS25yS19AfrabGTy5slc1z4L1hTyKNpXXZyb7YX0thnAVBBlGpbZgROQiRwKliatWPFIkIHpMfaGgPiM+Uk48+m+FArHu6GUr8A8FidnUiIr9TQd3Xn6Hz12xuJ/3ntGLqnTsKDKAYW0MmibiwwhHiUHPa4ZBTEUAOhkutbMe0TSSjofOe2uH6qM7F+J/AXmuWSdVQqXx3nK2fTdDJoHx2gArLQCaqgc3SJGoiiR/SMXtCr8WS8Gx/G56R1wZjO7KG5Mr6+AWpir3Q=</latexit>

Train very “wide” model

Order of tensor

TT-rankTensorizing Neural Networks (Novikov et al., NIPS 2015)39

Vector to Scalar Transformation via TN
Hidden layery = (wT h + b)
<latexit sha1_base64="AdrI1o6zSY5B7S5UGc2Z24usXmo=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQEcpMFXQjFN24rNAXtLUkaaYNTWaGJKOUoV/gR/gNbnXtTtyKS//EzLQL23og3JNz7uUmB4ecKe0431ZmZXVtfSO7mdva3tnds/cPGiqIJKF1EvBAtjBSlDOf1jXTnLZCSZHAnDbx6Cbxmw9UKhb4NT0OaVeggc88RpA2Us8udLCAY3gFO4oNBCom18f7GkzqEJ6mFZ/07LxTclLAZeLOSB7MUO3ZP51+QCJBfU04UqrtOqHuxkhqRjid5DqRoiEiIzSgbUN9JKjqxul3JrBglD70AmmOr2Gq/p2IkVBqLLDpFEgP1aKXiP957Uh7l92Y+WGkqU+mi7yIQx3AJBvYZ5ISzceGICKZeSskQyQR0SbBuS1YTEwm7mICy6RRLrlnpfLdeb5yPUsnC47AMSgCF1yACrgFVVAHBDyBF/AK3qxn6936sD6nrRlrNnMI5mB9/QI+W6AV</latexit>

Output(M ⇥ 1)<latexit sha1_base64="MDSipZiKAUGtWseRcifEtXMHECA=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosunEjVLAPmA4lk2ba0CQzJBmhlK78Bre6didu/RKX/olpOwvbeiBwOOdezs0JE860cd1vJ7e2vrG5ld8u7Ozu7R8UD4+aOk4VoQ0S81i1Q6wpZ5I2DDOcthNFsQg5bYXD26nfeqJKs1g+mlFCA4H7kkWMYGMlv3zfMUxQjbzzbrHkVtwZ0CrxMlKCDPVu8afTi0kqqDSEY619z01MMMbKMMLppNBJNU0wGeI+9S2V2OYE49nJE3RmlR6KYmWfNGim/t0YY6H1SIR2UmAz0MveVPzP81MTXQdjJpPUUEnmQVHKkYnR9P+oxxQlho8swUQxeysiA6wwMbalhZRQTGwn3nIDq6RZrXgXlerDZal2k7WThxM4hTJ4cAU1uIM6NIBADC/wCm/Os/PufDif89Gck+0cwwKcr1+4vpbv</latexit>

d-order Tensor

m m

Tensorization

TT-rank

TT Representation<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>r<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>wmm
M = mdm m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="GLrX03oUUrDXGW9cINC4Ff/+UxA=">AAAB/HicbVA9TwJBEJ3DL8Qv1NJmIzGxIndIoiXRxhIT+UjgQvaWPVjZ3bvs7mnIBX+DrdZ2xtb/Yuk/cYErBHzJJC/vzWRmXhBzpo3rfju5tfWNza38dmFnd2//oHh41NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGN1M/dYjVZpF8t6MY+oLPJAsZAQbKzW7gUBPqFcsuWV3BrRKvIyUIEO9V/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZtRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCa/8lMk4MVSS+aIw4chEaPo66jNFieFjSzBRzN6KyBArTIwNaGFLICY2E285gVXSrJS9i3LlrlqqXWfp5OEETuEcPLiEGtxCHRpA4AFe4BXenGfn3flwPuetOSebOYYFOF+/hMeVMg==</latexit>M<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="tF5plF15dsMELhjvTzdCGgMZ7Ww=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLERIpgPSM6wt7eXLNndO3b3hBCCv8FWazux9bdY+k/cJFeYxAcDj/dmmJkXJJxp47rfTm5ldW19I79Z2Nre2d0r7h80dJwqQusk5rFqBVhTziStG2Y4bSWKYhFw2gwGNxO/+USVZrF8MMOE+gL3JIsYwcZKzTt0hcRj2C2W3LI7BVomXkZKkKHWLf50wpikgkpDONa67bmJ8UdYGUY4HRc6qaYJJgPco21LJRZU+6PpuWN0YpUQRbGyJQ2aqn8nRlhoPRSB7RTY9PWiNxH/89qpiS79EZNJaqgks0VRypGJ0eR3FDJFieFDSzBRzN6KSB8rTIxNaG5LIMY2E28xgWXSqJS9s3Ll/rxUvc7SycMRHMMpeHABVbiFGtSBwABe4BXenGfn3flwPmetOSebOYQ5OF+/zaGVUw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>hM<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit><latexit sha1_base64="ywqRJkxyH+HCmiJcuw5VySAcw0s=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FGg8qNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvXxQNlPk=</latexit>

Weights= G1<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>r

Numberof parameters
p
d
2

d Mr
<latexit sha1_base64="+v8TApAf/RpCpHGGFBU2/tASY14=">AAACB3icbVDLSsNAFJ34rPXRqEs3g0VwVZIq6LLoxo1QwT4gjWUymbRDZyZxZiKUkA/wG9zq2p249TNc+idO2yxs64ELh3Pu5VxOkDCqtON8Wyura+sbm6Wt8vbO7l7F3j9oqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrid+54lIRWNxr8cJ8TkaCBpRjLSR+nYl7KlHqb3Qz25z+VDv21Wn5kwBl4lbkCoo0OzbP70wxiknQmOGlPJcJ9F+hqSmmJG83EsVSRAeoQHxDBWIE+Vn08dzeGKUEEaxNCM0nKp/LzLElRrzwGxypIdq0ZuI/3leqqNLP6MiSTUReBYUpQzqGE5agCGVBGs2NgRhSc2vEA+RRFibruZSAp6bTtzFBpZJu15zz2r1u/Nq46popwSOwDE4BS64AA1wA5qgBTBIwQt4BW/Ws/VufVifs9UVq7g5BHOwvn4BzvKZyA==</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>mGd

d-order tensor

H<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

m m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>mm
m m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>< w, h ><latexit sha1_base64="Axg69VjX69O3PJsUELEX0IW75G0=">AAACBHicbVA9SwNBEJ2LXzF+RS1tFoNgIeEuClqIBG0sI5gPSM6wt9lLluzeHbt7SjjS+htstbYTW/+Hpf/EveQKk/hgmMd7M8zwvIgzpW3728otLa+sruXXCxubW9s7xd29hgpjSWidhDyULQ8ryllA65ppTluRpFh4nDa94U3qNx+pVCwM7vUooq7A/YD5jGBtpIfLjifQ0wlK2+CqWyzZZXsCtEicjJQgQ61b/On0QhILGmjCsVJtx460m2CpGeF0XOjEikaYDHGftg0NsKDKTSZfj9GRUXrID6WpQKOJ+ncjwUKpkfDMpMB6oOa9VPzPa8fav3ATFkSxpgGZHvJjjnSI0ghQj0lKNB8Zgolk5ldEBlhiok1QM1c8MTaZOPMJLJJGpeyclit3Z6XqdZZOHg7gEI7BgXOowi3UoA4EJLzAK7xZz9a79WF9TkdzVrazDzOwvn4Bdd6X2w==</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit>=<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>< TT(G1 , · · · , Gd ), H ><latexit sha1_base64="+CLTasYnoQoLobMZcjyYlDMt0hQ=">AAACNHicbVDLSsNAFJ3UV31HXboZLEKFUhIVdCFSdGGXFVoVmhImk2k7dPJg5kYsIT/jR/gNbnUnuBFx6zc4rUG09cDAuefcy71zvFhwBZb1YhRmZufmF4qLS8srq2vr5sbmlYoSSVmLRiKSNx5RTPCQtYCDYDexZCTwBLv2Bucj//qWScWjsAnDmHUC0gt5l1MCWnLNkxMH2B2kzWZWdgICfUpEepG5dsWhfgSq8lv09yr4p65np65ZsqrWGHia2DkpoRwN13xz/IgmAQuBCqJU27Zi6KREAqeCZUtOolhM6ID0WFvTkARMddLxLzO8qxUfdyOpXwh4rP6eSEmg1DDwdOfoRjXpjcT/vHYC3eNOysM4ARbS70XdRGCI8Cgy7HPJKIihJoRKrm/FtE8koaCD/bPFCzKdiT2ZwDS52q/aB9X9y8NS7SxPp4i20Q4qIxsdoRqqowZqIYru0SN6Qs/Gg/FqvBsf360FI5/ZQn9gfH4BsCer+Q==</latexit>

40<latexit sha1_base64="91pa8qrcUAgEcxhJ3mIvPsSfh9c=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFN11WsA+YDiWTZtrQJDMkGaEMXfkNbnXtTtz6JS79EzNtF7b1QOBwzr3ckxMmnGnjut9OYWNza3unuFva2z84PCofn7R1nCpCWyTmseqGWFPOJG0ZZjjtJopiEXLaCcf3ud95okqzWD6aSUIDgYeSRYxgYyW/J7AZEcyzxrRfrrhVdwa0TrwFqcACzX75pzeISSqoNIRjrX3PTUyQYWUY4XRa6qWaJpiM8ZD6lkosqA6yWeQpurDKAEWxsk8aNFP/bmRYaD0RoZ3MI+pVLxf/8/zURLdBxmSSGirJ/FCUcmRilP8fDZiixPCJJZgoZrMiMsIKE2NbWroSirwTb7WBddKuVb2rau3hulK/W7RThDM4h0vw4Abq0IAmtIBADC/wCm/Os/PufDif89GCs9g5hSU4X79dx5f4</latexit>

Vector to Scalar Transformation via TN
Hidden layery = (wT h + b)
<latexit sha1_base64="7Yf6XVjcHb7gktfVNy1/e9BJ3/w=">AAACE3icbVDLSsNAFJ3UV62vqMtuBotQEUpSBd0IRTcuK/QFTSyT6aQdOpOEmYlSQhd+hN/gVtfuxK0f4NI/cdJmYVsPXO7hnHu5M8eLGJXKsr6N3Mrq2vpGfrOwtb2zu2fuH7RkGAtMmjhkoeh4SBJGA9JUVDHSiQRB3GOk7Y1uUr/9QISkYdBQ44i4HA0C6lOMlJZ6ZnEMr6Aj6YCjsuNx+HjfSNsQnnonPbNkVawp4DKxM1ICGeo988fphzjmJFCYISm7thUpN0FCUczIpODEkkQIj9CAdDUNECfSTaafmMBjrfShHwpdgYJT9e9GgriUY+7pSY7UUC56qfif142Vf+kmNIhiRQI8O+THDKoQponAPhUEKzbWBGFB9VshHiKBsNK5zV3x+ERnYi8msExa1Yp9VqnenZdq11k6eVAER6AMbHABauAW1EETYPAEXsAreDOejXfjw/icjeaMbOcQzMH4+gXvv5zb</latexit>

Output(M ⇥ 1)

Weights<latexit sha1_base64="MDSipZiKAUGtWseRcifEtXMHECA=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosunEjVLAPmA4lk2ba0CQzJBmhlK78Bre6didu/RKX/olpOwvbeiBwOOdezs0JE860cd1vJ7e2vrG5ld8u7Ozu7R8UD4+aOk4VoQ0S81i1Q6wpZ5I2DDOcthNFsQg5bYXD26nfeqJKs1g+mlFCA4H7kkWMYGMlv3zfMUxQjbzzbrHkVtwZ0CrxMlKCDPVu8afTi0kqqDSEY619z01MMMbKMMLppNBJNU0wGeI+9S2V2OYE49nJE3RmlR6KYmWfNGim/t0YY6H1SIR2UmAz0MveVPzP81MTXQdjJpPUUEnmQVHKkYnR9P+oxxQlho8swUQxeysiA6wwMbalhZRQTGwn3nIDq6RZrXgXlerDZal2k7WThxM4hTJ4cAU1uIM6NIBADC/wCm/Os/PufDif89Gck+0cwwKcr1+4vpbv</latexit>

TT-rankd-order Tensor

m m

Tensorization

TT Representation<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

w

m
m
d
m
M =m
m
m m
m
m
m
m

= G1

r

r

<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="GLrX03oUUrDXGW9cINC4Ff/+UxA=">AAAB/HicbVA9TwJBEJ3DL8Qv1NJmIzGxIndIoiXRxhIT+UjgQvaWPVjZ3bvs7mnIBX+DrdZ2xtb/Yuk/cYErBHzJJC/vzWRmXhBzpo3rfju5tfWNza38dmFnd2//oHh41NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGN1M/dYjVZpF8t6MY+oLPJAsZAQbKzW7gUBPqFcsuWV3BrRKvIyUIEO9V/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZtRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCa/8lMk4MVSS+aIw4chEaPo66jNFieFjSzBRzN6KyBArTIwNaGFLICY2E285gVXSrJS9i3LlrlqqXWfp5OEETuEcPLiEGtxCHRpA4AFe4BXenGfn3flwPuetOSebOYYFOF+/hMeVMg==</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>M<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="tF5plF15dsMELhjvTzdCGgMZ7Ww=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLERIpgPSM6wt7eXLNndO3b3hBCCv8FWazux9bdY+k/cJFeYxAcDj/dmmJkXJJxp47rfTm5ldW19I79Z2Nre2d0r7h80dJwqQusk5rFqBVhTziStG2Y4bSWKYhFw2gwGNxO/+USVZrF8MMOE+gL3JIsYwcZKzTt0hcRj2C2W3LI7BVomXkZKkKHWLf50wpikgkpDONa67bmJ8UdYGUY4HRc6qaYJJgPco21LJRZU+6PpuWN0YpUQRbGyJQ2aqn8nRlhoPRSB7RTY9PWiNxH/89qpiS79EZNJaqgks0VRypGJ0eR3FDJFieFDSzBRzN6KSB8rTIxNaG5LIMY2E28xgWXSqJS9s3Ll/rxUvc7SycMRHMMpeHABVbiFGtSBwABe4BXenGfn3flwPmetOSebOYQ5OF+/zaGVUw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

r

d Mr

r

<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Numberof parameterspd2<latexit sha1_base64="+v8TApAf/RpCpHGGFBU2/tASY14=">AAACB3icbVDLSsNAFJ34rPXRqEs3g0VwVZIq6LLoxo1QwT4gjWUymbRDZyZxZiKUkA/wG9zq2p249TNc+idO2yxs64ELh3Pu5VxOkDCqtON8Wyura+sbm6Wt8vbO7l7F3j9oqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrid+54lIRWNxr8cJ8TkaCBpRjLSR+nYl7KlHqb3Qz25z+VDv21Wn5kwBl4lbkCoo0OzbP70wxiknQmOGlPJcJ9F+hqSmmJG83EsVSRAeoQHxDBWIE+Vn08dzeGKUEEaxNCM0nKp/LzLElRrzwGxypIdq0ZuI/3leqqNLP6MiSTUReBYUpQzqGE5agCGVBGs2NgRhSc2vEA+RRFibruZSAp6bTtzFBpZJu15zz2r1u/Nq46popwSOwDE4BS64AA1wA5qgBTBIwQt4BW/Ws/VufVifs9UVq7g5BHOwvn4BzvKZyA==</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Gdm<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>hM<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit><latexit sha1_base64="ywqRJkxyH+HCmiJcuw5VySAcw0s=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FGg8qNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvXxQNlPk=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

H<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

d-order tensor<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="91pa8qrcUAgEcxhJ3mIvPsSfh9c=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFN11WsA+YDiWTZtrQJDMkGaEMXfkNbnXtTtz6JS79EzNtF7b1QOBwzr3ckxMmnGnjut9OYWNza3unuFva2z84PCofn7R1nCpCWyTmseqGWFPOJG0ZZjjtJopiEXLaCcf3ud95okqzWD6aSUIDgYeSRYxgYyW/J7AZEcyzxrRfrrhVdwa0TrwFqcACzX75pzeISSqoNIRjrX3PTUyQYWUY4XRa6qWaJpiM8ZD6lkosqA6yWeQpurDKAEWxsk8aNFP/bmRYaD0RoZ3MI+pVLxf/8/zURLdBxmSSGirJ/FCUcmRilP8fDZiixPCJJZgoZrMiMsIKE2NbWroSirwTb7WBddKuVb2rau3hulK/W7RThDM4h0vw4Abq0IAmtIBADC/wCm/Os/PufDif89GCs9g5hSU4X79dx5f4</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>w<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="E0sSRXFguO4RuUcPYFpQNC5WE9c=">AAAB+3icbVBNS8NAEJ34WetX1aOXxSJ4KkkV9Fj04rGCaQttKJvtpl26uwm7G6WE/AavevYmXv0xHv0nbtscbOuDgcd7M8zMCxPOtHHdb2dtfWNza7u0U97d2z84rBwdt3ScKkJ9EvNYdUKsKWeS+oYZTjuJoliEnLbD8d3Ubz9RpVksH80koYHAQ8kiRrCxkt8LBXruV6puzZ0BrRKvIFUo0OxXfnqDmKSCSkM41rrruYkJMqwMI5zm5V6qaYLJGA9p11KJBdVBNjs2R+dWGaAoVrakQTP170SGhdYTEdpOgc1IL3tT8T+vm5roJsiYTFJDJZkvilKOTIymn6MBU5QYPrEEE8XsrYiMsMLE2HwWtoQit5l4ywmskla95l3W6g9X1cZtkU4JTuEMLsCDa2jAPTTBBwIMXuAV3pzceXc+nM9565pTzJzAApyvXyuqlQg=</latexit>M<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>=<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>h< w, h >

G1

Gd

m

<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

d-order tensor<latexit sha1_base64="ywqRJkxyH+HCmiJcuw5VySAcw0s=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FGg8qNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvXxQNlPk=</latexit><latexit sha1_base64="Axg69VjX69O3PJsUELEX0IW75G0=">AAACBHicbVA9SwNBEJ2LXzF+RS1tFoNgIeEuClqIBG0sI5gPSM6wt9lLluzeHbt7SjjS+htstbYTW/+Hpf/EveQKk/hgmMd7M8zwvIgzpW3728otLa+sruXXCxubW9s7xd29hgpjSWidhDyULQ8ryllA65ppTluRpFh4nDa94U3qNx+pVCwM7vUooq7A/YD5jGBtpIfLjifQ0wlK2+CqWyzZZXsCtEicjJQgQ61b/On0QhILGmjCsVJtx460m2CpGeF0XOjEikaYDHGftg0NsKDKTSZfj9GRUXrID6WpQKOJ+ncjwUKpkfDMpMB6oOa9VPzPa8fav3ATFkSxpgGZHvJjjnSI0ghQj0lKNB8Zgolk5ldEBlhiok1QM1c8MTaZOPMJLJJGpeyclit3Z6XqdZZOHg7gEI7BgXOowi3UoA4EJLzAK7xZz9a79WF9TkdzVrazDzOwvn4Bdd6X2w==</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>< TT(G1 , · · · , Gd ), H ><latexit sha1_base64="+CLTasYnoQoLobMZcjyYlDMt0hQ=">AAACNHicbVDLSsNAFJ3UV31HXboZLEKFUhIVdCFSdGGXFVoVmhImk2k7dPJg5kYsIT/jR/gNbnUnuBFx6zc4rUG09cDAuefcy71zvFhwBZb1YhRmZufmF4qLS8srq2vr5sbmlYoSSVmLRiKSNx5RTPCQtYCDYDexZCTwBLv2Bucj//qWScWjsAnDmHUC0gt5l1MCWnLNkxMH2B2kzWZWdgICfUpEepG5dsWhfgSq8lv09yr4p65np65ZsqrWGHia2DkpoRwN13xz/IgmAQuBCqJU27Zi6KREAqeCZUtOolhM6ID0WFvTkARMddLxLzO8qxUfdyOpXwh4rP6eSEmg1DDwdOfoRjXpjcT/vHYC3eNOysM4ARbS70XdRGCI8Cgy7HPJKIihJoRKrm/FtE8koaCD/bPFCzKdiT2ZwDS52q/aB9X9y8NS7SxPp4i20Q4qIxsdoRqqowZqIYru0SN6Qs/Gg/FqvBsf360FI5/ZQn9gfH4BsCer+Q==</latexit>

41

H<latexit sha1_base64="91pa8qrcUAgEcxhJ3mIvPsSfh9c=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFN11WsA+YDiWTZtrQJDMkGaEMXfkNbnXtTtz6JS79EzNtF7b1QOBwzr3ckxMmnGnjut9OYWNza3unuFva2z84PCofn7R1nCpCWyTmseqGWFPOJG0ZZjjtJopiEXLaCcf3ud95okqzWD6aSUIDgYeSRYxgYyW/J7AZEcyzxrRfrrhVdwa0TrwFqcACzX75pzeISSqoNIRjrX3PTUyQYWUY4XRa6qWaJpiM8ZD6lkosqA6yWeQpurDKAEWxsk8aNFP/bmRYaD0RoZ3MI+pVLxf/8/zURLdBxmSSGirJ/FCUcmRilP8fDZiixPCJJZgoZrMiMsIKE2NbWroSirwTb7WBddKuVb2rau3hulK/W7RThDM4h0vw4Abq0IAmtIBADC/wCm/Os/PufDif89GCs9g5hSU4X79dx5f4</latexit>=<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

TN ContractionVector to Vector Transformation via TN
Hidden layerTh2 = (W h1 + b)
<latexit sha1_base64="vbyYqORwDxiIPWWbVv1/yM0qAVg=">AAACIXicbZDLSsNAFIYnXmu9RV26GSxCi1CSKuhGKLpxWaE3aGKYTKft0JkkzEyEEvoMPoTP4FbX7sSduPJNnKRZ2NYDAx//fw7nzO9HjEplWV/Gyura+sZmYau4vbO7t28eHLZlGAtMWjhkoej6SBJGA9JSVDHSjQRB3Gek449vU7/zSISkYdBUk4i4HA0DOqAYKS15ZsXxORx5NXgNHUmHHJVTofPQhDPDhmcZ+RXPLFlVKyu4DHYOJZBXwzN/nH6IY04ChRmSsmdbkXITJBTFjEyLTixJhPAYDUlPY4A4kW6SfWkKT7XSh4NQ6BcomKl/JxLEpZxwX3dypEZy0UvF/7xerAZXbkKDKFYkwLNFg5hBFcI0H9ingmDFJhoQFlTfCvEICYSVTnFui8+nOhN7MYFlaNeq9nm1dn9Rqt/k6RTAMTgBZWCDS1AHd6ABWgCDJ/ACXsGb8Wy8Gx/G56x1xchnjsBcGd+/d8uhLQ==</latexit>

Hidden layer(N ⇥ M )Weights<latexit sha1_base64="HeGff66rglnceMyNtgORFBnRcG0=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjRulgn3AdCiZNG1Dk8mQ3BHK0JXf4FbX7sStX+LSPzFtZ2FbDwQO59zLuTlhLLgB1/12VlbX1jc2c1v57Z3dvf3CwWHDqERTVqdKKN0KiWGCR6wOHARrxZoRGQrWDIc3E7/5xLThKnqEUcwCSfoR73FKwEp+6b4NXDKD7846haJbdqfAy8TLSBFlqHUKP+2uoolkEVBBjPE9N4YgJRo4FWycbyeGxYQOSZ/5lkbE5gTp9OQxPrVKF/eUti8CPFX/bqREGjOSoZ2UBAZm0ZuI/3l+Ar2rIOVRnACL6CyolwgMCk/+j7tcMwpiZAmhmttbMR0QTSjYluZSQjm2nXiLDSyTRqXsnZcrDxfF6nXWTg4doxNUQh66RFV0i2qojihS6AW9ojfn2Xl3PpzP2eiKk+0coTk4X7/mipcM</latexit>

Numberof parameters

Tensorization
M

d

M =m

<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>

TT Representationd<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>mm
2d-ordertensor<latexit sha1_base64="jsCvViEpXziPr7GYTUbXuMCt0Dk=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FKgzqNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvX/k7lOg=</latexit>

N

N =n<latexit sha1_base64="NprZbnjsLty/0NXDdUVSpVNUekY=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLGSCOYDkjPs7e0lq7t7x+6eEEL8DbZa24mt/8XSf+ImucIkPhh4vDfDzLwg4Uwb1/12ckvLK6tr+fXCxubW9k5xd6+h41QRWicxj1UrwJpyJmndMMNpK1EUi4DTZvB4NfabT1RpFss7M0ioL3BPsogRbKzUuLlA8j7sFktu2Z0ALRIvIyXIUOsWfzphTFJBpSEca9323MT4Q6wMI5yOCp1U0wSTR9yjbUslFlT7w8m1I3RklRBFsbIlDZqofyeGWGg9EIHtFNj09bw3Fv/z2qmJzv0hk0lqqCTTRVHKkYnR+HUUMkWJ4QNLMFHM3opIHytMjA1oZksgRjYTbz6BRdKolL2TcuX2tFS9zNLJwwEcwjF4cAZVuIYa1IHAA7zAK7w5z8678+F8TltzTjazDzNwvn4BeISVKw==</latexit><latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>dd M N r2
<latexit sha1_base64="k+oMM7QXifjnamxmcIy1KMjBoXI=">AAACCHicbVDLSsNAFJ3UV62vqEs3g0VwVZIq6LLoxo1SwT6gjWUymbRDZyZxZlIoIT/gN7jVtTtx61+49E+ctlnY1gMXDufcy7kcP2ZUacf5tgorq2vrG8XN0tb2zu6evX/QVFEiMWngiEWy7SNFGBWkoalmpB1LgrjPSMsfXk/81ohIRSPxoMcx8TjqCxpSjLSRerYddNWT1J3AS2/vMvlY7dllp+JMAZeJm5MyyFHv2T/dIMIJJ0JjhpTquE6svRRJTTEjWambKBIjPER90jFUIE6Ul04/z+CJUQIYRtKM0HCq/r1IEVdqzH2zyZEeqEVvIv7ndRIdXnopFXGiicCzoDBhUEdwUgMMqCRYs7EhCEtqfoV4gCTC2pQ1l+LzzHTiLjawTJrVintWqd6fl2tXeTtFcASOwSlwwQWogRtQBw2AwQi8gFfwZj1b79aH9TlbLVj5zSGYg/X1C3SfmiA=</latexit>m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="tF5plF15dsMELhjvTzdCGgMZ7Ww=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLERIpgPSM6wt7eXLNndO3b3hBCCv8FWazux9bdY+k/cJFeYxAcDj/dmmJkXJJxp47rfTm5ldW19I79Z2Nre2d0r7h80dJwqQusk5rFqBVhTziStG2Y4bSWKYhFw2gwGNxO/+USVZrF8MMOE+gL3JIsYwcZKzTt0hcRj2C2W3LI7BVomXkZKkKHWLf50wpikgkpDONa67bmJ8UdYGUY4HRc6qaYJJgPco21LJRZU+6PpuWN0YpUQRbGyJQ2aqn8nRlhoPRSB7RTY9PWiNxH/89qpiS79EZNJaqgks0VRypGJ0eR3FDJFieFDSzBRzN6KSB8rTIxNaG5LIMY2E28xgWXSqJS9s3Ll/rxUvc7SycMRHMMpeHABVbiFGtSBwABe4BXenGfn3flwPmetOSebOYQ5OF+/zaGVUw==</latexit>W

TT-rankpd<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

n

n

=

r
G1
<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

d

n

Gd
<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>n<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>N<latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>h1

N = nd<latexit sha1_base64="NprZbnjsLty/0NXDdUVSpVNUekY=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLGSCOYDkjPs7e0lq7t7x+6eEEL8DbZa24mt/8XSf+ImucIkPhh4vDfDzLwg4Uwb1/12ckvLK6tr+fXCxubW9k5xd6+h41QRWicxj1UrwJpyJmndMMNpK1EUi4DTZvB4NfabT1RpFss7M0ioL3BPsogRbKzUuLlA8j7sFktu2Z0ALRIvIyXIUOsWfzphTFJBpSEca9323MT4Q6wMI5yOCp1U0wSTR9yjbUslFlT7w8m1I3RklRBFsbIlDZqofyeGWGg9EIHtFNj09bw3Fv/z2qmJzv0hk0lqqCTTRVHKkYnR+HUUMkWJ4QNLMFHM3opIHytMjA1oZksgRjYTbz6BRdKolL2TcuX2tFS9zNLJwwEcwjF4cAZVuIYa1IHAA7zAK7w5z8678+F8TltzTjazDzNwvn4BeISVKw==</latexit>

d-order tensor

W T h1<latexit sha1_base64="L3726S0wPaAZMHE7GLy2w1g1F1Q=">AAACB3icbVDLSsNAFL2pr1ofjbp0M1gEVyWpgi6LblxW6AvaGCbTSTt0JgkzE6GUfoDf4FbX7sStn+HSP3HSZmFbD1zu4Zx7uZcTJJwp7TjfVmFjc2t7p7hb2ts/OCzbR8dtFaeS0BaJeSy7AVaUs4i2NNOcdhNJsQg47QTju8zvPFGpWBw19SShnsDDiIWMYG0k3y73A4E6j02U9ZHv+nbFqTpzoHXi5qQCORq+/dMfxCQVNNKEY6V6rpNob4qlZoTTWamfKppgMsZD2jM0woIqbzp/fIbOjTJAYSxNRRrN1b8bUyyUmojATAqsR2rVy8T/vF6qwxtvyqIk1TQii0NhypGOUZYCGjBJieYTQzCRzPyKyAhLTLTJaulKIGYmE3c1gXXSrlXdy2rt4apSv83TKcIpnMEFuHANdbiHBrSAQAov8Apv1rP1bn1Yn4vRgpXvnMASrK9f3diYkg==</latexit>

H1<latexit sha1_base64="nKqATXhVI8HwzzcJt8za2drnZZg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu17/VLZrbgzoFXi5aQMORr90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtK9f6qXLvN2ynAKZzBBXhwDTWoQwOaQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGT4Zic</latexit><latexit sha1_base64="biZCzotYpwHbJI17GwildibdgII=">AAAB/XicbVBNSwMxEJ31s9avqkcvwSJ4KrtV0GPRi8cK9gPapSRptg1NskuSFcpS/A1e9exNvPpbPPpPTNs92NYHA4/3ZpiZRxLBjfX9b29tfWNza7uwU9zd2z84LB0dN02casoaNBaxbhNsmOCKNSy3grUTzbAkgrXI6G7qt56YNjxWj3acsFDigeIRp9g6qdUlEg17Qa9U9iv+DGiVBDkpQ456r/TT7cc0lUxZKrAxncBPbJhhbTkVbFLspoYlmI7wgHUcVVgyE2azcyfo3Cl9FMXalbJopv6dyLA0ZiyJ65TYDs2yNxX/8zqpjW7CjKsktUzR+aIoFcjGaPo76nPNqBVjRzDV3N2K6BBrTK1LaGELkROXSbCcwCppVivBZaX6cFWu3ebpFOAUzuACAriGGtxDHRpAYQQv8Apv3rP37n14n/PWNS+fOYEFeF+/RImVnQ==</latexit>=

TT(G1 , · · · , Gd ) ⇥1,...,d H1<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="d4jmzzfyxXQpVdzRJhUlmj9ovMk=">AAACSHicbVDJSgNBEO2JW4xb1KOXxiBECGEmCnoMejDHCNkgE4aenk7S2LPQXSOGYX7Lj/APBC8e9OxNvNlZkCwWNP149R5V9dxIcAWm+Wpk1tY3Nrey27md3b39g/zhUUuFsaSsSUMRyo5LFBM8YE3gIFgnkoz4rmBt9+F23G8/Mql4GDRgFLGeTwYB73NKQFNOvm4De4Kk0UiLtk9gSIlI7lLHKmGbeiEo/c/R3jm2gftMOYlVssVE4KV/ipo2OvmCWTYnhVeBNQMFNKu6k3+3vZDGPguACqJU1zIj6CVEAqeCpTk7Viwi9IEMWFfDgOjxvWRyeYrPNOPhfij1CwBP2HlHQnylRr6rleMl1XJvTP7X68bQv+4lPIhiYAGdDurHAkOIxzFij0tGQYw0IFRyvSumQyIJBR32whTXT3Um1nICq6BVKVsX5cr9ZaF6M0sni07QKSoiC12hKqqhOmoiip7RG/pAn8aL8WV8Gz9TacaYeY7RQmUyv1ELszA=</latexit>

42Vector to Vector Transformation via TN
Hidden layerh2 = (W T h1 + b)
<latexit sha1_base64="vbyYqORwDxiIPWWbVv1/yM0qAVg=">AAACIXicbZDLSsNAFIYnXmu9RV26GSxCi1CSKuhGKLpxWaE3aGKYTKft0JkkzEyEEvoMPoTP4FbX7sSduPJNnKRZ2NYDAx//fw7nzO9HjEplWV/Gyura+sZmYau4vbO7t28eHLZlGAtMWjhkoej6SBJGA9JSVDHSjQRB3Gek449vU7/zSISkYdBUk4i4HA0DOqAYKS15ZsXxORx5NXgNHUmHHJVTofPQhDPDhmcZ+RXPLFlVKyu4DHYOJZBXwzN/nH6IY04ChRmSsmdbkXITJBTFjEyLTixJhPAYDUlPY4A4kW6SfWkKT7XSh4NQ6BcomKl/JxLEpZxwX3dypEZy0UvF/7xerAZXbkKDKFYkwLNFg5hBFcI0H9ingmDFJhoQFlTfCvEICYSVTnFui8+nOhN7MYFlaNeq9nm1dn9Rqt/k6RTAMTgBZWCDS1AHd6ABWgCDJ/ACXsGb8Wy8Gx/G56x1xchnjsBcGd+/d8uhLQ==</latexit>

Hidden layer

TensorizationM

M =m<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>m
2d-ordertensor<latexit sha1_base64="jsCvViEpXziPr7GYTUbXuMCt0Dk=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FKgzqNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvX/k7lOg=</latexit>

N =n

N

d

=

r
G1

Gd

n

<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>ddn<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit><latexit sha1_base64="NprZbnjsLty/0NXDdUVSpVNUekY=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLGSCOYDkjPs7e0lq7t7x+6eEEL8DbZa24mt/8XSf+ImucIkPhh4vDfDzLwg4Uwb1/12ckvLK6tr+fXCxubW9k5xd6+h41QRWicxj1UrwJpyJmndMMNpK1EUi4DTZvB4NfabT1RpFss7M0ioL3BPsogRbKzUuLlA8j7sFktu2Z0ALRIvIyXIUOsWfzphTFJBpSEca9323MT4Q6wMI5yOCp1U0wSTR9yjbUslFlT7w8m1I3RklRBFsbIlDZqofyeGWGg9EIHtFNj09bw3Fv/z2qmJzv0hk0lqqCTTRVHKkYnR+HUUMkWJ4QNLMFHM3opIHytMjA1oZksgRjYTbz6BRdKolL2TcuX2tFS9zNLJwwEcwjF4cAZVuIYa1IHAA7zAK7w5z8678+F8TltzTjazDzNwvn4BeISVKw==</latexit>

H1

d-order tensor<latexit sha1_base64="biZCzotYpwHbJI17GwildibdgII=">AAAB/XicbVBNSwMxEJ31s9avqkcvwSJ4KrtV0GPRi8cK9gPapSRptg1NskuSFcpS/A1e9exNvPpbPPpPTNs92NYHA4/3ZpiZRxLBjfX9b29tfWNza7uwU9zd2z84LB0dN02casoaNBaxbhNsmOCKNSy3grUTzbAkgrXI6G7qt56YNjxWj3acsFDigeIRp9g6qdUlEg17Qa9U9iv+DGiVBDkpQ456r/TT7cc0lUxZKrAxncBPbJhhbTkVbFLspoYlmI7wgHUcVVgyE2azcyfo3Cl9FMXalbJopv6dyLA0ZiyJ65TYDs2yNxX/8zqpjW7CjKsktUzR+aIoFcjGaPo76nPNqBVjRzDV3N2K6BBrTK1LaGELkROXSbCcwCppVivBZaX6cFWu3ebpFOAUzuACAriGGtxDHRpAYQQv8Apv3rP37n14n/PWNS+fOYEFeF+/RImVnQ==</latexit><latexit sha1_base64="nKqATXhVI8HwzzcJt8za2drnZZg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu17/VLZrbgzoFXi5aQMORr90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtK9f6qXLvN2ynAKZzBBXhwDTWoQwOaQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGT4Zic</latexit>mM<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>r<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>Wr<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>=<latexit sha1_base64="jsCvViEpXziPr7GYTUbXuMCt0Dk=">AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48VTFtoQ9lsN+3S3U3Y3Qgl5Dd41bM38eqP8eg/cdvmYFsfDDzem2FmXphwpo3rfjuljc2t7Z3ybmVv/+DwqHp80tZxqgj1Scxj1Q2xppxJ6htmOO0mimIRctoJJ/czv/NMlWaxfDLThAYCjySLGMHGSn4/FKgzqNbcujsHWideQWpQoDWo/vSHMUkFlYZwrHXPcxMTZFgZRjjNK/1U0wSTCR7RnqUSC6qDbH5sji6sMkRRrGxJg+bq34kMC62nIrSdApuxXvVm4n9eLzXRbZAxmaSGSrJYFKUcmRjNPkdDpigxfGoJJorZWxEZY4WJsfksbQlFbjPxVhNYJ+1G3buqNx6va827Ip0ynME5XIIHN9CEB2iBDwQYvMArvDm58+58OJ+L1pJTzJzCEpyvX/k7lOg=</latexit>N<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>

h1

G1

T

W h1

=
H

Gd

n

<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>M<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

d-order tensor<latexit sha1_base64="biZCzotYpwHbJI17GwildibdgII=">AAAB/XicbVBNSwMxEJ31s9avqkcvwSJ4KrtV0GPRi8cK9gPapSRptg1NskuSFcpS/A1e9exNvPpbPPpPTNs92NYHA4/3ZpiZRxLBjfX9b29tfWNza7uwU9zd2z84LB0dN02casoaNBaxbhNsmOCKNSy3grUTzbAkgrXI6G7qt56YNjxWj3acsFDigeIRp9g6qdUlEg17Qa9U9iv+DGiVBDkpQ456r/TT7cc0lUxZKrAxncBPbJhhbTkVbFLspoYlmI7wgHUcVVgyE2azcyfo3Cl9FMXalbJopv6dyLA0ZiyJ65TYDs2yNxX/8zqpjW7CjKsktUzR+aIoFcjGaPo76nPNqBVjRzDV3N2K6BBrTK1LaGELkROXSbCcwCppVivBZaX6cFWu3ebpFOAUzuACAriGGtxDHRpAYQQv8Apv3rP37n14n/PWNS+fOYEFeF+/RImVnQ==</latexit><latexit sha1_base64="L3726S0wPaAZMHE7GLy2w1g1F1Q=">AAACB3icbVDLSsNAFL2pr1ofjbp0M1gEVyWpgi6LblxW6AvaGCbTSTt0JgkzE6GUfoDf4FbX7sStn+HSP3HSZmFbD1zu4Zx7uZcTJJwp7TjfVmFjc2t7p7hb2ts/OCzbR8dtFaeS0BaJeSy7AVaUs4i2NNOcdhNJsQg47QTju8zvPFGpWBw19SShnsDDiIWMYG0k3y73A4E6j02U9ZHv+nbFqTpzoHXi5qQCORq+/dMfxCQVNNKEY6V6rpNob4qlZoTTWamfKppgMsZD2jM0woIqbzp/fIbOjTJAYSxNRRrN1b8bUyyUmojATAqsR2rVy8T/vF6qwxtvyqIk1TQii0NhypGOUZYCGjBJieYTQzCRzPyKyAhLTLTJaulKIGYmE3c1gXXSrlXdy2rt4apSv83TKcIpnMEFuHANdbiHBrSAQAov8Apv1rP1bn1Yn4vRgpXvnMASrK9f3diYkg==</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

N =n

h1

<latexit sha1_base64="k+oMM7QXifjnamxmcIy1KMjBoXI=">AAACCHicbVDLSsNAFJ3UV62vqEs3g0VwVZIq6LLoxo1SwT6gjWUymbRDZyZxZlIoIT/gN7jVtTtx61+49E+ctlnY1gMXDufcy7kcP2ZUacf5tgorq2vrG8XN0tb2zu6evX/QVFEiMWngiEWy7SNFGBWkoalmpB1LgrjPSMsfXk/81ohIRSPxoMcx8TjqCxpSjLSRerYddNWT1J3AS2/vMvlY7dllp+JMAZeJm5MyyFHv2T/dIMIJJ0JjhpTquE6svRRJTTEjWambKBIjPER90jFUIE6Ul04/z+CJUQIYRtKM0HCq/r1IEVdqzH2zyZEeqEVvIv7ndRIdXnopFXGiicCzoDBhUEdwUgMMqCRYs7EhCEtqfoV4gCTC2pQ1l+LzzHTiLjawTJrVintWqd6fl2tXeTtFcASOwSlwwQWogRtQBw2AwQi8gFfwZj1b79aH9TlbLVj5zSGYg/X1C3SfmiA=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit><latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>d M N r2

<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>n<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

N

p
d

m
<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>n<latexit sha1_base64="NprZbnjsLty/0NXDdUVSpVNUekY=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLGSCOYDkjPs7e0lq7t7x+6eEEL8DbZa24mt/8XSf+ImucIkPhh4vDfDzLwg4Uwb1/12ckvLK6tr+fXCxubW9k5xd6+h41QRWicxj1UrwJpyJmndMMNpK1EUi4DTZvB4NfabT1RpFss7M0ioL3BPsogRbKzUuLlA8j7sFktu2Z0ALRIvIyXIUOsWfzphTFJBpSEca9323MT4Q6wMI5yOCp1U0wSTR9yjbUslFlT7w8m1I3RklRBFsbIlDZqofyeGWGg9EIHtFNj09bw3Fv/z2qmJzv0hk0lqqCTTRVHKkYnR+HUUMkWJ4QNLMFHM3opIHytMjA1oZksgRjYTbz6BRdKolL2TcuX2tFS9zNLJwwEcwjF4cAZVuIYa1IHAA7zAK7w5z8678+F8TltzTjazDzNwvn4BeISVKw==</latexit><latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>

Numberof parameters

TT-rank<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="tF5plF15dsMELhjvTzdCGgMZ7Ww=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioI0QtLERIpgPSM6wt7eXLNndO3b3hBCCv8FWazux9bdY+k/cJFeYxAcDj/dmmJkXJJxp47rfTm5ldW19I79Z2Nre2d0r7h80dJwqQusk5rFqBVhTziStG2Y4bSWKYhFw2gwGNxO/+USVZrF8MMOE+gL3JIsYwcZKzTt0hcRj2C2W3LI7BVomXkZKkKHWLf50wpikgkpDONa67bmJ8UdYGUY4HRc6qaYJJgPco21LJRZU+6PpuWN0YpUQRbGyJQ2aqn8nRlhoPRSB7RTY9PWiNxH/89qpiS79EZNJaqgks0VRypGJ0eR3FDJFieFDSzBRzN6KSB8rTIxNaG5LIMY2E28xgWXSqJS9s3Ll/rxUvc7SycMRHMMpeHABVbiFGtSBwABe4BXenGfn3flwPmetOSebOYQ5OF+/zaGVUw==</latexit>

W

Weights<latexit sha1_base64="HeGff66rglnceMyNtgORFBnRcG0=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjRulgn3AdCiZNG1Dk8mQ3BHK0JXf4FbX7sStX+LSPzFtZ2FbDwQO59zLuTlhLLgB1/12VlbX1jc2c1v57Z3dvf3CwWHDqERTVqdKKN0KiWGCR6wOHARrxZoRGQrWDIc3E7/5xLThKnqEUcwCSfoR73FKwEp+6b4NXDKD7846haJbdqfAy8TLSBFlqHUKP+2uoolkEVBBjPE9N4YgJRo4FWycbyeGxYQOSZ/5lkbE5gTp9OQxPrVKF/eUti8CPFX/bqREGjOSoZ2UBAZm0ZuI/3l+Ar2rIOVRnACL6CyolwgMCk/+j7tcMwpiZAmhmttbMR0QTSjYluZSQjm2nXiLDSyTRqXsnZcrDxfF6nXWTg4doxNUQh66RFV0i2qojihS6AW9ojfn2Xl3PpzP2eiKk+0coTk4X7/mipcM</latexit>

TT Representation

d

m

d

<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>(N ⇥ M )TT(G1 , · · · , Gd ) ⇥1,...,d H1<latexit sha1_base64="d4jmzzfyxXQpVdzRJhUlmj9ovMk=">AAACSHicbVDJSgNBEO2JW4xb1KOXxiBECGEmCnoMejDHCNkgE4aenk7S2LPQXSOGYX7Lj/APBC8e9OxNvNlZkCwWNP149R5V9dxIcAWm+Wpk1tY3Nrey27md3b39g/zhUUuFsaSsSUMRyo5LFBM8YE3gIFgnkoz4rmBt9+F23G8/Mql4GDRgFLGeTwYB73NKQFNOvm4De4Kk0UiLtk9gSIlI7lLHKmGbeiEo/c/R3jm2gftMOYlVssVE4KV/ipo2OvmCWTYnhVeBNQMFNKu6k3+3vZDGPguACqJU1zIj6CVEAqeCpTk7Viwi9IEMWFfDgOjxvWRyeYrPNOPhfij1CwBP2HlHQnylRr6rleMl1XJvTP7X68bQv+4lPIhiYAGdDurHAkOIxzFij0tGQYw0IFRyvSumQyIJBR32whTXT3Um1nICq6BVKVsX5cr9ZaF6M0sni07QKSoiC12hKqqhOmoiip7RG/pAn8aL8WV8Gz9TacaYeY7RQmUyv1ELszA=</latexit>

43<latexit sha1_base64="nKqATXhVI8HwzzcJt8za2drnZZg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtUkYwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Pu17/VLZrbgzoFXi5aQMORr90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtK9f6qXLvN2ynAKZzBBXhwDTWoQwOaQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGT4Zic</latexit>

1

TN Contraction

Learning of TN Weights
h2 = (W T h1 + b)o2<latexit sha1_base64="vbyYqORwDxiIPWWbVv1/yM0qAVg=">AAACIXicbZDLSsNAFIYnXmu9RV26GSxCi1CSKuhGKLpxWaE3aGKYTKft0JkkzEyEEvoMPoTP4FbX7sSduPJNnKRZ2NYDAx//fw7nzO9HjEplWV/Gyura+sZmYau4vbO7t28eHLZlGAtMWjhkoej6SBJGA9JSVDHSjQRB3Gek449vU7/zSISkYdBUk4i4HA0DOqAYKS15ZsXxORx5NXgNHUmHHJVTofPQhDPDhmcZ+RXPLFlVKyu4DHYOJZBXwzN/nH6IY04ChRmSsmdbkXITJBTFjEyLTixJhPAYDUlPY4A4kW6SfWkKT7XSh4NQ6BcomKl/JxLEpZxwX3dypEZy0UvF/7xerAZXbkKDKFYkwLNFg5hBFcI0H9ingmDFJhoQFlTfCvEICYSVTnFui8+nOhN7MYFlaNeq9nm1dn9Rqt/k6RTAMTgBZWCDS1AHd6ABWgCDJ/ACXsGb8Wy8Gx/G56x1xchnjsBcGd+/d8uhLQ==</latexit><latexit sha1_base64="YA3YTF53qS01V1C9Vz2QU1pyR4E=">AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hd0o6DHoxWME84BkCbOT2WTIPJaZWSEswW/wqmdv4tVv8eifOEn2YBILGoqqbrq7ooQzY33/21tb39jc2i7sFHf39g8OS0fHTaNSTWiDKK50O8KGciZpwzLLaTvRFIuI01Y0upv6rSeqDVPy0Y4TGgo8kCxmBFsntbqRQKpX7ZXKfsWfAa2SICdlyFHvlX66fUVSQaUlHBvTCfzEhhnWlhFOJ8VuamiCyQgPaMdRiQU1YTY7d4LOndJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC099Rn2lKLB87golm7lZEhlhjYl1CC1siMXGZBMsJrJJmtRJcVqoPV+XabZ5OAU7hDC4ggGuowT3UoQEERvACr/DmPXvv3of3OW9d8/KZE1iA9/ULUS+VpQ==</latexit>‣ Loss: min L(W , x, y) ) min L({G , · · · , G }, x, y)‣ Gradients over TN core tensors<latexit sha1_base64="o/pLwqY8r6ARz1aT0Op+M4N/2Bw=">AAAClXicbVFdSxtBFJ1drVVrNa0PPvgyGASFEHaj0L4URIv6IPhBY4RsWGZnJ8ngfCwzd1vDsj+lP6yP/SfOxhWS6IVhDueeM/dyJskEtxAE/zx/afnDysfVtfVPG583txpfvt5bnRvKulQLbR4SYpnginWBg2APmWFEJoL1ksezqt/7zYzlWv2CScYGkowUH3JKwFFx428kuYqLKJG4V0aSwJgSUVyVB1OmhavrqYUnhzi646MxEGP0H1ybXuUXZRw6KU012NYsm5Z47s0Fy3uOqJwZGjeaQTuYFn4Lwho0UV03ceN/lGqaS6aACmJtPwwyGBTEAKeCletRbllG6CMZsb6DikhmB8U0xhLvOybFQ23cUYCn7KyjINLaiUycstrYLvYq8r1eP4fh90HBVZYDU/Rl0DAXGDSu/gSn3DAKYuIAoYa7XTEdE0MouJ+bm5LI0mUSLibwFtx32uFRu3N73Dw5rdNZRbtoDx2gEH1DJ+gS3aAuot6Sd+h1vCN/x//h//TPX6S+V3u20Vz518+iJ8mx</latexit>

1G1 ,··· ,GdWdd<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="PWeJA6IF0367g81E/pSbi6scmG0=">AAACtXicbVFdS8MwFE3r1/ye+uhLcAg+jXYK+iKIPqjgw/zYB6ylpFm6hSVNSVJhlPw0f4iP/hPTOcTOXQgczj0n93JunDGqtOd9Ou7K6tr6Rm1za3tnd2+/fnDYVSKXmHSwYEL2Y6QIoynpaKoZ6WeSIB4z0osnd2W/906koiJ909OMhByNUppQjLSlovpHkEiEiyBDUlPEAo70GCNWPBnzSxa/7L2JJsbAa1h1waU2GMQcjqOWWRTHvBgby1eVYrlSLCgrq0T1htf0ZgX/A38OGmBe7aj+FQwFzjlJNWZIqYHvZTosys8xI2YryBXJEJ6gERlYmCJOVFjMYjbw1DJDmAhpX6rhjP3rKBBXaspjqyy3VIu9klzWG+Q6uQoLmma5Jin+GZTkDGoBy5vBIZUEaza1AGFJ7a4Qj5GNStvLVqbEvMzEX0zgP+i2mv55s/V80bi5nadTA8fgBJwBH1yCG/AA2qADsNNwHp0X59W9dEN36CY/UteZe45ApVzxDWS83A0=</latexit>r@L@L @h2 @o2=@Gk@h2 @o2 @Gk

G1<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="ZLNEFqdz0hPzSQqT/my2G1muybo=">AAACDXicbVDLSsNAFJ3UV62vqLhyM1iEuilJFXRZdOOygn1Ak5bJZNIOnZmEmYlQQr/Bb3Cra3fi1m9w6Z84bbOwrQcuHM65l3M5QcKo0o7zbRXW1jc2t4rbpZ3dvf0D+/CopeJUYtLEMYtlJ0CKMCpIU1PNSCeRBPGAkXYwupv67SciFY3Fox4nxOdoIGhEMdJG6tsnXsBhG1ZEL/Q05URB3gsv+nbZqTozwFXi5qQMcjT69o8XxjjlRGjMkFJd10m0nyGpKWZkUvJSRRKER2hAuoYKZJL8bPb+BJ4bJYRRLM0IDWfq34sMcaXGPDCbHOmhWvam4n9eN9XRjZ9RkaSaCDwPilIGdQynXcCQSoI1GxuCsKTmV4iHSCKsTWMLKQGfmE7c5QZWSatWdS+rtYercv02b6cITsEZqAAXXIM6uAcN0AQYZOAFvII369l6tz6sz/lqwcpvjsECrK9fkuabJw==</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

Q1<latexit sha1_base64="Rm4XceLAlo1yWQZfjVksZUh/0hQ=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLBMwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Me17/VLZrbgzoFXi5aQMOer90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtKtXFVrt3m7RTgFM7gAjy4hhrcQx2aQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGiJ5il</latexit>rr<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>n<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

W (nd ⇥ md )<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

G1

Gd

n

<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>mm<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

d-order tensor

d-order tensorrr<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Gd

n

Qd

d-order tensor<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="3MFnF3TUwkZrFTYHOmHnfr6C7MY=">AAACA3icbVDLSsNAFL3xWeur6tJNsAiuSlIFXRbduGzBPqANZTKZtENnJnFmIpSQpd/gVtfuxK0f4tI/cdJmYVsPXDiccy/3cPyYUaUd59taW9/Y3Nou7ZR39/YPDitHxx0VJRKTNo5YJHs+UoRRQdqaakZ6sSSI+4x0/cld7nefiFQ0Eg96GhOPo5GgIcVIG8kbcKTHGLG0lQ2DYaXq1JwZ7FXiFqQKBZrDys8giHDCidCYIaX6rhNrL0VSU8xIVh4kisQIT9CI9A0ViBPlpbPQmX1ulMAOI2lGaHum/r1IEVdqyn2zmYdUy14u/uf1Ex3eeCkVcaKJwPNHYcJsHdl5A3ZAJcGaTQ1BWFKT1cZjJBHWpqeFLz7PTCfucgOrpFOvuZe1euuq2rgt2inBKZzBBbhwDQ24hya0AcMjvMArvFnP1rv1YX3OV9es4uYEFmB9/QLycJjY</latexit>rm<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>n<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

44

r

r

m
<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>n<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

Compression Performance‣ Model compression on ImageNet‣ SpeedTensorizing Neural Networks (Novikov et al., NIPS 2015)45

Ideal Case: TN to TN Transformation
Hidden layerh2 = (W T h1 + b)
<latexit sha1_base64="vbyYqORwDxiIPWWbVv1/yM0qAVg=">AAACIXicbZDLSsNAFIYnXmu9RV26GSxCi1CSKuhGKLpxWaE3aGKYTKft0JkkzEyEEvoMPoTP4FbX7sSduPJNnKRZ2NYDAx//fw7nzO9HjEplWV/Gyura+sZmYau4vbO7t28eHLZlGAtMWjhkoej6SBJGA9JSVDHSjQRB3Gek449vU7/zSISkYdBUk4i4HA0DOqAYKS15ZsXxORx5NXgNHUmHHJVTofPQhDPDhmcZ+RXPLFlVKyu4DHYOJZBXwzN/nH6IY04ChRmSsmdbkXITJBTFjEyLTixJhPAYDUlPY4A4kW6SfWkKT7XSh4NQ6BcomKl/JxLEpZxwX3dypEZy0UvF/7xerAZXbkKDKFYkwLNFg5hBFcI0H9ingmDFJhoQFlTfCvEICYSVTnFui8+nOhN7MYFlaNeq9nm1dn9Rqt/k6RTAMTgBZWCDS1AHd6ABWgCDJ/ACXsGb8Wy8Gx/G56x1xchnjsBcGd+/d8uhLQ==</latexit>

Weights

Hidden layerdd<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>d<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit><latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>

TT
Representationmn<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>r<latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>rr<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Q1

G1

Qd

<latexit sha1_base64="Rm4XceLAlo1yWQZfjVksZUh/0hQ=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLBMwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Me17/VLZrbgzoFXi5aQMOer90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtKtXFVrt3m7RTgFM7gAjy4hhrcQx2aQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGiJ5il</latexit>

Gdn<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

h1 (nd )<latexit sha1_base64="0oUJkVuCDmkiZhPvFRYImkaO990=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GWpG5cV7AOaUCaTSTt0MgkzE6GE7vwGt7p2J279EZf+iZM2C9t64MLhnHu5h+MnnClt299WaWNza3unvFvZ2z84PKoen3RVnEpCOyTmsez7WFHOBO1opjntJ5LiyOe050/ucr/3RKVisXjU04R6ER4JFjKCtZFcN8J6TDBHrWEwrNbsuj0HWidOQWpQoD2s/rhBTNKICk04Vmrg2In2Miw1I5zOKm6qaILJBI/owFCBI6q8bJ55hi6MEqAwlmaERnP170WGI6WmkW8284xq1cvF/7xBqsNbL2MiSTUVZPEoTDnSMcoLQAGTlGg+NQQTyUxWRMZYYqJNTUtf/GhmOnFWG1gn3Ubduao3Hq5rzVbRThnO4BwuwYEbaMI9tKEDBBJ4gVd4s56td+vD+lyslqzi5hSWYH39AluIl+c=</latexit>b(md )W (nd ⇥ md )<latexit sha1_base64="ZLNEFqdz0hPzSQqT/my2G1muybo=">AAACDXicbVDLSsNAFJ3UV62vqLhyM1iEuilJFXRZdOOygn1Ak5bJZNIOnZmEmYlQQr/Bb3Cra3fi1m9w6Z84bbOwrQcuHM65l3M5QcKo0o7zbRXW1jc2t4rbpZ3dvf0D+/CopeJUYtLEMYtlJ0CKMCpIU1PNSCeRBPGAkXYwupv67SciFY3Fox4nxOdoIGhEMdJG6tsnXsBhG1ZEL/Q05URB3gsv+nbZqTozwFXi5qQMcjT69o8XxjjlRGjMkFJd10m0nyGpKWZkUvJSRRKER2hAuoYKZJL8bPb+BJ4bJYRRLM0IDWfq34sMcaXGPDCbHOmhWvam4n9eN9XRjZ9RkaSaCDwPilIGdQynXcCQSoI1GxuCsKTmV4iHSCKsTWMLKQGfmE7c5QZWSatWdS+rtYercv02b6cITsEZqAAXXIM6uAcN0AQYZOAFvII369l6tz6sz/lqwcpvjsECrK9fkuabJw==</latexit><latexit sha1_base64="Kn5Q2wMppxyvaiol7MZeKqzglbg=">AAACA3icbVA9TwJBFHyHX4hfqKXNRmKCDblDEy2JNpaYCJjASfb29mDD7t65u2dCCKW/wVZrO2PrD7H0n7jAFQJO8pLJzHuZlwkSzrRx3W8nt7K6tr6R3yxsbe/s7hX3D5o6ThWhDRLzWN0HWFPOJG0YZji9TxTFIuC0FQyuJ37riSrNYnlnhgn1Be5JFjGCjZX8TiBQv+uhsnwIT7vFkltxp0DLxMtICTLUu8WfThiTVFBpCMdatz03Mf4IK8MIp+NCJ9U0wWSAe7RtqcSCan80fXqMTqwSoihWdqRBU/XvxQgLrYcisJsCm75e9Cbif147NdGlP2IySQ2VZBYUpRyZGE0aQCFTlBg+tAQTxeyviPSxwsTYnuZSAjG2nXiLDSyTZrXinVWqt+el2lXWTh6O4BjK4MEF1OAG6tAAAo/wAq/w5jw7786H8zlbzTnZzSHMwfn6BcR9l3o=</latexit>

Bd

B1<latexit sha1_base64="bwbYZJKlkn9SseHIoOjlCmRj900=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DLExjKCeUB2CbOTSTJkZnaZmRXCks5vsNXaTmz9EUv/xNlkC5N44MLhnHu5hxPGnGnjut9OYWNza3unuFva2z84PCofn7R1lChCWyTikeqGWFPOJG0ZZjjtxopiEXLaCSd3md95okqzSD6aaUwDgUeSDRnBxkq+L7AZE8xRo+/1yxW36s6B1omXkwrkaPbLP/4gIomg0hCOte55bmyCFCvDCKezkp9oGmMywSPas1RiQXWQzjPP0IVVBmgYKTvSoLn69yLFQuupCO1mllGvepn4n9dLzPA2SJmME0MlWTwaJhyZCGUFoAFTlBg+tQQTxWxWRMZYYWJsTUtfQjGznXirDayTdq3qXVVrD9eVeiNvpwhncA6X4MEN1OEemtACAjG8wCu8Oc/Ou/PhfC5WC05+cwpLcL5+AQs/l7Q=</latexit><latexit sha1_base64="3MFnF3TUwkZrFTYHOmHnfr6C7MY=">AAACA3icbVDLSsNAFL3xWeur6tJNsAiuSlIFXRbduGzBPqANZTKZtENnJnFmIpSQpd/gVtfuxK0f4tI/cdJmYVsPXDiccy/3cPyYUaUd59taW9/Y3Nou7ZR39/YPDitHxx0VJRKTNo5YJHs+UoRRQdqaakZ6sSSI+4x0/cld7nefiFQ0Eg96GhOPo5GgIcVIG8kbcKTHGLG0lQ2DYaXq1JwZ7FXiFqQKBZrDys8giHDCidCYIaX6rhNrL0VSU8xIVh4kisQIT9CI9A0ViBPlpbPQmX1ulMAOI2lGaHum/r1IEVdqyn2zmYdUy14u/uf1Ex3eeCkVcaKJwPNHYcJsHdl5A3ZAJcGaTQ1BWFKT1cZjJBHWpqeFLz7PTCfucgOrpFOvuZe1euuq2rgt2inBKZzBBbhwDQ24hya0AcMjvMArvFnP1rv1YX3OV9es4uYEFmB9/QLycJjY</latexit><latexit sha1_base64="tJ/clG65K51SzjaJ2Sy6glTQP/4=">AAACAHicbVA9TwJBEJ3DL8Qv1NJmIzHBhtyhiZZEG0tMBIxwkr29Pdiwu3fZ3TMhhMbfYKu1nbH1n1j6T1zgCgFfMsnLezOZmRcknGnjut9ObmV1bX0jv1nY2t7Z3SvuHzR1nCpCGyTmsboPsKacSdowzHB6nyiKRcBpKxhcT/zWE1WaxfLODBPqC9yTLGIEGys9dAKBgrJ4DE+7xZJbcadAy8TLSAky1LvFn04Yk1RQaQjHWrc9NzH+CCvDCKfjQifVNMFkgHu0banEgmp/NL14jE6sEqIoVrakQVP178QIC62HIrCdApu+XvQm4n9eOzXRpT9iMkkNlWS2KEo5MjGavI9CpigxfGgJJorZWxHpY4WJsSHNbQnE2GbiLSawTJrVindWqd6el2pXWTp5OIJjKIMHF1CDG6hDAwhIeIFXeHOenXfnw/mcteacbOYQ5uB8/QIrq5al</latexit>m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

(
= (

h2 =

r
G1
<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="6HXf1VNSoMXsSir2AN3cjG6FRsI=">AAAB/XicbVBNSwMxEJ31s9avqkcvwSJ4KrtV0GPRi8cK9gPapSRptg1NskuSFcpS/A1e9exNvPpbPPpPTNs92NYHA4/3ZpiZRxLBjfX9b29tfWNza7uwU9zd2z84LB0dN02casoaNBaxbhNsmOCKNSy3grUTzbAkgrXI6G7qt56YNjxWj3acsFDigeIRp9g6qdUlEg171V6p7Ff8GdAqCXJShhz1Xumn249pKpmyVGBjOoGf2DDD2nIq2KTYTQ1LMB3hAes4qrBkJsxm507QuVP6KIq1K2XRTP07kWFpzFgS1ymxHZplbyr+53VSG92EGVdJapmi80VRKpCN0fR31OeaUSvGjmCqubsV0SHWmFqX0MIWIicuk2A5gVXSrFaCy0r14apcu83TKcApnMEFBHANNbiHOjSAwghe4BXevGfv3fvwPueta14+cwIL8L5+AUYclZ4=</latexit><latexit sha1_base64="wzwKnz+HcTIy50z6HPzobCse668=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHubvWTN7t6xuyeEI/4GW63txNb/Yuk/cXO5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML6d+e0nqjSL5IOZxNQXeChZyAg2Vmr1NBsK3C9X3KqbAa0SLycVyNHol396g4gkgkpDONa667mx8VOsDCOcTku9RNMYkzEe0q6lEguq/TS7dorOrDJAYaRsSYMy9e9EioXWExHYToHNSC97M/E/r5uY8NpPmYwTQyWZLwoTjkyEZq+jAVOUGD6xBBPF7K2IjLDCxNiAFrYEYmoz8ZYTWCWtWtW7qNbuLyv1mzydIpzAKZyDB1dQhztoQBMIPMILvMKb8+y8Ox/O57y14OQzx7AA5+sXYGOVvQ==</latexit><latexit sha1_base64="w7xDm+1pHa7shFv3TXMR215Lgtw=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOQKtxFQcugjWUC5gOSI+xt9pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRBHaJhGPVC/AmnImadsww2kvVhSLgNNuMH2Y+91nqjSL5JNJY+oLPJYsZAQbK7Wqw3LFrbkLoE3i5aQCOZrD8s9gFJFEUGkIx1r3PTc2foaVYYTTWWmQaBpjMsVj2rdUYkG1ny0OnaErq4xQGClb0qCF+nciw0LrVAS2U2Az0eveXPzP6ycmvPMzJuPEUEmWi8KEIxOh+ddoxBQlhqeWYKKYvRWRCVaYGJvNypZAzGwm3noCm6RTr3nXtXrrptK4z9MpwgVcQhU8uIUGPEIT2kCAwgu8wpuTOu/Oh/O5bC04+cw5rMD5+gUOp5NG</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Gdn<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="JNr8sPIIoar8uht1lYfWLaQiPRk=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuyXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXzZk4w=</latexit>

+

Qd

Q1

Bd

B1
<latexit sha1_base64="bwbYZJKlkn9SseHIoOjlCmRj900=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DLExjKCeUB2CbOTSTJkZnaZmRXCks5vsNXaTmz9EUv/xNlkC5N44MLhnHu5hxPGnGnjut9OYWNza3unuFva2z84PCofn7R1lChCWyTikeqGWFPOJG0ZZjjtxopiEXLaCSd3md95okqzSD6aaUwDgUeSDRnBxkq+L7AZE8xRo+/1yxW36s6B1omXkwrkaPbLP/4gIomg0hCOte55bmyCFCvDCKezkp9oGmMywSPas1RiQXWQzjPP0IVVBmgYKTvSoLn69yLFQuupCO1mllGvepn4n9dLzPA2SJmME0MlWTwaJhyZCGUFoAFTlBg+tQQTxWxWRMZYYWJsTUtfQjGznXirDayTdq3qXVVrD9eVeiNvpwhncA6X4MEN1OEemtACAjG8wCu8Oc/Ou/PhfC5WC05+cwpLcL5+AQs/l7Q=</latexit><latexit sha1_base64="XmnsbflCSUflFiOq39oz+CqINvA=">AAAB93icbVBNSwMxEJ2tX7V+VT16CRZBEMpuFfRY9OKxBfsB7VKyabYNTbJLkhWWpb/Aq569iVd/jkf/iWm7B9v6YODx3gwz84KYM21c99spbGxube8Ud0t7+weHR+Xjk7aOEkVoi0Q8Ut0Aa8qZpC3DDKfdWFEsAk47weRh5neeqdIskk8mjakv8EiykBFsrNS8GpQrbtWdA60TLycVyNEYlH/6w4gkgkpDONa657mx8TOsDCOcTkv9RNMYkwke0Z6lEguq/Wx+6BRdWGWIwkjZkgbN1b8TGRZapyKwnQKbsV71ZuJ/Xi8x4Z2fMRknhkqyWBQmHJkIzb5GQ6YoMTy1BBPF7K2IjLHCxNhslrYEYmoz8VYTWCftWtW7rtaaN5X6fZ5OEc7gHC7Bg1uowyM0oAUEKLzAK7w5qfPufDifi9aCk8+cwhKcr18TYJNJ</latexit><latexit sha1_base64="0oUJkVuCDmkiZhPvFRYImkaO990=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GWpG5cV7AOaUCaTSTt0MgkzE6GE7vwGt7p2J279EZf+iZM2C9t64MLhnHu5h+MnnClt299WaWNza3unvFvZ2z84PKoen3RVnEpCOyTmsez7WFHOBO1opjntJ5LiyOe050/ucr/3RKVisXjU04R6ER4JFjKCtZFcN8J6TDBHrWEwrNbsuj0HWidOQWpQoD2s/rhBTNKICk04Vmrg2In2Miw1I5zOKm6qaILJBI/owFCBI6q8bJ55hi6MEqAwlmaERnP170WGI6WmkW8284xq1cvF/7xBqsNbL2MiSTUVZPEoTDnSMcoLQAGTlGg+NQQTyUxWRMZYYqJNTUtf/GhmOnFWG1gn3Ubduao3Hq5rzVbRThnO4BwuwYEbaMI9tKEDBBJ4gVd4s56td+vD+lyslqzi5hSWYH39AluIl+c=</latexit><latexit sha1_base64="if3oekf7lLMSq2v2gQpcxZkaDt0=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOgTbiLgpZBG8sEzAckR9jb7CVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiSK0RSIeqW6ANeVM0pZhhtNurCgWAaedYPIw8zvPVGkWySeTxtQXeCRZyAg2VmpeDcoVt+rOgdaJl5MK5GgMyj/9YUQSQaUhHGvd89zY+BlWhhFOp6V+ommMyQSPaM9SiQXVfjY/dIourDJEYaRsSYPm6t+JDAutUxHYToHNWK96M/E/r5eY8M7PmIwTQyVZLAoTjkyEZl+jIVOUGJ5agoli9lZExlhhYmw2S1sCMbWZeKsJrJN2repdV2vNm0r9Pk+nCGdwDpfgwS3U4REa0AICFF7gFd6c1Hl3PpzPRWvByWdOYQnO1y8QOpNH</latexit><latexit sha1_base64="3MFnF3TUwkZrFTYHOmHnfr6C7MY=">AAACA3icbVDLSsNAFL3xWeur6tJNsAiuSlIFXRbduGzBPqANZTKZtENnJnFmIpSQpd/gVtfuxK0f4tI/cdJmYVsPXDiccy/3cPyYUaUd59taW9/Y3Nou7ZR39/YPDitHxx0VJRKTNo5YJHs+UoRRQdqaakZ6sSSI+4x0/cld7nefiFQ0Eg96GhOPo5GgIcVIG8kbcKTHGLG0lQ2DYaXq1JwZ7FXiFqQKBZrDys8giHDCidCYIaX6rhNrL0VSU8xIVh4kisQIT9CI9A0ViBPlpbPQmX1ulMAOI2lGaHum/r1IEVdqyn2zmYdUy14u/uf1Ex3eeCkVcaKJwPNHYcJsHdl5A3ZAJcGaTQ1BWFKT1cZjJBHWpqeFLz7PTCfucgOrpFOvuZe1euuq2rgt2inBKZzBBbhwDQ24hya0AcMjvMArvFnP1rv1YX3OV9es4uYEFmB9/QLycJjY</latexit><latexit sha1_base64="Rm4XceLAlo1yWQZfjVksZUh/0hQ=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLBMwD0iWMDuZTYbMzK4zs0JYUvoNtlrbia0fYumfOJtsYRIPXDiccy/3cIKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Fd5refqNIskg9mElNf4KFkISPYWMnvCWxGBPO0Me17/VLZrbgzoFXi5aQMOer90k9vEJFEUGkIx1p3PTc2foqVYYTTabGXaBpjMsZD2rVUYkG1n85CT9G5VQYojJQdadBM/XuRYqH1RAR2Mwupl71M/M/rJia88VMm48RQSeaPwoQjE6GsATRgihLDJ5ZgopjNisgIK0yM7WnhSyCmthNvuYFV0qpWvMtKtXFVrt3m7RTgFM7gAjy4hhrcQx2aQOARXuAV3pxn5935cD7nq2tOfnMCC3C+fgGiJ5il</latexit>

TN Contraction+<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="wzwKnz+HcTIy50z6HPzobCse668=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHubvWTN7t6xuyeEI/4GW63txNb/Yuk/cXO5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML6d+e0nqjSL5IOZxNQXeChZyAg2Vmr1NBsK3C9X3KqbAa0SLycVyNHol396g4gkgkpDONa667mx8VOsDCOcTku9RNMYkzEe0q6lEguq/TS7dorOrDJAYaRsSYMy9e9EioXWExHYToHNSC97M/E/r5uY8NpPmYwTQyWZLwoTjkyEZq+jAVOUGD6xBBPF7K2IjLDCxNiAFrYEYmoz8ZYTWCWtWtW7qNbuLyv1mzydIpzAKZyDB1dQhztoQBMIPMILvMKb8+y8Ox/O57y14OQzx7AA5+sXYGOVvQ==</latexit>))<latexit sha1_base64="XmnsbflCSUflFiOq39oz+CqINvA=">AAAB93icbVBNSwMxEJ2tX7V+VT16CRZBEMpuFfRY9OKxBfsB7VKyabYNTbJLkhWWpb/Aq569iVd/jkf/iWm7B9v6YODx3gwz84KYM21c99spbGxube8Ud0t7+weHR+Xjk7aOEkVoi0Q8Ut0Aa8qZpC3DDKfdWFEsAk47weRh5neeqdIskk8mjakv8EiykBFsrNS8GpQrbtWdA60TLycVyNEYlH/6w4gkgkpDONa657mx8TOsDCOcTkv9RNMYkwke0Z6lEguq/Wx+6BRdWGWIwkjZkgbN1b8TGRZapyKwnQKbsV71ZuJ/Xi8x4Z2fMRknhkqyWBQmHJkIzb5GQ6YoMTy1BBPF7K2IjLHCxNhslrYEYmoz8VYTWCftWtW7rtaaN5X6fZ5OEc7gHC7Bg1uowyM0oAUEKLzAK7w5qfPufDifi9aCk8+cwhKcr18TYJNJ</latexit>

Bd

B1<latexit sha1_base64="bwbYZJKlkn9SseHIoOjlCmRj900=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DLExjKCeUB2CbOTSTJkZnaZmRXCks5vsNXaTmz9EUv/xNlkC5N44MLhnHu5hxPGnGnjut9OYWNza3unuFva2z84PCofn7R1lChCWyTikeqGWFPOJG0ZZjjtxopiEXLaCSd3md95okqzSD6aaUwDgUeSDRnBxkq+L7AZE8xRo+/1yxW36s6B1omXkwrkaPbLP/4gIomg0hCOte55bmyCFCvDCKezkp9oGmMywSPas1RiQXWQzjPP0IVVBmgYKTvSoLn69yLFQuupCO1mllGvepn4n9dLzPA2SJmME0MlWTwaJhyZCGUFoAFTlBg+tQQTxWxWRMZYYWJsTUtfQjGznXirDayTdq3qXVVrD9eVeiNvpwhncA6X4MEN1OEemtACAjG8wCu8Oc/Ou/PhfC5WC05+cwpLcL5+AQs/l7Q=</latexit><latexit sha1_base64="0oUJkVuCDmkiZhPvFRYImkaO990=">AAACAnicbVDLSsNAFL2pr1pfVZduBovgqiRV0GWpG5cV7AOaUCaTSTt0MgkzE6GE7vwGt7p2J279EZf+iZM2C9t64MLhnHu5h+MnnClt299WaWNza3unvFvZ2z84PKoen3RVnEpCOyTmsez7WFHOBO1opjntJ5LiyOe050/ucr/3RKVisXjU04R6ER4JFjKCtZFcN8J6TDBHrWEwrNbsuj0HWidOQWpQoD2s/rhBTNKICk04Vmrg2In2Miw1I5zOKm6qaILJBI/owFCBI6q8bJ55hi6MEqAwlmaERnP170WGI6WmkW8284xq1cvF/7xBqsNbL2MiSTUVZPEoTDnSMcoLQAGTlGg+NQQTyUxWRMZYYqJNTUtf/GhmOnFWG1gn3Ubduao3Hq5rzVbRThnO4BwuwYEbaMI9tKEDBBJ4gVd4s56td+vD+lyslqzi5hSWYH39AluIl+c=</latexit>

computationpp‣ Very efficientNr )) ConvenientO(dforMparallel
computing‣<latexit sha1_base64="if3oekf7lLMSq2v2gQpcxZkaDt0=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOgTbiLgpZBG8sEzAckR9jb7CVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiSK0RSIeqW6ANeVM0pZhhtNurCgWAaedYPIw8zvPVGkWySeTxtQXeCRZyAg2VmpeDcoVt+rOgdaJl5MK5GgMyj/9YUQSQaUhHGvd89zY+BlWhhFOp6V+ommMyQSPaM9SiQXVfjY/dIourDJEYaRsSYPm6t+JDAutUxHYToHNWK96M/E/r5eY8M7PmIwTQyVZLAoTjkyEZl+jIVOUGJ5agoli9lZExlhhYmw2S1sCMbWZeKsJrJN2repdV2vNm0r9Pk+nCGdwDpfgwS3U4REa0AICFF7gFd6c1Hl3PpzPRWvByWdOYQnO1y8QOpNH</latexit><latexit sha1_base64="w7xDm+1pHa7shFv3TXMR215Lgtw=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOQKtxFQcugjWUC5gOSI+xt9pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRBHaJhGPVC/AmnImadsww2kvVhSLgNNuMH2Y+91nqjSL5JNJY+oLPJYsZAQbK7Wqw3LFrbkLoE3i5aQCOZrD8s9gFJFEUGkIx1r3PTc2foaVYYTTWWmQaBpjMsVj2rdUYkG1ny0OnaErq4xQGClb0qCF+nciw0LrVAS2U2Az0eveXPzP6ycmvPMzJuPEUEmWi8KEIxOh+ddoxBQlhqeWYKKYvRWRCVaYGJvNypZAzGwm3noCm6RTr3nXtXrrptK4z9MpwgVcQhU8uIUGPEIT2kCAwgu8wpuTOu/Oh/O5bC04+cw5rMD5+gUOp5NG</latexit>=<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit><latexit sha1_base64="wzwKnz+HcTIy50z6HPzobCse668=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHubvWTN7t6xuyeEI/4GW63txNb/Yuk/cXO5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML6d+e0nqjSL5IOZxNQXeChZyAg2Vmr1NBsK3C9X3KqbAa0SLycVyNHol396g4gkgkpDONa667mx8VOsDCOcTku9RNMYkzEe0q6lEguq/TS7dorOrDJAYaRsSYMy9e9EioXWExHYToHNSC97M/E/r5uY8NpPmYwTQyWZLwoTjkyEZq+jAVOUGD6xBBPF7K2IjLDCxNiAFrYEYmoz8ZYTWCWtWtW7qNbuLyv1mzydIpzAKZyDB1dQhztoQBMIPMILvMKb8+y8Ox/O57y14OQzx7AA5+sXYGOVvQ==</latexit>(<latexit sha1_base64="w7xDm+1pHa7shFv3TXMR215Lgtw=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOQKtxFQcugjWUC5gOSI+xt9pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRBHaJhGPVC/AmnImadsww2kvVhSLgNNuMH2Y+91nqjSL5JNJY+oLPJYsZAQbK7Wqw3LFrbkLoE3i5aQCOZrD8s9gFJFEUGkIx1r3PTc2foaVYYTTWWmQaBpjMsVj2rdUYkG1ny0OnaErq4xQGClb0qCF+nciw0LrVAS2U2Az0eveXPzP6ycmvPMzJuPEUEmWi8KEIxOh+ddoxBQlhqeWYKKYvRWRCVaYGJvNypZAzGwm3noCm6RTr3nXtXrrptK4z9MpwgVcQhU8uIUGPEIT2kCAwgu8wpuTOu/Oh/O5bC04+cw5rMD5+gUOp5NG</latexit>d<latexit sha1_base64="eaK21q6M0GQvw97k4p1SVUerOQU=">AAACIHicbZDLSsNAFIYnXmu9RV26GSxC3ZSkFnRZdONGrWAv0MQymUzaoZOLMxOhhLyCD+EzuNW1O3GpO9/ESRvEtv4w8PGfczhnfidiVEjD+NQWFpeWV1YLa8X1jc2tbX1ntyXCmGPSxCELecdBgjAakKakkpFOxAnyHUbazvA8q7cfCBc0DG7lKCK2j/oB9ShGUlk9vWz5SA4wYsl1WnahJe657Lp2cpn+4lXK72pHPb1kVIyx4DyYOZRArkZP/7bcEMc+CSRmSIiuaUTSThCXFDOSFq1YkAjhIeqTrsIA+UTYyfhHKTxUjgu9kKsXSDh2/04kyBdi5DuqM7tfzNYy879aN5beqZ3QIIolCfBkkRczKEOYxQNdygmWbKQAYU7VrRAPEEdYqhCntjh+qjIxZxOYh1a1Yh5Xqje1Uv0sT6cA9sEBKAMTnIA6uAAN0AQYPIJn8AJetSftTXvXPiatC1o+swempH39AGDxpDo=</latexit><latexit sha1_base64="if3oekf7lLMSq2v2gQpcxZkaDt0=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoOgTbiLgpZBG8sEzAckR9jb7CVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiSK0RSIeqW6ANeVM0pZhhtNurCgWAaedYPIw8zvPVGkWySeTxtQXeCRZyAg2VmpeDcoVt+rOgdaJl5MK5GgMyj/9YUQSQaUhHGvd89zY+BlWhhFOp6V+ommMyQSPaM9SiQXVfjY/dIourDJEYaRsSYPm6t+JDAutUxHYToHNWK96M/E/r5eY8M7PmIwTQyVZLAoTjkyEZl+jIVOUGJ5agoli9lZExlhhYmw2S1sCMbWZeKsJrJN2repdV2vNm0r9Pk+nCGdwDpfgwS3U4REa0AICFF7gFd6c1Hl3PpzPRWvByWdOYQnO1y8QOpNH</latexit>

46

d

4

Challenges: Input to Hidden Layer via TN
TT Decomposition

Tensorizationx

d-order<latexit sha1_base64="OgRhWZSeaawCe9Kiw75R1El7XjM=">AAAB+3icbVBNS8NAEJ34WetX1aOXxSJ4KkkV9Fj04rGCaQttKJvtpl26uwm7G7GE/AavevYmXv0xHv0nbtscbOuDgcd7M8zMCxPOtHHdb2dtfWNza7u0U97d2z84rBwdt3ScKkJ9EvNYdUKsKWeS+oYZTjuJoliEnLbD8d3Ubz9RpVksH80koYHAQ8kiRrCxkt8LBXruV6puzZ0BrRKvIFUo0OxXfnqDmKSCSkM41rrruYkJMqwMI5zm5V6qaYLJGA9p11KJBdVBNjs2R+dWGaAoVrakQTP170SGhdYTEdpOgc1IL3tT8T+vm5roJsiYTFJDJZkvilKOTIymn6MBU5QYPrEEE8XsrYiMsMLE2HwWtoQit5l4ywmskla95l3W6g9X1cZtkU4JTuEMLsCDa2jAPTTBBwIMXuAV3pzceXc+nM9565pTzJzAApyvXy09lQk=</latexit>M<latexit sha1_base64="Kmi6kI52Sy6qSfrNJAybJLeAQJw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkZIwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CSOaTaw==</latexit>m<latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

G⇡m<latexit sha1_base64="bl+JnuNWfH9v44Hh7Y67uxnzVAE=">AAAB/XicbVBNSwMxEJ2tX7V+VT16CRbBU9mtgh6LXjxWsB/QLiWbpm1okg1JVixL8Td41bM38epv8eg/MW33YFsfDDzem2FmXqQ4M9b3v73c2vrG5lZ+u7Czu7d/UDw8apg40YTWScxj3YqwoZxJWrfMctpSmmIRcdqMRrdTv/lItWGxfLBjRUOBB5L1GcHWSc0OVkrHT91iyS/7M6BVEmSkBBlq3eJPpxeTRFBpCcfGtANf2TDF2jLC6aTQSQxVmIzwgLYdlVhQE6azcyfozCk91I+1K2nRTP07kWJhzFhErlNgOzTL3lT8z2sntn8dpkyqxFJJ5ov6CUc2RtPfUY9pSiwfO4KJZu5WRIZYY2JdQgtbIjFxmQTLCaySRqUcXJQr95el6k2WTh5O4BTOIYArqMId1KAOBEbwAq/w5j17796H9zlvzXnZzDEswPv6BV98llA=</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

1r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Gdm<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit><latexit sha1_base64="FGO9z/msaUot/55DudcBDsIGDxY=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuiXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BXtGk4s=</latexit>

Nonlinear Activation

Nonlinear Activation‣ Input data may not admit low-rank TT approximation (small r)‣ TT decomposition is computational costly‣ Nonlinear activation destroy TT format,<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

47

Feature Mapping‣

Mapping input data into TN representationx = [x1 , x2 , . . . , xd ]dInspired by “spin” vectors
in quantum systemT<latexit sha1_base64="BLWEx98A3/vcyb7W+XUHdBrmvUY=">AAACHXicbVC7TsMwFHXKq5RXgJHFoqrEUFVJQYIFqYKFsUh9SWmIHMdprToP2Q5qFfUL+Ai+gRVmNsSKGPkTnDYDbTmSreNz7tW9Pm7MqJCG8a0V1tY3NreK26Wd3b39A/3wqCOihGPSxhGLeM9FgjAakrakkpFezAkKXEa67ug287uPhAsahS05iYkdoEFIfYqRVJKjV/oBkkPXh+NraMGxY1bVVa/CPvMiKbKHZz+0HL1s1IwZ4Coxc1IGOZqO/tP3IpwEJJSYISEs04ilnSIuKWZkWuongsQIj9CAWIqGKCDCTmffmcKKUjzoR1ydUMKZ+rcjRYEQk8BVldnyYtnLxP88K5H+lZ3SME4kCfF8kJ8wKCOYZQM9ygmWbKIIwpyqXSEeIo6wVAkuTHGDqcrEXE5glXTqNfO8Vr+/KDdu8nSK4AScgjNggkvQAHegCdoAgyfwAl7Bm/asvWsf2ue8tKDlPcdgAdrXLz68oLY=</latexit><latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit>h(xi ) = cos<latexit sha1_base64="EYSo7ULy7+n5U+GE++M/M1QPVPY=">AAACWHicfZDPSxtBFMdfVm2MtTXao5fBUEighN0otBch1EuPKRgTyKxxdjKbDM7OLjNvS8Oy/17/B71599qeO/lxUCM+GObL930f7/GJMiUt+v5dxdva3nlX3a3tvd//8PGgfnh0ZdPccNHnqUrNMGJWKKlFHyUqMcyMYEmkxCC6vVj0B7+EsTLVlzjPRJiwqZax5AydNa7f0GwmSfP3WLbIOaFKxDgilKd2KZs0NowXNJNl0SldiBo5nWHrC7VSv5lYfeH15bje8Nv+ssimCNaiAevqjesPdJLyPBEauWLWjgI/w7BgBiVXotyjuRUZ47dsKkZOapYIGxZLEiX57JwJiVPjnkaydJ9OFCyxdp5ELpkwnNmXvYX5Wm+UY/wtLKTOchSarxbFuSKYkgVWMpFGcFRzJxg30t1K+Iw5NOjgP9sSJaVjErwksCmuOu3gtN35edbofl/T2YVjOIEmBPAVuvADetAHDn/gEf7Cv8q9B17Vq62iXmU98wmelXf0H/0ZtxQ=</latexit>

⇣⇡
2

⌘

xi , sin

⇣⇡
2

xi

⌘iT

d

d 7! 2<latexit sha1_base64="vGFQZt+tx/C7M+TKU/o+t9TybyY=">AAACA3icbVA9SwNBEJ3zM8avqKXNYhCswl0UtAzaWEYwH5CcYW9vL1myu3fu7gnhSOlvsNXaTmz9IZb+EzfJFSbxwcDjvRlm5gUJZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ThWhDRLzWLUDrClnkjYMM5y2E0WxCDhtBcObid96okqzWN6bUUJ9gfuSRYxgYyU/RF2BE21iVH0Ie6WyW3GnQMvEy0kZctR7pZ9uGJNUUGkIx1p3PDcxfoaVYYTTcbGbappgMsR92rFUYkG1n02PHqNTq4QoipUtadBU/TuRYaH1SAS2U2Az0IveRPzP66QmuvIzJpPUUElmi6KUI/vkJAEUMkWJ4SNLMFHM3orIACtMjM1pbksgxjYTbzGBZdKsVrzzSvXuoly7ztMpwDGcwBl4cAk1uIU6NIDAI7zAK7w5z8678+F8zlpXnHzmCObgfP0CyZCYHg==</latexit><latexit sha1_base64="con2OZWVyoEZ488qINDn0Gan+ps=">AAACQnicbZDLSgMxFIYzXmu9VV26CRah3ZSZKuhGKLoRVxXsBTqlZDKZNjQzGZIz0lL6Tj6EzyC4UrfuxK0L0xvY1gOBn+8/h5Pze7HgGmz71VpZXVvf2Extpbd3dvf2MweHVS0TRVmFSiFV3SOaCR6xCnAQrB4rRkJPsJrXvRn5tUemNJfRA/Rj1gxJO+IBpwQMamXu3HKH59yQQMcLcC+Pr7AbG9JrOXlXAg+ZnoHiDLjUl6AXXD/fymTtgj0uvCycqciiaZVbmQ/XlzQJWQRUEK0bjh1Dc0AUcCrYMO0mmsWEdkmbNYyMiFnXHIxvHuJTQ3wcSGVeBHhM/04MSKh1P/RM5+g4veiN4H9eI4HgsjngUZwAi+hkUZAIDBKPAsQ+V4yC6BtBqOLmr5h2iCIUTMxzW7xwaDJxFhNYFtViwTkrFO/Ps6XraTopdIxOUA456AKV0C0qowqi6Am9oDf0bj1bn9aX9T1pXbGmM0dorqyfX/VLsHA=</latexit>(x) = (x1 ) ⌦ (x2 ) ⌦ · · · ⌦ (xd )Local feature map

d-order

Rank-1
Tensor···=<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>(2 ⇥ 2 ⇥ · · · ⇥ 2)

‣ Different mapping for each feature is possible(x ) can be generalized to I -dimensional, e.g., (x) : d 7! I‣<latexit sha1_base64="FwT7i4aIOt19bPGP4EDr4cP+pJE=">AAACGXicbVC7SgNBFJ2Nrxhfq5Y2o0GITdhdBS2DNpYRzAOyS5idnSRDZh/M3BXCktqP8BtstbYTWytL/8RJsoJJPDBwOOdezp3jJ4IrsKwvo7Cyura+UdwsbW3v7O6Z+wdNFaeSsgaNRSzbPlFM8Ig1gINg7UQyEvqCtfzhzcRvPTCpeBzdwyhhXkj6Ee9xSkBLXfMYVxwXeMgUdvCMuDSIQf2KZ12zbFWtKfAysXNSRjnqXfPbDWKahiwCKohSHdtKwMuIBE4FG5fcVLGE0CHps46mEdE5Xjb9yhifaiXAvVjqFwGeqn83MhIqNQp9PRkSGKhFbyL+53VS6F15GY+SFFhEZ0G9VGCI8aQXHHDJKIiRJoRKrm/FdEAkoaDbm0vxw7HuxF5sYJk0nap9XnXuLsq167ydIjpCJ6iCbHSJaugW1VEDUfSIntELejWejDfj3fiYjRaMfOcQzcH4/AF9oZ/p</latexit><latexit sha1_base64="0k/G8pXVyMnPI1axHg0twfZsHHg=">AAACAHicbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+cDkCHubvWTJ7t6xuyeGI42/wVZrO7H1n1j6T9wkV5jEBwOP92aYmRfEnGnjut/Oyura+sZmbiu/vbO7t184OGzoKFGE1knEI9UKsKacSVo3zHDaihXFIuC0GQxvJn7zkSrNInlvRjH1Be5LFjKCjZUeOvGAodJTl511C0W37E6BlomXkSJkqHULP51eRBJBpSEca9323Nj4KVaGEU7H+U6iaYzJEPdp21KJBdV+Or14jE6t0kNhpGxJg6bq34kUC61HIrCdApuBXvQm4n9eOzHhlZ8yGSeGSjJbFCYcmQhN3kc9pigxfGQJJorZWxEZYIWJsSHNbQnE2GbiLSawTBqVsndertxdFKvXWTo5OIYTKIEHl1CFW6hBHQhIeIFXeHOenXfnw/mcta442cwRzMH5+gVgf5bG</latexit>id<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="vgMkLBUYSW87kbvrgj4lN2/0pqY=">AAACFnicbVDLSsNAFJ34rPUVdSnIYBHqpiRVUFwV3eiugn1AE8tkMmmHziRhZiKW0J0f4Te41bU7cevWpX/ipM3Cth4YOHPOvdx7jxczKpVlfRsLi0vLK6uFteL6xubWtrmz25RRIjBp4IhFou0hSRgNSUNRxUg7FgRxj5GWN7jK/NYDEZJG4Z0axsTlqBfSgGKktNQ1D5x6n5YdjlTfC+Dj8QX0of7FUkXw5t7vmiWrYo0B54mdkxLIUe+aP44f4YSTUGGGpOzYVqzcFAlFMSOjopNIEiM8QD3S0TREnEg3Hd8xgkda8WEQCf1CBcfq344UcSmH3NOV2cJy1svE/7xOooJzN6VhnCgS4smgIGFQH5mFAn0qCFZsqAnCgupdIe4jgbDS0U1N8fhIZ2LPJjBPmtWKfVKp3p6Wapd5OgWwDw5BGdjgDNTANaiDBsDgCbyAV/BmPBvvxofxOSldMPKePTAF4+sXF8Keng==</latexit>

Supervised Learning with Quantum-Inspired Tensor Networks (Stoudenmire et al., NIPS 2016)48

Classification Layer‣f (x) = hW, (x)i

Binary classification:

Number offeatures(x)<latexit sha1_base64="k0iYB7o0dTVlrJJEMXs6N6n1hiI=">AAACMnicbVDLSsNAFJ3UV62vqEs3g0WoICWpgiIIRTcuK9gHNKVMppN26GQSZiZiCfkXP8JvcKtb3RW3foSTNAvbemDgcM653DvHDRmVyrI+jcLK6tr6RnGztLW9s7tn7h+0ZBAJTJo4YIHouEgSRjlpKqoY6YSCIN9lpO2O71K//USEpAF/VJOQ9Hw05NSjGCkt9c1rr+L4SI1cDz6fwhvoMMSHjMBMxIjF7eTMaYzo35QjskzfLFtVKwNcJnZOyiBHo29OnUGAI59whRmSsmtboerFSCiKGUlKTiRJiPAYDUlXU458Intx9scEnmhlAL1A6McVzNS/EzHypZz4rk6ml8pFLxX/87qR8q56MeVhpAjHs0VexKAKYFoYHFBBsGITTRAWVN8K8QgJhJWudW6L6ye6E3uxgWXSqlXt82rt4aJcv83bKYIjcAwqwAaXoA7uQQM0AQYv4A28gw/j1fgypsb3LFow8plDMAfj5xfqKKns</latexit>

d-order<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>=<latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit>

R<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

2

d

W

<latexit sha1_base64="mZ4QwHEaK27ncx+qL7NPvaUR0wE=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKi+YDkDHt7e8mS3b1jd08IR36CrdZ2YuuvsfSfuEmuMIkPBh7vzTAzL0g408Z1v53C2vrG5lZxu7Szu7d/UD48auk4VYQ2Scxj1QmwppxJ2jTMcNpJFMUi4LQdjG6nfvuZKs1i+WjGCfUFHkgWMYKNlR5qT2G/XHGr7gxolXg5qUCORr/80wtjkgoqDeFY667nJsbPsDKMcDop9VJNE0xGeEC7lkosqPaz2akTdGaVEEWxsiUNmql/JzIstB6LwHYKbIZ62ZuK/3nd1ETXfsZkkhoqyXxRlHJkYjT9G4VMUWL42BJMFLO3IjLEChNj01nYEoiJzcRbTmCVtGpV76Jau7+s1G/ydIpwAqdwDh5cQR3uoAFNIDCAF3iFNydz3p0P53PeWnDymWNYgPP1C5qElCY=</latexit>

d-order<latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit>

R<latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit><latexit sha1_base64="akrtlxlD9vC7bP808GeCjqc6NX4=">AAAB+3icbVBNT8JAEJ3iF+IX6tHLRmLiibRookeiF49oLJBAJdvtFjZst83u1oQ0/Q1e9ezNePXHePSfuEAPAr5kkpf3ZjIzz084U9q2v63S2vrG5lZ5u7Kzu7d/UD08aqs4lYS6JOax7PpYUc4EdTXTnHYTSXHkc9rxx7dTv/NMpWKxeNSThHoRHgoWMoK1kdxG8PDUGFRrdt2eAa0SpyA1KNAaVH/6QUzSiApNOFaq59iJ9jIsNSOc5pV+qmiCyRgPac9QgSOqvGx2bI7OjBKgMJamhEYz9e9EhiOlJpFvOiOsR2rZm4r/eb1Uh9dexkSSairIfFGYcqRjNP0cBUxSovnEEEwkM7ciMsISE23yWdjiR7nJxFlOYJW0G3Xnot64v6w1b4p0ynACp3AODlxBE+6gBS4QYPACr/Bm5da79WF9zltLVjFzDAuwvn4BtoOUvg==</latexit><latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>‣<latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit>

Number ofparameters
2dR2

···

Multi-class model： fc (x) = hWc , (x)i, W = {Wc }K
c=1···(x) d-orderGcor<latexit sha1_base64="sBbHoFURRRZFLyZ22MJGU5sASKY=">AAACHnicbVDLSsNAFJ3UV62vqEs3g0VxVZIq6KZQdCO4qWAf0MQwmU7boTNJmJkIJeQP/Ai/wa2u3YlbXfonTtos+vDAwLnn3Mu9c/yIUaks68corKyurW8UN0tb2zu7e+b+QUuGscCkiUMWio6PJGE0IE1FFSOdSBDEfUba/ugm89tPREgaBg9qHBGXo0FA+xQjpSXPPHU4UkOMWNJOYc1JZkoPO6mX4JqdPt55ZtmqWBPAZWLnpAxyNDzz1+mFOOYkUJghKbu2FSk3QUJRzEhacmJJIoRHaEC6mgaIE+kmk/+k8EQrPdgPhX6BghN1diJBXMox93Vndq5c9DLxP68bq/6Vm9AgihUJ8HRRP2ZQhTALB/aoIFixsSYIC6pvhXiIBMJKRzi3xeepzsReTGCZtKoV+7xSvb8o16/zdIrgCByDM2CDS1AHt6ABmgCDZ/AK3sC78WJ8GJ/G17S1YOQzh2AOxvcf0lGjaw==</latexit><latexit sha1_base64="vjIrIQSTgoCX3HbggcbcghBX7+E=">AAACNnicbVDLSsNAFJ3UV62vqEs3g0WoICWpgm6UohuXFewDmhAm00k7dDIJMxOxhP6NH+E3uNWNG8GFuPUTnKRd2NYDA4dzzuXeOX7MqFSW9W4UlpZXVteK66WNza3tHXN3ryWjRGDSxBGLRMdHkjDKSVNRxUgnFgSFPiNtf3iT+e0HIiSN+L0axcQNUZ/TgGKktOSZV4GHK06I1MAP4OMxvIQOQ7zPCMxFjFjaHnv4BDqNAf0TdESe8syyVbVywEViT0kZTNHwzE+nF+EkJFxhhqTs2las3BQJRTEj45KTSBIjPER90tWUo5BIN83/OYZHWunBIBL6cQVz9e9EikIpR6Gvk9mhct7LxP+8bqKCCzelPE4U4XiyKEgYVBHMSoM9KghWbKQJwoLqWyEeIIGw0tXObPHDse7Enm9gkbRqVfu0Wrs7K9evp+0UwQE4BBVgg3NQB7egAZoAgyfwAl7Bm/FsfBhfxvckWjCmM/tgBsbPLziEq5g=</latexit>=<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="q25ZLOdJGxjZoJdm5kwZUWQQe4E=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aRHeqWyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXhAZjN</latexit>···<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

W(d+1)-order<latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>

K

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit><latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

Assumption:

Wc = TT(G1 , · · · , Gc (c), · · · , Gd , )<latexit sha1_base64="BrzgsKXNWfVf4ubqHo+I93X06DU=">AAACUXicbZBNa9tAEIbHykedpEnd9JjLElOwwRgpLbSXgmkO6dEFO06wjFiN1vaS1Qe7o1Ij9Nv6I3rqMYdc23NvXduiOE4Gln153hlmeMNMSUOu+6vm7Ozu7b+oHxwevTw+edV4fXpt0lyjGGKqUn0TciOUTMSQJClxk2nB41CJUXh3ufRH34Q2Mk0GtMjEJOazRE4lcrIoaNz6Mac5clWMygDZJ+aT+E7FYFC2/jtXZeB1mI9RSsb+Gxhb2O5UxiaPOqwdNJpu110Veyq8SjShqn7QuPejFPNYJISKGzP23IwmBdckUYny0M+NyDje8ZkYW5nwWJhJsYqgZG8tidg01fYlxFZ0c6LgsTGLOLSdyzvNtreEz3njnKYfJ4VMspxEgutF01wxStkyTxZJLZDUwgqOWtpbGc655kg29Udbwri0mXjbCTwV1xdd71334uv7Zu9zlU4dzuAcWuDBB+jBF+jDEBB+wAP8hj+1n7W/DjjOutWpVTNv4FE5R/8ALByz7g==</latexit>

Location isarbitrary‣ TN allows highly efficient optimization and computation
compression: 2 K 7! 2dR + KR , TT-rank (R) controls‣ Weight
complexity of TT representation, small R is expectedd22<latexit sha1_base64="ATdqUfFB9Wi5alvfCm9lJAo1yKs=">AAACEXicbVDLSgMxFM34rPU16krcBIsgCGVmFHRZdCN0U8U+oJ2WTCbThiaZIckIpRQ/wm9wq2t34tYvcOmfmLazsK0HQg7n3Mu99wQJo0o7zre1tLyyurae28hvbm3v7Np7+zUVpxKTKo5ZLBsBUoRRQaqaakYaiSSIB4zUg/7N2K8/EqloLB70ICE+R11BI4qRNlLHPvTaYRm2OEqUjqEX3rc9eAbL5uvYBafoTAAXiZuRAshQ6dg/rTDGKSdCY4aUarpOov0hkppiRkb5VqpIgnAfdUnTUIE4Uf5wcsIInhglhFEszRMaTtS/HUPElRrwwFRypHtq3huL/3nNVEdX/pCKJNVE4OmgKGXQXDvOA4ZUEqzZwBCEJTW7QtxDEmFtUpuZEvCRycSdT2CR1Lyie1707i4KpessnRw4AsfgFLjgEpTALaiAKsDgCbyAV/BmPVvv1of1OS1dsrKeAzAD6+sX8Ombvg==</latexit>

Supervised Learning with Quantum-Inspired Tensor Networks (Stoudenmire et al., NIPS 2016)49

Learning Algorithm: DMRG1‣GStandard:
computinggradient
over
each
core
tensor‣‣ Model:···(x)

E[`(W, x, y)] ) min E[`({G1 , . . . , Gd }, x, y)]Optimization: min
W
{G }
i

<latexit sha1_base64="c7vOjrU4YWTEA9CwzMPzNFOw4kU=">AAACrnicdVHZattAFB2pS5ZubvvYPAw1hRSMkdJA8hhSSgvJQxpiO+ARYjQa2UNmETNXTYzQb/Vf+tg/6dhWiJ2kFy4czrkb52alFA6i6E8QPnn67PnG5tb2i5evXr/pvH03dKayjA+YkcZeZtRxKTQfgADJL0vLqcokH2VXX+f66Be3Thh9AbOSJ4pOtCgEo+CptPObKKHTmigKU0ZlPWqaBc6y+lszJlzK3RWtt9QKfNPDt3D2OcHkXEymQK0117gdeDfye5MK8sjYtYq4R2RuwPVWyZz8b2Pa6Ub9aBH4IYhb0EVtnKWdvyQ3rFJcA5PUuXEclZDU1IJgkjfbpHK8pOyKTvjYQ00Vd0m98LfBnzyT48JYnxrwgl3tqKlybqYyXzk/0d3X5uRj2riC4jCphS4r4JotFxWVxGDw/Fk4F5YzkDMPKLPC34rZlFrKwL90bUumGu9JfN+Bh2C414+/9Pd+7nePjlt3NtEH9BHtohgdoCP0A52hAWLBTnAcnASnYRQOwyRMl6Vh0Pa8R2sRTv8BvAzWtw==</latexit><latexit sha1_base64="jcnIXMJ/yDzaghbatbadQOZkjMg=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTHeqWyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXqc5jT</latexit><latexit sha1_base64="AbBuSfOiY1GwlX/pOzx0lXn7Gb8=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHubvWTJ7t6xuyeGI4W/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRfEnGnjut/Oyura+sZmbiu/vbO7t184OGzoKFGE1knEI9UKsKacSVo3zHDaihXFIuC0GQxvJn7zkSrNInlvRjH1Be5LFjKCjZUeOvGAlZ663lm3UHTL7hRomXgZKUKGWrfw0+lFJBFUGsKx1m3PjY2fYmUY4XSc7ySaxpgMcZ+2LZVYUO2n04PH6NQqPRRGypY0aKr+nUix0HokAtspsBnoRW8i/ue1ExNe+SmTcWKoJLNFYcKRidDke9RjihLDR5Zgopi9FZEBVpgYm9HclkCMbSbeYgLLpFEpe+flyt1FsXqdpZODYziBEnhwCVW4hRrUgYCAF3iFN+fZeXc+nM9Z64qTzRzBHJyvX65wlmQ=</latexit>(x1 )<latexit sha1_base64="YBttwdOuOWY4x4aLW+83RArt7gM=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHt7e8mS3b1jd08MIYW/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRcknGnjut/Oyura+sZmbiu/vbO7t184OGzoOFWE1knMY9UKsKacSVo3zHDaShTFIuC0GQxuJn7zkSrNYnlvhgn1Be5JFjGCjZUeOkmflZ664Vm3UHTL7hRomXgZKUKGWrfw0wljkgoqDeFY67bnJsYfYWUY4XSc76SaJpgMcI+2LZVYUO2PpgeP0alVQhTFypY0aKr+nRhhofVQBLZTYNPXi95E/M9rpya68kdMJqmhkswWRSlHJkaT71HIFCWGDy3BRDF7KyJ9rDAxNqO5LYEY20y8xQSWSaNS9s7LlbuLYvU6SycHx3ACJfDgEqpwCzWoAwEBL/AKb86z8+58OJ+z1hUnmzmCOThfv/7slpc=</latexit>

i

(xd )

<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>

f (x) =

<latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit>

W<latexit sha1_base64="70ygt9g97CI1R34hK/NUeX0VGK8=">AAACB3icbVDLSsNAFL2pr1ofjbp0M1iEuilJFXQjFN24rGAf0IYymU7aoZNJmJmIpfQD/Aa3unYnbv0Ml/6JkzYL23rgwuGce7mH48ecKe0431ZubX1jcyu/XdjZ3dsv2geHTRUlktAGiXgk2z5WlDNBG5ppTtuxpDj0OW35o9vUbz1SqVgkHvQ4pl6IB4IFjGBtpJ5dDMrdEOuhH6CnM3SNenbJqTgzoFXiZqQEGeo9+6fbj0gSUqEJx0p1XCfW3gRLzQin00I3UTTGZIQHtGOowCFV3mQWfIpOjdJHQSTNCI1m6t+LCQ6VGoe+2UxDqmUvFf/zOokOrrwJE3GiqSDzR0HCkY5Q2gLqM0mJ5mNDMJHMZEVkiCUm2nS18MUPp6YTd7mBVdKsVtzzSvX+olS7ydrJwzGcQBlcuIQa3EEdGkAggRd4hTfr2Xq3PqzP+WrOym6OYAHW1y+ybZh1</latexit>

R<latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit>

R<latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit><latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>‣ DMRG:

G1

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

Step 1: Merge two adjacent cores<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit><latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>@G(i:i+1)

i, i+1K···@f<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit>

Step 2: Compute gradient over block cores

···
G1

Gd

<latexit sha1_base64="VgcWjQSuKkCImFieBiiUNqVXT24=">AAACKXicbVDLSsNAFJ3UV62vqEs3g0WoCCWpguKq6EKXFewDmlAm00k7dCYJMxOhhHyHH+E3uNW1O3Un/oiTNqBtPTBwOOdezp3jRYxKZVkfRmFpeWV1rbhe2tjc2t4xd/daMowFJk0cslB0PCQJowFpKqoY6USCIO4x0vZG15nffiBC0jC4V+OIuBwNAupTjJSWeqbt+ALhxImQUBQx6Ke/3OFIDTFiyU3aSyr0kp7Yx2naM8tW1ZoALhI7J2WQo9Ezv5x+iGNOAoUZkrJrW5FykywEM5KWnFiSCOERGpCupgHiRLrJ5GspPNJKH/qh0C9QcKL+3UgQl3LMPT2ZXSvnvUz8z+vGyr9wExpEsSIBngb5MYMqhFlPsE8FwYqNNUFYUH0rxEOku1K6zZkUj2ed2PMNLJJWrWqfVmt3Z+X6Vd5OERyAQ1ABNjgHdXALGqAJMHgEz+AFvBpPxpvxbnxORwtGvrMPZmB8/wBvfKfU</latexit>

Gd

G1<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

@`

@`
@f
=
@G(i:i+1)@f @G(i:i+1)

Gd<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="bXRFaA0jsgJ15m1jyx88jjMN+oQ=">AAACGHicbVC7TsMwFL3hWcorwMhiUSExVUlBgrGChbFI9CE1UeW4TmvVTiLbQaqirHwE38AKMxtiZWPkT3DaStCWI1k6Puc+7BMknCntOF/Wyura+sZmaau8vbO7t28fHLZUnEpCmyTmsewEWFHOItrUTHPaSSTFIuC0HYxuCr/9QKVicXSvxwn1BR5ELGQEayP1bOSFEpPMS7DUDHPkUc7z32uY9+yKU3UmQMvEnZEKzNDo2d9ePyapoJEmHCvVdZ1E+1kxkHCal71U0QSTER7QrqERFlT52eQnOTo1Sh+FsTQn0mii/u3IsFBqLAJTKbAeqkWvEP/zuqkOr/yMRUmqaUSmi8KUIx2jIhbUZ5ISzceGYCKZeSsiQ2yi0Sa8uS2BKDJxFxNYJq1a1T2v1u4uKvXrWTolOIYTOAMXLqEOt9CAJhB4hGd4gVfryXqz3q2PaemKNes5gjlYnz+XSaE+</latexit>@`@f<latexit sha1_base64="L4XP49jWAVKCwsi2/Z5CHy8aoV8=">AAACgHichVFbS8MwGE3rbc5bvbyJEhzCRJjtFJSBMPRBHxXcBdZS0izdwtILSSqM0nf/oo/+AX+D6daHXUQ/CJyc8518HydezKiQpvmp6Sura+sbpc3y1vbO7p6xf9AWUcIxaeGIRbzrIUEYDUlLUslIN+YEBR4jHW/0mOudd8IFjcI3OY6JE6BBSH2KkVSUa3zYPkc4tWPEJUUM2oSxbOYaIDnEiKVPmZtWaYNeWhdZBu/h3zY/W9D9f990jYpZMycFl4FVgAoo6sU1vux+hJOAhBIzJETPMmPppPkQzEhWthNBYoRHaEB6CoYoIMJJJ4ll8FwxfehHXJ1Qwgk760hRIMQ48FRnvq1Y1HLyN62XSP/OSWkYJ5KEeDrITxiUEczjh33KCZZsrADCnKpdIR4ilZVUnzQ3xQvyTKzFBJZBu16zrmv115tK86FIpwSOwRmoAgvcgiZ4Bi+gBTD41o60E+1U1/WqfqVb01ZdKzyHYK70xg8u5MRQ</latexit>

Supervised Learning with Quantum-Inspired Tensor Networks (Stoudenmire et al., NIPS 2016)
1DMRG: Density Matrix Renormalization Group50

Learning Algorithm: DMRG
Step 3: Update block core tensors by gradient

@`

@`
@f
=
@G(i:i+1)@f @G(i:i<latexit sha1_base64="L4XP49jWAVKCwsi2/Z5CHy8aoV8=">AAACgHichVFbS8MwGE3rbc5bvbyJEhzCRJjtFJSBMPRBHxXcBdZS0izdwtILSSqM0nf/oo/+AX+D6daHXUQ/CJyc8518HydezKiQpvmp6Sura+sbpc3y1vbO7p6xf9AWUcIxaeGIRbzrIUEYDUlLUslIN+YEBR4jHW/0mOudd8IFjcI3OY6JE6BBSH2KkVSUa3zYPkc4tWPEJUUM2oSxbOYaIDnEiKVPmZtWaYNeWhdZBu/h3zY/W9D9f990jYpZMycFl4FVgAoo6sU1vux+hJOAhBIzJETPMmPppPkQzEhWthNBYoRHaEB6CoYoIMJJJ4ll8FwxfehHXJ1Qwgk760hRIMQ48FRnvq1Y1HLyN62XSP/OSWkYJ5KEeDrITxiUEczjh33KCZZsrADCnKpdIR4ilZVUnzQ3xQvyTKzFBJZBu16zrmv115tK86FIpwSOwRmoAgvcgiZ4Bi+gBTD41o60E+1U1/WqfqVb01ZdKzyHYK70xg8u5MRQ</latexit>···<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>

i, i+1G1

Gd<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit>

Step 4: TT-rank is chosen adaptivelyWi<latexit sha1_base64="7jgIarwgB8SsQKCxEZTIYA0exD8=">AAACAXicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmUF+4DpUDJppg1NMkOSEcrQld/gVtfuxK1f4tI/MdPOwrYeCBzOuZd7csKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMnvCWxGBPOsM+1Xqm7NnQGtEq8gVSjQ7Fd+eoOYpIJKQzjW2vfcxAQZVoYRTqflXqppgskYD6lvqcSC6iCbRZ6ic6sMUBQr+6RBM/XvRoaF1hMR2sk8ol72cvE/z09NdBNkTCapoZLMD0UpRyZG+f/RgClKDJ9YgoliNisiI6wwMbalhSuhyDvxlhtYJe16zbus1R+uqo3bop0SnMIZXIAH19CAe2hCCwjE8AKv8OY8O+/Oh/M5H11zip0TWIDz9Qt1c5gH</latexit>

2 3<latexit sha1_base64="I6pTNfLRQHL9ABSqfh4pRCUtjzM=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRrVfKnsVbw53nfg5KUOOer/00xvENBUoDeVE667vJSbIiDKMcpwWe6nGhNAxGWLXUkkE6iCbHzp1L60ycKNY2ZLGnat/JzIitJ6I0HYKYkZ61ZuJ/3nd1ER3QcZkkhqUdLEoSrlrYnf2tTtgCqnhE0sIVcze6tIRUYQam83SllBMbSb+agLrpFWt+NeVauOmXLvP0ynAOVzAFfhwCzV4hDo0gQLCC7zCmzNx3p0P53PRuuHkM2ewBOfrFx5lk1A=</latexit>

1

2

3, 4, 5

1, 2

<latexit sha1_base64="s1uo5+ZHf/3rPtKcDB6n9aoTqrw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe4SQcugjWUC5gOSI+xt5pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRFFo04hHqhcQDZxJaBtmOPRiBUQEHLrB9GHud59BaRbJJ5PG4AsylixklBgrterDcsWtugvgTeLlpIJyNIfln8EoookAaSgnWvc9NzZ+RpRhlMOsNEg0xIROyRj6lkoiQPvZ4tAZvrLKCIeRsiUNXqh/JzIitE5FYDsFMRO97s3F/7x+YsI7P2MyTgxIulwUJhybCM+/xiOmgBqeWkKoYvZWTCdEEWpsNitbAjGzmXjrCWySTq3q1au11k2lcZ+nU0QX6BJdIw/dogZ6RE3URhQBekGv6M1JnXfnw/lcthacfOYcrcD5+gUf+JNR</latexit><latexit sha1_base64="NPqPz4i79G+LweZfI6lhLjKaDMk=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgIeEuCloGbSwjmg9IjrC32UuW7O4du3tCOPITbLW2E1t/jaX/xL3kCpP4YODx3gwz84KYM21c99sprK1vbG4Vt0s7u3v7B+XDo5aOEkVok0Q8Up0Aa8qZpE3DDKedWFEsAk7bwfgu89vPVGkWyScziakv8FCykBFsrPToXdT65YpbdWdAq8TLSQVyNPrln94gIomg0hCOte56bmz8FCvDCKfTUi/RNMZkjIe0a6nEgmo/nZ06RWdWGaAwUrakQTP170SKhdYTEdhOgc1IL3uZ+J/XTUx446dMxomhkswXhQlHJkLZ32jAFCWGTyzBRDF7KyIjrDAxNp2FLYGY2ky85QRWSatW9S6rtYerSv02T6cIJ3AK5+DBNdThHhrQBAJDeIFXeHNS5935cD7nrQUnnzmGBThfv/tCk8E=</latexit>

3<latexit sha1_base64="I6pTNfLRQHL9ABSqfh4pRCUtjzM=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRrVfKnsVbw53nfg5KUOOer/00xvENBUoDeVE667vJSbIiDKMcpwWe6nGhNAxGWLXUkkE6iCbHzp1L60ycKNY2ZLGnat/JzIitJ6I0HYKYkZ61ZuJ/3nd1ER3QcZkkhqUdLEoSrlrYnf2tTtgCqnhE0sIVcze6tIRUYQam83SllBMbSb+agLrpFWt+NeVauOmXLvP0ynAOVzAFfhwCzV4hDo0gQLCC7zCmzNx3p0P53PRuuHkM2ewBOfrFx5lk1A=</latexit>

R0

4<latexit sha1_base64="rdlpyfk1gl7a0CrCqjBan2ByBGo=">AAAB+HicbVA9TwJBEJ3DL8Qv1NJmIzFakTs00ZJoY4lEPhK4kL1lDzbs7l1290zwwj+w1drO2PpvLP0nLnCFgC+Z5OW9mczMC2LOtHHdbye3tr6xuZXfLuzs7u0fFA+PmjpKFKENEvFItQOsKWeSNgwznLZjRbEIOG0Fo7up33qiSrNIPppxTH2BB5KFjGBjpXr9vFcsuWV3BrRKvIyUIEOtV/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZpRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCW/8lMk4MVSS+aIw4chEaPo26jNFieFjSzBRzN6KyBArTIwNZ2FLICY2E285gVXSrJS9y3Ll4apUvc3SycMJnMIFeHANVbiHGjSAQAgv8ApvzrPz7nw4n/PWnJPNHMMCnK9fs2+ToQ==</latexit><latexit sha1_base64="n1k1qrO7uLoUYDoXrNsgfjCIwqQ=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlht8vlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1CxzSk08=</latexit>

i, i+1<latexit sha1_base64="n1k1qrO7uLoUYDoXrNsgfjCIwqQ=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlht8vlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1CxzSk08=</latexit>

4R<latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit><latexit sha1_base64="BxzUdPfK3U+ieVXKGeH7OtGNOes=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjIR8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1gcj/zO8+gNIvkk0lj8AUZSRYySoyVmo+DcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsJbP2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/S5Pp4jO0Dm6RB66QXX0gBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QtQxZNw</latexit><latexit sha1_base64="s1uo5+ZHf/3rPtKcDB6n9aoTqrw=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe4SQcugjWUC5gOSI+xt5pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRFFo04hHqhcQDZxJaBtmOPRiBUQEHLrB9GHud59BaRbJJ5PG4AsylixklBgrterDcsWtugvgTeLlpIJyNIfln8EoookAaSgnWvc9NzZ+RpRhlMOsNEg0xIROyRj6lkoiQPvZ4tAZvrLKCIeRsiUNXqh/JzIitE5FYDsFMRO97s3F/7x+YsI7P2MyTgxIulwUJhybCM+/xiOmgBqeWkKoYvZWTCdEEWpsNitbAjGzmXjrCWySTq3q1au11k2lcZ+nU0QX6BJdIw/dogZ6RE3URhQBekGv6M1JnXfnw/lcthacfOYcrcD5+gUf+JNR</latexit><latexit sha1_base64="JkqmLeNkd5bHVANb2wXSWqzazvk=">AAAB+3icbVBNT8JAEJ36ifiFevTSSEw8ENICRo9ELx4xsUACDdkuW9iwu212tyak6W/wqmdvxqs/xqP/xAV6EPAlk7y8N5OZeUHMqNKO821tbG5t7+wW9or7B4dHx6WT07aKEomJhyMWyW6AFGFUEE9TzUg3lgTxgJFOMLmf+Z1nIhWNxJOexsTnaCRoSDHSRvLqlUblelAqO1VnDnuduDkpQ47WoPTTH0Y44URozJBSPdeJtZ8iqSlmJCv2E0VihCdoRHqGCsSJ8tP5sZl9aZShHUbSlND2XP07kSKu1JQHppMjPVar3kz8z+slOrz1UyriRBOBF4vChNk6smef20MqCdZsagjCkppbbTxGEmFt8lnaEvDMZOKuJrBO2rWqW6/WHhvl5l2eTgHO4QKuwIUbaMIDtMADDBRe4BXerMx6tz6sz0XrhpXPnMESrK9f5ZaUOg==</latexit>

1

R

Before update<latexit sha1_base64="IjrKspETuJsfTI4pOd2MkQAkP0U=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjWUC5gOSI+xt9pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRBHaJhGPVC/AmnImadsww2kvVhSLgNNuMH2Y+91nqjSL5JNJY+oLPJYsZAQbK7Xqw3LFrboLoE3i5aQCOZrD8s9gFJFEUGkIx1r3PTc2foaVYYTTWWmQaBpjMsVj2rdUYkG1ny0OnaErq4xQGClb0qCF+nciw0LrVAS2U2Az0eveXPzP6ycmvPMzJuPEUEmWi8KEIxOh+ddoxBQlhqeWYKKYvRWRCVaYGJvNypZAzGwm3noCm6RTq3o31VqrXmnc5+kU4QIu4Ro8uIUGPEIT2kCAwgu8wpuTOu/Oh/O5bC04+cw5rMD5+gUhi5NS</latexit><latexit sha1_base64="IjrKspETuJsfTI4pOd2MkQAkP0U=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjWUC5gOSI+xt9pIlu3vH7p5wHPkFtlrbia0/x9J/4ia5wiQ+GHi8N8PMvCDmTBvX/XYKW9s7u3vF/dLB4dHxSfn0rKOjRBHaJhGPVC/AmnImadsww2kvVhSLgNNuMH2Y+91nqjSL5JNJY+oLPJYsZAQbK7Xqw3LFrboLoE3i5aQCOZrD8s9gFJFEUGkIx1r3PTc2foaVYYTTWWmQaBpjMsVj2rdUYkG1ny0OnaErq4xQGClb0qCF+nciw0LrVAS2U2Az0eveXPzP6ycmvPMzJuPEUEmWi8KEIxOh+ddoxBQlhqeWYKKYvRWRCVaYGJvNypZAzGwm3noCm6RTq3o31VqrXmnc5+kU4QIu4Ro8uIUGPEIT2kCAwgu8wpuTOu/Oh/O5bC04+cw5rMD5+gUhi5NS</latexit>

R0<latexit sha1_base64="rdlpyfk1gl7a0CrCqjBan2ByBGo=">AAAB+HicbVA9TwJBEJ3DL8Qv1NJmIzFakTs00ZJoY4lEPhK4kL1lDzbs7l1290zwwj+w1drO2PpvLP0nLnCFgC+Z5OW9mczMC2LOtHHdbye3tr6xuZXfLuzs7u0fFA+PmjpKFKENEvFItQOsKWeSNgwznLZjRbEIOG0Fo7up33qiSrNIPppxTH2BB5KFjGBjpXr9vFcsuWV3BrRKvIyUIEOtV/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZpRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCW/8lMk4MVSS+aIw4chEaPo26jNFieFjSzBRzN6KyBArTIwNZ2FLICY2E285gVXSrJS9y3Ll4apUvc3SycMJnMIFeHANVbiHGjSAQAgv8ApvzrPz7nw4n/PWnJPNHMMCnK9fs2+ToQ==</latexit>

R05<latexit sha1_base64="jIJE66YSYFqnTR7q/kRwRKAYCPU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6iomXQxjIB8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1g8jDzO8+gNIvkk0lj8AUZSRYySoyVmjeDcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsI7P2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/T5Pp4jO0Dm6RB66RXX0iBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QsjHpNT</latexit><latexit sha1_base64="rdlpyfk1gl7a0CrCqjBan2ByBGo=">AAAB+HicbVA9TwJBEJ3DL8Qv1NJmIzFakTs00ZJoY4lEPhK4kL1lDzbs7l1290zwwj+w1drO2PpvLP0nLnCFgC+Z5OW9mczMC2LOtHHdbye3tr6xuZXfLuzs7u0fFA+PmjpKFKENEvFItQOsKWeSNgwznLZjRbEIOG0Fo7up33qiSrNIPppxTH2BB5KFjGBjpXr9vFcsuWV3BrRKvIyUIEOtV/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZpRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCW/8lMk4MVSS+aIw4chEaPo26jNFieFjSzBRzN6KyBArTIwNZ2FLICY2E285gVXSrJS9y3Ll4apUvc3SycMJnMIFeHANVbiHGjSAQAgv8ApvzrPz7nw4n/PWnJPNHMMCnK9fs2+ToQ==</latexit>

5<latexit sha1_base64="jIJE66YSYFqnTR7q/kRwRKAYCPU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6iomXQxjIB8wHJEfY2c8mS3b1jd084jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoii0aMQj1Q2IBs4ktAwzHLqxAiICDp1g8jDzO8+gNIvkk0lj8AUZSRYySoyVmjeDcsWtunPgdeLlpIJyNAbln/4wookAaSgnWvc8NzZ+RpRhlMO01E80xIROyAh6lkoiQPvZ/NApvrDKEIeRsiUNnqt/JzIitE5FYDsFMWO96s3E/7xeYsI7P2MyTgxIulgUJhybCM++xkOmgBqeWkKoYvZWTMdEEWpsNktbAjG1mXirCayTdq3qXVVrzetK/T5Pp4jO0Dm6RB66RXX0iBqohSgC9IJe0ZuTOu/Oh/O5aC04+cwpWoLz9QsjHpNT</latexit>G̃(i,i+1)fW

Truncated SVD<latexit sha1_base64="hktwy/0v9dBXeIhQqi50eAHJD6c=">AAACD3icbVDLSsNAFJ34rPUVdeHCzWARXJWkCrosunFZwT6gCWUyuWmHziRhZqKUkI/wG9zq2p249RNc+icmbRa29cCFwzn3cu89XsyZ0pb1baysrq1vbFa2qts7u3v75sFhR0WJpNCmEY9kzyMKOAuhrZnm0IslEOFx6Hrj28LvPoJULAof9CQGV5BhyAJGic6lgXnsPDEfNOM+pI4gekQJT7tZNjBrVt2aAi8TuyQ1VKI1MH8cP6KJgFBTTpTq21as3ZRIzSiHrOokCmJCx2QI/ZyGRIBy0+kDGT7LFR8Hkcwr1Hiq/p1IiVBqIry8s7hRLXqF+J/XT3Rw7aYsjBMNIZ0tChKOdYSLNLDPJFDNJzkhVLL8VkxHRBKq88zmtniiyMReTGCZdBp1+6LeuL+sNW/KdCroBJ2ic2SjK9REd6iF2oiiDL2gV/RmPBvvxofxOWtdMcqZIzQH4+sX9dudvw==</latexit><latexit sha1_base64="0JTiPl8XNrVFAzHgLrxZNfRlhTw=">AAACFXicbVDLSsNAFJ3UV62vqEsRgkWoKCWpgi6LLnRZwT6gCWEymbRDJ5MwMxFKyMqP8Bvc6tqduHXt0j9x0mZhWw9cOJxzL/fe48WUCGma31ppaXllda28XtnY3Nre0Xf3OiJKOMJtFNGI9zwoMCUMtyWRFPdijmHoUdz1Rje5333EXJCIPchxjJ0QDhgJCIJSSa5+aEtCfZzaIZRDBGl6m2VuWiNn5NQ6yVy9atbNCYxFYhWkCgq0XP3H9iOUhJhJRKEQfcuMpZNCLgmiOKvYicAxRCM4wH1FGQyxcNLJG5lxrBTfCCKuikljov6dSGEoxDj0VGd+rZj3cvE/r5/I4MpJCYsTiRmaLgoSasjIyDMxfMIxknSsCEScqFsNNIQcIqmSm9nihXkm1nwCi6TTqFvn9cb9RbV5XaRTBgfgCNSABS5BE9yBFmgDBJ7AC3gFb9qz9q59aJ/T1pJWzOyDGWhfvxZen0Q=</latexit>iR0<latexit sha1_base64="rdlpyfk1gl7a0CrCqjBan2ByBGo=">AAAB+HicbVA9TwJBEJ3DL8Qv1NJmIzFakTs00ZJoY4lEPhK4kL1lDzbs7l1290zwwj+w1drO2PpvLP0nLnCFgC+Z5OW9mczMC2LOtHHdbye3tr6xuZXfLuzs7u0fFA+PmjpKFKENEvFItQOsKWeSNgwznLZjRbEIOG0Fo7up33qiSrNIPppxTH2BB5KFjGBjpXr9vFcsuWV3BrRKvIyUIEOtV/zp9iOSCCoN4VjrjufGxk+xMoxwOil0E01jTEZ4QDuWSiyo9tPZpRN0ZpU+CiNlSxo0U/9OpFhoPRaB7RTYDPWyNxX/8zqJCW/8lMk4MVSS+aIw4chEaPo26jNFieFjSzBRzN6KyBArTIwNZ2FLICY2E285gVXSrJS9y3Ll4apUvc3SycMJnMIFeHANVbiHGjSAQAgv8ApvzrPz7nw4n/PWnJPNHMMCnK9fs2+ToQ==</latexit>

After update

Supervised Learning with Quantum-Inspired Tensor Networks (Stoudenmire et al., NIPS 2016)51

Discussions
Advantages：‣ Single and wide layer with comparable accuracy‣ Highly efficient model compression and computation‣ Better interpretability due to multilinear nature
a simple and efficient way to represent data input‣ Provide
as TN format
Challenges:
SGD with DMRG algorithm is non-trivial, batch‣ Mixing
gradient is needed
hyper-parameters: dimension of nonlinear mapping,‣ MoreTT-ranks
52

Exponential Machines‣ Mapping of input datax = [x1 , x2 , . . . , xd ]T<latexit sha1_base64="BLWEx98A3/vcyb7W+XUHdBrmvUY=">AAACHXicbVC7TsMwFHXKq5RXgJHFoqrEUFVJQYIFqYKFsUh9SWmIHMdprToP2Q5qFfUL+Ai+gRVmNsSKGPkTnDYDbTmSreNz7tW9Pm7MqJCG8a0V1tY3NreK26Wd3b39A/3wqCOihGPSxhGLeM9FgjAakrakkpFezAkKXEa67ug287uPhAsahS05iYkdoEFIfYqRVJKjV/oBkkPXh+NraMGxY1bVVa/CPvMiKbKHZz+0HL1s1IwZ4Coxc1IGOZqO/tP3IpwEJJSYISEs04ilnSIuKWZkWuongsQIj9CAWIqGKCDCTmffmcKKUjzoR1ydUMKZ+rcjRYEQk8BVldnyYtnLxP88K5H+lZ3SME4kCfF8kJ8wKCOYZQM9ygmWbKIIwpyqXSEeIo6wVAkuTHGDqcrEXE5glXTqNfO8Vr+/KDdu8nSK4AScgjNggkvQAHegCdoAgyfwAl7Bm/asvWsf2ue8tKDlPcdgAdrXLz68oLY=</latexit>

p 1 T
2(xi ) = [1, xi , xi , . . . , xi ]<latexit sha1_base64="JdpqkdUXm1NoHSf6/q3fEYSDtkE=">AAACJ3icbVDLSgMxFM3UV62vUZdugkWoUOtMFXQjFN24rNAXzExLJs20oZkHSUYsQz/Dj/Ab3Oraneiyf2I6nYWtHkg4Oede7s1xI0aFNIxvLbeyura+kd8sbG3v7O7p+wctEcYckyYOWcg7LhKE0YA0JZWMdCJOkO8y0nZHdzO//Ui4oGHQkOOIOD4aBNSjGEkl9fRzOxrS0lOPnsIbyyxDxdKrWy1Dm/VDKebPJDozJ0630dOLRsVIAf8SMyNFkKHe06d2P8SxTwKJGRLCMo1IOgnikmJGJgU7FiRCeIQGxFI0QD4RTpJ+bAJPlNKHXsjVCSRM1d8dCfKFGPuuqvSRHIplbyb+51mx9K6dhAZRLEmA54O8mEEZwllKsE85wZKNFUGYU7UrxEPEEZYqy4Uprj9RmZjLCfwlrWrFvKhUHy6LtdssnTw4AsegBExwBWrgHtRBE2DwDF7BG3jXXrQP7VP7mpfmtKznECxAm/4AZgakZw==</latexit>

d 7! pd(x) = (x1 ) ⌦ (x2 ) ⌦ · · · ⌦ (xd )<latexit sha1_base64="con2OZWVyoEZ488qINDn0Gan+ps=">AAACQnicbZDLSgMxFIYzXmu9VV26CRah3ZSZKuhGKLoRVxXsBTqlZDKZNjQzGZIz0lL6Tj6EzyC4UrfuxK0L0xvY1gOBn+8/h5Pze7HgGmz71VpZXVvf2Extpbd3dvf2MweHVS0TRVmFSiFV3SOaCR6xCnAQrB4rRkJPsJrXvRn5tUemNJfRA/Rj1gxJO+IBpwQMamXu3HKH59yQQMcLcC+Pr7AbG9JrOXlXAg+ZnoHiDLjUl6AXXD/fymTtgj0uvCycqciiaZVbmQ/XlzQJWQRUEK0bjh1Dc0AUcCrYMO0mmsWEdkmbNYyMiFnXHIxvHuJTQ3wcSGVeBHhM/04MSKh1P/RM5+g4veiN4H9eI4HgsjngUZwAi+hkUZAIDBKPAsQ+V4yC6BtBqOLmr5h2iCIUTMxzW7xwaDJxFhNYFtViwTkrFO/Ps6XraTopdIxOUA456AKV0C0qowqi6Am9oDf0bj1bn9aX9T1pXbGmM0dorqyfX/VLsHA=</latexit><latexit sha1_base64="jKCu50Y6XVxcycv1aQqZ+Bmzx08=">AAACAnicbVA9SwNBEJ2LXzF+RS1tDoNgFe6ioGXQxjKC+YDcGfb29pIlu3vL7p4QQjp/g63WdmLrH7H0n7hJrjCJDwYe780wMy+SjGrjed9OYW19Y3OruF3a2d3bPygfHrV0milMmjhlqepESBNGBWkaahjpSEUQjxhpR8Pbqd9+IkrTVDyYkSQhR31BE4qRsVIQBxxJbVJXPsa9csWrejO4q8TPSQVyNHrlnyBOccaJMJghrbu+J004RspQzMikFGSaSISHqE+6lgrEiQ7Hs5sn7plVYjdJlS1h3Jn6d2KMuNYjHtlOjsxAL3tT8T+vm5nkOhxTITNDBJ4vSjLm2ienAbgxVQQbNrIEYUXtrS4eIIWwsTEtbIn4xGbiLyewSlq1qn9Rrd1fVuo3eTpFOIFTOAcfrqAOd9CAJmCQ8AKv8OY8O+/Oh/M5by04+cwxLMD5+gXSeZgy</latexit>‣ p-degree polynomial function(x1 )<latexit sha1_base64="AbBuSfOiY1GwlX/pOzx0lXn7Gb8=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHubvWTJ7t6xuyeGI4W/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRfEnGnjut/Oyura+sZmbiu/vbO7t184OGzoKFGE1knEI9UKsKacSVo3zHDaihXFIuC0GQxvJn7zkSrNInlvRjH1Be5LFjKCjZUeOvGAlZ663lm3UHTL7hRomXgZKUKGWrfw0+lFJBFUGsKx1m3PjY2fYmUY4XSc7ySaxpgMcZ+2LZVYUO2n04PH6NQqPRRGypY0aKr+nUix0HokAtspsBnoRW8i/ue1ExNe+SmTcWKoJLNFYcKRidDke9RjihLDR5Zgopi9FZEBVpgYm9HclkCMbSbeYgLLpFEpe+flyt1FsXqdpZODYziBEnhwCVW4hRrUgYCAF3iFN+fZeXc+nM9Z64qTzRzBHJyvX65wlmQ=</latexit><latexit sha1_base64="YBttwdOuOWY4x4aLW+83RArt7gM=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHt7e8mS3b1jd08MIYW/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRcknGnjut/Oyura+sZmbiu/vbO7t184OGzoOFWE1knMY9UKsKacSVo3zHDaShTFIuC0GQxuJn7zkSrNYnlvhgn1Be5JFjGCjZUeOkmflZ664Vm3UHTL7hRomXgZKUKGWrfw0wljkgoqDeFY67bnJsYfYWUY4XSc76SaJpgMcI+2LZVYUO2PpgeP0alVQhTFypY0aKr+nRhhofVQBLZTYNPXi95E/M9rpya68kdMJqmhkswWRSlHJkaT71HIFCWGDy3BRDF7KyJ9rDAxNqO5LYEY20y8xQSWSaNS9s7LlbuLYvU6SycHx3ACJfDgEqpwCzWoAwEBL/AKb86z8+58OJ+z1hUnmzmCOThfv/7slpc=</latexit>

···

(x)

f (x) = hW, (x)i(xd )<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit><latexit sha1_base64="7hYzq4AhBpa0Pb7Iq9kVg5m2Yvg=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRtIvlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1C3//k44=</latexit>p<latexit sha1_base64="7hYzq4AhBpa0Pb7Iq9kVg5m2Yvg=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRtIvlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1C3//k44=</latexit>p<latexit sha1_base64="m1ksc1aV+naOnCUX0CgvTyqCF4M=">AAACLnicbVDLSsNAFJ3UV62vqEs3g0WoICWpom6EohuXFewDmlIm00k7dDIJMxOxhPyJH+E3uNW14ELEnZ/hJM3Cth4YOJxzLvfOcUNGpbKsD6OwtLyyulZcL21sbm3vmLt7LRlEApMmDlggOi6ShFFOmooqRjqhIMh3GWm745vUbz8QIWnA79UkJD0fDTn1KEZKS33z3Ks4PlIj14OPx1cOQ3zISKZgxOJ2cuI0RvRPxBFZom+WraqVAS4SOydlkKPRN7+dQYAjn3CFGZKya1uh6sVIKIoZSUpOJEmI8BgNSVdTjnwie3H2vwQeaWUAvUDoxxXM1L8TMfKlnPiuTqaHynkvFf/zupHyLnsx5WGkCMfTRV7EoApgWhYcUEGwYhNNEBZU3wrxCAmEla50ZovrJ7oTe76BRdKqVe3Tau3urFy/ztspggNwCCrABhegDm5BAzQBBk/gBbyCN+PZeDc+ja9ptGDkM/tgBsbPL1mvqUQ=</latexit>W

d-order tensor<latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>

Example:f (x) = W000 + W100 x1 + W010 x2 + W001 x3+ W110 x1 x2 + W101 x1 x3 + W011 x2 x3

d = 3, p = 2<latexit sha1_base64="vJp2KUGXx8QWdKoOQZbMMRrWoJQ=">AAAB/nicbVA9SwNBEJ3zM8avqKXNYhAsJNwlgjaBoI1lBPMByRH2NnvJkt29Y3dPCEfA32CrtZ3Y+lcs/SdukitM4oOBx3szzMwLYs60cd1vZ219Y3NrO7eT393bPzgsHB03dZQoQhsk4pFqB1hTziRtGGY4bceKYhFw2gpGd1O/9USVZpF8NOOY+gIPJAsZwcZK7X61coniarlXKLoldwa0SryMFCFDvVf46fYjkggqDeFY647nxsZPsTKMcDrJdxNNY0xGeEA7lkosqPbT2b0TdG6VPgojZUsaNFP/TqRYaD0Wge0U2Az1sjcV//M6iQlv/JTJODFUkvmiMOHIRGj6POozRYnhY0swUczeisgQK0yMjWhhSyAmNhNvOYFV0iyXvEqp/HBVrN1m6eTgFM7gAjy4hhrcQx0aQIDDC7zCm/PsvDsfzue8dc3JZk5gAc7XL/5ulWM=</latexit><latexit sha1_base64="QHOTLk3OztfA9vEcpO7pabUqGQI=">AAAC1XicdZFLSwMxEMez67u+qh69BIulIpSkFfQiiF48VrBW6JYlm2bbYPZBkhXLsjfx6ufz4MFvYna3B/twIDD5/fOfCTNeLLjSCH1Z9srq2vrG5lZle2d3b796cPikokRS1qWRiOSzRxQTPGRdzbVgz7FkJPAE63kvd7nee2VS8Sh81JOYDQIyCrnPKdEGudVvv+EERI89H76dwfo1LG6UiLSXuSlCKIPnswzn7M3F8xzhgrcWOMI5bztOpT5fqrTgJS5cuozUXmxUSK3/auKsLGl0t1pDTVQEXEzwNKmBaXTc6o8zjGgSsFBTQZTqYxTrQUqk5lSwrOIkisWEvpAR65s0JAFTg7RYQwZPDRlCP5LmhBoW9K8jJYFSk8AzL/Mfq3kth8u0fqL9q0HKwzjRLKRlIz8RUEcw3ykccsmoFhOTECq5+SukYyIJ1WbzM128IDMzwfMTWEyeWk3cbrYeLmo3t9PpbIJjcAIaAINLcAPuQQd0AbXuLG5JS9k9O7Pf7Y/yqW1NPUdgJuzPX4/Z3MU=</latexit>+ W111 x1 x2 x3‣ Interactions of features are fully captured
weights of interactions and increases exponentially‣ Wwithcontains
the dimension of input<latexit sha1_base64="7jgIarwgB8SsQKCxEZTIYA0exD8=">AAACAXicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmUF+4DpUDJppg1NMkOSEcrQld/gVtfuxK1f4tI/MdPOwrYeCBzOuZd7csKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMnvCWxGBPOsM+1Xqm7NnQGtEq8gVSjQ7Fd+eoOYpIJKQzjW2vfcxAQZVoYRTqflXqppgskYD6lvqcSC6iCbRZ6ic6sMUBQr+6RBM/XvRoaF1hMR2sk8ol72cvE/z09NdBNkTCapoZLMD0UpRyZG+f/RgClKDJ9YgoliNisiI6wwMbalhSuhyDvxlhtYJe16zbus1R+uqo3bop0SnMIZXIAH19CAe2hCCwjE8AKv8OY8O+/Oh/M5H11zip0TWIDz9Qt1c5gH</latexit>

Exponential Machines (Novikov et al., ICLR workshop 2017)53

Exponential Machines
f (x) =

p
X

i1 =1

<latexit sha1_base64="8TtkiTgyf9YwWTf9jfHMltm+0Bs=">AAACZnicbZHPS9xAFMcn0dbVtroqxUMvjy4FC3ZJVNDLgtiLRwtdV9isYTKZ6JD5EWYm4hLyX3rx6M2/wZOTGKFqHwx8+b7P4818Jyk4MzYI7jx/YfHDx6Xe8sqnz19W1/rrG2dGlZrQMVFc6fMEG8qZpGPLLKfnhaZYJJxOkvx3059cU22Ykn/tvKAzgS8lyxjB1llxX2bbkcD2Ksng5ieMIDKliCsWh6OwvqiKOiKpsubFTTsX2hmCeTWpW3on4g23A46pAaJCqzSu8gZP4SbOLxyU/wrruD8IhkFb8F6EnRigrk7j/n2UKlIKKi3h2JhpGBR2VmFtGeG0XolKQwtMcnxJp05KLKiZVW0uNfxwTgqZ0u5IC63770SFhTFzkTiyeY9522vM//Wmpc0OZxWTRWmpJM+LspKDVdCEDCnTlFg+dwITzdxdgVxhjYl1X/FqSyKaTMK3CbwXZ7vDcG+4+2d/cHTcpdND39B3tI1CdICO0Ak6RWNE0C169Ba8Re/BX/W/+lvPqO91M5voVfnwBJcxue4=</latexit>

···

p
X

id =1

Wi1 ,...,id<latexit sha1_base64="cWevHXNBXC8pGezPMZ62PmQtf3c=">AAACBHicbVC7SgNBFL0bXzG+opY2g0GITdiNgjZC0MYygnlAsobZyWwyZGZ3mZkVw5LWb7DV2k5s/Q9L/8TZZAuTeODC4Zx7uYfjRZwpbdvfVm5ldW19I79Z2Nre2d0r7h80VRhLQhsk5KFse1hRzgLa0Exz2o4kxcLjtOWNblK/9UilYmFwr8cRdQUeBMxnBGsjPfjlrsB66Pno6RRd9Yolu2JPgZaJk5ESZKj3ij/dfkhiQQNNOFaq49iRdhMsNSOcTgrdWNEIkxEe0I6hARZUuck09QSdGKWP/FCaCTSaqn8vEiyUGgvPbKYZ1aKXiv95nVj7l27CgijWNCCzR37MkQ5RWgHqM0mJ5mNDMJHMZEVkiCUm2hQ198UTE9OJs9jAMmlWK85ZpXp3XqpdZ+3k4QiOoQwOXEANbqEODSAg4QVe4c16tt6tD+tztpqzsptDmIP19Qvc8Zga</latexit>

xikk 1k=1

dO(dpd )

p2
dpr
<latexit sha1_base64="pU2PjrkgjHl13g5p0Jmde7X6ajg=">AAAB+3icbVBNS8NAEJ34WetX1aOXxSJ4KkkV9Fj04rGCaQttLJvNpl26uwm7G6GU/AavevYmXv0xHv0nbtscbOuDgcd7M8zMC1POtHHdb2dtfWNza7u0U97d2z84rBwdt3SSKUJ9kvBEdUKsKWeS+oYZTjupoliEnLbD0d3Ubz9TpVkiH804pYHAA8liRrCxkh+l6qner1TdmjsDWiVeQapQoNmv/PSihGSCSkM41rrruakJJlgZRjjNy71M0xSTER7QrqUSC6qDyezYHJ1bJUJxomxJg2bq34kJFlqPRWg7BTZDvexNxf+8bmbim2DCZJoZKsl8UZxxZBI0/RxFTFFi+NgSTBSztyIyxAoTY/NZ2BKK3GbiLSewSlr1mndZqz9cVRu3RTolOIUzuAAPrqEB99AEHwgweIFXeHNy5935cD7nrWtOMXMCC3C+fgFLuJUc</latexit>O(dpr2 )<latexit sha1_base64="Ncs2odeEPLv30N5cprHQ6fqXqeU=">AAACCnicbVDLTsJAFJ3iC/FVcelmIjHBDWnRRJdEN+7ERJAEKplOpzBhZtrMTI2k4Q/8Bre6dmfc+hMu/ROn0IWAJ7nJyTn35p4cP2ZUacf5tgorq2vrG8XN0tb2zu6evV9uqyiRmLRwxCLZ8ZEijArS0lQz0oklQdxn5N4fXWX+/SORikbiTo9j4nE0EDSkGGkj9e1yjyM9xIilN5NqEMuH+knfrjg1Zwq4TNycVECOZt/+6QURTjgRGjOkVNd1Yu2lSGqKGZmUeokiMcIjNCBdQwXiRHnpNPsEHhslgGEkzQgNp+rfixRxpcbcN5tZUrXoZeJ/XjfR4YWXUhEnmgg8exQmDOoIZkXAgEqCNRsbgrCkJivEQyQR1qauuS8+n5hO3MUGlkm7XnNPa/Xbs0rjMm+nCA7BEagCF5yDBrgGTdACGDyBF/AK3qxn6936sD5nqwUrvzkAc7C+fgFwFJqd</latexit>(xd )<latexit sha1_base64="YBttwdOuOWY4x4aLW+83RArt7gM=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHt7e8mS3b1jd08MIYW/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRcknGnjut/Oyura+sZmbiu/vbO7t184OGzoOFWE1knMY9UKsKacSVo3zHDaShTFIuC0GQxuJn7zkSrNYnlvhgn1Be5JFjGCjZUeOkmflZ664Vm3UHTL7hRomXgZKUKGWrfw0wljkgoqDeFY67bnJsYfYWUY4XSc76SaJpgMcI+2LZVYUO2PpgeP0alVQhTFypY0aKr+nRhhofVQBLZTYNPXi95E/M9rpya68kdMJqmhkswWRSlHJkaT71HIFCWGDy3BRDF7KyJ9rDAxNqO5LYEY20y8xQSWSaNS9s7LlbuLYvU6SycHx3ACJfDgEqpwCzWoAwEBL/AKb86z8+58OJ+z1hUnmzmCOThfv/7slpc=</latexit>(x)<latexit sha1_base64="bxjCNYAtkLf7rw+8941N/QkfQ3A=">AAACCXicbVDLTsJAFJ3iC/FVdOlmIjHBDWnRRJdEN+7ERJAEKplOpzBhZtrMTDWk4Qv8Bre6dmfc+hUu/ROn0IWAJ7nJyTn35p4cP2ZUacf5tgorq2vrG8XN0tb2zu6eXd5vqyiRmLRwxCLZ8ZEijArS0lQz0oklQdxn5N4fXWX+/SORikbiTo9j4nE0EDSkGGkj9e1yjyM9xIilN5NqED8EJ3274tScKeAycXNSATmaffunF0Q44URozJBSXdeJtZciqSlmZFLqJYrECI/QgHQNFYgT5aXT6BN4bJQAhpE0IzScqn8vUsSVGnPfbGZB1aKXif953USHF15KRZxoIvDsUZgwqCOY9QADKgnWbGwIwpKarBAPkURYm7bmvvh8YjpxFxtYJu16zT2t1W/PKo3LvJ0iOARHoApccA4a4Bo0QQtg8ARewCt4s56td+vD+pytFqz85gDMwfr6Bd92mlM=</latexit><latexit sha1_base64="vvu3qg7c77IblFA7Hx1hwTeORjU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKi+YDkDHt7e8mS3b1jd08IR36CrdZ2YuuvsfSfuEmuMIkPBh7vzTAzL0g408Z1v53C2vrG5lZxu7Szu7d/UD48auk4VYQ2Scxj1QmwppxJ2jTMcNpJFMUi4LQdjG6nfvuZKs1i+WjGCfUFHkgWMYKNlR6Sp7BfrrhVdwa0SrycVCBHo1/+6YUxSQWVhnCsdddzE+NnWBlGOJ2UeqmmCSYjPKBdSyUWVPvZ7NQJOrNKiKJY2ZIGzdS/ExkWWo9FYDsFNkO97E3F/7xuaqJrP2MySQ2VZL4oSjkyMZr+jUKmKDF8bAkmitlbERlihYmx6SxsCcTEZuItJ7BKWrWqd1Gt3V9W6jd5OkU4gVM4Bw+uoA530IAmEBjAC7zCm5M5786H8zlvLTj5zDEswPn6BfyalGQ=</latexit>

With TN···<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit>r r

WComputation

Normal(x1 )<latexit sha1_base64="AbBuSfOiY1GwlX/pOzx0lXn7Gb8=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHubvWTJ7t6xuyeGI4W/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRfEnGnjut/Oyura+sZmbiu/vbO7t184OGzoKFGE1knEI9UKsKacSVo3zHDaihXFIuC0GQxvJn7zkSrNInlvRjH1Be5LFjKCjZUeOvGAlZ663lm3UHTL7hRomXgZKUKGWrfw0+lFJBFUGsKx1m3PjY2fYmUY4XSc7ySaxpgMcZ+2LZVYUO2n04PH6NQqPRRGypY0aKr+nUix0HokAtspsBnoRW8i/ue1ExNe+SmTcWKoJLNFYcKRidDke9RjihLDR5Zgopi9FZEBVpgYm9HclkCMbSbeYgLLpFEpe+flyt1FsXqdpZODYziBEnhwCVW4hRrUgYCAF3iFN+fZeXc+nM9Z64qTzRzBHJyvX65wlmQ=</latexit>

f (x) =dY

Memory<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>

W = TT(G1 , . . . , Gd )<latexit sha1_base64="Uy48ySauEjTpxtagi75L+zemdQk=">AAACNXicbVDJSgNBEO2JW4xb1KOXxiBEkDATBb0Egh70GCEbZELo6ekkTXoWumvEMMzX+BF+g1c9efAgiFd/wc6CZPFB04/3qqiq54SCKzDNdyO1srq2vpHezGxt7+zuZfcP6iqIJGU1GohANh2imOA+qwEHwZqhZMRzBGs4g5uR33hgUvHAr8IwZG2P9Hze5ZSAljrZku0R6FMi4kaCS9gG9ghxtZrk//TbpGOdYVu4ASj9z8juaSebMwvmGHiZWFOSQ1NUOtlP2w1o5DEfqCBKtSwzhHZMJHAqWJKxI8VCQgekx1qa+sRjqh2Pz0zwiVZc3A2kfj7gsTrbERNPqaHn6MrRlmrRG4n/ea0IulftmPthBMynk0HdSGAI8Cgz7HLJKIihJoRKrnfFtE8koaCTnZvieInOxFpMYJnUiwXrvFC8v8iVr6fppNEROkZ5ZKFLVEZ3qIJqiKIn9IJe0ZvxbHwYX8b3pDRlTHsO0RyMn1/7DKwS</latexit>

r r<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

f (x) =
<latexit sha1_base64="SGRA8W6YC4+GRO5ENiAqdQWQE04=">AAAC23icdVFNb9QwEHXCV1m+FjhysVghbQ+s4gUJLpUqOAC3IrFtpU0aOfZk16qdRPYEdhXlxA1x5cdxRPwRnG0q6LaMZOn5vZk3o5ms0sphFP0MwmvXb9y8tXN7cOfuvfsPhg8fHbqytgJmotSlPc64A60KmKFCDceVBW4yDUfZ6dtOP/oM1qmy+ITrChLDF4XKleDoqXT4Ox/HhuMyy+lql+5RGmvIcRy72qSNStkea08qukrVSfd7ztpNtuC6edembO65hMZWLZa4S2MhS3RbDvLcQXYOcsuhka33kH89Bn6G+IuSgEpLaPrZfKrv9j9het74almmw1E0iTZBLwPWgxHp4yAd/oplKWoDBQrNnZuzqMKk4RaV0NAO4tpBxcUpX8Dcw4IbcEmzOUZLn3lG0ry0/hVIN+y/FQ03zq1N5jO7Id221pFXafMa89dJo4qqRijEWaO81hRL2l2WSmVBoF57wIVVflYqltxygf7+F7pkpvU7YdsbuAwOpxP2YjL9+HK0/6bfzg55Qp6SMWHkFdkn78kBmRERfAjKYBWswyT8Gn4Lv5+lhkFf85hciPDHHxrQ5to=</latexit>

p
X

i1 =1

!

xii1 1 G1 [i1 ] · · ·

p
X

id =1

xidd 1 Gd [id ]<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>!

e 1G
e2 ···Ged=G

TN representation is highly efficient for very large d, p‣
‣ Low-rank structure of TN (i.e., TT-rank) controls expressive power<latexit sha1_base64="M0650FM65GnYMpebDPRAVAszLFA=">AAAB93icbVBNS8NAEN34WetX1aOXxSJ4KkkV9Fj04rEF+wFtKJvNpF26uwm7GyGE/gKvevYmXv05Hv0nbtscbOuDgcd7M8zMCxLOtHHdb2djc2t7Z7e0V94/ODw6rpycdnScKgptGvNY9QKigTMJbcMMh16igIiAQzeYPMz87jMozWL5ZLIEfEFGkkWMEmOlVjisVN2aOwdeJ15BqqhAc1j5GYQxTQVIQznRuu+5ifFzogyjHKblQaohIXRCRtC3VBIB2s/nh07xpVVCHMXKljR4rv6dyInQOhOB7RTEjPWqNxP/8/qpie78nMkkNSDpYlGUcmxiPPsah0wBNTyzhFDF7K2Yjoki1NhslrYEYmoz8VYTWCedes27rtVbN9XGfZFOCZ2jC3SFPHSLGugRNVEbUQToBb2iNydz3p0P53PRuuEUM2doCc7XL20bk4I=</latexit><latexit sha1_base64="7hYzq4AhBpa0Pb7Iq9kVg5m2Yvg=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRtIvlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1C3//k44=</latexit>

Exponential Machines (Novikov et al., ICLR workshop 2017)54

epartment of Computer Science, Brown University
School of Computing, University of Utah
Model Xjera
Compression
alabs.com,
Labs, Pte.Ltdvia Different TN Models‣ Various DL models + Various TF models…

rful seth highcompuparamnt comeos andproposeFigureArchitectureof BT-LSTM.
The redundant dense
Block 1:term
decomposition(Ye et al., 2018)rm tenconnections between input and hidden state is replaced byameters
Tesnor ring decomposition (Pan et al., 2019)
low-rank BT representation.
mpared…
tensor-Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al. CVPR 2018)
Memory(LSTM)
introduces
numberof gatesandpassesNN (BT-Compressing
recurrent neuralnetworks
with tensor ringafor
action recognition(Pan etal. AAAI
2019)
information with element-wise operations [14]. This ime rank),
55
provement
drasticallyalleviates
the gradient vanishing ise origi-

Model Compression: Tensor Contraction
Standard CNN

followed by fully-connected layers discards multilinear structure‣ Flattening
in the activations and require many parameters
Tensor Contraction Layer (TCL)

Tensor Regression Networks

ImageNet (VGG and ResNet)
65% reduction of parameters

Tensor Regression Layer (TRL)

multilinear structure by TCL, low-rankness on weights by TRL‣ Preserving‣ Reduce parameters while maintaining or increasing accuracy
Tensor Regression Networks (Kossaifi et al., JMLR 2020)56

Tensor Contraction Layer
Tensor Contraction(I1 ⇥ I2 ⇥ I3 )(R1 ⇥ R2 ⇥ R3 )<latexit sha1_base64="a7UZcdvsHI2O7S8HLMuwi0lpNQc=">AAACEXicbVDLSgMxFM34rPU16krcBItQN2WmFXRZdGN3FewD2mHIpJk2NMkMSUYoQ/Ej/Aa3unYnbv0Cl/6JaTuIbT0QOJxzL+fmBDGjSjvOl7Wyura+sZnbym/v7O7t2weHTRUlEpMGjlgk2wFShFFBGppqRtqxJIgHjLSC4c3Ebz0QqWgk7vUoJh5HfUFDipE2km8fF2u+29WUEwVrfvmXVc59u+CUnCngMnEzUgAZ6r793e1FOOFEaMyQUh3XibWXIqkpZmSc7yaKxAgPUZ90DBXIJHnp9AtjeGaUHgwjaZ7QcKr+3UgRV2rEAzPJkR6oRW8i/ud1Eh1eeSkVcaKJwLOgMGFQR3DSB+xRSbBmI0MQltTcCvEASYS1aW0uJeBj04m72MAyaZZLbqVUvrsoVK+zdnLgBJyCInDBJaiCW1AHDYDBI3gGL+DVerLerHfrYza6YmU7R2AO1ucP6amcWg==</latexit><latexit sha1_base64="axLk9tdNgaC4xFpEVO3AJ8l6Cmc=">AAACEXicbVDLSgMxFM34rPU16krcBItQN2WmFXRZdOOyFvuAdhgyaaYNTTJDkhHKUPwIv8Gtrt2JW7/ApX9i2g5iWw8EDufcy7k5Qcyo0o7zZa2srq1vbOa28ts7u3v79sFhU0WJxKSBIxbJdoAUYVSQhqaakXYsCeIBI61geDPxWw9EKhqJez2KicdRX9CQYqSN5NvHxbrvdjXlRMG6X/5llXPfLjglZwq4TNyMFECGmm9/d3sRTjgRGjOkVMd1Yu2lSGqKGRnnu4kiMcJD1CcdQwUySV46/cIYnhmlB8NImic0nKp/N1LElRrxwExypAdq0ZuI/3mdRIdXXkpFnGgi8CwoTBjUEZz0AXtUEqzZyBCEJTW3QjxAEmFtWptLCfjYdOIuNrBMmuWSWymV7y4K1eusnRw4AaegCFxwCargFtRAA2DwCJ7BC3i1nqw36936mI2uWNnOEZiD9fkDFZicdQ==</latexit>

Low-dimensional
representation

Activation tensor

V(k) 2 RRk ⇥Ik<latexit sha1_base64="A/7apjfWqHOUIavTAEpDMGcgPIA=">AAACJnicbVDLSsNAFJ34rPUVdelmsAh1U5Iq6LLoRne12Ac0bZhMJ+2QySTMTIQS8hd+hN/gVtfuRNzpnzhpu7CtBwYO59zLuXO8mFGpLOvLWFldW9/YLGwVt3d29/bNg8OWjBKBSRNHLBIdD0nCKCdNRRUjnVgQFHqMtL3gJvfbj0RIGvEHNY5JL0RDTn2KkdKSa1acEKmR58NWPy0HZxl0KIdTzUsbWT9tuIGjaEgkvHODzDVLVsWaAC4Te0ZKYIa6a/44gwgnIeEKMyRl17Zi1UuRUBQzkhWdRJIY4QANSVdTjnRSL538K4OnWhlAPxL6cQUn6t+NFIVSjkNPT+YXy0UvF//zuonyr3op5XGiCMfTID9hUEUwLwkOqCBYsbEmCAuqb4V4hATCSlc5l+KFeSf2YgPLpFWt2OeV6v1FqXY9a6cAjsEJKAMbXIIauAV10AQYPIEX8ArejGfj3fgwPqejK8Zs5wjMwfj+BQ8OphE=</latexit>

Projection factors

W

O

T

<latexit sha1_base64="smBngnAvs95iqsCISPy3O+FGRCU=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r9CFMx5JJM21oMhmSjFBKV36DW127E7d+iUv/xEw7C9t6IHA4517uyQkTzrRx3W+nsLa+sblV3C7t7O7tH5QPj9paporQFpFcqocQa8pZTFuGGU4fEkWxCDnthKPbzO88UaWZjJtmnNBA4EHMIkawsZLfFdgMwwh1Hpu9csWtujOgVeLlpAI5Gr3yT7cvSSpobAjHWvuem5hggpVhhNNpqZtqmmAywgPqWxpjQXUwmUWeojOr9FEklX2xQTP178YEC63HIrSTWUS97GXif56fmug6mLA4SQ2NyfxQlHJkJMr+j/pMUWL42BJMFLNZERlihYmxLS1cCcXUduItN7BK2rWqd1Gt3V9W6jd5O0U4gVM4Bw+uoA530IAWEJDwAq/w5jw7786H8zkfLTj5zjEswPn6BZTEl3k=</latexit>⇣

vec(X 0 ) = V(0) ⌦ · · · ⌦ V(N )<latexit sha1_base64="k1zrLPd2+wOs1+rQYErSn2Ww8iI=">AAACa3icbVHLTsMwEHTCq7wL3ICDRYVoL1UCSHBBQnDhhECipVJTKsfdtBZOHNkbRBXlQznyAXwDuG0OvFayNJ6d9Y7GYSqFQc97c9y5+YXFpcryyura+sZmdWu7bVSmObS4kkp3QmZAigRaKFBCJ9XA4lDCY/h8Pek/voA2QiUPOE6hF7NhIiLBGVqqXzUBwivmL8CLehAzHHEmaeeoQS9oICHCGRlGtP2U171GESgUMZiADxSa8kK/a26tRovhCBtPD/++3aD9as1retOif4Ffghop665ffQ8GimcxJMglM6breyn2cqZRcAnFSpAZSBl/ZkPoWpgwa6qXT8Mp6KFlBjRS2p4E6ZT9PpGz2JhxHFrlxKP53ZuQ//W6GUbnvVwkaYaQ8NmiKJMUFZ0kTQdCA0c5toBxLaxXykdMM472P35sCePCZuL/TuAvaB83/ZPm8f1p7fKqTKdC9sgBqROfnJFLckPuSItw8kY+nYqz7Hy4O+6uuz+Tuk45s0N+lHv4BUTBu4Q=</latexit>

Low-rank regularizer<latexit sha1_base64="mVjbfruuo+ozZX+Xb1xpVdOrsbI=">AAACIHicbVDLSsNAFJ34rPUVdelmsAh1U5Iq6LLoRldWsQ9oSphMJ+2QyYOZG6GE/IIf4Te41bU7cak7/8TpY2FbD1w4nHMv997jJYIrsKwvY2l5ZXVtvbBR3Nza3tk19/abKk4lZQ0ai1i2PaKY4BFrAAfB2olkJPQEa3nB1chvPTKpeBw9wDBh3ZD0I+5zSkBLrll2QgIDSkR2mzuC+VB2Ehn33CzI8Y0b3LuBI3l/ACeuWbIq1hh4kdhTUkJT1F3zx+nFNA1ZBFQQpTq2lUA3IxI4FSwvOqliCaEB6bOOphEJmepm449yfKyVHvZjqSsCPFb/TmQkVGoYerpzdL+a90bif14nBf+im/EoSYFFdLLITwWGGI/iwT0uGQUx1IRQyfWtmA6IJBR0iDNbvDDXmdjzCSySZrVin1aqd2el2uU0nQI6REeojGx0jmroGtVRA1H0hF7QK3ozno1348P4nLQuGdOZAzQD4/sXdw2kRg==</latexit>⌘T

vec(X )

O
<latexit sha1_base64="+lQbaUymmn0AU8umG0MPb4iXHrE=">AAACH3icbVDLSsNAFJ3Ud31VXboZLGLdlKQKuhTd6Moq9gFNCZPppB0yk4SZG6GEfIIf4Te41bU7cdulf+L0sdDqgQuHc+7l3nv8RHANtj2yCguLS8srq2vF9Y3Nre3Szm5Tx6mirEFjEau2TzQTPGIN4CBYO1GMSF+wlh9ejf3WI1Oax9EDDBPWlaQf8YBTAkbySkeuJDCgRGS3uStYABVXp9LLwhzfeOG9F7qK9wdw7JXKdtWeAP8lzoyU0Qx1r/Tl9mKaShYBFUTrjmMn0M2IAk4Fy4tuqllCaEj6rGNoRCTT3WzyUI4PjdLDQaxMRYAn6s+JjEith9I3nePz9bw3Fv/zOikE592MR0kKLKLTRUEqMMR4nA7uccUoiKEhhCpubsV0QBShYDL8tcWXucnEmU/gL2nWqs5JtXZ3Wr64nKWzivbRAaogB52hC3SN6qiBKHpCL+gVvVnP1rv1YX1OWwvWbGYP/YI1+garcKPc</latexit>!

Y

I k RkX

I k RkkkParameter! reduction

TCLs require fewer parameters and less computation, while‣ preserving
the multilinear structure of the activation tensor
57Speed-up and Compression of CNNs

‣ 4-order kernel tensor is represented by CPD
ImageNet (AlexNet)
4x speedup
1% accuracy drop

Kernel Size

Number of Parameters& Computations

Input channels(d ⇥ d ⇥ S ⇥ T )<latexit sha1_base64="q1o+/lA8Q/OfYC7HssYWrMfqTec=">AAACE3icbVDLSgMxFM3UV62vUZfdBItQN2WmCrosunFZsS9oh5LJZNrQJDMkGaEMXfgRfoNbXbsTt36AS//EtB3Bth4I93DuvZyb48eMKu04X1ZubX1jcyu/XdjZ3ds/sA+PWipKJCZNHLFIdnykCKOCNDXVjHRiSRD3GWn7o5tpv/1ApKKRaOhxTDyOBoKGFCNtpL5dLAc9TTlR8LfeZ7Vx1rdLTsWZAa4SNyMlkKHet797QYQTToTGDCnVdZ1YeymSmmJGJoVeokiM8AgNSNdQgYyPl84+MYGnRglgGEnzhIYz9e9GirhSY+6bSY70UC33puJ/vW6iwysvpSJONBF4bhQmDOoIThOBAZUEazY2BGFJza0QD5FEWJvcFlx8PjGZuMsJrJJWteKeV6p3F6XadZZOHhTBCSgDF1yCGrgFddAEGDyCZ/ACXq0n6816tz7mozkr2zkGC7A+fwCsY53t</latexit>

Spatial

ST d2

Output channels<latexit sha1_base64="d0GvZE/m9JhpzljatTq4GGNh8Zs=">AAAB+3icbVBNT8JAEJ3iF+IX6tHLRmLiibRookeiF48YKZJAJdvtFjZst83u1oQ0/Q1e9ezNePXHePSfuEAPAr5kkpf3ZjIzz084U9q2v63S2vrG5lZ5u7Kzu7d/UD086qg4lYS6JOax7PpYUc4EdTXTnHYTSXHkc/roj2+n/uMzlYrFoq0nCfUiPBQsZARrI7kP7eCpMajW7Lo9A1olTkFqUKA1qP70g5ikERWacKxUz7ET7WVYakY4zSv9VNEEkzEe0p6hAkdUedns2BydGSVAYSxNCY1m6t+JDEdKTSLfdEZYj9SyNxX/83qpDq+9jIkk1VSQ+aIw5UjHaPo5CpikRPOJIZhIZm5FZIQlJtrks7DFj3KTibOcwCrpNOrORb1xf1lr3hTplOEETuEcHLiCJtxBC1wgwOAFXuHNyq1368P6nLeWrGLmGBZgff0C7hSU4Q==</latexit>

R(S + 2d + T )<latexit sha1_base64="Tkr2eVZRT1k4WRgbSVMcYji5fS4=">AAAB/3icbVA9TwJBEJ3DL8Qv1NLmIjHBkJA7NNGSaGOJCoKBC9nbW2DD7t5ld8+EXCj8DbZa2xlbf4ql/8QFrhDwJZO8vDeTmXl+xKjSjvNtZVZW19Y3spu5re2d3b38/sGDCmOJSQOHLJQtHynCqCANTTUjrUgSxH1Gmv7weuI3n4hUNBR1PYqIx1Ff0B7FSBvp8a54X6oEpfppN19wys4U9jJxU1KAFLVu/qcThDjmRGjMkFJt14m0lyCpKWZknOvEikQID1GftA0ViBPlJdODx/aJUQK7F0pTQttT9e9EgrhSI+6bTo70QC16E/E/rx3r3qWXUBHFmgg8W9SLma1De/K9HVBJsGYjQxCW1Nxq4wGSCGuT0dwWn49NJu5iAsvkoVJ2z8qV2/NC9SpNJwtHcAxFcOECqnADNWgABg4v8Apv1rP1bn1Yn7PWjJXOHMIcrK9fe8yVpA==</latexit>

Speeding-up convolutional neural networks using fine-tuned cp-decomposition (Lebedev et al. ICLR 2015)
58

CNN with Low-rank Regularization‣ Standard: Pre-trained network + Tensor Approximation with fine-tuning‣ Re-parametrize kernel naturally enforces low-rank constraint‣ Convolutional kernel in CNN is 4D N:tensor
Number of output,
C: number of input feature maps

‣ Train low-rank CNNs from scratch
H 2 RN ⇥1⇥d⇥K<latexit sha1_base64="u9utEueRm5dCrLVrmnxOrI6lxAo=">AAACL3icbVDLSgMxFM3UV62vqks3wSK4KjNVqMuim4IgVewDOrUkmbQNzWSGJCOUoZ/iR/gNbnUtbkSX/oWZ6Sxs64GQk3Pv5dwcHHKmtG1/WLmV1bX1jfxmYWt7Z3evuH/QUkEkCW2SgAeyg5GinAna1Exz2gklRT7mtI3HV0m9/UilYoG415OQ9nw0FGzACNJG6herro/0iCAO6xC6TMD0jXF8N32Ib1zNfKqgk90ezMj1tF8s2WU7BVwmTkZKIEOjX/x2vYBEPhWacKRU17FD3YuR1IxwOi24kaIhImM0pF1DBTI+vTj94BSeGMWDg0CaIzRM1b8TMfKVmvjYdCbrq8VaIv5X60Z6cNGLmQgjTQWZGQ0iDnUAk7SgxyQlmk8MQUQysyskIyQR0SbTORfsJ5k4iwksk1al7JyVK7fnpdpllk4eHIFjcAocUAU1UAcN0AQEPIEX8ArerGfr3fq0vmatOSubOQRzsH5+AVXhqS4=</latexit>

V 2 RK⇥d⇥1⇥C<latexit sha1_base64="esW73EZomT9ztIyGfXl9cqV5Ejw=">AAACL3icbVC7TsMwFHV4lvIKMLJYVEhMVVKQyljRBYmlIPqQmlA5jtNadZzIdpCqKJ/CR/ANrDAjFgQjf4HTZqAtR7J8fO69OtfHixmVyrI+jJXVtfWNzdJWeXtnd2/fPDjsyCgRmLRxxCLR85AkjHLSVlQx0osFQaHHSNcbN/N695EISSN+ryYxcUM05DSgGCktDcy6EyI1wojBDoQO5XD69rz0LntIbxxFQyKhX9w2LEgzG5gVq2pNAZeJXZAKKNAamN+OH+EkJFxhhqTs21as3BQJRTEjWdlJJIkRHqMh6WvKkfZx0+kHM3iqFR8GkdCHKzhV/06kKJRyEnq6M19fLtZy8b9aP1HBpZtSHieKcDwzChIGVQTztKBPBcGKTTRBWFC9K8QjJBBWOtM5Fy/MM7EXE1gmnVrVPq/Wbi8qjasinRI4BifgDNigDhrgGrRAG2DwBF7AK3gzno1349P4mrWuGMXMEZiD8fMLXlKpMQ==</latexit>‣ Lower expressive power due to low-rank constraints
speedup, sometimes even improved accuracy (91.31% on‣ Significant
CIFAR-10)
Convolutional neural networks with low-rank regularization (Tai et al., ICLR 2016)
59

CNN Compression by Tucker
Convolution Layer

Size of Kernel K

Input channels<latexit sha1_base64="CJYO4HwvDT9s1sMFJtv9mJhgyFk=">AAACAHicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoRnBTwT6wHUomzbShSWZIMkIZuvEb3Oranbj1T1z6J2baWdjWA4HDOfdyT04Qc6aN6347K6tr6xubha3i9s7u3n7p4LCpo0QR2iARj1Q7wJpyJmnDMMNpO1YUi4DTVjC6yfzWE1WaRfLBjGPqCzyQLGQEGys9dgU2Q4I5uuuVym7FnQItEy8nZchR75V+uv2IJIJKQzjWuuO5sfFTrAwjnE6K3UTTGJMRHtCOpRILqv10mniCTq3SR2Gk7JMGTdW/GykWWo9FYCezhHrRy8T/vE5iwis/ZTJODJVkdihMODIRyr6P+kxRYvjYEkwUs1kRGWKFibElzV0JxMR24i02sEya1Yp3XqneX5Rr13k7BTiGEzgDDy6hBrdQhwYQkPACr/DmPDvvzofzORtdcfKdI5iD8/UL5QGXGQ==</latexit>(H ⇥ W ⇥ S)
<latexit sha1_base64="8n3gCJXix0a7hPmMVe7oS7/K+nc=">AAACC3icbVDLTgIxFO3gC/E1ytJNIzHBDZlBE10S3bDEKI8EJqRTOtDQ6UzaOyZkwif4DW517c649SNc+icWmIWAJ2l6cs69OTfHjwXX4DjfVm5jc2t7J79b2Ns/ODyyj09aOkoUZU0aiUh1fKKZ4JI1gYNgnVgxEvqCtf3x3cxvPzGleSQfYRIzLyRDyQNOCRipbxfL9R7wkGnczv6Hi75dcirOHHiduBkpoQyNvv3TG0Q0CZkEKojWXdeJwUuJAk4FmxZ6iWYxoWMyZF1DJTE5Xjo/forPjTLAQaTMk4Dn6t+NlIRaT0LfTIYERnrVm4n/ed0Eghsv5TJOgEm6CAoSgSHCsybwgCtGQUwMIVRxcyumI6IIBdPXUoofTk0n7moD66RVrbiXler9Val2m7WTR6foDJWRi65RDdVRAzURRRP0gl7Rm/VsvVsf1udiNGdlO0W0BOvrF1ozmoI=</latexit>

Yt = X ⇤ K t<latexit sha1_base64="zQWWMv0LWSoyvheiUHYxraNpp7g=">AAACIHicbVDLSgMxFM34rPU16tJNsAjFRZmpgm6EohvBTQX7kHYYMmmmDc08SO4IZegv+BF+g1tduxOXuvNPTNtBbeuBwLnn3Mu9OV4suALL+jAWFpeWV1Zza/n1jc2tbXNnt66iRFJWo5GIZNMjigkeshpwEKwZS0YCT7CG178c+Y17JhWPwlsYxMwJSDfkPqcEtOSaxXZAoEeJwHcu4HP8Uzbx0W9x7YJrFqySNQaeJ3ZGCihD1TW/2p2IJgELgQqiVMu2YnBSIoFTwYb5dqJYTGifdFlL05AETDnp+EdDfKiVDvYjqV8IeKz+nUhJoNQg8HTn6EY1643E/7xWAv6Zk/IwToCFdLLITwSGCI/iwR0uGQUx0IRQyfWtmPaIJBR0iFNbvGCoM7FnE5gn9XLJPi6Vb04KlYssnRzaRweoiGx0iiroClVRDVH0gJ7QM3oxHo1X4814n7QuGNnMHpqC8fkNUnKiYA==</latexit>(H ⇥ W ⇥ T )<latexit sha1_base64="fdcqHwxDOXfrOJvUSM6EyQd4Iek=">AAACC3icbVDLTgIxFL2DL8TXKEs3jcQEN2QGTXRJdMMSE14JTEindKCh05m0HZMJ4RP8Bre6dmfc+hEu/RMLzELAkzQ9OefenJvjx5wp7TjfVm5re2d3L79fODg8Oj6xT8/aKkokoS0S8Uh2fawoZ4K2NNOcdmNJcehz2vEnD3O/80SlYpFo6jSmXohHggWMYG2kgV0s1/uahVShTvY3rwZ2yak4C6BN4makBBkaA/unP4xIElKhCcdK9Vwn1t4US80Ip7NCP1E0xmSCR7RnqMAmx5sujp+hS6MMURBJ84RGC/XvxhSHSqWhbyZDrMdq3ZuL/3m9RAd33pSJONFUkGVQkHCkIzRvAg2ZpETz1BBMJDO3IjLGEhNt+lpJ8cOZ6cRdb2CTtKsV97pSfbwp1e6zdvJwDhdQBhduoQZ1aEALCKTwAq/wZj1b79aH9bkczVnZThFWYH39AlvHmoM=</latexit>(d ⇥ d ⇥ S ⇥ T )t 2 [1, T ]<latexit sha1_base64="q1o+/lA8Q/OfYC7HssYWrMfqTec=">AAACE3icbVDLSgMxFM3UV62vUZfdBItQN2WmCrosunFZsS9oh5LJZNrQJDMkGaEMXfgRfoNbXbsTt36AS//EtB3Bth4I93DuvZyb48eMKu04X1ZubX1jcyu/XdjZ3ds/sA+PWipKJCZNHLFIdnykCKOCNDXVjHRiSRD3GWn7o5tpv/1ApKKRaOhxTDyOBoKGFCNtpL5dLAc9TTlR8LfeZ7Vx1rdLTsWZAa4SNyMlkKHet797QYQTToTGDCnVdZ1YeymSmmJGJoVeokiM8AgNSNdQgYyPl84+MYGnRglgGEnzhIYz9e9GirhSY+6bSY70UC33puJ/vW6iwysvpSJONBF4bhQmDOoIThOBAZUEazY2BGFJza0QD5FEWJvcFlx8PjGZuMsJrJJWteKeV6p3F6XadZZOHhTBCSgDF1yCGrgFddAEGDyCZ/ACXq0n6816tz7mozkr2zkGC7A+fwCsY53t</latexit><latexit sha1_base64="du8lp/Blnad9de9SCx/avWsaDwk=">AAAB/3icbVA9SwNBEJ2LXzF+RS1tFoNgIeEuCloGbSwj5EsuR9jb7CVLdveO3T0hhBT+Blut7cTWn2LpP3GTXGESHww83pthZl6YcKaN6347ubX1jc2t/HZhZ3dv/6B4eNTUcaoIbZCYx6odYk05k7RhmOG0nSiKRchpKxzeTf3WE1WaxbJuRgkNBO5LFjGCjZUeTYdJ37uoB91iyS27M6BV4mWkBBlq3eJPpxeTVFBpCMda+56bmGCMlWGE00mhk2qaYDLEfepbKrGgOhjPDp6gM6v0UBQrW9Kgmfp3YoyF1iMR2k6BzUAve1PxP89PTXQTjJlMUkMlmS+KUo5MjKbfox5TlBg+sgQTxeytiAywwsTYjBa2hGJiM/GWE1glzUrZuyxXHq5K1dssnTycwCmcgwfXUIV7qEEDCAh4gVd4c56dd+fD+Zy35pxs5hgW4Hz9Atdlln4=</latexit>

Spatial(d ⇥ d ⇥ S)

Output channels<latexit sha1_base64="KFEUSdli744KSmMjudwUbLERHGw=">AAACC3icbVDLSgMxFM3UV62v0S7dBItQN2WmCrosunFZ0T6gHUomk2lDk8yQZIRh6Cf4DW517U7c+hEu/RPTdgTbeiDkcM69nMvxY0aVdpwvq7C2vrG5Vdwu7ezu7R/Yh0dtFSUSkxaOWCS7PlKEUUFammpGurEkiPuMdPzxzdTvPBKpaCQedBoTj6OhoCHFSBtpYJerQV9TThT8/e/PBnbFqTkzwFXi5qQCcjQH9nc/iHDCidCYIaV6rhNrL0NSU8zIpNRPFIkRHqMh6RkqkMnxstnxE3hqlACGkTRPaDhT/25kiCuVct9McqRHatmbiv95vUSHV15GRZxoIvA8KEwY1BGcNgEDKgnWLDUEYUnNrRCPkERYm74WUnw+MZ24yw2skna95p7X6ncXlcZ13k4RHIMTUAUuuAQNcAuaoAUwSMEzeAGv1pP1Zr1bH/PRgpXvlMECrM8fnQ+aqw==</latexit>

Ranks are‣ Tensor
hyper-parameters

Tucker 2 decomposition

K = G ⇥ 3 VS ⇥ 4 VT

trade-off‣ Control
between kernel size<latexit sha1_base64="eK2WxzRstfPNAu/bBMI0z4pyPoA=">AAACOHicbZDLSsNAFIYnXmu9RV26GSyCq5K0Bd0IRRcKbir2Bk0Jk+mkHTqThJmJUEJfx4fwGdwquNOVuPUJnLRBevHAwH++cw7nzO9FjEplWe/Gyura+sZmbiu/vbO7t28eHDZlGAtMGjhkoWh7SBJGA9JQVDHSjgRB3GOk5Q2v03rrkQhJw6CuRhHpctQPqE8xUhq5ZtXhSA0wYvAOXsK/5AY6inIi3fKUeT5sug8Zq8ywumsWrKI1Cbgs7EwUQBY11/x0eiGOOQkUZkjKjm1FqpsgoShmZJx3YkkihIeoTzpaBkiv7CaTn47hqSY96IdCv0DBCZ2dSBCXcsQ93ZmeKBdrKfyv1omVf9FNaBDFigR4usiPGVQhTG2DPSoIVmykBcKC6lshHiCBsNLmzm3x+Fh7Yi86sCyapaJdLpbuK4XqVeZODhyDE3AGbHAOquAW1EADYPAEXsAreDOejQ/jy/ietq4Y2cwRmAvj5xcRhKvu</latexit>

and accuracy losssize
‣ Kernelcorresponds to speed
and model size

3.68x speedup for VGG

Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (Kim et al., ICLR 2016)60

TN Decompositions for CNN Kernels‣ Interpret algorithms as different TN decompositions‣ Explore unexplored TN decompositions
Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks (Hayashi et al., NeurIPS 2019)61

Is Weight Kernel Low-rank?
Low-rank regularizer
W = TD(G1 , . . . , GN )<latexit sha1_base64="vT2p8d22NelW/zJqAdd30N8BsaI=">AAACNXicbVDJSgNBEO2JW4xb1KOXxiBEkDATBb0Eggp6kgjZIBNCT6eTNOlZ6K4RwzBf40f4DV715MGDIF79BTuLkMUHTT/eq6KqnhMIrsA0343E0vLK6lpyPbWxubW9k97dqyo/lJRVqC98WXeIYoJ7rAIcBKsHkhHXEazm9K+Gfu2BScV9rwyDgDVd0vV4h1MCWmqlC7ZLoEeJwDVcwDawR4jK1zHO/unRTdyyTrAt2j4o/U/Jd8etdMbMmSPgRWJNSAZNUGqlP+22T0OXeUAFUaphmQE0IyKBU8HilB0qFhDaJ13W0NQjLlPNaHRmjI+00sYdX+rnAR6p0x0RcZUauI6uHG6p5r2h+J/XCKFz0Yy4F4TAPDoe1AkFBh8PM8NtLhkFMdCEUMn1rpj2iCQUdLIzUxw31plY8wkskmo+Z53m8vdnmeLlJJ0kOkCHKIssdI6K6BaVUAVR9IRe0Ct6M56ND+PL+B6XJoxJzz6agfHzC3kFqzQ=</latexit>

Random shuffling (Rs) low-rank
W = R(TD(G1 , . . . , GN ))CIFAR-10 Noisy Data<latexit sha1_base64="vjC57x1el9O1ghrqXthSbBiKpEM=">AAACOXicbVDLSsQwFE3H93vUpZvgICjI0KqgG8EX6EpUnAdMS0nTzBhMm5LcikPp9/gRfoNbxaU7cesPmI5d6OiBkMM593LvPUEiuAbbfrUqI6Nj4xOTU9Mzs3PzC9XFpaaWqaKsQaWQqh0QzQSPWQM4CNZOFCNRIFgruD0u/NYdU5rL+Br6CfMi0ot5l1MCRvKrh25E4IYSgVt4H1/hdRfYPWTXJ7mhpZWd5r6ziV0RStDm/yGfb2z41ZpdtwfAf4lTkhoqceFX39xQ0jRiMVBBtO44dgJeRhRwKlg+7aaaJYTekh7rGBqTiGkvG5ya4zWjhLgrlXkx4IH6syMjkdb9KDCVxZp62CvE/7xOCt09L+NxkgKL6fegbiowSFzkhkOuGAXRN4RQxc2umN4QRSiYdH9NCaLcZOIMJ/CXNLfqznZ963KndnBUpjOJVtAqWkcO2kUH6AxdoAai6AE9oWf0Yj1ab9a79fFdWrHKnmX0C9bnF4KZrB8=</latexit>

performs better in‣ RsTDhigher compression rate

‣ More robust to noisy data
kernels have inherent‣ CNN
general low-rank structure
Low-rank embedding of kernels in convolutional neural networks under random shuffling (Li et al., ICASSP 2019)62

Generalization of Compressed CNN
matrices/kernels‣ Weight
of well-trained models are
not necessarily low-rank
weight‣ Re-parametrizes
tensors as CPD
layer reduces the‣ CPgeneralization error

K=<latexit sha1_base64="Cii5IEYHYKkxMQuB23l4P2+9thM=">AAACWHicbVDLSsQwFL1T3+9Rl26Cg6CboVVBN4LoRhBExVFhOtY0TTWYNCW5FYbS3/MfdOfera7NjIM46oHAybnnPjhxLoVF33+ueSOjY+MTk1PTM7Nz8wv1xaVLqwvDeItpqc11TC2XIuMtFCj5dW44VbHkV/HDYa9+9ciNFTq7wG7OO4reZSIVjKKTovptqCjeMyrJMdkjoS1UVJq9oLo5J6F0YxIaGdL3xGn5WEXmplwPNioSahSKWxKyRKP9/g0bT5wxqjf8pt8H+UuCAWnAAKdR/TVMNCsUz5BJam078HPslNSgYJJX02FheU7ZA73jbUcz6hZ3yn4SFVlzSkJSbdzLkPTVnx0lVdZ2VeycvVPt71pP/K/WLjDd7ZQiywvkGftalBaSoCa9WEkiDGcou45QZoS7lbB7aihDF/7QllhVLpPgdwJ/yeVmM9hqbp5tN/YPBulMwgqswjoEsAP7cASn0AIGT/AG7/BRe/HAm/CmvqxebdCzDEPwlj4BtJq2Rw==</latexit>

R
X
r=1

Generalization
Error

‣

(1)
(N )
v
⌦
·
·
·
⌦
v
r r
r

Empirical
Loss
CP Rank

Higher compression ->smaller generalization
error bound

Understanding Generalization in Deep Learning via Tensor Methods (Li et al., AISTATS 2020)63

Tensor Networks for Theoretical Study of DNNs
Theoretical understanding of DNNs is limited
How to analyze expressive power of DNNs?
How is depth efficiency of DNNs?
Generalization error bound
…

‣
‣
‣
‣

Tensor Networks can be used as a tool for theoretical analysis of
DNNs
Relation and equivalence between TNs and DNNs
Expressive power of network can be measured by rank of tensor
networks
Expressive analysis of DNNs by analyzing TN ranks
Provide valuable insights on how they can be improved

‣
‣
‣
‣

64

Equivalence between CNNs and TNs‣ Shallow ConvNet corresponds to CP decomposition
Score function:‣ Deep CNN network corresponds to hierarchical Tucker decomposition

hierarchical Tucker
Convolutional Rectifier Networks as Generalized Tensor Decompositions (Cohen et al., ICML 2016)65

Deep Learning and Quantum Entanglement(linear activations + product pooling layers introducing‣ ConvAC
non-linearity) vs. CNN (ReLU + average/max pooling)‣ Tensor Network rank as capacity of model‣ Experiment on “inductive bias” of model architecture

Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to
Network Design (Levine et al., ICLR 2018)
66

Theoretical Understanding of CNN
CNNs are exponentially more expressive than their‣ Deepshallow counterpart
efficiency: deep network of polynomial size = shallow‣ Depthnetwork of exponential size
rectifier networks are universal with max‣ Convolutional
pooling, but not with average pooling(linear activation + product pooling) have an‣ SimNets
advantage over the prevalent convolutional rectifier networks

On the expressive power of deep learning: A tensor analysis (Cohen et al., COLT 2016)
67

Relations between TNs and DNNs

‣ Representing an image as sequential data
‣ Representation mapping‣ Feature tensor
Local feature map
d-order···=<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

Expressive Power of Recurrent Neural Networks (Khrulkov et al., ICLR 2018)68

Multilinear Unit in Tensorial Networks

Expressive Power of Recurrent Neural Networks (Khrulkov et al., ICLR 2018)
69

Relations between TNs and DNNs

TT-Network
Score function:

Expressive Power of Recurrent Neural Networks (Khrulkov et al., ICLR 2018)
70

Theoretical Results

Given a network of width r shown in a column, what is the upperbound on the width of equivalent networks shown in rows

‣ TT is exponentially more expressive than CP
shallow network of exponentially large width is required to‣ Amimica recurrent neural network
Expressive Power of Recurrent Neural Networks (Khrulkov et al., ICLR 2018)71

Multimodal Learning
Language

Sentiment Intensity“This movie is sick”?

VisualSpeak’s behaviours
Speaker’s behaviors① Intra-modal Interactions② Cross-modal Interactions③ Computational Efficiency

Sentimental intensity

Smile

Acoustic

Unimodal

BimodalTrimodalsentimental classification (Acoustic, Visual, Language)‣ Multimodal
Multimodal
Sentiment and Emotion Analysis

Loud

Ti

time
Multimodal
representation
Multimodal Representation(Multimodal Fusion)Multimodal fusion‣ Visual question answering (Image + Language)72

VQA: Visual Question Answering
an image and a natural language question about the image,‣ Giventhe task is to provide an accurate natural language answer.

Joint
Representation
VQA: Visual Question Answering (Agrawal et al., ICCV 2015)73

Types of Multimodel Fusion
Data-level fusion

Decision-level fusion

Shared Representation

Feature-level fusion

Acoustic

Acoustic

Acoustic

Acoustic

VisualConcatenation

VisualVisualElement-wise multiplication⇤

Element-wise add+<latexit sha1_base64="XmnsbflCSUflFiOq39oz+CqINvA=">AAAB93icbVBNSwMxEJ2tX7V+VT16CRZBEMpuFfRY9OKxBfsB7VKyabYNTbJLkhWWpb/Aq569iVd/jkf/iWm7B9v6YODx3gwz84KYM21c99spbGxube8Ud0t7+weHR+Xjk7aOEkVoi0Q8Ut0Aa8qZpC3DDKfdWFEsAk47weRh5neeqdIskk8mjakv8EiykBFsrNS8GpQrbtWdA60TLycVyNEYlH/6w4gkgkpDONa657mx8TOsDCOcTkv9RNMYkwke0Z6lEguq/Wx+6BRdWGWIwkjZkgbN1b8TGRZapyKwnQKbsV71ZuJ/Xi8x4Z2fMRknhkqyWBQmHJkIzb5GQ6YoMTy1BBPF7K2IjLHCxNhslrYEYmoz8VYTWCftWtW7rtaaN5X6fZ5OEc7gHC7Bg1uowyM0oAUEKLzAK7w5qfPufDifi9aCk8+cwhKcr18TYJNJ</latexit><latexit sha1_base64="UGzvece26doSutsuA0nqqa/5wsM=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoMgFuEuCloGbSwTMB+QHGFvs5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYrQFol4pLoB1pQzSVuGGU67saJYBJx2gsnDzO88U6VZJJ9MGlNf4JFkISPYWKl5NShX3Ko7B1onXk4qkKMxKP/0hxFJBJWGcKx1z3Nj42dYGUY4nZb6iaYxJhM8oj1LJRZU+9n80Cm6sMoQhZGyJQ2aq38nMiy0TkVgOwU2Y73qzcT/vF5iwjs/YzJODJVksShMODIRmn2NhkxRYnhqCSaK2VsRGWOFibHZLG0JxNRm4q0msE7atap3Xa01byr1+zydIpzBOVyCB7dQh0doQAsIUHiBV3hzUufd+XA+F60FJ585hSU4X78RzZNI</latexit>

74

VisualTensor Fusion Network for Multimodal Learning‣ Trilinear fusion: linear, bilinear and trilinear interactions‣ Capture cross-modality nonlinear interaction informationhaha ⌦ hv
Acousticha ⌦ hlhv

hl

Languagehv ⌦ hlVisualha ⌦ hv ⌦ hl

‣ Exponential increase of dimensionality and complexity
Tensor Fusion Network for Multimodal Sentiment Analysis (Zadeh et al., EMNLP 2017)75

Low-rank Multimodal Fusion‣ Efficient computation based low-rank representation
Low-rank Representationdmdy<latexit sha1_base64="t2LYD8xgiRioV6JSXZugtMhzBsQ=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjWVE8wHJEfb29pIlu3vH7p4QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXpBwpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZxqghtkZjHqhtgTTmTtGWY4bSbKIpFwGknGN/N/M4zVZrF8slMEuoLPJQsYgQbKz2GAzEoV9yqOwdaJ15OKpCjOSj/9MOYpIJKQzjWuue5ifEzrAwjnE5L/VTTBJMxHtKepRILqv1sfuoUXVglRFGsbEmD5urfiQwLrScisJ0Cm5Fe9Wbif14vNdGNnzGZpIZKslgUpRyZGM3+RiFTlBg+sQQTxeytiIywwsTYdJa2BGJqM/FWE1gn7VrVu6rWHuqVxm2eThHO4BwuwYNraMA9NKEFBIbwAq/w5mTOu/PhfC5aC04+cwpLcL5+AfldlGI=</latexit><latexit sha1_base64="no9bFJyhz/cGY4Im+yrNq3CM538=">AAAB+XicbVBNS8NAEJ34WetX1aOXxSJ4KkkV9Fj04rGi/YA2lM1m0y7d3YTdjRBCf4JXPXsTr/4aj/4Tt20OtvXBwOO9GWbmBQln2rjut7O2vrG5tV3aKe/u7R8cVo6O2zpOFaEtEvNYdQOsKWeStgwznHYTRbEIOO0E47up33mmSrNYPpksob7AQ8kiRrCx0mM4yAaVqltzZ0CrxCtIFQo0B5WffhiTVFBpCMda9zw3MX6OlWGE00m5n2qaYDLGQ9qzVGJBtZ/PTp2gc6uEKIqVLWnQTP07kWOhdSYC2ymwGellbyr+5/VSE934OZNJaqgk80VRypGJ0fRvFDJFieGZJZgoZm9FZIQVJsams7AlEBObibecwCpp12veZa3+cFVt3BbplOAUzuACPLiGBtxDE1pAYAgv8ApvTu68Ox/O57x1zSlmTmABztcvDFCUbg==</latexit>

Computation complexity

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors (Liu et al., ACL 2018)76

Multimodal Tucker Fusion
huge dimensionality issue‣ Bilinear fusion suffer from
2048
1010

2000

2048

‣

Tensor decomposition of model parameter T<latexit sha1_base64="OmxF5vpQ0OMfp3brjJ/Rloi4QXY=">AAACAHicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmWFvrAdSibNtKFJZkgyQhm68Rvc6tqduPVPXPonZtpZ2NYDgcM593JPThBzpo3rfjtr6xubW9uFneLu3v7BYenouKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7L/PYTVZpFsmEmMfUFHkoWMoKNlR57ApsRwRw1+qWyW3FnQKvEy0kZctT7pZ/eICKJoNIQjrXuem5s/BQrwwin02Iv0TTGZIyHtGupxIJqP50lnqJzqwxQGCn7pEEz9e9GioXWExHYySyhXvYy8T+vm5jwxk+ZjBNDJZkfChOOTISy76MBU5QYPrEEE8VsVkRGWGFibEkLVwIxtZ14yw2skla14l1Wqg9X5dpt3k4BTuEMLsCDa6jBPdShCQQkvMArvDnPzrvz4XzOR9ecfOcEFuB8/QLzLJci</latexit>

TuckerDecomposition:
Tensor Rank is key to balance expressivity and complexity
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (Ben-younes et al., ICCV 2017)77

Single-block Attention by Tucker Decomposition

Tensorized Transformer

e building the Single-block attention, it is necessary to propose the theorem 3.1. The theorem isly related to attributes of Single-block attention function by Tucker decomposition [35].
rem 3.1. Let e1 , . . . , en be basis vectors from the vector space S. Assume that these vectors. , en are linear independent and Q,K,V can be linearly represented by this set of basis vectors.
BTD model
Single/multi-headattention
utput ofThethe attentionfunction in Eq. 2 can be represented
by a linear combinationof the set ofbasis vectors.
ܶଵܰ‫ݐ݈݅݌ݏ‬ܿ‫ݐܽܿ݊݋‬Attention(Q, K, V ) = (e1 , . .ܰ. , eܰn )M,
(4)
ܰ
ࣲ
݀
ܶଶ
ࣲ
1
n⇥d
‫ݐݑ݌ݐݑ݋‬eM 2R
is a coefficientmatrix, andof thesematrices ܰ(i.e., Q,‫)݄( כ‬K, and V ).
ȉ
ȉ d is a dimension
ܰ

‣

‣

ܴଷ

݀ଶ

ൎ

݀ଷ

(ଵ)
݀ଵ ࣲଵ

ȉ

ȉ

ܴଵ

(ଶ)
ࣲଵ
2

1

࣡ଵ

݀ଶ

ܴଶ

+‫ ڮ‬+

(ଵ)
݀ଵ ࣲ௉

(ଷ)
௉

‫ڮ‬

ܴଵ

3

݀ଷ

3

ȉ࣡ ȉ

2

1

(ଶ)

ࣲ௉

ܶ௡

ܴଶ

݄

݀ଶ

. The proof
can be found in Supplementary Materials B. ܸ
ࣛ
௉

݀
ܴ

+

+ ‫ ڮ‬+

݃ଶ

‫ڮ‬

ܰ
݃ଵ

ܹை

‫ݐݑ݌ݐݑ݋‬

݀ଷ

ଵ

ܴଷ

(ଷ)
ଵ

݃௛

he representation of Block-Term tensor decomposition for a 3-order tensor.݀ A 2
݀gure2(left),
itisaschematic
diagramabout
theSingle-blockattention.
First, we assume that theparameters
a 3-order tensor, and can be approximated by P Tucker decomposition. P is the CP ܴ
sharingܴܰ ܴ=R =R
,Rkey
mappedinto
threefactorofthreegroupsܰ
are thevalue
Tuckercan
rank, berespectively.
In thispaper,
we assume
thatmatrices
R=R
2 , R3and1
2 of3 .which are composed
LinearLinearLinearSum of Tucker decompositionhogonal basis vectors. Three factor matrices are Q, K and V . After that, wecan‫ ܭ‬construct
ܳ
ܸ
‫ܭ‬
ܳ
࣡
Block-Term(i.e.,
TensorSingle-block
Decomposition attention) by initializing a 3-order diagonal tensor (trainable)
wandattentionhseisthethe
In letter
Figure2 (left),
R whichis thecanrank
about
tensor, N is the length of a sequence, andEulerG.
scriptA to denotea tensorbe thoughtof asthea multi-array.
Figure 2: (left) Single-block attention using Tucker decomposition. (right) Multi-linear atten
ctor
a matrix are aof
1-order
tensor and2-order
tensor, respectively.
Theonelement
inattention
a tensor decomposition.
he anddimension
matrix.
The
functionofSingle-block
can becomputed
based onbased
Block-Term
Interaction
among(Q)uery,(K)eyand(V)ariable
for
each
head
r is denoted as Ad ,...,d . In the geometric representation of a tensor, 3-order tensor can
er-decomposition
as isfollows:
d by a cube. After that, therea related concept named tensor slice that will be used
In orderto compress
Tensor and some other related concepts are showed in Supplementary
Materials
A. the multi-head mechanism, we propose a multi-linear attention constructed
AttenT D (G; Q, K, V ) =G•1 Q•
V decomposition. This attention uses the idea of parameters sharing, i.e., sha
Block-Term
2 K•3tensor

‣

1

n

Tensor Decomposition (BTD) Block-Term tensor decomposition is
a combinationof multiple blocks, shown in Figure 2 (right). After that, the compression ra
factormatrices acrossd⇥...⇥d1n
I A
J2relatively
M lowersition [6] and Tucker decomposition [35]. Given a n-order tensor
RX. Acomplexity have been analyzed.
and
X
X
(5)
nsor can be decomposed into P block terms by the method named BTD. •z is denoted as=
Gijm
Qi byKTucker
j VDecompositionmor product on the z-th order [22] and z 2 {1, . . . , d}. Each term contains•z betweena
3.1 Single-block
Attention(k)
R1 ⇥...⇥Rd
m=1
j=1
and d factor matrices Xi 2 Rdk ⇥Rk , wherei=1
i 2 [1,
P ] and
k 2 [1, d].
i 2R
Before building the Single-block attention, it is necessary to propose the theorem 3.1. The theore
ion of BTD decomposition is as follows:
relatedcore
to attributesof Single-blockattentionfunctionby Tucker•decomposition[35].
e G is a core tensor. i, j and m are the indexescloselyof thetensor.
is theouter
product.
z isP
Theorem78
3.1. Let e1 , . . . , en be basis vectors from the vector space S. Assume that these vec
X
Ma,definition
Xindian,etal.
“A
Tenderized
Transformerfor
Language
Modeling.”,
Neurips
2019
(1)
(d)
ame
in
Eq.
1.
Q
,
K
and
V
are
columnvectorsfrommatrices
Q,beK
andrepresented
V , where2
e1 , . . . , en are linear
and Q,K,V
canlinearlyby this set of basis vecA=
Gi •1 Xi •2iXi •3j. . . •d Xi k(1) independentImperfect Multimodal Time Series Data
Imperfect data:
Incomplete due to sensor failure
Corrupted by random or structured noises‣‣

How to learn robust representation from imperfect multimodal data?

data: multimodal fused‣ Cleantensor exhibits low-rankness
across time and modality
and incomplete data
‣ Noisy
breaks low-rank structure79

Multimodal Learning with Low-rank Regularization

Tensor fusion (Rank-1 tensor)

Low-rank regularizer

Upper bounds on nuclear norm

Low-rankness regularizer improves robustness to imperfect data
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization(Liang et al., ACL 2019)80

High-order Tensor Fusion‣ Expressive power of tensor fusion is limited‣ From first-order to high-oder intra-modal and cross-modal feature interactions‣ Tensor Polynomial Pooling (PTP)
Feature Interactions
Modality 1‣ Linear‣ Bilinear‣ Trilinear‣ Intra-modal
Modality 2‣ High-order
F = f ⌦ f ⌦ ··· ⌦ f|{z}Dimensionality increases exponentially with P
Order

P-order(md)

3
2
Example: F1,1,1 = f1 , F1,2,1 = f1 f2 ,F1,2,3 = f1 f2 f3 , F1,2,2 = f1 f22

P<latexit sha1_base64="+30Spn0nonFZCs0WSP54egitTCA=">AAAB/HicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosunFZwT6gHSWTybSxSWZIMkIZ6je41bU7ceu/uPRPTB8L23ogcDjnXO7NCRLOtHHdbye3srq2vpHfLGxt7+zuFfcPmjpOFaENEvNYtQOsKWeSNgwznLYTRbEIOG0Fg+ux33qiSrNY3plhQn2Be5JFjGBjpWZZhKf39Ydiya24E6Bl4s1ICWaw+Z9uGJNUUGkIx1p3PDcxfoaVYYTTUaGbappgMsA92rFUYkG1n02uHaETq4QoipV90qCJ+nciw0LroQhsUmDT14veWPzP66QmuvQzJpPUUEmmi6KUIxOj8ddRyBQlhg8twUQxeysifawwMbaguS2BGNlOvMUGlkmzWvHOKtXb81LtatZOHo7gGMrgwQXU4Abq0AACj/ACr/DmPDvvzofzOY3mnNnMIczB+foFZ3eVIA==</latexit>

Number ofmodality<latexit sha1_base64="BQJo0G/BnhGJ0/JHApGEW/esbBc=">AAACgXicbVFrS8MwFE07H3O+quInQYNDUBij7QSFIQwF8eME94B1K2mWbmHpgyQVRtkP8Cf60V/gXzDdJux1ww2He87NvZx4MaNCmua3pue2tnd283uF/YPDo2Pj5LQpooRj0sARi3jbQ4IwGpKGpJKRdswJCjxGWt7oJeNbn4QLGoUfchyTboAGIfUpRlKVXOPLCZAcYsTgq5taJXUm8AlC37V6lRJ0qnCZtxd4W9220jiFVU3lX5MpVG5+yV5U9WzXKJplcxpwHVhzUATzqLvGj9OPcBKQUGKGhOhYZiy7KeKSYkYmBScRJEZ4hAako2CIAiK66dSyCbxRlT70I64ylHBaXexIUSDEOPCUMltbrHJZcRPXSaT/2E1pGCeShHg2yE8YlBHM/Id9ygmWbKwAwpyqXSEeIo6wVL+0NMULJsoTa9WBddC0y1albL/fF2vPc3fy4AJcg1tggQdQA2+gDhoAg1/tXLvUrvScfqebuj2T6tq85wwshV79AwPet5M=</latexit>

Dimensionof feature

Deep Multimodal Multilinear Fusion with High-order Polynomial Pooling (Hou et al., NeurIPS 2019)
81

z2
f

⌦

·

f

⇥

W1

⇥

⇥

W2

⇥

W3

W4

⇥

W 5 W out

Tensor Polynomial Pooling (PTP)
Low-rank tensor networkz

P-order
Concatenate
tensor product1T
fT = [1, zT
1 , z2 ]

f

f

z2
f

Tensorcontractionf⌦

W ⇡ G1·fz

G2<latexit sha1_base64="ez/egD5Ah7VfKUfoOfc3pUxYzAA=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DJooWUE84DsEmYnk2TIzOwyMyuEJZ3fYKu1ndj6I5b+ibPJFibxwIXDOfdyDyeMOdPGdb+dwtr6xuZWcbu0s7u3f1A+PGrpKFGENknEI9UJsaacSdo0zHDaiRXFIuS0HY5vM7/9RJVmkXw0k5gGAg8lGzCCjZV8X2AzIpiju16tV664VXcGtEq8nFQgR6NX/vH7EUkElYZwrHXXc2MTpFgZRjidlvxE0xiTMR7SrqUSC6qDdJZ5is6s0keDSNmRBs3UvxcpFlpPRGg3s4x62cvE/7xuYgbXQcpknBgqyfzRIOHIRCgrAPWZosTwiSWYKGazIjLCChNja1r4Eoqp7cRbbmCVtGpV76Jae7is1G/ydopwAqdwDh5cQR3uoQFNIBDDC7zCm/PsvDsfzud8teDkN8ewAOfrFxS7l7o=</latexit><latexit sha1_base64="nZQJ9axZnLypIQT4ogoHYUpd2Gg=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DJooWUE84DsEmYns8mQmdllZlYIIZ3fYKu1ndj6I5b+ibPJFibxwIXDOfdyDydMONPGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFWENknMY9UJsaacSdo0zHDaSRTFIuS0HY5uM7/9RJVmsXw044QGAg8kixjBxkq+L7AZEszRXc/rlStu1Z0BrRIvJxXI0eiVf/x+TFJBpSEca9313MQEE6wMI5xOS36qaYLJCA9o11KJBdXBZJZ5is6s0kdRrOxIg2bq34sJFlqPRWg3s4x62cvE/7xuaqLrYMJkkhoqyfxRlHJkYpQVgPpMUWL42BJMFLNZERlihYmxNS18CcXUduItN7BKWrWqd1GtPVxW6jd5O0U4gVM4Bw+uoA730IAmEEjgBV7hzXl23p0P53O+WnDym2NYgPP1CxMol7k=</latexit>

GP

Gn<latexit sha1_base64="vhZ5svoA1tUJockdOza6jRi10xU=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DJooWUE84DsEmYns8mQmdllZlYIIZ3fYKu1ndj6I5b+ibPJFibxwIXDOedyLydMONPGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFWENknMY9UJsaacSdo0zHDaSRTFIuS0HY5uM7/9RJVmsXw044QGAg8kixjBxkq+L7AZEszRXa/RK1fcqjsDWiVeTiqQw+Z//H5MUkGlIRxr3fXcxAQTrAwjnE5LfqppgskID2jXUokF1cFk9vMUnVmlj6JY2ZEGzdS/GxMstB6L0CazH/Wyl4n/ed3URNfBhMkkNVSS+aEo5cjEKCsA9ZmixPCxJZgoZn9FZIgVJsbWtHAlFFPbibfcwCpp1areRbX2cFmp3+TtFOEETuEcPLiCOtxDA5pAIIEXeIU359l5dz6cz3m04OQ7x7AA5+sXQ/WX2A==</latexit><latexit sha1_base64="SD9GE0cc51EvjG/PszEWr8mFv2w=">AAACAnicbVC7SgNBFL0bXzG+opY2g0GwCrtR0DJooWUE84DsEmYns8mQmdllZlYIIZ3fYKu1ndj6I5b+ibPJFibxwIXDOfdyDydMONPGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFWENknMY9UJsaacSdo0zHDaSRTFIuS0HY5uM7/9RJVmsXw044QGAg8kixjBxkq+L7AZEszRXU/2yhW36s6AVomXkwrkaPTKP34/Jqmg0hCOte56bmKCCVaGEU6nJT/VNMFkhAe0a6nEgupgMss8RWdW6aMoVnakQTP178UEC63HIrSbWUa97GXif143NdF1MGEySQ2VZP4oSjkyMcoKQH2mKDF8bAkmitmsiAyxwsTYmha+hGJqO/GWG1glrVrVu6jWHi4r9Zu8nSKcwCmcgwdXUId7aEATCCTwAq/w5jw7786H8zlfLTj5zTEswPn6BXMvl/Y=</latexit>

Low-rank tensor network

Number of Parameters

O(md)

f
<latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit>

P

Gn

f

f

<latexit sha1_base64="1CH/E/k7vw4iob5ofCdhFnqOU70=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aQne6WyW3GnQMvEy0kZctR7pZ9uPyKJoNIQjrXueG5s/BQrwwink2I30TTGZIQHtGOpxIJqP52GnqBTq/RRGCk70qCp+vcixULrsQjsZhZSL3qZ+J/XSUx45adMxomhkswehQlHJkJZA6jPFCWGjy3BRDGbFZEhVpgY29Pcl0BMbCfeYgPLpFmteOeV6v1FuXadt1OAYziBM/DgEmpwB3VoAIFHeIFXeHOenXfnw/mcra44+c0RzMH5+gXyUpjY</latexit><latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit><latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit><latexit sha1_base64="5sG7oX987gzGTpwkyDANse4nVso=">AAACCXicbVDLTsJAFL3FF+ILdOlmIjHBDWnRRJdEN+7ERJAEKplOpzBhpm1mphpC+AK/wa2u3Rm3foVL/8QpdCHgSW5ycs65uTfHizlT2ra/rdzK6tr6Rn6zsLW9s7tXLO23VJRIQpsk4pFse1hRzkLa1Exz2o4lxcLj9N4bXqX+/SOVikXhnR7F1BW4H7KAEayN1CuWugLrAcEc3aCK8E8eGr1i2a7aU6Bl4mSkDBlM/qfrRyQRNNSEY6U6jh1rd4ylZoTTSaGbKBpjMsR92jE0xIIqdzx9fYKOjeKjIJJmQo2m6t+NMRZKjYRnkumjatFLxf+8TqKDC3fMwjjRNCSzQ0HCkY5Q2gPymaRE85EhmEhmfkVkgCUm2rQ1d8UTE9OJs9jAMmnVqs5ptXZ7Vq5fZu3k4RCOoAIOnEMdrqEBTSDwBC/wCm/Ws/VufVifs2jOynYOYA7W1y+UFJmE</latexit>rG<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="1h/TU896uDFwcMUPY8XDmR6Idps=">AAACA3icbVDLSsNAFL2pr1pfVZduBovgqiRV0GXRha6kgn1AG8pkOmmHTiZxZiKUkKXf4FbX7sStH+LSP3HSZmFbD1w4nHMv93C8iDOlbfvbKqysrq1vFDdLW9s7u3vl/YOWCmNJaJOEPJQdDyvKmaBNzTSnnUhSHHictr3xdea3n6hULBQPehJRN8BDwXxGsDaS2wuwHhHMk5u0f9cvV+yqPQVaJk5OKpCj0S//9AYhiQMqNOFYqa5jR9pNsNSMcJqWerGiESZjPKRdQwUOqHKTaegUnRhlgPxQmhEaTdW/FwkOlJoEntnMQqpFLxP/87qx9i/dhIko1lSQ2SM/5kiHKGsADZikRPOJIZhIZrIiMsISE216mvviBanpxFlsYJm0alXnrFq7P6/Ur/J2inAEx3AKDlxAHW6hAU0g8Agv8Apv1rP1bn1Yn7PVgpXfHMIcrK9fv/KYuA==</latexit>f2

O(mdr P )r
G<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>N<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit>

G1

2<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit><latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit>f<latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit>f<latexit sha1_base64="3unYc+gg+jjNjU3rPvctdCKp8+M=">AAACC3icbVDLSsNAFJ3UV62vaJduBotQNyWpgi6LbtxZwT6gjWUymbRDZyZhZiKE0k/wG9zq2p249SNc+idO2ixs64ELh3Pu5R6OHzOqtON8W4W19Y3NreJ2aWd3b//APjxqqyiRmLRwxCLZ9ZEijArS0lQz0o0lQdxnpOOPbzK/80SkopF40GlMPI6GgoYUI22kgV3uc6RHGDF4B6s8kI/15tnArjg1Zwa4StycVECO5sD+6QcRTjgRGjOkVM91Yu1NkNQUMzIt9RNFYoTHaEh6hgrEifIms/BTeGqUAIaRNCM0nKl/LyaIK5Vy32xmUdWyl4n/eb1Eh1fehIo40UTg+aMwYVBHMGsCBlQSrFlqCMKSmqwQj5BEWJu+Fr74fGo6cZcbWCXtes09r9XvLyqN67ydIjgGJ6AKXHAJGuAWNEELYJCCF/AK3qxn6936sD7nqwUrvymDBVhfv+0Xmjw=</latexit><latexit sha1_base64="1HTzLuAxmXMW+d2dbRI3rP/zAWs=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkIU9ssVt+rOgFaJl5MK5Gj0yz+9QUQSQaUhHGvd9dzY+ClWhhFOp6VeommMyRgPaddSiQXVfjoLPEVnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhtZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEwgIeIFXeHOenXfnw/mcjxacfOcYFuB8/QJDLJbC</latexit>‣ Tensor network representation of W is crucial<latexit sha1_base64="87/8Kl/wmGaKS2Y3MMIQsH7aBa4=">AAACAHicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ae2Q8mkmTY0yQxJRihDN36DW127E7f+iUv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzIpijdr9ccavuDGiVeDmpQI5Gv/zTG0QkEVQawrHWXc+NjZ9iZRjhdFrqJZrGmIzxkHYtlVhQ7aezxFN0ZpUBCiNlnzRopv7dSLHQeiICO5kl1MteJv7ndRMTXvspk3FiqCTzQ2HCkYlQ9n00YIoSwyeWYKKYzYrICCtMjC1p4UogprYTb7mBVdKqVb2Lau3+slK/ydspwgmcwjl4cAV1uIMGNIGAhBd4hTfn2Xl3PpzP+WjByXeOYQHO1y/35Zcl</latexit>

Deep Multimodal Multilinear Fusion with High-order Polynomial Pooling (Hou et al., NeurIPS 2019)82

Output
modality

Audio Video Text

O

Input
modality

Intermediate
modality

Outputmodality

Hierarchical Polynomial Fusion Network (HPFN)
H1-1

T4
PTP
Interactions
between modality and PTP
time
steps are modeled explicitly
T6‣

Audio Video Text

O(a) one-layer HPFN

T8

PTP

Output layer(b) two-layer HPFN

PTP resembles convolutional filter

Au

T8

T1
T2
T3
T4
T5
T6
T7
T8

Output layer

Input layer(a) one-layer HPFN

Input
modality

Figure 2: (a) An illustrative example of a fusion
Intermediate‘window’ size is Output[8 ⇥ 3]. (b) An example of two‘window’ has size
[4 ⇥ 3] with stride step size 2
modality
modality
Input
Inte‘window’ with size [3 ⇥ 1]
coversalltheintermed
modalitym
H2-1 H2-2 H2-3
O
for the ‘1st’ column
index
offeature
nodes
in
the
Audio Video Text

Intermediate
modality

Audio Video Text
T1
T2
T3
T4
T5
T6
T7
T8

PTP

H1-1 H1-2

H1-3

PTP

T2

78

T4

T4
PTP

T6

80
81

T8

Input layer

79

T8

Hidden layer 1

82

3.1

T1

High-order polynomial
tensor pooling
(P
T2
T2

T3
T4
PTP of a T4
The objective
PTP block is to efficientlymeT5
joint compact representationz by leveraging
the e
T6
T6
1 illustrates the flowchartof operations in a PTP b
T7
T8 into a lo{zm }M
concatenated together
m=1 are first T8
T8
Input layer zT
Hidde
[1, z
12···M =Hidden layer 2
layer
83
Then, a degreeOutputof P polynomialfeature tensor Z
84
product of the concatenated feature vector z1···M

Figure 3: An example of a three-layer HPFN.
83

H1-1 HTime‣

Outputmodality

T1
T2
T8
T3
T4
T5
Hidden layer
1
T6
T7
T8

Local temporal-modality correlations
Output layer
Input layer
Input layer
are fused and multi-layers
are designed
T8

Input
modality
PTP

Time

T1
T2
T3
T4
T5
T6
T7
T8

O

Time

‣

Audio Video Text

Time

T1
T2
T3
T4
T5
T6
T7
T8

Time

Time

Input
modality85

Z P = z12···M ⌦1 z12{⌦p }P
p=1 are the tensor product operators. NoticeHigher-Order Tensor RNNs

Tensor-Power Recurrent Model‣of functions that satisfies certain regularity conditions. For such functions, tensor-trai
Mappingof hidden
state:
tensor power
representations
preserve
the degree-p
weak di↵erentiability
and yield a compact representation.
The following theorem characterizes the representation power of HOT-RNN, viewed as
one-layer hidden neural network, in terms of 1) the regularity of the target function f , 2
the dimension of the input space, 3) the tensor train rank and 4) the order of the tensor:‣

Theorem 1 Let the target function f 2 Hµk be a Hölder continuous function defined on
Tensor-powerrecurrent
model
input domain I = I1 ⇥ · · · ⇥ Id , with bounded derivatives up to order k and finite Fourie
magnitude distribution Cf . A single layer HOT-RNN with h hidden units, fˆ can approximat
f with approximation error ✏ at most:✓ Yu et al.
◆
1
d 1
2
k
Approximation error ✏ 
Cf
+
C(k)p
(9
k
1
h
(k 1)(r + 1)
R
where Cf = |!|1 |fˆ(!)d!|, d is the dimension of(TT-)ranks
the function, i.e., the size of the stat
space, r is the tensor-train rank, p is the degree of the higher-order polynomials i.e., thorder of the tensor, and C(k) is the coefficient of the spectral expansion of f .

‣Remarks: The result above shows that the number of weights required to approximat
the target function f is dictated by its regularity (i.e., its Hölder-continuity order k). Th
expressiveness (b)of HOT-RNN
the selection of(d)theHOT-RNN
rank r and thedegre(a) RNN
MRNN is driven(c)byHORNN(e)polynomial
HOT-LSTM
p; moreover, it improves for functions with increasing regularity. Compared with “first-orde
84
RNNs,(right
HOT-RNNs
are exponentiallymore
powerful
for versus
large
rank:
if the orderFigure11:regularLong-term2) predictions
forhigherdi↵erentthegroundtruth
Yu, Rose,
et al. "Long-term
forecastingusing
ordermodels
tensor (red)RNNs."
JMLR(2019)

Tensor-Power Recurrent Model‣ Extension to spatio-temporal recurrent modelC(t−1)

Preprocessing Module

Conv. Tensor-Train Module

P(1)H(t−1:t−3)

*

~

H (1)

*

*

H (2)

*

X(t )

**

5x5

K (3)

~

*

H (3)

W

3x3 G(1)=K (1)

H(t )

+

LSTM

K (2)

~

P(3)
H(t−3:t−5)

tanh

+

P(2)
H(t−2:t−4)

C(t )

+

*

7x7

*

3x3

G(2)
+ Addition

*

3x3

G(3)

* Convolution
Element-wise productSigmoidtanh Hyperbolic tangent

85 for Spatio-temporal Learning.”, Neurips 2020
Su,
Jiahao,etal.
"Convolutional
Tensor-Train
LSTM
Figure 1: Convolutional Tensor-Train LSTM. The preprocessing module first groups the previous

Software for Tensor Networks‣ Google TensorNetwork (https://github.com/google/TensorNetwork)‣ Tensorly (http://tensorly.org/)‣ ITensor (https://itensor.org)‣ TeNPy (https://tenpy.readthedocs.io/)‣ Tntorch (https://tntorch.readthedocs.io/)‣ TT-Toolbox (https://github.com/oseledets/TT-Toolbox)
network machine learning (TNML) (https://github.com/‣ Tensor
emstoudenmire/TNML)‣ TT_RNN (https://github.com/Tuyki/TT_RNN)‣ TensorNet (https://github.com/Bihaqo/TensorNet)‣ More …
86Part 3
Frontier and Future Direction

87TNs for Function Approximation
Supervised learning‣ Identify a nonlinear function from training dataset‣

The function space is modeled as tensor

N discrete inputs
and an output

Tensor Completion

Training points are observed entries

Inference = Missing value prediction‣ Supervised learning to tensor completion
Nonlinear System Identification via Tensor Completion (Kargas et al. AAAI 2020)88

Learning Low-rank Approximation to Function‣ Optimization problem of tensor completion
Fidelity

Regularization

Low-rank CP representation as constraint‣ Pros.
‣ Non-parametric model without specific form of function‣ Complexity of function = Tensor Rank, can be learned from dataset‣ Rank-1 can be a complex nonlinear function89

A Simple Exampleh(x2 )<latexit sha1_base64="QRppmZ64BWuP80OlSlT0ps5bSCQ=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoMQm3AXBS2DNpYRzAckR9jb7CVLdveO3T0xHMHfYKu1ndj6Wyz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaPbqd96pEqzSD6YcUx9gQeShYxgY6UWGpafetXzXrHkVtwZ0CrxMlKCDPVe8afbj0giqDSEY607nhsbP8XKMMLppNBNNI0xGeEB7VgqsaDaT2fnTtCZVfoojJQtadBM/TuRYqH1WAS2U2Az1MveVPzP6yQmvPZTJuPEUEnmi8KEIxOh6e+ozxQlho8twUQxeysiQ6wwMTahhS2BmNhMvOUEVkmzWvEuKtX7y1LtJksnDydwCmXw4ApqcAd1aACBEbzAK7w5z8678+F8zltzTjZzDAtwvn4BqjKVPA==</latexit>

-1

1

-1

1

Highly nonlinear1h(x1 )<latexit sha1_base64="D2s3wGEO21IxWAuVv9wube4VkOI=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoMQm3AXBS2DNpYRzAckR9jb7CVLdveO3T0xHMHfYKu1ndj6Wyz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaPbqd96pEqzSD6YcUx9gQeShYxgY6UWGpafet55r1hyK+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NzJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvspk3FiqCTzRWHCkYnQ9HfUZ4oSw8eWYKKYvRWRIVaYGJvQwpZATGwm3nICq6RZrXgXler9Zal2k6WThxM4hTJ4cAU1uIM6NIDACF7gFd6cZ+fd+XA+5605J5s5hgU4X7+onpU7</latexit>-1

Rank = 1

1-1

y = f (x1 , x2 )f (x1 , x2 ) = h(x1 )h(x2 )<latexit sha1_base64="Og1vm1VWuy5f6fIHgIqtmyM7vaw=">AAACE3icbVDLSgMxFM3UV62vUZfdBIvQgpSZUdCNUHTjsoJ9QDsMmTTThmYeJBlpGbrwI/wGt7p2J279AJf+iZnpLGzrgZCTc+7l3hw3YlRIw/jWCmvrG5tbxe3Szu7e/oF+eNQWYcwxaeGQhbzrIkEYDUhLUslIN+IE+S4jHXd8m/qdR8IFDYMHOY2I7aNhQD2KkVSSo5e96sQxz+DEsWrwGo7SVy27rJqjV4y6kQGuEjMnFZCj6eg//UGIY58EEjMkRM80ImkniEuKGZmV+rEgEcJjNCQ9RQPkE2En2Sdm8FQpA+iFXJ1Awkz925EgX4ip76pKH8mRWPZS8T+vF0vvyk5oEMWSBHg+yIsZlCFME4EDygmWbKoIwpyqXSEeIY6wVLktTHH9mcrEXE5glbStunlet+4vKo2bPJ0iKIMTUAUmuAQNcAeaoAUweAIv4BW8ac/au/ahfc5LC1recwwWoH39AsVRm4c=</latexit><latexit sha1_base64="sKGac9mIipVWQgM2LY6ATyuDKTY=">AAACB3icbVDLSgMxFM3UV62Pjrp0EyxCBSkzVdCNUHTjsoJ9QDsMmTTThiaZIclIy9AP8Bvc6tqduPUzXPonpu0sbOuBC4dz7uVcThAzqrTjfFu5tfWNza38dmFnd2+/aB8cNlWUSEwaOGKRbAdIEUYFaWiqGWnHkiAeMNIKhndTv/VEpKKReNTjmHgc9QUNKUbaSL5dHMMbGJZHvnsOR371zLdLTsWZAa4SNyMlkKHu2z/dXoQTToTGDCnVcZ1YeymSmmJGJoVuokiM8BD1ScdQgThRXjp7fAJPjdKDYSTNCA1n6t+LFHGlxjwwmxzpgVr2puJ/XifR4bWXUhEnmgg8DwoTBnUEpy3AHpUEazY2BGFJza8QD5BEWJuuFlICPjGduMsNrJJmteJeVKoPl6XabdZOHhyDE1AGLrgCNXAP6qABMEjAC3gFb9az9W59WJ/z1ZyV3RyBBVhfv8h8l+U=</latexit>

90Challenges‣ Discretization of features is necessary‣ Sample complexity of TNs? Is it comparable with DNNsof parameters (latent cores in TNs vs. weight‣ Numberparameters in DNNs)

91

TNs for Probability Graphical Model‣ Probability mass function = Non-negative Tensor
P (X1 , X2 , . . . , XN )<latexit sha1_base64="ixwxAsr2TryIB7Ti2QhlMSiNLpU=">AAACEHicbVDLSsNAFJ3UV62vqBvBzWARKpSSVEGXRTeupIJtA20Ik8mkHTqThJmJUEr9CL/Bra7diVv/wKV/4qTNwrYeuHDmnHu5c4+fMCqVZX0bhZXVtfWN4mZpa3tnd8/cP2jLOBWYtHDMYuH4SBJGI9JSVDHiJIIg7jPS8Yc3md95JELSOHpQo4S4HPUjGlKMlJY886hZcTy7Ch2vXoU9FsRKZo+7M88sWzVrCrhM7JyUQY6mZ/70ghinnEQKMyRl17YS5Y6RUBQzMin1UkkShIeoT7qaRogT6Y6nF0zgqVYCGMZCV6TgVP07MUZcyhH3dSdHaiAXvUz8z+umKrxyxzRKUkUiPFsUpgyqGGZxwIAKghUbaYKwoPqvEA+QQFjp0Oa2+HyiM7EXE1gm7XrNPq/V7y/Kjes8nSI4BiegAmxwCRrgFjRBC2DwBF7AK3gzno1348P4nLUWjHzmEMzB+PoFuPKbEg==</latexit>‣ Tensor decomposition (CPD)

Xn 2 [1 : d]<latexit sha1_base64="kxnKRW4V5ODpKEdWLG0mRsrYqYM=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKilXQxjKCeUCyhNnZ2WTIzOw6MyuEJaXfYKu1ndj6IZb+iZNkC5N44MLhnHs5lxMknGnjut/Oyura+sZmYau4vbO7t186OGzqOFWENkjMY9UOsKacSdowzHDaThTFIuC0FQxvJ37riSrNYvlgRgn1Be5LFjGCjZX8dk+iLpOo412Hfq9UdivuFGiZeDkpQ456r/TTDWOSCioN4Vjrjucmxs+wMoxwOi52U00TTIa4TzuWSiyo9rPp02N0apUQRbGyIw2aqn8vMiy0HonAbgpsBnrRm4j/eZ3URFd+xmSSGirJLChKOTIxmjSAQqYoMXxkCSaK2V8RGWCFibE9zaUEYmw78RYbWCbNasU7r1TvL8q1m7ydAhzDCZyBB5dQgzuoQwMIPMILvMKb8+y8Ox/O52x1xclvjmAOztcvIkeXtQ==</latexit>Ph<latexit sha1_base64="fa+BpWWPN3rh++DRpWfE7xUzdQ0=">AAACAHicbVDLSgMxFL1TX7W+qi7dBIvgqszUgi6LblxWsA9sB8mkmTY0yQxJRihDN36DW127E7f+iUv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj9o6ShShLRLxSHUDrClnkrYMM5x2Y0WxCDjtBOObzO88UaVZJO/NJKa+wEPJQkawsdJDX2AzIpij5mO54lbdGdAq8XJSgRx2/qc/iEgiqDSEY617nhsbP8XKMMLptNRPNI0xGeMh7VkqsaDaT2eJp+jMKgMURso+adBM/buRYqH1RAR2Mkuol71M/M/rJSa88lMm48RQSeaHwoQjE6Hs+2jAFCWGTyzBRDGbFZERVpgYW9LClUBMbSfecgOrpF2rehfV2l290rjO2ynCCZzCOXhwCQ24hSa0gICEF3iFN+fZeXc+nM/5aMHJd45hAc7XL+zglx4=</latexit>=

N-order<latexit sha1_base64="Vy5/Vls8b+i8k3uIgMh4PXUpcAI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioI0QtLFMwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmDzM/M4zKM0i+WTSGHxBRpKFjBJjpebdoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/P0ymiM3SOLpGHblAdPaIGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CL7aTWw==</latexit>

X1

X2 · · · XN<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>

Joint Probability:
CPD:<latexit sha1_base64="W1HEcjnS4A7UOWwgdGq5kfeHT3Q=">AAAB/HicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHubvWTN7t6xuyeEI/4GW63txNb/Yuk/cXO5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML6d+e0nqjSL5IOZxNQXeChZyAg2Vmr1dCL6ql+uuFU3A1olXk4qkKPRL//0BhFJBJWGcKx113Nj46dYGUY4nZZ6iaYxJmM8pF1LJRZU+2l27RSdWWWAwkjZkgZl6t+JFAutJyKwnQKbkV72ZuJ/Xjcx4bWfMhknhkoyXxQmHJkIzV5HA6YoMXxiCSaK2VsRGWGFibEBLWwJxNRm4i0nsEpatap3Ua3dX1bqN3k6RTiBUzgHD66gDnfQgCYQeIQXeIU359l5dz6cz3lrwclnjmEBztcvgZSV0g==</latexit>

P (X1 , X2 , . . . , XN ) =<latexit sha1_base64="JeYNVNDq6cwUnIUZ+O1f6BQgmUA=">AAACRXicbZBNS8NAEIY39avWr6pHL4tFaKGUpAp6EYpevFgq2A9oSthst+3S3STsboSS9lf5I/wNPapnb+JVN2kO2jqw8Mw7M8zs6waMSmWacyOztr6xuZXdzu3s7u0f5A+PWtIPBSZN7DNfdFwkCaMeaSqqGOkEgiDuMtJ2x7dxvf1EhKS+96gmAelxNPTogGKktOTk7xvFjmOVYceplqHN+r6ScVIvQXgNbRlyJxrNYKM4KsGkcwpTqk412DgeSPJ6nDv5glkxk4CrYKVQAGk0nPyr3fdxyImnMENSdi0zUL0ICUUxI7OcHUoSIDxGQ9LV6CFOZC9Kvj2DZ1rpw4Ev9PMUTNTfExHiUk64qzs5UiO5XIvF/2rdUA2uehH1glARDy8WDUIGlQ9jD2GfCoIVm2hAWFB9K8QjJBBW2uk/W1w+055Yyw6sQqtasc4r1YeLQu0mdScLTsApKAILXIIauAMN0AQYPIM5eAPvxovxYXwaX4vWjJHOHIM/YXz/AGiorPg=</latexit>

Px1 ,x2 ,...,xN =<latexit sha1_base64="WkhMi45Gy2G8cqdyhCCCtlbj6zE=">AAACX3icbVFdS8MwFM3qx+bUWfVJfAkOYcIY7RT0RZj64tOY4DZhmyVNsy0saUuSykbpT/QH+Ljf4KuC6boH3bwQOPecc2/CiRsyKpVlfeSMjc2t7Xxhp7i7t186MA+POjKIBCZtHLBAvLhIEkZ90lZUMfISCoK4y0jXnTykeveNCEkD/1nNQjLgaOTTIcVIacoxRy0nnjp2FU6dehX2mRcomTbNBMJb2JcRd2KRaEGv9JAj4N1rXLEvkmxKK2lfz3q9ILXidMeCb2Z8U/scs2zVrEXBdWAvQRksq+WY874X4IgTX2GGpOzZVqgGMRKKYkaSYj+SJER4gkakp6GPOJGDeBFIAs8148FhIPTxFVywvydixKWccVc7OVJjuaql5H9aL1LDm0FM/TBSxMfZRcOIQRXANF3oUUGwYjMNEBZUvxXiMRIIK/0Hf25xeZqJvZrAOujUa/Zlrf50VW7cL9MpgFNwBirABtegAR5BC7QBBu/gE3yB79zcyBslw8ysRm45cwz+lHHyAx60te4=</latexit>

X

X
r

X
h

(N )
Ar

(1)
Ar
<latexit sha1_base64="H2nnqi7rsnjcTVLlp6NgM3yxpEo=">AAAB/3icbVA9TwJBEJ3DL8Qv1NJmIzHBhtyhiZaojSUm8mHgJHvLHmzYvbvs7pmQC4W/wVZrO2PrT7H0n7gHVwj4kkle3pvJzDwv4kxp2/62ciura+sb+c3C1vbO7l5x/6CpwlgS2iAhD2Xbw4pyFtCGZprTdiQpFh6nLW90k/qtJyoVC4N7PY6oK/AgYD4jWBvp4aonH5OyczrpFUt2xZ4CLRMnIyXIUO8Vf7r9kMSCBppwrFTHsSPtJlhqRjidFLqxohEmIzygHUMDLKhyk+nBE3RilD7yQ2kq0Giq/p1IsFBqLDzTKbAeqkUvFf/zOrH2L92EBVGsaUBmi/yYIx2i9HvUZ5ISzceGYCKZuRWRIZaYaJPR3BZPpJk4iwksk2a14pxVqnfnpdp1lk4ejuAYyuDABdTgFurQAAICXuAV3qxn6936sD5nrTkrmzmEOVhfv5o6llg=</latexit>r<latexit sha1_base64="lhP+/I3P/BmLVSxNzHyMOIx8j7U=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXUxkoimA9JzrC32UuW7N4du3tCOFL4G2y1thNbf4ql/8S95AqT+GDg8d4MM/O8iDOlbfvbWlpeWV1bz23kN7e2d3YLe/sNFcaS0DoJeShbHlaUs4DWNdOctiJJsfA4bXrD69RvPlGpWBjc61FEXYH7AfMZwdpID5dd+ZiUbk/G3ULRLtsToEXiZKQIGWrdwk+nF5JY0EATjpVqO3ak3QRLzQin43wnVjTCZIj7tG1ogAVVbjI5eIyOjdJDfihNBRpN1L8TCRZKjYRnOgXWAzXvpeJ/XjvW/oWbsCCKNQ3IdJEfc6RDlH6PekxSovnIEEwkM7ciMsASE20ymtniiTQTZz6BRdKolJ3TcuXurFi9ytLJwSEcQQkcOIcq3EAN6kBAwAu8wpv1bL1bH9bntHXJymYOYAbW1y/IG5Z1</latexit>···<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>

P (h)P (X1 |h)P (X2 |h) · · · P (XN |h)

(1)
(2)
(N )
A
A
·
·
·
A
r x1 ,r x2 ,rxN ,r‣ Conditional dependency controlled by tensor rank
Expressive power of tensor-network factorizations for probabilistic modeling (Glasser et al.NuerIPS, 2019)92

Rank vs. Dependency

93Why use TN for Graphical Models‣ Efficient computation in:‣ Marginalization, normalization (partition function)‣ Maximum log-likelihood, MAP estimate‣ Conditional distributions and sampling‣…

94

Example‣ Factorize probability tensor in MPS/TR format‣ The marginal distribution is

Summation simply performed on core tensor
95

Example‣ Normalization
Summation over all core tensors‣ Conditional distribution

Summation over all core tensors except k-th
96

Unsupervised Generative ModelingHan et al., Phys. Rev. X 2018

‣ Modeling joint probability distribution of binary2 variables
G1r<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>p(v) =<latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit>

Gdr· · ·<latexit sha1_base64="f87RNn8QQzIlB9zcaTAp8Gw4vqE=">AAACA3icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy60GUF+4A2lMlk0g6dSeLMRCghS7/Bra7diVs/xKV/4qTNwrYeuHA4517u4XgxZ0rb9re1srq2vrFZ2ipv7+zu7VcODtsqSiShLRLxSHY9rChnIW1ppjntxpJi4XHa8cY3ud95olKxKHzQk5i6Ag9DFjCCtZHcvsB6RDBPb7OBP6hU7Zo9BVomTkGqUKA5qPz0/YgkgoaacKxUz7Fj7aZYakY4zcr9RNEYkzEe0p6hIRZUuek0dIZOjeKjIJJmQo2m6t+LFAulJsIzm3lItejl4n9eL9HBlZuyME40DcnsUZBwpCOUN4B8JinRfGIIJpKZrIiMsMREm57mvngiM504iw0sk3a95pzX6vcX1cZ10U4JjuEEzsCBS2jAHTShBQQe4QVe4c16tt6tD+tztrpiFTdHMAfr6xfilJjO</latexit><latexit sha1_base64="p7b/XR6tAd9mhLDe/vXcc5ryw8g=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlhuqXyl7Fm8NdJ35OypCj3i/99AYxTQVKQznRuut7iQkyogyjHKfFXqoxIXRMhti1VBKBOsjmh07dS6sM3ChWtqRx5+rfiYwIrScitJ2CmJFe9Wbif143NdFdkDGZpAYlXSyKUu6a2J197Q6YQmr4xBJCFbO3unREFKHGZrO0JRRTm4m/msA6aVUr/nWl2rgp1+7zdApwDhdwBT7cQg0eoQ5NoIDwAq/w5kycd+fD+Vy0bjj5zBkswfn6BYMlk5A=</latexit><latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="+QkZgccLf3P1dj9tFIPa5/DLE6M=">AAACB3icbVDLSsNAFL2pr1ofjbp0M1iEuilJFXQjFN24rGAf0IYymU7aoTNJmJkUSugH+A1ude1O3PoZLv0Tp20WtvXAhcM593IPx485U9pxvq3cxubW9k5+t7C3f3BYtI+OmypKJKENEvFItn2sKGchbWimOW3HkmLhc9ryR/czvzWmUrEofNKTmHoCD0IWMIK1kXp2MS53BdZDP0DjC3SLenbJqThzoHXiZqQEGeo9+6fbj0giaKgJx0p1XCfWXoqlZoTTaaGbKBpjMsID2jE0xIIqL50Hn6Jzo/RREEkzoUZz9e9FioVSE+GbzVlIterNxP+8TqKDGy9lYZxoGpLFoyDhSEdo1gLqM0mJ5hNDMJHMZEVkiCUm2nS19MUXU9OJu9rAOmlWK+5lpfp4VardZe3k4RTOoAwuXEMNHqAODSCQwAu8wpv1bL1bH9bnYjVnZTcnsATr6xe/iZh9</latexit>···<latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit>

v1<latexit sha1_base64="I6pTNfLRQHL9ABSqfh4pRCUtjzM=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRrVfKnsVbw53nfg5KUOOer/00xvENBUoDeVE667vJSbIiDKMcpwWe6nGhNAxGWLXUkkE6iCbHzp1L60ycKNY2ZLGnat/JzIitJ6I0HYKYkZ61ZuJ/3nd1ER3QcZkkhqUdLEoSrlrYnf2tTtgCqnhE0sIVcze6tIRUYQam83SllBMbSb+agLrpFWt+NeVauOmXLvP0ynAOVzAFfhwCzV4hDo0gQLCC7zCmzNx3p0P53PRuuHkM2ewBOfrFx5lk1A=</latexit>/Z<latexit sha1_base64="IInzBaVCvCR/c0OLJa6qvWrf2fA=">AAAB+HicbVA9TwJBEJ3DL8Qv1NJmIzGxwjs00ZJoY4lGPiJcyN6yBxt29y67eyZ44R/Yam1nbP03lv4TF7hCwJdM8vLeTGbmBTFn2rjut5NbWV1b38hvFra2d3b3ivsHDR0litA6iXikWgHWlDNJ64YZTluxolgEnDaD4c3Ebz5RpVkkH8wopr7AfclCRrCx0v3ZY7dYcsvuFGiZeBkpQYZat/jT6UUkEVQawrHWbc+NjZ9iZRjhdFzoJJrGmAxxn7YtlVhQ7afTS8foxCo9FEbKljRoqv6dSLHQeiQC2ymwGehFbyL+57UTE175KZNxYqgks0VhwpGJ0ORt1GOKEsNHlmCimL0VkQFWmBgbztyWQIxtJt5iAsukUSl75+XK3UWpep2lk4cjOIZT8OASqnALNagDgRBe4BXenGfn3flwPmetOSebOYQ5OF+/zHyTsQ==</latexit>

vd<latexit sha1_base64="3fI/p7NP0864kfzcnDdkkhkUyHI=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtxFQcugjWVE8wHJEfY2e8mS3b1jdy8QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXphwpo3nfTuFjc2t7Z3ibmlv/+DwyD0+aeo4VYQ2SMxj1Q6xppxJ2jDMcNpOFMUi5LQVju5nfmtMlWaxfDaThAYCDySLGMHGSk/jnt9zy17FmwOtEz8nZchR77k/3X5MUkGlIRxr3fG9xAQZVoYRTqelbqppgskID2jHUokF1UE2P3WKLqzSR1GsbEmD5urfiQwLrScitJ0Cm6Fe9Wbif14nNdFtkDGZpIZKslgUpRyZGM3+Rn2mKDF8YgkmitlbERlihYmx6SxtCcXUZuKvJrBOmtWKf1WpPl6Xa3d5OkU4g3O4BB9uoAYPUIcGEBjAC7zCm5M5786H87loLTj5zCkswfn6BbdjlDg=</latexit><latexit sha1_base64="hvl/x6jXEI3HmHQyzyHMYHKGexU=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKi+YDkCHt7e8mS3b1jdy8QjvwEW63txNZfY+k/cZNcYRIfDDzem2FmXpBwpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tJxqghtkpjHqhNgTTmTtGmY4bSTKIpFwGk7GN3P/PaYKs1i+WwmCfUFHkgWMYKNlZ7G/bBfrrhVdw60TrycVCBHo1/+6YUxSQWVhnCsdddzE+NnWBlGOJ2WeqmmCSYjPKBdSyUWVPvZ/NQpurBKiKJY2ZIGzdW/ExkWWk9EYDsFNkO96s3E/7xuaqJbP2MySQ2VZLEoSjkyMZr9jUKmKDF8YgkmitlbERlihYmx6SxtCcTUZuKtJrBOWrWqd1WtPV5X6nd5OkU4g3O4BA9uoA4P0IAmEBjAC7zCm5M5786H87loLTj5zCkswfn6BQe7lGs=</latexit>‣‣ Efficient direct sampling method to generate a sample bit by bitd{G}
Minimizing negative log-likelihood to learn parameters i i=1<latexit sha1_base64="VdNdPz18XROtz9qz+hw+/8DRGtM=">AAACEXicbVDLSsNAFJ3UV62vqCtxEyyCq5JUQTdC0YUuK9gHNDFMJtN26MwkzEyEEoIf4Te41bU7cesXuPRPnLRZ2NYDFw7n3Mu99wQxJVLZ9rdRWlpeWV0rr1c2Nre2d8zdvbaMEoFwC0U0Et0ASkwJxy1FFMXdWGDIAoo7weg69zuPWEgS8Xs1jrHH4ICTPkFQack3D9zUZVANEaTpTeYTN/NTculkD6FvVu2aPYG1SJyCVEGBpm/+uGGEEoa5QhRK2XPsWHkpFIogirOKm0gcQzSCA9zTlEOGpZdOXsisY62EVj8SuriyJurfiRQyKccs0J35uXLey8X/vF6i+hdeSnicKMzRdFE/oZaKrDwPKyQCI0XHmkAkiL7VQkMoIFI6tZktAct0Js58AoukXa85p7X63Vm1cVWkUwaH4AicAAecgwa4BU3QAgg8gRfwCt6MZ+Pd+DA+p60lo5jZBzMwvn4BxEeeHA==</latexit>

ri , i = 1, . . . , drmax<latexit sha1_base64="sUA/x5xv7uQQ6TJyc64dMPJEWjc=">AAACC3icbVDLSsNAFJ3UV62vaJduBovgopSkCroRim5cVrAPaEOYTCbt0JlJmJkIIfQT/Aa3unYnbv0Il/6J0zYLbT1w4XDOvZzLCRJGlXacL6u0tr6xuVXeruzs7u0f2IdHXRWnEpMOjlks+wFShFFBOppqRvqJJIgHjPSCye3M7z0SqWgsHnSWEI+jkaARxUgbyber0qd1SK/dOhyyMNaqDkPfrjkNZw64StyC1ECBtm9/D8MYp5wIjRlSauA6ifZyJDXFjEwrw1SRBOEJGpGBoQJxorx8/vwUnholhFEszQgN5+rvixxxpTIemE2O9FgtezPxP2+Q6ujKy6lIUk0EXgRFKYM6hrMmYEglwZplhiAsqfkV4jGSCGvT15+UgE9NJ+5yA6uk22y4543m/UWtdVO0UwbH4AScARdcgha4A23QARhk4Bm8gFfryXqz3q2PxWrJKm6q4A+szx9QwZne</latexit><latexit sha1_base64="uAmvLEnCnnKRcvnctKQpMLQoiSE=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHubvWTJ7t6xuyeGI/gbbLW2E1t/i6X/xL3kCpP4YODx3gwz84KYM21c99sprK1vbG4Vt0s7u3v7B+XDo5aOEkVok0Q8Up0Aa8qZpE3DDKedWFEsAk7bwfg289uPVGkWyQcziakv8FCykBFsrNRW/VTgp2m/XHGr7gxolXg5qUCORr/80xtEJBFUGsKx1l3PjY2fYmUY4XRa6iWaxpiM8ZB2LZVYUO2ns3On6MwqAxRGypY0aKb+nUix0HoiAtspsBnpZS8T//O6iQmv/ZTJODFUkvmiMOHIRCj7HQ2YosTwiSWYKGZvRWSEFSbGJrSwJRBZJt5yAqukVat6F9Xa/WWlfpOnU4QTOIVz8OAK6nAHDWgCgTG8wCu8Oc/Ou/PhfM5bC04+cwwLcL5+AYdblmk=</latexit>

97

Continuous Distribution by TNs‣ Gaussian mixture distribution
Mixture coefficient
discrete variable‣ Multi-dimensional Gaussian mixture distribution
Mixture coefficient is a discrete vector
Probability is N-dimensional tensor
A prior of a googol gaussians: a tensor ring induced prior for generative models (Kuznetsov et al., NeurIPS 2019)98

Prior Distribution Modeled by TN
of mixture coefficient p(z) is‣ Probability
TR representation‣ Tensor Ring Induced Prior

A prior of a googol gaussians: a tensor ring induced prior for generative models (Kuznetsov et al., NeurIPS 2019)99

Learning Relevant Features of Data

‣ Unsupervised learning with reduced order of TN representation‣ Supervised learning for the top classification layer
Classification

Orthogonalcore tensors

Kernel PCA

TN Representation:<latexit sha1_base64="CnWUV4NsgZH9D7Sz2ehPCFHo6Hg=">AAACM3icbVDLSgMxFM3UV62vqks3wSK0mzJTBQURim5cVrAP6JSSyWTa0MxkSO5IS+nH+BF+g1tdiivFrf9g+kBs64XAuefcy8k9Xiy4Btt+s1Irq2vrG+nNzNb2zu5edv+gpmWiKKtSKaRqeEQzwSNWBQ6CNWLFSOgJVvd6N2O9/sCU5jK6h0HMWiHpRDzglICh2tlLt9LleTck0PUC3C/gK+zGhum3nQJ2JfCQaexSX4L+7aa6X2hnc3bRnhReBs4M5NCsKu3sh+tLmoQsAiqI1k3HjqE1JAo4FWyUcRPNYkJ7pMOaBkbE2LWGkyNH+MQwPg6kMi8CPGH/bgxJqPUg9Mzk+Bq9qI3J/7RmAsFFa8ijOAEW0alRkAgMEo8Twz5XjIIYGECo4uavmHaJIhRMrnMuXjgymTiLCSyDWqnonBZLd2e58vUsnTQ6Qscojxx0jsroFlVQFVH0iJ7RC3q1nqx369P6mo6mrNnOIZor6/sHTGeqFg==</latexit>

Local feature mapping:
Input data:(x) = (x1 ) ⌦ · · · ⌦ (xd )<latexit sha1_base64="bsKIMmbxuUZZAISzpXsoB3l+iAg=">AAACHHicbZDLSsNAFIYn9VbrLerSzWAVKpSSVEFBhKIblxV6gySWyXTSDp1MwsxELKEv4EP4DG517U7cCi59E6eXhW39YeDjP+dwzvx+zKhUlvVtZJaWV1bXsuu5jc2t7R1zd68ho0RgUscRi0TLR5IwykldUcVIKxYEhT4jTb9/M6o3H4iQNOI1NYiJF6IupwHFSGmrbR65cY8WHtv0BF5Bxy5Cjd59rQjdS0hdyrXV8dpm3ipZY8FFsKeQB1NV2+aP24lwEhKuMENSOrYVKy9FQlHMyDDnJpLECPdRlzgaOQqJ9NLxb4bwWDsdGERCP67g2P07kaJQykHo684QqZ6cr43M/2pOooILL6U8ThTheLIoSBhUERxFAztUEKzYQAPCgupbIe4hgbDSAc5s8cOhzsSeT2ARGuWSfVoq353lK9fTdLLgAByCArDBOaiAW1AFdYDBE3gBr+DNeDbejQ/jc9KaMaYz+2BGxtcvTjqflw==</latexit>(xi ) = [1, xi ]T , i 2 [1, d]

x = [x1 , . . . , xd ]T<latexit sha1_base64="D4cd/XMcx25zk2f5SU8NipNwbZ0=">AAACFnicbVDLSsNAFJ3UV62vqEtBBovgopSkCroRim5cVugLmhgmk0k7dPJgZiItoTs/wm9wq2t34tatS//ESZuFbT1w4XDOvdx7jxszKqRhfGuFldW19Y3iZmlre2d3T98/aIso4Zi0cMQi3nWRIIyGpCWpZKQbc4ICl5GOO7zN/M4j4YJGYVOOY2IHqB9Sn2IkleTox1aA5MD14Qhew97IMSsW8yIpKiPHsx+ajl42qsYUcJmYOSmDHA1H/7G8CCcBCSVmSIieacTSThGXFDMyKVmJIDHCQ9QnPUVDFBBhp9M/JvBUKR70I64qlHCq/p1IUSDEOHBVZ3a1WPQy8T+vl0j/yk5pGCeShHi2yE8YlBHMQoEe5QRLNlYEYU7VrRAPEEdYqujmtrjBRGViLiawTNq1qnlerd1flOs3eTpFcAROwBkwwSWogzvQAC2AwRN4Aa/gTXvW3rUP7XPWWtDymUMwB+3rF3k8nts=</latexit>

Learning Relevant Features of Data with Multi-scale Tensor Networks (Stoudenmire et al. 2018)100

TN for Image Classification(x1 )<latexit sha1_base64="AbBuSfOiY1GwlX/pOzx0lXn7Gb8=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHubvWTJ7t6xuyeGI4W/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRfEnGnjut/Oyura+sZmbiu/vbO7t184OGzoKFGE1knEI9UKsKacSVo3zHDaihXFIuC0GQxvJn7zkSrNInlvRjH1Be5LFjKCjZUeOvGAlZ663lm3UHTL7hRomXgZKUKGWrfw0+lFJBFUGsKx1m3PjY2fYmUY4XSc7ySaxpgMcZ+2LZVYUO2n04PH6NQqPRRGypY0aKr+nUix0HokAtspsBnoRW8i/ue1ExNe+SmTcWKoJLNFYcKRidDke9RjihLDR5Zgopi9FZEBVpgYm9HclkCMbSbeYgLLpFEpe+flyt1FsXqdpZODYziBEnhwCVW4hRrUgYCAF3iFN+fZeXc+nM9Z64qTzRzBHJyvX65wlmQ=</latexit>

f (x) = hW, (x)i<latexit sha1_base64="YBttwdOuOWY4x4aLW+83RArt7gM=">AAAB/3icbVA9SwNBEJ3zM8avqKXNYhBiE+6ioGXQxjKC+ZDkCHt7e8mS3b1jd08MIYW/wVZrO7H1p1j6T9wkV5jEBwOP92aYmRcknGnjut/Oyura+sZmbiu/vbO7t184OGzoOFWE1knMY9UKsKacSVo3zHDaShTFIuC0GQxuJn7zkSrNYnlvhgn1Be5JFjGCjZUeOkmflZ664Vm3UHTL7hRomXgZKUKGWrfw0wljkgoqDeFY67bnJsYfYWUY4XSc76SaJpgMcI+2LZVYUO2PpgeP0alVQhTFypY0aKr+nRhhofVQBLZTYNPXi95E/M9rpya68kdMJqmhkswWRSlHJkaT71HIFCWGDy3BRDF7KyJ9rDAxNqO5LYEY20y8xQSWSaNS9s7LlbuLYvU6SycHx3ACJfDgEqpwCzWoAwEBL/AKb86z8+58OJ+z1hUnmzmCOThfv/7slpc=</latexit>

···

(x)

(xd )

<latexit sha1_base64="jKCu50Y6XVxcycv1aQqZ+Bmzx08=">AAACAnicbVA9SwNBEJ2LXzF+RS1tDoNgFe6ioGXQxjKC+YDcGfb29pIlu3vL7p4QQjp/g63WdmLrH7H0n7hJrjCJDwYe780wMy+SjGrjed9OYW19Y3OruF3a2d3bPygfHrV0milMmjhlqepESBNGBWkaahjpSEUQjxhpR8Pbqd9+IkrTVDyYkSQhR31BE4qRsVIQBxxJbVJXPsa9csWrejO4q8TPSQVyNHrlnyBOccaJMJghrbu+J004RspQzMikFGSaSISHqE+6lgrEiQ7Hs5sn7plVYjdJlS1h3Jn6d2KMuNYjHtlOjsxAL3tT8T+vm5nkOhxTITNDBJ4vSjLm2ienAbgxVQQbNrIEYUXtrS4eIIWwsTEtbIn4xGbiLyewSlq1qn9Rrd1fVuo3eTpFOIFTOAcfrqAOd9CAJmCQ8AKv8OY8O+/Oh/M5by04+cwxLMD5+gXSeZgy</latexit><latexit sha1_base64="4mP99hm5bAeMAV6d/EL5DmUaAXI=">AAAB/HicbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxjKC+YDkCHt7e8ma3dtjd04IR/wNtlrbia3/xdJ/4ia5wiQ+GHi8N8PMvCAR3IDrfjuFtfWNza3idmlnd2//oHx41DIq1ZQ1qRJKdwJimOAxawIHwTqJZkQGgrWD0e3Ubz8xbbiKH2CcMF+SQcwjTglYqdWjoQLTL1fcqjsDXiVeTiooR6Nf/umFiqaSxUAFMabruQn4GdHAqWCTUi81LCF0RAasa2lMJDN+Nrt2gs+sEuJIaVsx4Jn6dyIj0pixDGynJDA0y95U/M/rphBd+xmPkxRYTOeLolRgUHj6Og65ZhTE2BJCNbe3YjokmlCwAS1sCeTEZuItJ7BKWrWqd1Gt3V9W6jd5OkV0gk7ROfLQFaqjO9RATUTRI3pBr+jNeXbenQ/nc95acPKZY7QA5+sXcw+VyQ==</latexit><latexit sha1_base64="/TUEiEF4/6+MyLSF+cOwuHlZqFY=">AAACCHicbVDLSsNAFL2pr1pfUZduBotQNyWpgi6LblxWsA9oQplMJ+3QySTMTIql9Af8Bre6didu/QuX/omTNgttPXDhcM693MMJEs6Udpwvq7C2vrG5Vdwu7ezu7R/Yh0ctFaeS0CaJeSw7AVaUM0GbmmlOO4mkOAo4bQej28xvj6lULBYPepJQP8IDwUJGsDZSz7a9xpChihdhPQxC9Hjes8tO1ZkDrRI3J2XI0ejZ314/JmlEhSYcK9V1nUT7Uyw1I5zOSl6qaILJCA9o11CBI6r86Tz5DJ0ZpY/CWJoRGs3V3xdTHCk1iQKzmSVUy14m/ud1Ux1e+1MmklRTQRaPwpQjHaOsBtRnkhLNJ4ZgIpnJisgQS0y0KevPlyCamU7c5QZWSatWdS+qtfvLcv0mb6cIJ3AKFXDhCupwBw1oAoExPMMLvFpP1pv1bn0sVgtWfnMMf2B9/gAAqZk5</latexit><latexit sha1_base64="7hYzq4AhBpa0Pb7Iq9kVg5m2Yvg=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRtIvlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1C3//k44=</latexit>p<latexit sha1_base64="m1ksc1aV+naOnCUX0CgvTyqCF4M=">AAACLnicbVDLSsNAFJ3UV62vqEs3g0WoICWpom6EohuXFewDmlIm00k7dDIJMxOxhPyJH+E3uNW14ELEnZ/hJM3Cth4YOJxzLvfOcUNGpbKsD6OwtLyyulZcL21sbm3vmLt7LRlEApMmDlggOi6ShFFOmooqRjqhIMh3GWm745vUbz8QIWnA79UkJD0fDTn1KEZKS33z3Ks4PlIj14OPx1cOQ3zISKZgxOJ2cuI0RvRPxBFZom+WraqVAS4SOydlkKPRN7+dQYAjn3CFGZKya1uh6sVIKIoZSUpOJEmI8BgNSVdTjnwie3H2vwQeaWUAvUDoxxXM1L8TMfKlnPiuTqaHynkvFf/zupHyLnsx5WGkCMfTRV7EoApgWhYcUEGwYhNNEBZU3wrxCAmEla50ZovrJ7oTe76BRdKqVe3Tau3urFy/ztspggNwCCrABhegDm5BAzQBBk/gBbyCN+PZeDc+ja9ptGDkM/tgBsbPL1mvqUQ=</latexit>W

d-order tensor<latexit sha1_base64="v9ydZp9xFzlXLmdmcHIRqQI/Afk=">AAACAnicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYB2SXMDuZTYbMzC4zs0JY0vkNtlrbia0/YumfOJtsYRIPXDiccy/3cMKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMlHvsBmRDDPOtN+perW3BnQKvEKUoUCzX7lxx/EJBVUGsKx1j3PTUyQYWUY4XRa9lNNE0zGeEh7lkosqA6yWeYpOrfKAEWxsiMNmql/LzIstJ6I0G7mEfWyl4v/eb3URDdBxmSSGirJ/FGUcmRilBeABkxRYvjEEkwUs1kRGWGFibE1LXwJRd6Jt9zAKmnXa95lrf5wVW3cFu2U4BTO4AI8uIYG3EMTWkAggRd4hTfn2Xl3PpzP+eqaU9ycwAKcr1/OOpgx</latexit>‣ Global structure information lost by flattening image‣ How to apply TN to large image task?
‣ Only flatten local small patches‣ Hierarchical TN model
Tensor Networks for Medical Image Classification (Selvan et al., MIDL 2020)
101

d 7! p<latexit sha1_base64="7hYzq4AhBpa0Pb7Iq9kVg5m2Yvg=">AAAB93icbVA9SwNBEJ3zM8avqKXNYRCswl0UtAzaWCZgPiA5wt5mLlmyu3fs7gnhyC+w1dpObP05lv4TN8kVJvHBwOO9GWbmhQln2njet7OxubW9s1vYK+4fHB4dl05OWzpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2OH2Z++xmVZrF8MpMEA0GGkkWMEmOlRtIvlb2KN4e7TvyclCFHvV/66Q1imgqUhnKiddf3EhNkRBlGOU6LvVRjQuiYDLFrqSQCdZDND526l1YZuFGsbEnjztW/ExkRWk9EaDsFMSO96s3E/7xuaqK7IGMySQ1KulgUpdw1sTv72h0whdTwiSWEKmZvdemIKEKNzWZpSyimNhN/NYF10qpW/OtKtXFTrt3n6RTgHC7gCny4hRo8Qh2aQAHhBV7hzZk4786H87lo3XDymTNYgvP1C3//k44=</latexit>pd

TN for Image Classification
global structure by squeeze‣ Captureoperation‣ Single layer to multi-layer TNs
performance with‣ Comparable
DenseNet but fewer parameters

Tensor Networks for Medical Image Classification (Selvan et al., MIDL 2020)102

Supervised Learning with Projected Entangled Pair Stateslayer PEPS for supervised‣ One
learning‣ CNN + PEPS as learning model
7% parameters of CNN+MLP

From Probabilistic Graphical Models to Generalized Tensor Networks for Supervised Learning(Glasser, arXiv 2019)
Supervised Learning with Projected Entangled Pair States (Chen et al., arXiv 2020)
103

Machine Learning by Hierarchical Tensor Networks‣ Supervised learning with tree tensor networks

Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure (Liu et al., 2019)104

Learning Tensor Network StructureStandard models may not be the most compressive one‣ How to choose an optimal tensor network structure is necessary‣ Can we learn an optimal TN structure by data?
given an 9-order tensor, there are more than 68‣ Challenge:
BILLION candidates
105

Optimization Problem‣ The TN structures can be fully described by its adjacency matrice
Collection of core tensors

Adjacency matrix‣ Find an optimal A such thatcompression ratio such that approximation error is‣ Maximize
upper-bounded106

Experiment and Discussion
natural images are random selected from LIVE dataset [Sheikh et al.,‣ 102006], and reshaped as order-8 tensors
The learned topology have more complex structures than simple ones like‣ line,tree or cycles.

structures could significantly boost the expressive power of TN‣ Optimal
decomposition‣ Near-optimal structures are sufficient to outperform the simple ones
Evolutionary Topology Search for Tensor Network Decomposition (Li et al., ICML 2020)107

Discussions‣ TNs are tools for data modeling, representing model
parameters and function approximation‣ Theory shows TNs have expressive power similar to DNNs
‣ Practically, performance of TNs for ML tasks is less attractive‣ If TNs can be developed as a general ML model？
‣ Are there any advantages of TNs from perspective ofrobustness and interpretability？

108

Acknowledgements

Chao Li(RIKEN AIP)

Ming Hou(RIKEN AIP)

Ming Hou

Xiao-yang Liu（Columbia University）

Andrzej Cichocki
mputer Science, Laval University, Quebec, Canada, G1V 0A6(Skoltech)
1 (418) 5599097 • B ming.hou.1@ulaval.canCesar F. Caiafa(CONICET)

Nationality: China

Tatsuya Yokota
(NIT)
109

Andong Wang
(GDUT)