BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Towards a Regression using Tensors
Ming Hou

February 27, 2014

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Outline1

Background
Linear Regression
Tensorial Data Analysis

2Tensor Basics
Definition
Tensor Operation
Tensor Decomposition

3

Generalized Linear Tensor Regression
Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

4

Future Work Plan
Future Work Plan
Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Linear Regression
Tensorial Data Analysis

Classical Linear Regression

Predicte.g. speed,road conditions, weather ⇒ traffic accidents rates
Identify the key predictors
e.g mental disease status ⇒ the regions of brain
Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Linear Regression
Tensorial Data Analysis

Multi-Dimensional Array Data (Tensors)
Neuroscience
EEG data: (time × frequency × electrodes)
fMRI data: (time × x axis × y axis × z axis)

Visionimage (video) data:(pixel × illumination × expression × viewpoints)

Chemistryfluorescence excitation-emission data:(samples × emission × excitation)

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Linear Regression
Tensorial Data Analysis

Brain Imaging Data Analysis
Mental health disorders are difficult to diagnose and treat
Physiology of brain is not well understood
Neuroimaging can explain the brain physiology
Several types of neuroimaging EEG MRI fMRI

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Linear Regression
Tensorial Data Analysis

Brain Imaging Data Analysis using Regression
Goal is to find association between brain images and clinicaloutcomes.
Formulate as regression problem
clinical outcome as response
brain image (multi-dimensional array) as tensor predictor

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Linear Regression
Tensorial Data Analysis

Limitation of Classical Regression
Naive approach: turning an image array as vector predictor
e.g. a fMRI image: 4D array with size 256 × 256 × 256 × 100
yields a huge number of parameters (167 millions!)
ignores spatial and temporal correlation

New method: treat each fMRI observation as one tensor
predictor in regression model

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

What is Tensor?

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work PlanDefinition
Tensor Operation
Tensor Decomposition

What is Tensor? con’t

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

What is Tensor? con’t
A tensor is formally denoted as X ∈ RI1 ×I2 ×...×IN
generalization of vector and matrix
represented as multi-dimensional array

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

FibersMing HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

SlicesMing Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

Matricization (Unfolding)
Convert a tensor to a matrix

Tube fibers are rearranged into the columns of a matrix

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

Matricization (Unfolding) Example

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

The n-Mode Multiplication
Let X ∈ RI ×J×K , B ∈ RM×J , the 2-mode product of X with B isdefined by
Y = X ×2 B ∈ RI ×M×KElementwise
yimk =X

xijk bmjj

In matrix form
Y(2) = BX(2)Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor OperationTensor Decomposition

The n-Mode Multiplication Example

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

Rank-1 Tensor
3-way outer product
X =a◦b◦c
Elementwise
xijk = ai bj ck

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

CANDECOMP/PARAFAC Decomposition

X ≈RX

λr ar ◦ br ◦ crr =1

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

CANDECOMP/PARAFAC Decomposition con’t

Define factor matrix A ∈ RI ×R , B ∈ RJ×R and C ∈ RK ×R
X ≈RX

λr ar ◦ br ◦ cr ≡ [λ; A, B, C]r =1

xijk ≈

RX

λr air bjr ckr

r =1
Ming HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Definition
Tensor Operation
Tensor Decomposition

Tucker decomposition

Defined by factor matrix A ∈ RI ×R , B ∈ RJ×S and C ∈ RK ×T ,and core tensor G ∈ RR×S×TX ≈ G ×1 A ×2 B ×3 C ≡ [G; A, B, C]
xijk =R X
S XTX

grst air bjs cktr =1 r =1 r =1
Ming HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

Generalized Linear Regression Model
The standard linear regression model x ∈ Rp , y = β T x + α + ε,ε ∼ N (0, σ 2 ) can be writtenµ = βT x + αy ∼ N (µ, σ 2 )where µ = E(Y |x)
A generalized linear regression model (GLM) extends this tog (µ) = β T x + αy ∼ EF(µ, φ)EF(µ, φ) is any exponential family distribution (e.g. Normal,Poisson, Binomial)g (·) is any smooth monotonic link function
β T x + α(= η) is the linear predictor
Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression ModelAttention Deficit Hyperactivity Disorder Data Analysis

Generalized Linear Regression Model con’t
In classical GLM Y belongs to an exponential family with PMF
p(y |θ, φ) = exp{y θ − b(θ)+ c(y , φ)}a(φ)

The GLM relates x ∈ Rp to the mean µ = E(Y |x) byg (µ) = η = α + β T x
The GLM for the matrix predictor X given byg (µ) = η = α + γ T z + β1T Xβ2

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

Generalized Linear Regression Model for Tensor Predictor
The GLM with the systematic part for tensor predictor given byg (µ) = η = α + γ T z+ < B, X >D-dimensional tensor predictor X ∈ Rp1 ×···×pD
D-dimensional coefficient tensor B ∈ Rp1 ×···×pD
Q
B has D
d=1 pd parameters, which is ultrahigh dimensionaland far exceeds sample size

Ming HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

Generalized Linear CP Tensor Regression
Univariate outcome Y belongs to exponential familyTensor covariate X ∈ Rp1 ×···×pD
Assume coefficient tensor B has a rank-R decomposition[B1 , ..., BD ] where Bd ∈ Rpd ×R
Generalized linear CP tensor regression model (Zhou et al.
2013) with the systematic part given byg (µ) = η = α + γ T z+ <RX

(r )

(r )

β1 ◦ · · · ◦ βD , X >r =1= α + γ T z+ < (BDMing Hou···

B1 )1R , vec(X ) >Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression ModelAttention Deficit Hyperactivity Disorder Data Analysis

Generalized Linear CP Tensor Regression con’t
Generalized linear CP tensor regression model given byg (µ) = η = α + γ T z+ < (BD···

B1 )1R , vec(X ) >substantial
PD reduction in dimensionality to the scale ofR × d=1 pde.g For a 128-by-128-by-128 MRI image, the dimensionality
reduce from 2,097,157 to 1157 using rank-3 decomposition
Zhou et al.(2013) showed that this low rank tensor model
could provide a sound recovery of many low rank signals

Ming HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

Estimation

Given n iid data {(yi , Xi , zi ), i = 1, ..., n} the log-likelihood`(α, γ, B1 , ..., BD ) =nX
yi θ − b(θ)
i=1a(φ)+nXc(yi , φ)i=1find the parameters (α, γ, B1 , ..., BD ) that maximizes this function

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression ModelAttention Deficit Hyperactivity Disorder Data Analysis

Estimation con’t
Generalized linear CP tensor regression model given byg (µ) = η = α + γ T z+ < (BD···

B1 )1R , vec(X ) >A key observation is although g (µ) is not linear in (B1 , ..., BD )jointly, it is linear in each Bd separatelyWhen updating Bd ∈ Rpd ×R , the inner product part can be writtenas< Bd , X(d) (BD · · · Bd+1 Bd−1 · · · B1 ) >this yields the block relaxation algorithm, which converges to astationary point
Ming HouTowards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

Sparsity Regularization

Maximize a regularized log-likelihood function`(α, γ, B1 , ..., BD ) −pd
D X
R XX(r )Pλ (|βdi |, ρ)

d=1 r =1 i=1

scalar penalty function Pλ (|β|, ρ)
power family Pλ (|x|, ρ) = ρ|β|λ , λ ∈ (0, 2]
in particular lasso (λ = 1)

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Generalized Linear Tensor Regression Model
Attention Deficit Hyperactivity Disorder Data Analysis

ADHD-200 Data Results[taken from (Zhou et al. 2013)]
Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Future Work Plan

Future Work Plan
Extending the linear CP/Tucker tensor regression model tothe linear H-Tucker tensor regression model
like CP model, the number of parameters is free fromexponential dependence on D
preserve the flexibility of Tucker model

Comparing the performance of different tensor regression
models (Tucker, H-Tucker) when applying different
regularization approaches (sparsity regularization, trace normregularization)

Ming Hou

Towards a Regression using Tensors

BackgroundTensor Basics
Generalized Linear Tensor Regression
Future Work Plan

Future Work Plan

Future Work Plan con’t

Finding the appropriate model and algorithm to address themulti-block tensor regression problems
Combing the kernel concept and partial least squares (PLS)
techniques to deal with tensor (multi-block tensor) regression
problem
Applying tensor regression approaches listed above to theapplications such as neuroimaing data analysis, brain signal
data analysis to test if the improved performance can beachieved

Ming Hou

Towards a Regression using Tensors