TPFN: Applying Outer Product along Time to Multimodal Sentiment
Analysis Fusion on Incomplete Data

Binghua Li‡†§

Chao Li†§Feng Duan‡

Ning Zheng†

Qibin Zhao†

Email Address: chao.li@riken.jp§ Equal Contribution† RIKEN Center for Advanced Intelligence Project (AIP),Tokyo, Japan‡ College of Artificial Intelligence, Nankai University, Tianjin, China
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

1 / 13

Multimodal Sentiment Analysis with Incomplete Data
Missing entries
Highly Positive

Text Text
Text

3𝑓

Audio-3
Highly Negative
VideoChao Li (RIKEN-AIP)

Time-steps
ECCV2020

chao.li@riken.jp

2 / 13

Prior Arts

Out-product-based Methods: TFN (A. Zadeh et al, 2017), LMF (Z. Liu et al, 2018), HFFN (S.
Mai et al, 2019), HPFN (M. Hou et al, 2019), T2FN (P.P. Liang et al 2019).

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

3 / 13

Prior Arts

Out-product-based Methods: TFN (A. Zadeh et al, 2017), LMF (Z. Liu et al, 2018), HFFN (S.
Mai et al, 2019), HPFN (M. Hou et al, 2019), T2FN (P.P. Liang et al 2019).

REMARK
Outer-product riches the modal interaction in the data fusion phase.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

3 / 13

Prior Arts

Out-product-based Methods: TFN (A. Zadeh et al, 2017), LMF (Z. Liu et al, 2018), HFFN (S.
Mai et al, 2019), HPFN (M. Hou et al, 2019), T2FN (P.P. Liang et al 2019).

REMARK
Outer-product riches the modal interaction in the data fusion phase.
Most of the studies assume that the collected data is complete and ideal.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

3 / 13

Prior Arts

Out-product-based Methods: TFN (A. Zadeh et al, 2017), LMF (Z. Liu et al, 2018), HFFN (S.
Mai et al, 2019), HPFN (M. Hou et al, 2019), T2FN (P.P. Liang et al 2019).

REMARK
Outer-product riches the modal interaction in the data fusion phase.
Most of the studies assume that the collected data is complete and ideal.

Question
How to accurately give the prediction when the dataset (or features) are incomplete?

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

3 / 13

Cont’dT2FN (P.P. Liang et al 2019): “Learning representations from imperfect time series data via
tensor rank regularization”.
OUTPUT(3) Tensor Weights.

W(4) Low-rankness of the fused features.
!

(2) Sum among time-steps.

(1) Modal out-product

!

INPUT
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

4 / 13

Cont’d
T2FN (P.P. Liang et al 2019): “Learning representations from imperfect time series data via
tensor rank regularization”.
OUTPUT(3) Tensor Weights.

W

NOTE(4) Low-rankness of the fused features.
!

Explicit out-products are applied.

(2) Sum among time-steps.

(1) Modal out-product

!

INPUT
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

4 / 13

Cont’dT2FN (P.P. Liang et al 2019): “Learning representations from imperfect time series data via
tensor rank regularization”.
OUTPUT(3) Tensor Weights.

WNOTE(4) Low-rankness of the fused features.
!

(2) Sum among time-steps.

Explicit out-products are applied.
Time-steps are averaged in each sample.

(1) Modal out-product

!

INPUT
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

4 / 13

Cont’d
T2FN (P.P. Liang et al 2019): “Learning representations from imperfect time series data via
tensor rank regularization”.
OUTPUT(3) Tensor Weights.

WNOTE(4) Low-rankness of the fused features.
!

(2) Sum among time-steps.

Explicit out-products are applied.
Time-steps are averaged in each sample.
High-dimensional tensor weights.

(1) Modal out-product

!

INPUT
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

4 / 13

Cont’dT2FN (P.P. Liang et al 2019): “Learning representations from imperfect time series data via
tensor rank regularization”.
OUTPUT(3) Tensor Weights.

WNOTE(4) Low-rankness of the fused features.
!

(2) Sum among time-steps.

Explicit out-products are applied.
Time-steps are averaged in each sample.
High-dimensional tensor weights.

Goal of Our work
(1) Modal out-product

Implicit out-product.
Higher-order statistics of temporal dynamic.

!

Dimension reduction of the tensor weights.

INPUT
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

4 / 13

One Step Further along T2FN: Time Product Fusion Network (TPFN)CP Decomposition
T1

LSTM

T1
T2

W
1

T1

T2

T2T3

TPFN consists of:

1T3LSTM(a) Pre-process LSTM

T2(b) Fusion: TPFoutput(d) Low-rank regularization

I just watched this aweek ago mostspecifically July 27…

LSTM(c) Inference: LIM(a) Pre-process LSTM(b) Modal Fusion: TPF(c) Inference Network: LIM(d) Efficient Low-rank Regularization
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

5 / 13

TPF: Outer Product along Temporal Domain
Text Text
Text

2. Window sliding

AudioVideo

3. Summation
0. Concatenation
1Fused Features

1. outer-productChao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

6 / 13

TPF: Outer Product along Temporal Domain
Text Text
Text

2. Window sliding

AudioVideo

3. Summation
0. Concatenation
1

Fused Features

1. outer-product

Are the explicit out-products really necessary?
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

6 / 13

The Answer Is No. – Low-rank Inference Module (LIM)
Low-rank decomposition

Complexity

Concatenated features
Weights

LIM:

fused features

whereLow-rank

weights

M
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

7 / 13

Low-rank Regularization before Outer-products
In (P.P. Liang et al 2019), it is pointed out that the low-rank regularization on the fused features
M is of importance in the case of incomplete data.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

8 / 13

Low-rank Regularization before Outer-products
In (P.P. Liang et al 2019), it is pointed out that the low-rank regularization on the fused features
M is of importance in the case of incomplete data.

REMARKIn T2FN (P.P. Liang et al 2019), the Frobenius norm of M is regularized. It is therefore impliedthat the explicit outer-products are inevitable in the model.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

8 / 13

Low-rank Regularization before Outer-products
In (P.P. Liang et al 2019), it is pointed out that the low-rank regularization on the fused features
M is of importance in the case of incomplete data.

REMARKIn T2FN (P.P. Liang et al 2019), the Frobenius norm of M is regularized. It is therefore impliedthat the explicit outer-products are inevitable in the model.

Our Claim
Briefly speaking, the nuclear norm (i.e., convex surrogate of the rank) of M is upper-boundedby the Frobenius norm of the concatenated features from all modalities and time-steps.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

8 / 13

Low-rank Regularization before Outer-products
In (P.P. Liang et al 2019), it is pointed out that the low-rank regularization on the fused features
M is of importance in the case of incomplete data.

REMARKIn T2FN (P.P. Liang et al 2019), the Frobenius norm of M is regularized. It is therefore impliedthat the explicit outer-products are inevitable in the model.

Our Claim
Briefly speaking, the nuclear norm (i.e., convex surrogate of the rank) of M is upper-boundedby the Frobenius norm of the concatenated features from all modalities and time-steps.

REMARKThe regularization term is applied before the outer-product. The explicit out-products is avoided
in our model.
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

8 / 13

Experiments: Datasets and Settings

Datasets: MOSI (A. Zadeh et al, 2016) and MOSEI (A. Zadeh et al, 2018).

MOSEI
MOSI

Number of Samples
TrainValTest
15,290 2,291 4,832
1,284229686

Size of Features
Acoustic Visual Language
7435300
520300Missing Pattern
Random Drop (RD)Structured Drop (SR): Randomly drop the whole modalities from each time-step.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

9 / 13

Performance (ACC-2) on CMU-MOSITFPN/reg: TPFN without low-rank regularization.
Task

RDSDMethod
TFN
LMF
HPFN
T2FN
TPFN/reg(ours)
TPFN(ours)
TFN
LMF
HPFN
T2FN
TPFN/reg(ours)
TPFN(ours)
Chao Li (RIKEN-AIP)

Low
0.73610.73460.75650.77690.76380.79150.73170.73460.74630.74780.76820.7594

Medium0.71720.73170.69820.71130.75940.76090.68800.71280.71860.71420.72880.7434High
0.44750.52180.55680.59620.58450.65590.57580.5976
0.61510.61370.61510.6516

ECCV2020

Params759,424
2,288
4,622,039
19,737
8,652
19,488
390,784792
1,168,247
19,737
11,3607,344
chao.li@riken.jp

10 / 13Performance (ACC-2) on CMU-MOSEITFPN/reg: TPFN without low-rank regularization.
Task

RD

SD

Method
TFN
LMF
HPFN
T2FN
TPFN/reg(ours)
TPFN(ours)
TFN
LMF
HPFN
T2FN
TPFN/reg(ours)
TPFN(ours)
Chao Li (RIKEN-AIP)

Low
0.71950.73070.73710.73940.73750.74110.72950.73130.73110.73500.74370.7386

Medium0.71930.72330.71890.73820.72970.73670.71210.70670.72450.72950.73010.7382High
0.67050.66840.71190.71040.71560.73340.69680.70570.70030.71730.70070.7301

ECCV2020

Params1,353,856
1,208
1,295,895
18,785
14,24016,842759,424
1,304
1,296,423
9,945
7,056
5,796
chao.li@riken.jp

11 / 13

Take-away Messages

The outer-product is important to obtain rich interaction among modalities.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

12 / 13

Take-away Messages

The outer-product is important to obtain rich interaction among modalities.
The benefit by outer-products is also given through time-steps.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

12 / 13

Take-away Messages

The outer-product is important to obtain rich interaction among modalities.
The benefit by outer-products is also given through time-steps.
Low-rank tensor decomposition is a promising tool to tackle the dimension exploration issue
by outer-products.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

12 / 13

Take-away Messages

The outer-product is important to obtain rich interaction among modalities.
The benefit by outer-products is also given through time-steps.
Low-rank tensor decomposition is a promising tool to tackle the dimension exploration issue
by outer-products.
The low-rank regularization on the fused features works well to cope with the incompletenessissue.

Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

12 / 13

TPFN: Applying Outer Product along Time to Multimodal Sentiment
Analysis Fusion on Incomplete Data

Binghua Li‡†§

Chao Li†§Feng Duan‡

Ning Zheng†

Qibin Zhao†

Email Address: chao.li@riken.jp§ Equal Contribution† RIKEN Center for Advanced Intelligence Project (AIP),Tokyo, Japan‡ College of Artificial Intelligence, Nankai University, Tianjin, China
Chao Li (RIKEN-AIP)ECCV2020

chao.li@riken.jp

13 / 13