Expectation Propagation fort-Exponential Family Using q-Algebra
Futoshi Futami
Issei Sato
Masashi Sugiyama
University of Tokyo University of Tokyo
RIKEN
RIKEN
RIKEN
University of Tokyo

2017, Nov.8 IBIS

Gaussian distribution• Plays central role in ML• E.g. Gaussian process
Sensitive to outliers
likelihood

Prior

1

Example of Gaussian process (GP) regression

If an outlier exists...
GP is sensitive to outliers...
2

Student-t distribution

Student-t (v=2) and Gaussian3

GP example（Student-t likelihood）
Replace Gaussian likelihood with Student-t

Student-t likelihood is less affected by outliers

Importance of Student-t as an alternative to Gaussian4

The problem of dealing with Student-t
Student-t is difficult to handle...
• Not a member of the Exponential family.
• We cannot use many useful relations whichmake Gaussian easy to use.
• Need to approximate Student–t further byanother distribution which is easy to handle（variational inference, EP, Laplace）

How to use Student-t directly as analternative to Gaussian?

5t-exponential family[N. Ding and S. V. N. Vishwanathan, 2010,2011]

Extend Exponential Familyt = 1.5• Deformed exponential

6Approximation for t-exponential family[N. Ding and S. V. N. Vishwanathan, 2010,2011]

exponential family

t-exponential family

Assumed Density Filtering

Assumed Density Filtering

Online algorithm

Online algorithm• The performance heavily depends on theorder in which data is processed.

• Sometimes disastrous results are obtained.

7

Expectation Propagation[T. Minka, 2001]

exponential family

Assumed Density Filtering
Data: permutation dependent

Property ofPropertyof Exponential
exponentialfamilyfamily• A product of same exponential
families yields the same unnormalized exponential family

Calculation throughnatural parameter

Expectation Propagation• Calculation can be performed
through natural parameter or
expected sufficient statistics

Data: permutation independent8

Approximation for t-exponential family[T. Minka, 2001][N. Ding and S. V. N. Vishwanathan, 2010,2011]

exponential family

t-exponential family

Assumed Density Filtering

Assumed Density Filtering

Data: permutation dependent

Data: permutation dependent

Calculation throughnatural parameter

Expectation Propagation
Data: permutation independent9

Problem of t-Exponential family

A product of t-exponential families does not yield
the same un-normalized t-exponential family

Calculation cannot be performed through naturalparameter or expected sufficient statistics

10Expectation Propagation[T. Minka, 2001][N. Ding and S. V. N. Vishwanathan, 2010,2011]

exponential family

t-exponential family

Assumed Density Filtering

Assumed Density Filtering

Data: permutation dependent

Data: permutation dependent

Calculation throughnatural parameter

Calculation throughnatural parameter

Expectation Propagation

Expectation Propagation

Data: permutation independentData: permutation independent11

Our Contribution
t-exponential family

Assumed Density Filtering
Data: permutation dependent①

Calculation throughnatural parameter②
Expectation Propagation

By using q-algebra①
Enabled natural parameterbased calculationin t-exponential family②
Extend ADF to EP in
t-exponential family

Data: permutation independent12

q-Algebra[H. Suyari and M. Tsukada, 2005] [L. Nivanen, A. Le Mehaute, and Q. A. Wang, 2003]

We can treat the pseudo additivity efficiently
by using q-algebra

13

Numerical ExperimentComparison of binary classification by using
Student-t Process and Gaussian Process

Student-t Process classifier

Gaussian Process classifierpriorlikelihood[H. C. Kim and Z. Ghahramani 2008 ]① Toy datasets

How outliers affect the decision boundary② UCI datasets
Compare the performance14

Experiment① toy experiment

Student–t is lessaffected by outliers
than Gaussian15

Experiment② UCI benchmark data

Better performance compared to Gaussian
process on many datasets
16

Summaryt-exponential family

Assumed Density Filtering
Data: permutation dependent①Calculation throughnatural parameter②Expectation Propagation

By using q-algebra①
Enabled natural parameterbased calculationin t-exponential family②
Extend ADF to EP in
t-exponential family

Data: permutation independent17