リンクされたイメージを表示できません。ファイルが移動または削除されたか、名前が変更された可能性があります。リンクに正しいファイル名と場所が指定されていることを確認してください。

Approximate Bayesian Inference Team

Mohammad Emtiyaz KhanGoals and Challenges
Long-Term Goal: To discover fundamental principles of learning, and design AI systems
that continue to learn and improve throughout their life (like humans)
Living beings can learn throughout their life from small chunks of data in a non-stationary
world, but deep learning requires a large amount of data from a stationary world. Ourcurrent research focuses on reducing this gap.

Standard Deep Learning

Deep Continual Learning

Main Idea: We use Bayesian principles to enable human-like learning of deep networks.
- New methods for uncertainty estimation in deep networks- Convert deep networks to Gaussian process to use them as prior for life-long learning.

From Deep Neural Networks to Gaussian Processes

Uncertainty Estimation for Large Deep Networks

Using Gaussian posterior approximations, we can convert Neuralnetwork to Gaussian processes (GP). This enables us to use
deep networks as functional priors.

• A new scalable algorithm, called VOGN, to estimate uncertainty
for large deep-learning problems.

Step A: Find a Gaussian
posteriorapproximation

AB

Step B: Find a linear
model whose posterior
is equal to the Gaussian
approximation

C• For the first time, we can train ResNet-18 on ImageNet with 128
GPUs. We achieve similar performance to Adam/SGD in about
the same number of epochs while preserving the benefits ofBayesian principles:• predictive probabilities are well-calibrated• uncertainties on out-of-distribution data are improved• continual-learning performance is boosted

Step C: Find a GP
whose predictions are
the same as those ofthe linear model.

Using this approach, we can identify important data examples.
MNIST
Not important
Important

CIFAR
Not important
Important
Bayesian Uncertainty Estimation in Image Segmentation

Segmentation (left) and its predictive entropy (right) on Cityscapes dataset
Approximate Inference Turns Deep Networks into Gaussian Processes. Khan ME, Immer A,Abedi E, Korzepa M (NeurIPS 2019).

Practical Deep Learning with Bayesian Principles. Osawa K, Swaroop S, Jain A, Eschenhagen R,
Turner RE, Yokota R, Khan ME (NeurIPS 2019).

A New Bayesian Learning Rule for
Mixture of Exponential Family

Inference Networks for GaussianProcess Models

A Generalization Bound for Online
Variational Inference• Bayesian Learning rule is a generallearning rule from which many
learning-algorithms can be derived• Deep learning (SGD and Adam)• Least-squares, Kalman filters etc.
• And many more..
• We extended the application of rule tomixture of exponential family, enabling
us to derive, e.g., an ensemble
Newton method.

• This work uses neural networks toapproximate the posterior distributionof Gaussian process models.

• This work derives new bounds formethods that perform online
variational inference.

• The main idea is to use a ”functional”
mirror descent algorithm.

• We considers a variety of suchalgorithms and showed that thegeneralization error is bounded,even though each step of thealgorithm is approximate.

Fast and Simple Natural-Gradient Variational Inference with
Mixture of Exponential-family Approximations. Lin W, KhanME, Schmidt M (ICML 2019).

• The updates can be approximatedusing a neural networks as theposterior distribution.

Scalable Training of Inference Networks for GaussianProcess Models. Shi J, Khan ME, Zhu J (ICML 2019).

• Best paper award at ACML 2019.
A Generalization Bound For Online Variational Inference.
Chérief-Abdellatif, BE, Alquier, P and Khan, ME (ACML2019).