FY2023/2023年度

Approximate Bayesian Inference Team
Mohammad Emtiyaz Khan
近似ベイズ推論チーム

Overview and Goals
Goal: to design AI systems that can learn and improve continually throughout theirlives, just like humans and animals. Currently, deep learning (DL) requires a large
amount of data which is costly and rigid, leading to a system that is unable to quickly
adapt. We aim to fix this with a new principle which we call Bayes-duality principles.

Standard Deep Learning

Continual lifelong learning

Summary for the year 2023 (paper number shown in green boxes)
1. We propose memory-perturbation to understand sensitivity of algorithms to data.
2. We improve continual learning on ImageNet by adding memory (<10% of data).
3. We use memory to improve sequential learning in sparse Gaussian Process.
4. We give a simple recipe to derive variational Bayes updates.
5. We improve sample efficiency of RL by using a functional regularization.
6. We propose a structured inference method for Neural Processes.
7. We discuss challenges in annotating African languages.
8. We infer properties of Markov chains with sparse connection graphs.
9. We recover known sampling algorithms by means of information geometry.
10.We propose a reduction method for identity testing of Markov chains.

Memory-Perturbation Equation

Improving Continual Learning by Gradient Reconstructions

Problem: Understanding model’s sensitivity to training data is crucial, but can bechallenging and costly, especially during training (including existing work on influence
measures, for example, Koh and Liang (2017)).

Problem: There are many well-known continual learning algorithms that work well,
but is there a principled way to combine them?

Contribution: We use Bayesian principles to derive the Memory Perturbation
Equation (MPE) which unifies and generalizes sensitivity measures to a wide-varietyof algorithms and unravels useful properties regarding their sensitivities.

Contribution: Following Khan & Swaroop (2021), we design a regularizer thatfaithfully reconstruct gradients of past data by combining Memory Replay, ElasticWeight Consolidation, and Functional Regularization.

Deviation in posterior parameters (Δ) = Natural gradient of the example removed
Δ ∝ PredError x PredVariance

Sensitivities can be used to predict
generalization error by using training data alone

Result: We often obtain > 80%of the batch performance byusing a memory of < 10% ofthe past data.
On ImageNet-1000, by using
7.5% past data, we achieve
around 76% accuracy, 11% lessthan batch accuracy 87%.

FMNIST, LeNet5

CIFAR10, Resnet20

1. Nickl, Xu*, Tailor*, Moellenhoff, Khan, The Memory Perturbation Equation: Understanding
Model’s Sensitivity to Data, NeurIPS 2023.

2. Daxberger, Swaroop, Osawa, Yokota, Turner, Hernandez-Lobato, Khan, Improving Continual
Learning by Accurate Gradient Reconstruction of the Past, TMLR 2023.

Other Works
Sequential learning with Gaussian processes: We present a method to avoid
accumulation of errors during sequential learning by using the recently proposed dual
sparse variational GP. The method enables accurate inference for generic likelihoods
and improves learning by actively building and updating a memory of past data.

Tagging for Diverse African Languages: We discuss the challenges in annotating POS
for African languages using the Universal Dependencies guidelines and conduct
extensive part-of-speech baseline experiments using various models.

3. Chang, Verma, John, Solin, Khan, Memory-based dual GP for sequential learning. ICML 2023

7. Dione, Adelani, Nabende, Alabi, Sindane, Buzaaba, Muhammad, Emezue, Ogayo, Aremu, Gitau,
MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages, ACL 2023

A three-step Recipe for Variational Bayes: We give a simple recipe to identify theposterior form by explicitly looking for linearity with respect to expectations of wellknown distributions. This enables us to directly write the update by simply “readingoff” the terms in front of those expectations.

Estimation of Mixing Properties of Markov Chains: We derive confidence intervals tolearn transition matrices of Markov chains, yielding improved estimators for some ofits mixing properties. Notably, our analysis goes beyond the worst-case scenario byleveraging the sparsity of the connection graph.

4. Khan ME. Variational Bayes Made Easy, AABI 2023

8. Wolfer, Empirical and Instance-Dependent Estimation of Markov Chain and Mixing Time,
Scandinavian Journal of Statistics 2023

Replacing Target Networks by Functional Regularization: We propose an explicit
Functional Regularization to replace target networks and empirically demonstrate itleads to better sample efficiency and performance improvements.

Reversiblizations of Markov Chains: We propose systematic projection schemes toreversiblize Markov chains. Various divergences lead to many popular Markov Chain
Monte Carlo algorithms, including a recently popularized Barker dynamics algorithm.

5. Piche, Thomas, Marino, Pardinas, Marconi, Pal, Khan, Bridging the Gap Between Target
Networks and Functional Regularization, TLMR 2023

9. Choi, Wolfer, Systematic Approaches to Generate Reversiblizations of Markov Chains,IEEE Transactions on Information Theory 2023

Structured inference networks for Neural Processes: We enrich the latent variableof Neural Processes with structured priors (e.g. with multiple modes, heavy-tails,etc.) and provide a framework that directly translates such distributional
assumptions into an aggregation strategy for the context set.
6. Tailor, Khan, Nalisnick, Exploiting inferential structure in neural processes, UAI 2023

Identity Testing of Markov Chains: We show that the problem of identity testing forreversible Markov chains can be reduced to the same for symmetric Markov chains.
Our streamlined approach not only recovers the state-of-the-art sample complexity forthe problem but also extends to related problems.
10. Wolfer, Watanabe, Geometric Reduction for Identity Testing of Reversible Markov Chains,Geometric Science of Information 2023