FY2022/2022年度

Approximate Bayesian Inference Team
Mohammad Emtiyaz Khan
近似ベイズ推論チーム

Overview and Goals
Goal: AI systems that can learn and improve continually throughout their lives, just like humans and animals. Currently, deeplearning (DL) requires a large amount of data which is costly and rigid, leading to a system that is unable to quickly adapt. We
aim to fix this with a new learning paradigm based on Bayesian principles.
Summary of our research highlights in the year 2022 (paper number shown in green boxes)-1 We show a robust deep-learning method is related to Bayes and add uncertainty to it (top 75/5000 papers at ICLR2023)
2- We extend the Bayesian learning rule using Lie-groups, simplifying gradient computations and eliminating retractions.
3- We design a generally unimprovable procedure for relaxation time estimation in non-reversible Markov chains.
-4 We simplify momentum-based Riemannian optimization over positive-semi-definite matrix submanifold.
5- We use low-rank matrix completion techniques to reconstruct partially-observed high-dimensional time series.
-6-10 Analysis of MMD estimation, Neural processes, A Dataset for African Languages, Deviation inequalities, and more …!

Standard Deep Learning

Deep Continual Learning

SAM as an Optimal Relaxation of Bayes

The Lie-Group Bayesian Learning Rule

Problem: Sharpness-Aware Minimization (SAM) by Foret et al. improve significantly
over SGD but the reasons behind its success are unclear.

Problem: Many popular algorithms can be derived from the Bayesian learning ruleof Khan and Rue (2021) but the rule can be difficult to apply, e.g., gradients aredifficult to compute, and steps can lead to invalid distributions (e.g., –ve variances).

Solution: We show that SAM is equivalent to an optimal relaxation of Bayes obtainedby using Fenchel biconjugate (left figure). SAM can be seen as “smoothing” theobjective using a “posterior variance” and always upper-bounding Bayes (right).
Our paper [1] is among top-5% of all accepted papers (75 out of 5000 submissions)Contribution #2: Our Bayesian-SAM improves
accuracy over SGD (by 8%), SAM-Adam (by 10%),and Adam (by 22%), but also improves AUROC(for CIFAR-100 using ResNet-20 with 270Kparams).
Accuracy
AUROC+8%+10%+22%

Solution: We extend the rule by using Lie-groups which solves the above problems:
gradients can always be obtained by reparameterizations, and steps always stay on
the manifold. Fisher computation is also simplified and only need to be done once.
Conceptual algorithm:

Result: Reduces to the update [3] when applied to exp. fam. and linearized.
SAM isoverconfident(narrow whiteregion)

Additive

MultiplicativeAffine

Multiplicative Learning (nll=0.058)

Bayesian-SAM
fixes this(more blurryaway from thedata)

1. Moellenhoff, Khan, SAM as an Optimal Relaxation of Bayes, ICLR 2023 (oral, notable top-5%)

Additive Learning (nll=0.083)

2. Kiral, Moellenhoff, Khan, The Lie-Group Bayesian Learning Rule, AISTATS 2023.

Estimation of Relaxation Time in Non-Reversible Markov Chains
Problem: Estimate the relaxation time of an unknown, ergodic, non-reversible Markovchain from a single trajectory of length 𝒎𝒎 started from an arbitrary state.
Application: This is useful in reinforcement learning, Markov Chain Monte Carlodiagnostics, and deriving generalization bounds with Markov-dependent data.

Contributions: Prove there exists an estimatorforsuch that for any ergodic chain, when

where is the minimum stationary probability, it holds w.h.p. that
Upper bound matches known lower bounds, thus generally unimprovable.
 Design algorithm with fully empirical confidence intervals that decay in.
.

3. Wolfer and Kontorovich, Improved Estimation of Relaxation Time in Non-reversible Markov Chains, arXiv 2022.

Other Works
Simplify momentum-based Riemannian optimization: We consider the specific caseof a submanifold containing symmetric positive-definite matrices. The method uses ageneralized version of local coordinates which “trivializes” the Fisher matrix.
4. Lin, Duruisseaux, Leok, Nielsen, Khan, Schmidt, Practical Structured Riemannian Optimization
with Momentum by using Generalized Normal Coordinates, NeurIPS 2022 Workshop on
Symmetry and Geometry in Neural Representation

High-dimensional time series completion: We use low-rank matrix completion
techniques to reconstruct partially observed high-dimensional time series and showthat periodicity or smoothness can even lead to faster rates than in the independentsetting.
5. Alquier, Marie, Rosier, Tight risk bound for high dimensional time series completion, EJS 2022

Improving Neural Process: We improve test-time inference for Neural Processes byincorporating and exploiting graphical-model structure among context points.
7. Tailor, Khan, Nalisnick, Exploiting Inference Structure in Neural Processes, UAI 2022 Workshop on
Tractable Probabilistic Modeling

Name-Entity Recognition (NER) dataset for Sub-Saharan African languages: We create
the largest human-annotated dataset called MasakhaNER 2.0, and analyze featuresthat contribute to cross-lingual transfer, giving large gains for 0-shot learning.
8. Buzaaba with many others, MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity
Recognition, EMNLP 2022

Deviation inequalities for stochastic approximation by averaging: We establish
deviation inequalities for separately Lipschitz functions of Markov chains belonging toa certain class we define, and which includes models of stochastic approximation byaveraging and non-averaging.

Finite sample properties of parametric MMD estimation: We tackle the problem ofuniversal estimation using a minimum distance estimator based on Maximum Mean
Discrepancy, and we show its robustness to both dependence and presence ofoutliers.

Prioritization of minibatches: We give empirical support for the hypothesis thatimproving calibration can help in prioritizing minibatches during training.

6. Chérief-Abdellatif, Alquier, Finite Sample Properties of Parametric MMD Estimation:
Robustness to Misspecification and Dependence, Bernoulli 2022

10. Tata, Gudur, Chennupati, Khan, Can calibration improve sample prioritization, NeurIPS 2022
Workshop on Has It Trained Yet

9. Fan, Alquier, Doukhan, Deviation inequalities for stochastic approximation by averaging,SPA 2022