Query: bayesian neural networks reinforcement learning

Candidate papers:

abstract: Monte Carlo methods are revolutionizing the on line analysis of data in fields as diverse as financial modeling, target tracking and computer vision. These methods, appearing under the names of bootstrap filters, condensation, optimal Monte Carlo filters, particle filters and survival of the fittest, have made it possible to solve numerically many complex, non standard problems that were previously intractable. This book presents the first comprehensive treatment of these techniques, including convergence results and applications to tracking, guidance, automated target recognition, aircraft navigation, robot navigation, econometrics, financial modeling, neural networks, optimal control, optimal filtering, communications, reinforcement learning, signal enhancement, model averaging and selection, computer vision, semiconductor design, population biology, dynamic Bayesian networks, and time series analysis. This will be of great value to students, researchers and practitioners, who have some basic knowledge of probability. Arnaud Doucet received the Ph. D. degree from the University of Paris XI Orsay in 1997. From 1998 to 2000, he conducted research at the Signal Processing Group of Cambridge University, UK. He is currently an assistant professor at the Department of Electrical Engineering of Melbourne University, Australia. His research interests include Bayesian statistics, dynamic models and Monte Carlo methods. Nando de Freitas obtained a Ph.D. degree in information engineering from Cambridge University in 1999. He is presently a research associate with the artificial intelligence group of the University of California at Berkeley. His main research interests are in Bayesian statistics and the application of on line and batch Monte Carlo methods to machine learning. Neil Gordon obtained a Ph.D. in Statistics from Imperial College, University of London in 1993. He is with the Pattern and Information Processing group at the Defence Evaluation and Research Agency in the United Kingdom. His research interests are in time series, statistical data analysis, and pattern recognition with a particular emphasis on target tracking and missile guidance.
author_names: A Doucet

Nando de Freitas

Neil J Gordon
title: Sequential Monte Carlo Methods in Practice
venue: Statistics for Engineering and Information Science
year: 2001

abstract: Scalable and effective exploration remains a key challenge in reinforcement learning (RL) While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME) an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.
author_names: Rein Houthooft

Xi Chen

Yan Duan

John Schulman

Filip De Turck

P Abbeel
title: VIME: Variational Information Maximizing Exploration
venue: NIPS
year: 2016

abstract: Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state of the art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state of the art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high dimensional toy functions, support vector machines, feed forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.
author_names: Stefan Falkner

Aaron Klein

Frank Hutter
title: BOHB: Robust and Efficient Hyperparameter Optimization at Scale
venue: ICML
year: 2018

abstract: Bayesian optimization is a prominent method for optimizing expensive to evaluate black box functions that is widely applied to tuning the hyperparameters of machine learning algorithms. Despite its successes, the prototypical Bayesian optimization approach using Gaussian process models does not scale well to either many hyperparameters or many function evaluations. Attacking this lack of scalability and flexibility is thus one of the key challenges of the field. We present a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible. We obtain scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness we improve via a scale adaptation. Experiments including multi task Bayesian optimization with 21 tasks, parallel optimization of deep neural networks and deep reinforcement learning show the power and flexibility of this approach.
author_names: Jost Tobias Springenberg

Aaron Klein

Stefan Falkner

Frank Hutter
title: Bayesian Optimization with Robust Bayesian Neural Networks
venue: NIPS
year: 2016

abstract: Sample complexity and safety are major challenges when learning policies with reinforcement learning for real world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model based methods where the real world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.
author_names: Aravind Rajeswaran

Sarvjeet Ghotra

Sergey Levine

Balaraman Ravindran
title: EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
venue: ICLR
year: 2017

abstract: Providing a broad but in depth introduction to neural network and machine learning in a statistical framework, this book provides a single, comprehensive resource for study and further research. All the major popular neural network models and statistical learning approaches are covered with examples and exercises in every chapter to develop a practical working understanding of the content. Each of the twenty five chapters includes state of the art descriptions and important research results on the respective topics. The broad coverage includes the multilayer perceptron, the Hopfield network, associative memory models, clustering models and algorithms, the radial basis function network, recurrent neural networks, principal component analysis, nonnegative matrix factorization, independent component analysis, discriminant analysis, support vector machines, kernel methods, reinforcement learning, probabilistic and Bayesian networks, data fusion and ensemble learning, fuzzy sets and logic, neurofuzzy models, hardware implementations, and some machine learning topics. Applications to biometric/bioinformatics and data mining are also included. Focusing on the prominent accomplishments and their practical aspects, academic and technical staff, graduate students and researchers will find that this provides a solid foundation and encompassing reference for the fields of neural networks, pattern recognition, signal processing, machine learning, computational intelligence, and data mining.
author_names: Ke-Lin Du

M N Shanmukha Swamy
title: Neural Networks and Statistical Learning
year: 2013

abstract: Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. Bayesian methods provide a general framework to quantify uncertainty. However, because of model misspecification and the use of approximate inference, Bayesian uncertainty estimates are often inaccurate for example, a 90% credible interval may not contain the true outcome 90% of the time. Here, we propose a simple procedure for calibrating any regression algorithm; when applied to Bayesian and probabilistic models, it is guaranteed to produce calibrated uncertainty estimates given enough data. Our procedure is inspired by Platt scaling and extends previous work on classification. We evaluate this approach on Bayesian linear regression, feedforward, and recurrent neural networks, and find that it consistently outputs well calibrated credible intervals while improving performance on time series forecasting and model based reinforcement learning tasks.
author_names: Volodymyr Kuleshov

Nathan Fenner

Stefano Ermon
title: Accurate Uncertainties for Deep Learning Using Calibrated Regression
venue: ICML
year: 2018

abstract: Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting.
author_names: Carlos Riquelme

G Tucker

Jasper Snoek
title: Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
venue: ICLR
year: 2018

abstract: We present an algorithm for model based reinforcement learning that combines Bayesian neural networks (BNNs) with random roll outs and stochastic optimization for policy learning. The BNNs are trained by minimizing \alpha$ divergences, allowing us to capture complicated statistical patterns in the transition dynamics, e.g. multi modality and heteroskedasticity, which are usually missed by other common modeling approaches. We illustrate the performance of our method by solving a challenging benchmark where model based approaches usually fail and by obtaining promising results in a real world scenario for controlling a gas turbine.
author_names: Stefan Depeweg

Jose Miguel Hernandez-Lobato

Finale Doshi-Velez

Steffen Udluft
title: Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks
venue: ICLR
year: 2017

abstract: Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini batch by implicitly sampling pseudo independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor of 4 cost reduction on Amazon Web Services.
author_names: Yeming Wen

Paul Vicol

Jimmy Ba

Dustin Tran

Roger B Grosse
title: Flipout: Efficient Pseudo Independent Weight Perturbations on Mini Batches
venue: ICLR
year: 2018