Alan Fern – Deep Flat MDPs for Offline Model-Based Reinforcement Learning
Channel: PRL Workshop – Planning and Reinforcement Learning
Date: 23/10/2020

Alan Fern, Oregon State University
Title: Deep Flat MDPs for Offline Model-Based Reinforcement Learning

Invited talk at the workshop "Bridging the Gap Between AI Planning and Reinforcement Learning (PRL)" – ICAPS 2020
https://icaps20subpages.icaps-conference.org/workshops/prl/

Abstract:
While there has been a growing interest in model-based RL, it is rare to see optimal, or near-optimal, planning incorporated into actual implementations. This is especially true in the offline RL setting, where a model must be estimated from a static set of experience data. In this talk, we will describe our recent efforts to push the integration of deep representation learning with near-optimal planners for offline RL. In particular, we will introduce the Deep Averagers with Costs MDP (DAC-MDP) as a principled way to leverage optimal planners for flat/tabular MDP representations derived from continuous deep representations. Using a planner based on a GPU-implementation of value iteration, we demonstrate scalability to complex image-based environments such as Atari with relatively simple representations derived from offline model-free learners. We also illustrate potential use-cases of our planning-based approach for zero-shot adaptation to changes in the environment and optimization objectives. We will end with a discussion of future directions suggested by the work for further integration of symbolic planners and RL.