Interactive Visualization for Debugging RL

1 INTRODUCTION . Machine learning systems have made impressive advances due to their ability to learn high dimensional models from large amounts of data ( LeCun et al. , 2015 ) . However , high dimensional models are hard to understand and trust ( Doshi-Velez & Kim , 2017 ) . Many tools exist for addressing this challenge in the supervised learning setting , which find usage in tracking metrics ( Abadi et al. , 2015 ; Satyanarayan et al. , 2017 ) , generating graphs of model internals ( Wongsuphasawat et al. , 2018 ) , and visualizing embeddings ( van der Maaten & Hinton , 2008 ) . However , there is no corresponding set of tools for the reinforcement learning setting . At first glance , it appears we may repurpose existing tools for this task . However , we quickly run into limitations , that arise due to the intent with which these tools were designed . Reinforcement learning ( RL ) is a more interactive science ( Neftci & Averbeck , 2019 ) compared to supervised learning , due to a stronger feedback loop between the researcher and the agent . Whereas supervised learning involves a static dataset , RL often entails collecting new data . To fully understand an RL algorithm , we must understand the effect it has on the data collected . Note that in supervised learning , the learned model has no effect on a fixed dataset . Visualization systems are important for overcoming these challenges . At their core visualization systems , consist of two components : representation and interaction . Representation is concerned with how data is mapped to a representation and then rendered . Interaction is concerned with the dialog between the user and the system as the user explores the data to uncover insights ( Yi et al. , 2007 ) . Though appearing to be disparate , these two processes have a symbiotic influence on each other . The tools we use for representation affect how we interact with the system , and our interaction affects the representations that we create . Thus , while designing visualization systems , it is important to think about the application domain from which the data originates , in this case , reinforcement learning . Using existing tools we can plot descriptive metrics such as cumulative reward , TD-error , and action values , to name a few . However , it is harder to pose and easily answer questions such as : - How does the agent state-visitation distribution change as training progresses ? - What effect do noteworthy , influential states have on the policy ? 1An interactive ( anonymized ) demo of the system can be found at https : //vizarel-demo . herokuapp.com - Are there repetitive patterns across space and time that result in the observed agent behavior ? These are far from an exhaustive list of questions that a researcher may pose while training agent policies , but are chosen to illustrate the limitations created by our current set of tools that prevent us from being able to easily answer such questions . This paper describes our attempt at constructing Vizarel 2 , an interactive visualization system to help interpret RL algorithms , debug RL policies , and help RL researchers pose and answer questions of this nature . Towards these goals , we identify features that an interactive system for interpretable reinforcement learning should encapsulate and build a prototype of these ideas . We complement this by providing a walkthrough example of how this system could fit into the RL debugging workflow and be used in a real scenario to debug a policy . 2 RELATED WORK . As we have argued in the introduction , existing visualization tools for machine learning primarily focus on the supervised learning setting . However , the process of designing and debugging RL algorithms requires a different set of tools , that can complement the strengths and overcome the weaknesses of offerings in the current ecosystem . In the rest of this section , we highlight aspects of prior work upon which our system builds . To the best of our knowledge , there do not exist visualization systems built for interpretable reinforcement learning that effectively address the broader goals we have identified . There exists prior work , aspects of which are relevant to features which the current system encapsulates , that we now detail . Visual Interpretability Related work for increasing understanding in machine learning models using visual explanations includes : feature visualization in neural networks ( Olah et al. , 2017 ; Simonyan et al. , 2014 ; Zeiler & Fergus , 2013 ) , visual analysis tools for variants of machine learning models ( Strobelt et al. , 2017 ; Kahng et al. , 2017 ; Kapoor et al. , 2010 ; Krause et al. , 2016 ; Yosinski et al . ) , treating existing methods as composable building blocks for user interfaces ( Olah et al. , 2018 ) , and visualization techniques for increasing explainability in reinforcement learning ( Rupprecht et al. , 2020 ; Atrey et al. , 2020 ; McGregor et al. , 2015 ) Explaining agent behavior There exists related work that tries to explain agent behavior . Amir & Amir summarize agent behavior by displaying important trajectories . van der Waa et al . ( 2018 ) introduce a method to provide contrastive explanations between user derived and agent learned policies . Huang et al . ( 2017 ) show maximally informative examples to guide the user towards understanding 2Vizarel is a portmanteau of visualization + reinforcement learning . the agent objective function . Hayes & Shah ( 2017 ) present algorithms and a system for robots to synthesize policy descriptions and respond to human queries . Explainable reinforcement learning Puiutta & Veith ( 2020 ) provide a survey of techniques for explainable reinforcement learning . Related work in this theme includes Puri et al . ( 2020 ) ; Reddy et al . ( 2019 ) ; Calvaresi et al . ( 2019 ) ; Juozapaitis et al . ( 2019 ) ; Sequeira & Gervasio ( 2020 ) ; Fukuchi et al . ( 2017 ) ; Madumal et al . ( 2020 ) Similar to Amir & Amir ; van der Waa et al . ( 2018 ) ; Huang et al . ( 2017 ) , this work is motivated by the aim to provide the researcher relevant information to explore a possible space of solutions while debugging the policy . Similar to Hayes & Shah ( 2017 ) , we present a functioning system that can respond to human queries to provide explanations . However , in contrast , the interactive system we present is built around the RL training workflow , and designed to evolve beyond the explanatory use case to complement the existing ecosystem of tools ( Abadi et al. , 2015 ; Satyanarayan et al. , 2017 ) . In contrast to the techniques surveyed in Puiutta & Veith ( 2020 ) , the contribution here is not on any single technique to increase interpretability , but a whole suite of visualizations built on an extensible platform to help researchers better design and debug RL agent policies for their task . 3 PRELIMINARIES . We use the standard reinforcement learning setup ( Sutton & Barto , 2018 ) . An agent interacting with an environment at discrete timesteps t , receiving a scalar reward r ( st , at ) ∈ R. The agent ’ s behavior is defined by a policy π , which maps states s ∈ S , to a probability distribution over actions , π : S → P ( A ) . The environment can be stochastic , which is modeled by a Markov decision process with a state space S , action space A ∈ Rn , an initial state distribution p ( s0 ) , a transition function p ( st+1 | st , a ) , and a reward function r ( st , at ) . The future discounted return from a state st and action at is defined as Rt = ∑T i=t γ i−tr ( st , at ) , with a discount factor γ ∈ [ 0 , 1 ] . We use a replay buffer ( Mnih et al. , 2013 ; Lin ) to store the agent ’ s experiences et = ( st , at , rt , st+1 ) in a buffer B = { e0 , e1 , ... , eT } . 4 VIZAREL : A TOOL FOR INTERACTIVE VISUALIZATION OF RL This section describes how our interactive visualization system ( Vizarel ) , is currently designed . The system offers different views that allow the user to analyze agent policies along spatial and temporal dimensions ( described later in further detail ) . The tool consists of a set of viewports , that provide the user with different representations of the data , contingent on the underlying data stream . Viewports are generated by chaining together different visualization elements , such as : 1. image buffers : visualize observation spaces ( image and non-image based ) 2. line plots : visualize sequentially ordered data , such as action values or rewards across time 3. scatter plots : to visualize embedding spaces or compare tensors along specified dimensions 4. histograms : visualize frequency counts of specified tensors or probability distributions The current implementation provides core viewports ( detailed further ) , but can easily be extended by the user to generate additional viewports to explore different visualization ideas . This design naturally leads to the idea of an ecosystem of plugins that could be integrated into the core system , and distributed for use among a community of users to support different visualization schemes and algorithms . For example , the user could combine image buffers and line plots in novel ways to create a viewport to visualize the the state-action value function Sutton & Barto ( 2018 ) . In the rest of this section , we provide details and distinguish between two types viewports currently implemented in Vizarel : temporal viewports and spatial viewports . Discussion on viewports beyond these has been deferred to the appendix . Comprehensive information on adding new viewports is beyond the scope of the paper , but has been described at length in the system documentation3 . 4.1 TEMPORAL VIEWS . Temporal views are oriented around visualizing the data stream ( e.g . images , actions , rewards ) as a sequence of events ordered along the time dimension . We have implemented three types of temporal viewports : state viewports , action viewports , and reward viewports , which we now detail . 4.1.1 STATE VIEWPORT . For visualization , we can classify states as either image-based or non-image based . The type of observation space influences the corresponding viewport used for visualization . We provide two examples that illustrate how these differing observation spaces can result in different viewports . Consider a non-image based observation space , such as that for the inverted pendulum task . Here , the state vector ~s = { sin ( θ ) , cos ( θ ) , θ̇ } , where θ is the angle which the pendulum makes with the vertical . We can visualize the state vector components individually , which provides insight into how states vary across episode timesteps ( Figure 2 ) . Since images are easier for humans to interpret , we can generate an additional viewport using image buffers , that tracks changes in state space to the corresponding changes in image space . Having this simultaneous visualization is useful since it now enables us to jump back and forth between the state representation which the agent receives , and the corresponding image representation , by simply hovering over the desired timestep in the state viewport . 3Vizarel is planned to be released as an open source tool For environments that have higher dimensional state spaces , such as that of a robotic arm with multiple degrees of freedom , we can visualize individual state components . However , since this may not be intuitive , we can also generate an additional viewport to display an image rendering of the environment to help increase interpretability .

Summary: The paper deals with debugging of black-box deep reinforcement learning (RL) agents to better understand and fix their policies. The authors propose diverse tools for, among others, visualizing the state space in terms of calculated statistics, analyzing the taken actions across learning episodes or exploring the replay buffer. The authors also propose a workflow for using the proposed tools. The resulting Vizarel tool is evaluated in terms of an exemplary walkthrough.