Interactive Visualization for Debugging RL

1 INTRODUCTION . Machine learning systems have made impressive advances due to their ability to learn high dimensional models from large amounts of data ( LeCun et al. , 2015 ) . However , high dimensional models are hard to understand and trust ( Doshi-Velez & Kim , 2017 ) . Many tools exist for addressing this challenge in the supervised learning setting , which find usage in tracking metrics ( Abadi et al. , 2015 ; Satyanarayan et al. , 2017 ) , generating graphs of model internals ( Wongsuphasawat et al. , 2018 ) , and visualizing embeddings ( van der Maaten & Hinton , 2008 ) . However , there is no corresponding set of tools for the reinforcement learning setting . At first glance , it appears we may repurpose existing tools for this task . However , we quickly run into limitations , that arise due to the intent with which these tools were designed . Reinforcement learning ( RL ) is a more interactive science ( Neftci & Averbeck , 2019 ) compared to supervised learning , due to a stronger feedback loop between the researcher and the agent . Whereas supervised learning involves a static dataset , RL often entails collecting new data . To fully understand an RL algorithm , we must understand the effect it has on the data collected . Note that in supervised learning , the learned model has no effect on a fixed dataset . Visualization systems are important for overcoming these challenges . At their core visualization systems , consist of two components : representation and interaction . Representation is concerned with how data is mapped to a representation and then rendered . Interaction is concerned with the dialog between the user and the system as the user explores the data to uncover insights ( Yi et al. , 2007 ) . Though appearing to be disparate , these two processes have a symbiotic influence on each other . The tools we use for representation affect how we interact with the system , and our interaction affects the representations that we create . Thus , while designing visualization systems , it is important to think about the application domain from which the data originates , in this case , reinforcement learning . Using existing tools we can plot descriptive metrics such as cumulative reward , TD-error , and action values , to name a few . However , it is harder to pose and easily answer questions such as : - How does the agent state-visitation distribution change as training progresses ? - What effect do noteworthy , influential states have on the policy ? 1An interactive ( anonymized ) demo of the system can be found at https : //vizarel-demo . herokuapp.com - Are there repetitive patterns across space and time that result in the observed agent behavior ? These are far from an exhaustive list of questions that a researcher may pose while training agent policies , but are chosen to illustrate the limitations created by our current set of tools that prevent us from being able to easily answer such questions . This paper describes our attempt at constructing Vizarel 2 , an interactive visualization system to help interpret RL algorithms , debug RL policies , and help RL researchers pose and answer questions of this nature . Towards these goals , we identify features that an interactive system for interpretable reinforcement learning should encapsulate and build a prototype of these ideas . We complement this by providing a walkthrough example of how this system could fit into the RL debugging workflow and be used in a real scenario to debug a policy . 2 RELATED WORK . As we have argued in the introduction , existing visualization tools for machine learning primarily focus on the supervised learning setting . However , the process of designing and debugging RL algorithms requires a different set of tools , that can complement the strengths and overcome the weaknesses of offerings in the current ecosystem . In the rest of this section , we highlight aspects of prior work upon which our system builds . To the best of our knowledge , there do not exist visualization systems built for interpretable reinforcement learning that effectively address the broader goals we have identified . There exists prior work , aspects of which are relevant to features which the current system encapsulates , that we now detail . Visual Interpretability Related work for increasing understanding in machine learning models using visual explanations includes : feature visualization in neural networks ( Olah et al. , 2017 ; Simonyan et al. , 2014 ; Zeiler & Fergus , 2013 ) , visual analysis tools for variants of machine learning models ( Strobelt et al. , 2017 ; Kahng et al. , 2017 ; Kapoor et al. , 2010 ; Krause et al. , 2016 ; Yosinski et al . ) , treating existing methods as composable building blocks for user interfaces ( Olah et al. , 2018 ) , and visualization techniques for increasing explainability in reinforcement learning ( Rupprecht et al. , 2020 ; Atrey et al. , 2020 ; McGregor et al. , 2015 ) Explaining agent behavior There exists related work that tries to explain agent behavior . Amir & Amir summarize agent behavior by displaying important trajectories . van der Waa et al . ( 2018 ) introduce a method to provide contrastive explanations between user derived and agent learned policies . Huang et al . ( 2017 ) show maximally informative examples to guide the user towards understanding 2Vizarel is a portmanteau of visualization + reinforcement learning . the agent objective function . Hayes & Shah ( 2017 ) present algorithms and a system for robots to synthesize policy descriptions and respond to human queries . Explainable reinforcement learning Puiutta & Veith ( 2020 ) provide a survey of techniques for explainable reinforcement learning . Related work in this theme includes Puri et al . ( 2020 ) ; Reddy et al . ( 2019 ) ; Calvaresi et al . ( 2019 ) ; Juozapaitis et al . ( 2019 ) ; Sequeira & Gervasio ( 2020 ) ; Fukuchi et al . ( 2017 ) ; Madumal et al . ( 2020 ) Similar to Amir & Amir ; van der Waa et al . ( 2018 ) ; Huang et al . ( 2017 ) , this work is motivated by the aim to provide the researcher relevant information to explore a possible space of solutions while debugging the policy . Similar to Hayes & Shah ( 2017 ) , we present a functioning system that can respond to human queries to provide explanations . However , in contrast , the interactive system we present is built around the RL training workflow , and designed to evolve beyond the explanatory use case to complement the existing ecosystem of tools ( Abadi et al. , 2015 ; Satyanarayan et al. , 2017 ) . In contrast to the techniques surveyed in Puiutta & Veith ( 2020 ) , the contribution here is not on any single technique to increase interpretability , but a whole suite of visualizations built on an extensible platform to help researchers better design and debug RL agent policies for their task . 3 PRELIMINARIES . We use the standard reinforcement learning setup ( Sutton & Barto , 2018 ) . An agent interacting with an environment at discrete timesteps t , receiving a scalar reward r ( st , at ) ∈ R. The agent ’ s behavior is defined by a policy π , which maps states s ∈ S , to a probability distribution over actions , π : S → P ( A ) . The environment can be stochastic , which is modeled by a Markov decision process with a state space S , action space A ∈ Rn , an initial state distribution p ( s0 ) , a transition function p ( st+1 | st , a ) , and a reward function r ( st , at ) . The future discounted return from a state st and action at is defined as Rt = ∑T i=t γ i−tr ( st , at ) , with a discount factor γ ∈ [ 0 , 1 ] . We use a replay buffer ( Mnih et al. , 2013 ; Lin ) to store the agent ’ s experiences et = ( st , at , rt , st+1 ) in a buffer B = { e0 , e1 , ... , eT } . 4 VIZAREL : A TOOL FOR INTERACTIVE VISUALIZATION OF RL This section describes how our interactive visualization system ( Vizarel ) , is currently designed . The system offers different views that allow the user to analyze agent policies along spatial and temporal dimensions ( described later in further detail ) . The tool consists of a set of viewports , that provide the user with different representations of the data , contingent on the underlying data stream . Viewports are generated by chaining together different visualization elements , such as : 1. image buffers : visualize observation spaces ( image and non-image based ) 2. line plots : visualize sequentially ordered data , such as action values or rewards across time 3. scatter plots : to visualize embedding spaces or compare tensors along specified dimensions 4. histograms : visualize frequency counts of specified tensors or probability distributions The current implementation provides core viewports ( detailed further ) , but can easily be extended by the user to generate additional viewports to explore different visualization ideas . This design naturally leads to the idea of an ecosystem of plugins that could be integrated into the core system , and distributed for use among a community of users to support different visualization schemes and algorithms . For example , the user could combine image buffers and line plots in novel ways to create a viewport to visualize the the state-action value function Sutton & Barto ( 2018 ) . In the rest of this section , we provide details and distinguish between two types viewports currently implemented in Vizarel : temporal viewports and spatial viewports . Discussion on viewports beyond these has been deferred to the appendix . Comprehensive information on adding new viewports is beyond the scope of the paper , but has been described at length in the system documentation3 . 4.1 TEMPORAL VIEWS . Temporal views are oriented around visualizing the data stream ( e.g . images , actions , rewards ) as a sequence of events ordered along the time dimension . We have implemented three types of temporal viewports : state viewports , action viewports , and reward viewports , which we now detail . 4.1.1 STATE VIEWPORT . For visualization , we can classify states as either image-based or non-image based . The type of observation space influences the corresponding viewport used for visualization . We provide two examples that illustrate how these differing observation spaces can result in different viewports . Consider a non-image based observation space , such as that for the inverted pendulum task . Here , the state vector ~s = { sin ( θ ) , cos ( θ ) , θ̇ } , where θ is the angle which the pendulum makes with the vertical . We can visualize the state vector components individually , which provides insight into how states vary across episode timesteps ( Figure 2 ) . Since images are easier for humans to interpret , we can generate an additional viewport using image buffers , that tracks changes in state space to the corresponding changes in image space . Having this simultaneous visualization is useful since it now enables us to jump back and forth between the state representation which the agent receives , and the corresponding image representation , by simply hovering over the desired timestep in the state viewport . 3Vizarel is planned to be released as an open source tool For environments that have higher dimensional state spaces , such as that of a robotic arm with multiple degrees of freedom , we can visualize individual state components . However , since this may not be intuitive , we can also generate an additional viewport to display an image rendering of the environment to help increase interpretability .

Summary: 		a. This article contributes a framework and tool for visualising data collected from a policy during RL training. It also contributes a set of views for visualising RL training data that could be used in general. Finally, the paper contributes a high-level workflow and a set of example use cases for how this tool might impact debugging a training session.