Interactive Visualization for Debugging RL
Authors: Shuby Deshpande
Benjamin Eysenbach
Jeff Schneider
Venue:  (2021)
Keywords: Reinforcement Learning
Interpretability
Visualization

Abstract: Visualization tools for supervised learning (SL) allow users to interpret, introspect, and gain an intuition for the successes and failures of their models. While reinforcement learning (RL) practitioners ask many of the same questions while debugging agent policies, existing tools aren't a great fit for the RL setting as these tools address challenges typically found in the SL regime. Whereas SL involves a static dataset, RL often entails collecting new data in challenging environments with partial observability, stochasticity, and non-stationary data distributions. This necessitates the creation of alternate visual interfaces to help us better understand agent policies trained using RL. In this work, we design and implement an interactive visualization tool for debugging and interpreting RL. Our system identifies and addresses important aspects missing from existing tools such as (1) visualizing alternate state representations (different from those seen by the agent) that researchers could use while debugging RL policies; (2) interactive interfaces tailored to metadata stored while training RL agents (3) a conducive workflow designed around RL policy debugging. We provide an example workflow of how this system could be used, along with ideas for future extensions.

Under review as a conference paper at ICLR 2021
INTERACTIVE VISUALIZATION FOR DEBUGGING RL
Anonymous authors
Paper under double-blind review
ABSTRACT
Visualization tools for supervised learning (SL) allow users to interpret, intro- speet, and gain an intuition for the successes and failures their models. While reinforcement learning (RL) practitioners ask many of the same questions while debugging agent policies, existing tools aren't It a great fit for the RL setting as these tools address challenges typically found in the SL regime. Whereas SL involves a static dataset, RL often entails collecting new data in challenging environments with partial observability, stochasticity, and non-stationary data distributions. This necessitates the creation of alternate visual interfaces to help us better understand agent policies trained using RL. In this work, we design and implement an interac- tive visualization tool for debugging and interpreting RL. Our system identifies and addresses important aspects missing from existing tools such as (1) visual- izing alternate state representations (different from those seen by the agent) that researchers could use while debugging RI. policies; (2) interactive interfaces tai- lored to metadata stored while training RI agents (3) conducive workflow de- signed around RL policy debugging. We provide an example workflow of how this system could be used, along with ideas for future extensions.
INTRODUCTION
Machine learning systems have made impressive advances due to their ability to learn high dimen- sional models from large amounts of data (LeCun al.) (2015). However, high dimensional models
are hard to understand and trust (Doshi Velez & Kim 2017). Many tools exist for addressing this challenge in the supervised learning setting, which find usage in tracking metrics (Abadi et al. [215]] Satyanarayan et al. (017)) generating graphs of model internals (Wongsuphasawat et al. 2018)) and visualizing embeddings (van der Maaten Hinton, 2008). However, there no corresponding set of tools for the reinforcement learning setting. At first glance, it appears we may repurpose exist- ing tools for this task. However, we quickly run into limitations, that arise due to the intent with which these tools were designed. Reinforcement learning (RL) is more interactive science (Neftci) & Averbeck 2019) compared to supervised learning, due to a stronger feedback loop between the researcher and the agent. Whereas supervised learning involves static dataset, RL often entails col- lecting new data. To fully understand an RL algorithm, we must understand the effect has on the data collected. Note that in supervised learning, the learned model has no effect on fixed dataset.
Visualization systems are important for overcoming these challenges. At their core visualization systems, consist of two components: representation and interaction. Reprexentation is concerned with how data is mapped to representation and then rendered. Interaction is concerned with the dialog between the user and the system as the user explores the data to uncover insights (Yi al. 2007]. Though appearing to be disparate, these two processes have symbiotic influence on each other. The tools we use for representation affect how we interact with the system, and our interaction affects the representations that we create. Thus, while designing visualization systems, it is impor- tant to think about the application domain from which the data originates, in this case, reinforcement learning.
Using existing tools we can plot descriptive metrics such as cumulative reward, TD-error, and action values, to name a few. However, it is harder to pose and casily answer questions such as:
How does the agent state-visitation distribution change as training progresses?
What effect do noteworthy, influential states have on the policy?
'An interactive (anonymized) demo of the system be found at https:///e zarel demo
herokuapp.com

---

Under review as a conference paper at ICLR 2021
(a) Representative Tool (Tensorboard)
(b) Vizarel
Tool Comparison Contrasting between a representative tool for debugging RL in the existing ecosystem (L), and Vizarel (R), highlights the difference design intent between both systems. The former was designed for the supervised learning setting, and has shown promise for use in reinforcement learning. However, we argue that there exists a large space of unexplored interfaces that could help aid the process of debugging RL algorithms and trained policies. We explore one such solution that is designed around the spatial and temporal aspects of training RL agents. This approach might help increase understanding, interpretability, and thereby serve complement to tools in the existing ecosystem.
Figure 1:
Are there repetitive patterns across space and time that result in the observed agent behavior?
These are far from an exhaustive list of questions that a researcher may pose while training agent policies, but are chosen to illustrate the limitations created by our current set tools that prevent us from being able to easily answer such questions.
This paper describes our attempt at constructing Vizarel 2 an interactive visualization system to help interpret RL algorithms, debug RL policies, and help RL researchers pose and answer questions of this nature. Towards these goals, we identify features that an interactive system for interpretable reinforcement learning should encapsulate and build prototype of these ideas. We complement this by providing a walkthrough example of how this system could fit into the RI. debugging workflow and be used in real scenario debug policy.
RELATED WORK
As we have argued in the introduction, existing visualization tools for machine learning primarily focus on the supervised learning setting. However, the process of designing and debugging Rí. algorithms requires different of tools, that can complement the strengths and overcome the weaknesses of offerings in the current ecosystem. In the rest of this section, we highlight aspects of prior work upon which our system builds. To the best of our knowledge, there do not exist visu- alization systems built for interpretable reinforcement learning that effectively address the broader goals we have identified. There exists prior work, aspects of which are relevant to features which the current system encapsulates, that we now detail.
Visual Interpretability Related work for increasing understanding in machine learning models us- ing visual explanations includes: feature visualization in neural networks (Olah et al. 2017; Si- monyan et al. 2014 Zeiler & Fergus 2013), visual analysis tools for variants of machine learning
models (Strobelt et al.) 2017 Kahng al. 2017 Kapoor et al.) 2010 Krause al. 2016 Yosin-
ski et ).. treating existing methods as composable building blocks for user interfaces (Olah et al.
2018), and visualization techniques for increasing explainability in reinforcement learning (Rup-
precht et al. 2020) Atrey et al. 2020 McGiregor et al. 2015)
Explaining agent behavior There exists related work that tries to explain agent behavior. Amir & Amirriimmarize agent behavior by displaying important trajectories. yan der Waa et al.) (0088) intro- duce method provide contrastive explanations between user derived and agent learned policies. show maximally informative examples to guide the user towards understanding
Huang et al. 2017)
Vizarel is portmanteau of visualization + reinforcement learning.
2

---

Under review as a conference paper at ICLR 2021
VO0
Figure 2: State Action Viewports (T) Visualizing the state viewport for the inverted pendulum task. This representation overlayed with another state viewport similar (b). provides the with better intuition about the correspondence between states and actions for non image state spaces.
(B) Visualizing the action viewport for the Pong environment (Bellemare et al. 2013). Hovering over instantancous timesteps dynamically updates the state viewport (4.1.l and shows the corresponding rendered image for the selected state. This representation provides the user with intuition about the agent policy, and could help subsequent debugging.
the agent objective function. HHaes & Shah (2017) present algorithms and system for robots to synthesize policy descriptions and respond to human queries.
Explainable reinforcement learning Puiutta & Veith (2020) provide survey of techniques for explainable reinforcement learning. Related work this theme includes Puri al. (2020); Reddy
et al. 2019); Calvaresi et al. [019; et al. (2019); Sequeira & Gervasio (2020); Fukuchi
ct al.
2017); Madumal et al. (2020)
Similar to Amir & Amir; van der Waa et al. (2018) Huang et al. (201)), this work is motivated by the aim to provide the researcher relevant information to explore a possible space of solutions while debugging the policy. Similar to |Haye & Shah (2017), present functioning system that can respond to human queries to provide explanations. However, in contrast, the interactive system we present built around the RI. training workflow, and designed evolve heyond the explanatory use case to complement the existing ecosystem tools (Abadi et al.) 2015; Satyanarayan et al.) (007)). In contrast to the techniques surveyed in Puiutta Veith (2020). the contribution here is not on any single technique to increase interpretability, but whole suite of visualizations built on an extensible platform help researchers better design and debug RL agent policies for their task.
3 PRELIMINARIES
We use the standard reinforcement learning setup (Sutton & Barto 2018). An agent interacting with an environment at discrete timesteps 6, receiving scalar reward r(s,, az) E R. The agent's behavior is defined by a policy which maps states E S, to probability distribution over actions, क (AA). The environment can be stochastic, which is modeled by a Markov decision process with state space S, action space A E R", an initial state distribution p(so), a transition function p(s++1 St,a). a). and a reward function r(st; at). The future discounted return from a state & and action is defined as R => -f
az), with discount factor E [0, 1]. We use replay buffer (Mnih et al. 2013 Lin) to store the agent's experiences et
(8t, Dty Tty 8441) in a buffer
B (Co, C1,....
VIZAREL: A TOOL FOR INTERACTIVE VISUALIZATION OF RL
This section describes how our interactive visualization system (Vizarel), is currently designed. The system offers different views that allow the user analyze agent policies along spatial and temporal dimensions (described later in further detail). The tool consists of set viewports, that provide the user with different representations of the data, contingent the underlying data stream. Viewports are generated by chaining together different visualization elements, such as:
1. image buffers: visualize observation spaces (image and non-image based)
3

---

Under review as a conference paper at ICLR 2021
Replay Buffar
Stb Space Veepot
Projeded Din
Projecedd Dim
Figure 3: Replay Buffer Viewport Projecting the contents of the replay buffer into 2D space for easier navigation, and clustering states based similarity. This viewport provides visual representation for replay buffer diversity and can help in subsequent debugging. Hovering over points the replay buffer dynami- cally updates the generated state viewport (4.1.I) and shows the rendered image for the corresponding state (animation depicted using overlay).
2. line plots: visualize sequentially ordered data, such as action values or rewards across time
3. scatter plots: to visualize embedding spaces or compare tensors along specified dimensions
4. histograms: visualize frequency counts of specified tensors probability distributions
The current implementation provides core viewports (detailed further), but can easily be extended by the user generate additional viewports to explore different visualization ideas. This design naturally leads to the idea of an ecosystem of plugins that could be integrated into the core system, and distributed for use among community of users to support different visualization schemes and algorithms. For example, the user could combine image buffers and line plots in novel ways to create viewport to visualize the the state-action value function /Sutton & Barto (2018). In the rest of this section, we provide details and distinguish between two types viewports currently implemented in Vizarel: temporal viewports and spatial viewports. Discussion on viewports beyond these has been deferred to the appendix. Comprehensive information on adding new viewports is beyond the scope of the paper, but has been described at length in the system documentatior
4.1 TEMPORAL VIEWS
Temporal views are oriented around visualizing the data stream (e.g. images, actions, rewards) as a sequence of events ordered along the time dimension. We have implemented three types of temporal viewports: state viewports, action viewports, and reward viewports, which we now detail.
4.1.1 STATE VIEWPORT
For visualization, classify states either image-based non-image based. The type of observation space influences the corresponding viewport used for visualization. We provide two examples that illustrate how these differing observation spaces can result in different viewports. Consider non-image based observation space, such as that for the inverted pendulum task. Here, the state vector z = (fi cos((), 0}. where 0 is the angle which the pendulum makes with the vertical.
We can visualize the state vector components individually, which provides insight into how states vary across episode timesteps (Figure (}. Since images are easier for humans to interpret, we can generate an additional viewport using image buffers, that tracks changes in state space to the cor- responding changes in image space. Having this simultaneous visualization useful since it now enables to jump back and forth between the state representation which the agent receives, and the corresponding image representation, by simply hovering over the desired timestep in the state viewport.
Vizarel is planned he released as open source tool
4

---

Under review as a conference paper at ICLR 2021
Roplay Suffer
R.
300
Figure 4: Distribution Viewport Using the lasso tool to select group of points (dashed gray line) in the replay buffer viewport (4.2.]) dynamically updates (dashed red line) the distribution viewport 442.22 by computing and plotting the distribution of values for the specified tensor (e.g. actions or rewards).
For environments that have higher dimensional state spaces, such as that of a robotic arm with multiple degrees of freedom, we can visualize individual state components. However, since this may not be intuitive, we can also generate an additional viewport display an image rendering of the environment to help increase interpretability.
4.1.2 ACTION VIEWPORT
The action viewport is used to visualize how the actions chosen by the agent vary across the episode (Figure[). Consider the Pong environment (Bellemare al. 2013), where the action at timestep corresponds to the direction in which the paddle should move. A visualization such as the one shown in Figure 2 can be generated to show the correspondence between actions and states. This allows the user to easily identify states marked by sudden action transitions, and thus aid debugging. This idea can easily be extended agents with stochastic actions, where we could generate a viewport using histograms to visualize the change in action distribution over time.
For higher dimensional action spaces we can use a technique similar to the one used for state view- ports, to generate multiple viewports that track individual action dimensions, for example joint torques for a multilink robot.
4.1.3 REWARD VIEWPORT
The reward viewport used to visualize how the rewards received by an agent vary across the episode. A user can look at the reward viewport together with the state viewport to understand and find patterns across state transitions that result high reward. For many environments, the reward function consists of components weighted by different coefficients. These individual components are often easier to interpret since they are usually correspond to a physically motivated quantity tied to agent behaviors that we wish to either reward or penalize. For example, in autonomous driving environments the reward can be formulated as a function of speed, collision penalties, and the distance from optimal trajectory (Agarwal al.)
In situations where we have access to these reward components, we can generate multiple viewports each of which visualize different components of the reward function. The viewports discussed so far can be combined to provide the user more insight into the correspondence between states (state viewport), actions (action viewport), and the components of the reward function (reward viewport) that the agent is attempting to maximize. Such visualization could help alert researchers to reward hacking (Amodei et 2016).
4.2 SPATIAL VIEWS
Spatial views oriented around visualizing the data stream spatial distribution of events. We have implemented three types of spatial viewports: replay buffer viewports, distribution viewports, and trajectory viewports, that now describe.
5

---

Under review as a conference paper at ICLR 2021
BOXK
61
100
Figure 5: Trajectory Viewport Selecting points in the replay buffer viewport (4.2. causes the trajectory viewport 4.2.3) dynamically update and plot the absolute normalized TD error values over the length of the trajectory. Hovering over points in the trajectory viewport, allows the user view rendering of the state corresponding to that timestep the generated state viewport 4.1.I).
4.2.1 REPLAY BUFFER VIEWPORT
As formulated in 3 the replay buffer stores the agent's experiences (s2, 0s, 8(41) in buffer B {eo, C1, er,....r Vi E [0, For off-policy algorithms, the replay buffer of crucial importance, since it effectively serves as online dataset for agent policy updates (Fu et al. 2020). In the supervised learning setting, there exist tools to visualize datasets, that provide the user with an intuition for the underlying data distribution. The replay buffer viewport aims to provide similar intuitions for the reinforcement learning setting by visualizing the distribution of data samples the replay buffer.
Since the individual elements of the replay buffer are at least four-dimensional vector et, this rules out the possibility of generating viewports to visualize data in the original space. We can instead visualize the data samples by transforming the points (van der Maaten & Hinton| (2008) to lower- dimensional representation. This technique helps visualize the distribution of samples in the replay buffer, which is a visual representation of the replay buffer diversity (de Bruin et al.).
The size of the replay buffer can be quite large (Zhang & Sutton 2018), which lead to difficulties while navigating the space of points visualized in the replay buffer viewport. To nudge (Thaler & Sunstein 2009) the user towards investigating samples of higher potential interest, we scale the size of points in proportion to the absolute normalized TD error (Sutton & Barto 2018), which has been used past work (Schaul al.) (016)) as a measure of sample priority during experience replay.
Moreover, the replay buffer viewport can be combined with the state viewport to simultancously visualize an image rendering of the state, by tracking changes as the user hovers over points in the replay buffer viewport (Figure 3).
4.2.2 DISTRIBUTION VIEWPORT
The distribution viewport (Figure complements the replay buffer viewport by allowing the user to select clusters of data samples and ask questions regarding the distribution of action, rewards, and other relevant tensors for the selected group of points.
Users might ask questions like:
What is the distribution of actions the agent took for these groups of similar states?
What the distribution of rewards for the state action transitions?
What is the overall diversity of states which the agent has visited?
the updates to the agent policy result in better task performance, the entropy of the action distribu- tion should reduce over time (discounting any external annealing caused due to exploration), which
can be easily verified through this viewport. In the limit, the distribution of actions for group of similar points should converge a Dirac distribution, since the optimal policy for infinite horizon discounted MDP is a stationary distribution (Puterman) 2005). In practice, observing the distribution converging around the mean value could indicate a promising policy training experiment.

---

Under review as a conference paper at ICLR 2021
Failure learn?
Make guided changes
Checkpoint policy
Train Policy
Vizarel
Visualize
Eval (Policy Rollout)
Generate
Debug
Viewports
Vizarel
Visarel
Dashboard
UbiS
Figure 6: Vizarel Workflow Diagram Typical steps during policy debugging, and how the designed system fits into this workflow. The system takes as input policy saved during checkpoint and evaluates the policy through a specified number of rollouts. This data is then visualized through viewports specified by the user, that used for debugging the policy through making guided changes.
For lalti-dimensional action spaces, the viewport could be repurposed to display the variance of the action distribution, plot different projections of the action distribution, or more sophisticated techniques such as projection pursuit (Huber 1985).
4.2.3 TRAJECTORY VIEWPORT
fusion of the components from the spatial and temporal views leads to a spatio-temporal view, an example of which is the trajectory viewport (Figure The replay buffer viewport alone visualizes the spatial nature of the points in the replay buffer but does not display the temporal nature of trajectories. Being able switch between spatial and temporal views is crucial when understanding and debugging policies. This supported by selecting points the replay buffer viewport, which then retrieves the corresponding trajectory.
The X coordinate in the trajectory viewport represents the timestep, and the Y coordinate is the absolute TD error, normalized to lie within [0, 1]. Hovering over points in the trajectory viewport retrieves an image rendering of the corresponding state in the state viewport. This correspondence enables the user easily navigate through and visualize action sequences in the trajectory that consistently have high TD error, thus speeding up debugging of policies.
WALKTHROUGH
We now detail an example workflow of how the system can be used in real scenario. Figure 6, illustrates how Vizarel fits into an RL re- searcher's policy debugging workflow. Train- ing a successful agent policy often requires multiple iterations of changing algorithm hy- perparameters.
To speed up and increase the intuitiveness of this process, the researcher can load stored checkpoint of the policy into the system, and evaluate a specified number of policy rollouts. found that there should be enough rollouts to ensure sufficient coverage of the state space, since this influences the scope of questions which can be posed during debug- ging (e.g. through the replay buffer viewport). These rollouts then be visualized and inter- acted with through specifying the required data streams and generating different viewports.
Empirically,
Figure 7: Opatio-Tempont Interaction Visual- izing the replay buffer viewport (4.2.l) (spatial view), and trajectory viewport 4.2.3) (temporal view), along with overlays to independently track image renderings of states in both as state space viewport (4.1.1) Nav- igating between these viewports allows the user to ob- serve both agent spatial and temporal behavior, which could facilitate better insights during debugging.
Figure 7 shows an example of replay buffer, state, and trajectory viewports generated for

---

Under review as a conference paper at ICLR 2021
Figure 8: Comparing TD error along an agent trajectory Visualizing the trajectory viewport (.2.3 allows the user to compare the TD error at different timesteps along the trajectory, along with the associated state viewport (4.1.]). An example interaction is visualized here by hovering over regions of potential interest trajectory viewport. This simultaneous view allows the user easily compare and draw similarities between action sequences which cause large changes in TD error.
policy trained using DDPG on the HalfCheetah
task. The high variance in the TD error suggests the presence of critic overestimation bias (Thrun & Schwartz). which could be remedied by using algorithms known to reduce the impact of this issue (Fujimoto al. 2018. Hasseli 2010). Figure&Jshows how the user the agent trajectory. Hovering over regions of potential interest in the trajectory viewport allows the user to find action sequences that cause high variance in the TD error. similar technique could be used to visualize clusters of states in the replay buffer space with high TD error (Figure This approach could enable the user identify patterns states across space or time that persistently have high TD error, and design methods to mitigate this (Amodei al. 2016).
compare the TD error along
Another approach the user could take is to generate a distribution viewport (Figure 4.. and identify the distribution of actions in the vicinity of states with high TD error. If similar states persistently have a higher action and/or reward variance, this suggests that the usage of variance reduction tech- niques could help learning (Schulman et al., 2018 Romoff al. 2018). Once promising avenues for modification have been identified, the user can make guided changes, and retrain the policy.
6 CONCLUSION
In this paper, have introduced visualization tool, Vizarel, that helps interpret and debug RL algorithms. Existing tools which we use to gain insights into our agent policies and RL algorithms constrained by design choices that were made for the supervised learning framework. To that end, we identified features that an interactive system for debugging and interpreting RL algorithms should encapsulate, built instantiation of this system which plan release as open source tool, and provided walkthrough of an example workflow of how the system could be used.
There are multiple features under development that contribute towards both the core system. One feature is the integration of additional data streams such as saliency maps (Greydanus et al. 2018) to complement the state viewport. Another is designing the capability use the system domains that lack a visual component, for example in healthcare YYu et al.. (022)) or education (Reddy et al.) An extension to add search capabilities that allow the user to easily traverse, query, and identify regions of interest in the replay buffer viewport.
Vizarel suggests number of avenues for future research. First, we hypothesize that it could help design metrics that better capture priority during experience replay (Schaul et al. (016). Second, it could help the researchers create safety mechanisms early on in the training process through identi- fying patterns in agent failure conditions (Amodei et al. 2016). Another possible research direction this tool catalyzes is the construction reproducible visualizations through plugins integrated into the core system.
We anticipate that the best features yet to be built will emerge through iterative feedback, deploy- ment, and usage in the broader reinforcement learning and interpretability research communities.
8

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
SVD is supported by the CMU Argo AI Center for Autonomous Vehicle Research. BE is supported by the Fannie and John Hertz Foundation and the National Science Foundation (DGE1745016). Any opinions, findings, recommendations, and conclusions expressed this material are those of the author(s) and do not reflect the views of funding agencies.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens. Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: / available from tensorflow.org.
:/wps:///wwww./ /www wwnsorflww org/ Software
Tanmay Agarwal, Hitesh Arora, Tanvir Parhar, Shubhankar Deshpande, and Jeff Schneider, Learn- ing to Drive using Waypoints. pp. 7.
Dan Amir and Ofra Amir HIGHLIGHTS: Summarizing Agent Behavior to People. pp. 9.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete Problems in Al Safety. /606. [es], July 2016. arXiv: 1606.06565.
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory explanatory: Counterfactual analysis of saliency maps for deep reinforcement learnin In International Conference on Learn- ing Representations, 2020. URL http:: /ttpsss/s////eeeerriii http//ppnnreeeeeeeeeeeeeee net. forum pofoormmid=rk111 3ml BFDB
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Envi- ronment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013. ISSN 1076-9757. doi: 10.1613/jair.391 arXiv: 2707.4708.
Davide Calvaresi, Amro Najjar, Michael Schumacher, and Kary Främling. Explainable, Transparent Autonomous Agents and Multi-Agent Systems: First International Workshop, EXTRAAMAS 2019, Montreal, QC, Canada, May 13-14, 2019, Revised Selected Papers, volume 11763. Springer Nature, 2019.
Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuska. Experience Selection in Deep Rein- forcement Learning for Control. pp. 56.
Finale Doshi- and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning. arXiv: 08608 [cs, stat], March 2017. arXiv: 1702.08608.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning. arXiv:2004.072//9 (cs, stat], June 2020. arXiv: 2004.07219.
Scout Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. arXiv:1802.00447. [cs, stat]. October 2018. arXiv: 1802.09477.
Yosuke Fukuchi, Masahiko Osawa, Hiroshi Yamakawa, and Michita Imai. Autonomous self-
explanation of behavior for interactive reinforcement learning agents. Proceedings of the 5th In-
ternational Conference on Human Agent Interaction, Oct 2017. doi: 10.1145/31253993115544
URL http: www.doi.oooooo. /ttpr///..111/3115533333
Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and Understanding Atari Agents, arXiv:1711.001: [cs], September 2018. arXiv: 1711.00138.
Hado V. Hasselt. Double Q-learning, In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2613- 2621. Curran Associates, Inc., 2010.

---

Under review as a conference paper at ICLR 2021
Bradley Hayes and Julie A. Shah. Improving Robot Controller Transparency Through Autonomous Policy Explanation. In Proceedings of the 2017 ACM/IEEE International Conference on Human- Robot Interaction, pp. 303-312, Vienna Austria, March 2017. ACM. ISBN 978-1-4503-4336-7 doi: 10.1 00.1145/29098243022222
Sandy H. Huang, David Held, Pieter Abbeel, and Anca D. Dragan. Enabling Robots to Communi- cate their Objectives. Robotics: Science and Systems XIII, July 2017. doi: 10.15607/RSS.2017 XIII.059. arXiv: 1702.03465.
Peter J. Huber. Projection Pursuit. The Annals of Statistics, 13(2):435-47) June 1985. ISSN 0090-5364. doi: 10.1214/aos/ 176349519.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi Velez. Explainable rein- forcement learning via reward decomposition. In ICCA//ECAI Workshop Explainable Artificial Intelligence, 2019.
Minsuk Kahng, Pierr Y. Andrews, Aditya Kalro, and Duen Horng Chau. ActiVis: Visual Ex- ploration of Industry-Scale Deep Neural Network Models. arXiv:1704.01942 [cs, stat], August 2017. arxiv: 1704.01942.
Ashish Kapoor, Bongshin Lec, Desney Tan, and Eric Horvitz. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference Human Factors in Com- puting Systems, CHI '10 pp. 1343-1352, Atlanta, Georgia, USA, April 2010. Association for Computing Machinery. ISBN 978- 177--1605558-929-) doi: 10.1145/1753326.175555 1753529.
Josua Krause, Adam Perer, and Kenney Ng. Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models. In Proceedings of the 2016 CHI Conference Hu- man Factors in Computing Systems, CHI "16, pp. 5686-5697, San Jose, California, USA, May 2016. Association for Computing Machinery. ISBN 978-1-4503-3362-7, doi: 10.1145/2858036. 2858529.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553))3364444 May 2015. ISSN 0028-0836, 1476-4687. doi: 10. 1008/nature14530
Long-Ji Lin. Self-improving reactive agents based reinforcement leamiing planning and teaching. pp. 29.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Distal explanations for model-free explainable reinforcement learning, 2020.
Sean McGregor, Hailey Buckingham, Thomas G Dietterich, Rachel Houtman, Claire Montgomery, and Ronald Metoyer. Facilitating testing and debugging of markov decision processes with inter- active visualization. In 2015 IEEE Symposium Visual Languages and Human-Centric Com- puting (VLJHCC), pp. 53-61. IEEE, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv: [cs], December 2013. arXiv: 1312.5602.
Emre O. Neftci and Bruno B. Averbeck. Reinforcement learning in artificial and biological systems. Nature Machine Intelligence, 1(3):133-143, March 2019. ISSN 2522-5839. doi: (0.10088/42255-01990000
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(11):e7, November 2017. ISSN 2476-0757. doi: 10.23915/dissti..0000
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The Building Blocks of Interpretability. Distill, 3(3)e10, March 2018.
ISSN 2476-0757. doi: 10.2391 10.23915/distill.oo 5/distill.00010..
Erika Puiutta and Eric Veith. Explainable reinforcement learning:
survey. arXiv preprint
a7Xiv:2005.06647, 2020.
10

---

Under review as a conference paper at ICLR 2021
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2020. URL
https: httpps////peeeeeee....r www.neeeiiieeeee.......e.m...... www.:nntt/foummmmmmm forum? 1155m
Martin L. Puterman. Markov decision processes: discrele stochastic dynamic programming. Wiley series probability and statistics. Wiley-Interscience, Hoboken, NJ, 2005. ISBN 978-0-471- 72782-8. OCI.C: 254152847.
Siddharth Reddy, Sergey Levine, and Anca Dragan. Accelerating Human Learning with Deep Re- inforcement Learning. pp. 9.
Siddharth Reddy, Anca D Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning human objectives by evaluating hypothetical behavior. arXiv preprint 1rXiv:1912.055522, 2019.
Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavel, and Joelle Pineau. Re- ward Estimation for Variance Reduction in Deep Reinforcement Learning. arXiv:1805.03339 [es, stat], November 2018. arXiv: 1805.03359.
Christian Rupprecht, Cyril Ibrahim, and Christopher J. Pal. Finding and visualizing weaknesses of deep reinforcement learning agents. In International Conference on Learning Representations, 2020. /RRLitttps:/ppeeeeiieee.. wwtp://www.eeee.ooo...eooe/e//a.
Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. Vega-Lite: A Grammar of Interactive Graphics. IEEE Transactions on Visualization and Computer Graphics, 23(1):341-350, January 2017. ISSN 1077-2626. doi: 10. 1099TVVC
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. arXiv:1511.05952 [es], February 2016. arXiv: 1511.05952.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. 17X.0:1506.02438 [cs]. October 2018. arXiv: 1506.02438.
Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable exppliinableriiifooeeeeemttttta learn- ing: Understanding agents' capabilities and limitations. Artificial Intelligence, 288:103367, Nov 2020. ISSN 0004-3702. doi: 10.
103367. URL
1016/j. www.rtttttt........mmm
2020. 103367
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv:1312.6034 [cs], April 2014.
arXiv: 1312.6034.
Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M. Rush. LST- MVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks.
arXiv:1606.0744 07661 [cs], October 2017. arXiv: 1606.07461.
Richard S. Sutton and Andrew Barto. Reinforcement learning: an introduction. Adaptive com- putation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-039924-6.
Richard H. Thaler and Cass R. Sunstein. Nudge: improving decisions about health, wealth, and happiness. Penguin Books, New York, rev. and expanded ed edition, 2009. ISBN 978-0-14- 311526-7.
Sebastian Thrun and Anton Schwartz. Issues in Using Function Approximation for Reinforcement Learning. pp. 9.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. 9:2579-2605, 2088.
Jasper der Waa. Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerinex. Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences. arXiv: [es, stat), July 2018. arXiv: 1807 607.08706.

---

Under review as a conference paper at ICLR 2021
Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B. Viegas, and Martin Wattenberg. Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow. IEEE Transactions Visualization and Computer doi: 10. N0.109//VCG.2217.2448777
Graphics, 24(1):1-12, January 2018. ISSN 1077-2626.
Ji Soo Yi, Youn ah Kang. John Stasko, and J.A. Jacko. Toward a Deeper Understanding of the Role of Interaction Information Visualization. JEEE Transactions on Visualization and Computer Graphics, 13(6):1224-1 12311 November 2007. ISSN 1077-2626, 1941-0506, 2160-9306. doi:
10.1109/TVCG..007.77 70515.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Networks Through Deep Visualization. pp. 12.
Understanding Neural
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement Learning in Healthcare: A Survey. arXiv: (ato:1908,07999 [cs], April 2020. arXiv: 1908.08796.
Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks.
arXiv:1311.290 [cs]. November 2013. arXiv: 1311.2901.
Shangtong Zhang and Richard S. Sutton. Deeper Look Experience Replay. arXiv: [cs], April 2018. arXiv: 1712.01275.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
ADDITIONAL VIEWPORT DETAILS
TENSOR COMPARISON VIEWPORT
For environments that have higher dimensional action spaces, it is hard for the user to understand how neighboring points in the replay buffer viewport differ. This becomes especially relevant for diagnosing clusters of points that have a higher TD error. The tensor comparison viewport (Figure 9 enables the user to easily select points and then compare them along the dimensions of interest, which for example could be actions. Dimensions that have a standard deviation beyond a speci- fied threshold are automatically highlighted, which enables the user to focus the dimensions of interest.
Replay Buffer
Tensor Compartion Viewport
PRODOUE
Figure 9: Tensor Comparison Viewport Selecting points (dashed green line) in replay buffer viewport (4.2.1). and generating (dashed red line) the tensor comparison viewport, allows the user to compare tensors (e.g. actions or states), where dimensions of higher variance are automatically highlighted. This could lead to faster debugging in environments where each dimension corresponds physically intuitive quantities.
CONSTRUCTING VIEWPORTS
We describe how a user could create new viewport through an example. However, we defer an extended discussion the system documentation, since reading the source code and give the user more insight. As mentioned in Section 4 viewports are generated by chaining together different visualization elements. Various viewports we introduced along with the visualization elements they make use of are:
1. image buffers: state viewport
2. line plots: action viewport, reward viewport, trajectory viewport
3. scatter plots: replay buffer viewport
4. histograms: distribution viewport
Note that these primitives are not fixed and are bound to change the creation of different viewports necessitates their expansion. However, we've found them be good starting point to provide the minimal functionality required to construct new viewports. For example the construction of a saliency map viewport, could be done using an image buffer and line plot.
There exist utilities in the system to handle the rollout of agent policies, and storage of generated metadata. However, the user would still need provide code to generate saliency maps Once the metadata has been generated, the user specifies which viewports to generate (e.g. core viewports and custom viewports) along with a visual layout for the dashboard. The system then generates an interactive interface with the specified viewports and layout, that the user can use to debug the agent policy and perform further analysis.
This viewport could further be incorporated as a plugin or extension to the core system and dis- tributed community of users in the future. The exact details for this not concrete yet, since we expect there to emerge a robust process through iterative design changes, as the tool finds broader usage.
for which there exist open source tools
13

---

Under review as a conference paper at ICLR 2021
ALGOITHTM
REPLAY BUFFER VIEWPORT: PROJECTION
Algorithm 1: Procedure to generate samples for the replay buffer viewport
Result:
Data samples from replay bufter transformed from original space R= to lower dimensional space IR where n. >>
Load data samples d, from replay buffer into memory:
Where, d, (s,, ag T,, $1+1), (0, T') and T' is the episode timestep;
for i=0;; T; i++ do
// create data matrix from replay buffer samples; data[i] = d;;
end
transform compute wwwppte transform(data, type); where, type [PCA, TSNE, UMAP] reduced_pooin transform(data);
REPLAY BUFFER VIEWPORT: SAMPLE INFLUENCE VISUALIZATION
Algorithm 2:
Procedure compute visual size of data samples in the replay buffer viewport Visual size data samples in replay buffer viewport transformed proportion of their influence the agent policy
Result:
Load data samples d, from replay buffer into memory:
Where, d, (si, di, Ti, 81+1), i E (0, T) and T is the episode timestep;
Load sample metadata my into memory;
Where, m, stores the TD error for each sample (for relevant algorithms);
for i=0; << T; i++ do data[i] m;
end
min = min(data);
d_max max(data)),
for i=0; i<, T: i++ do
|/ p[i] stores the normalized TD error value, and (cmmin, c.max) is the range to which values are mapped;
p[i] -d min min
d.mas-d_min d-min (c_max c_min) + (c_min);
|| r[i] stores the radius of the point be plotted in the replay buffer viewport;
rl
r[i]
end
14

---

Under review as a conference paper at ICLR 2021
Replay Bulfer
0
19
62
ProckatOe11
Figure 10: Visualizing the replay buffer for hard exploration tasks Tasks such as Montezuma's revenge are classic examples of hard exploration tasks. Here show how the replay buffer viewport, help visualize the distribution of data samples the replay buffer.
ALTERNATE ENVIRONMENTS
HARD EXPLORATION
We've also experimented with using this tool for procedural, hard-exploration tasks such as Mon- tezuma's 's Revenge. Figure T0 shows how the replay buffer viewport can be used visualize the distribution of data samples in the replay buffer. Since, the observations returned from the environ- ment are images, we extract the embeddings computed by the feature extraction model the agent policy, and use these for the projection technique described in the Appendix (Algorithms).
USER STUDY
We conducted user study cases a tool such as the one described and implemented in this paper would serve.
[ from RL users for feedback and an extended evaluation of potential use
These questions were of both numerical and subjective type. We now list both types along with preliminary results
NUMERICAL QUESTIONS
On scale of 0-10, do you think Vizarel would help you identify bugs in your RL algo- rithms? Average: 7.5
On scale 0-10, you think Vizarel would help you identify improvements your RI. algorithms? Average: 7.0
On a scale of 0-10,, do you think Vizarel would help you understand whether your RI. algorithms are working as intended? Average: 8.0
SUBJECTIVE QUESTIONS (SELECTED)
Are there specific settings where you think this tool might help answer questions that you might otherwise not easily been able to?
To show the effects of changes in reward function coefficients
Surfacing important points in the agent trajectory history
Which features do you think are missing and would be a useful addition to have?
Add the capability to search over the replay buffer viewport and filter events based on search criteria.
Provide a documented approach to load agent policies
More details on how to create plugins
Results from this are preliminary, the survey is still in progress
15

---

Under review as a conference paper at ICLR 2021
Display agent trajectories in the replay buffer viewport
Future work along this direction would include creating test scenarios for debugging, and running an A/B test for users, contrasting their experience with existing tools vs the proposed tool along dimensions of efficacy in debugging RI. algorithms.
PERFORMANCE
We've run measured preliminary performance metrics to help provide insight into how much over- head running this system would create" These numbers were collected for a visualization of an agent trained using DDPG the HalfChectah-v2 task, about 35% of the way to task completion. The vizarel interface was generated on a machine with an Intel(R) Xeon(R) CPU ES-2620 v4 2. 10GHzz processor, with a 4TB HDD, and 128 GB RAM.
Tasks
Time
Generate Viewports
45 sec
Load Dashboard
sec
Policy Rollouts
minutes
Logging Overhead Fraction (Relative Tensorboard)
Note that the policy rollout time is conditional the length of the episode trajectory. These were collected for the HalfCheetah-ve v2 task
"Note that these numbers may not be indicative of the final numbers, since the endebase is under active development
16


Reviews:
[{"review_id": "uK47hEbGcwu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with debugging of black-box deep reinforcement learning (RL) agents to better understand and fix their policies. The authors propose diverse tools for, among others, visualizing the state space in terms of calculated statistics, analyzing the taken actions across learning episodes or exploring the replay buffer. The authors also propose a workflow for using the proposed tools. The resulting Vizarel tool is evaluated in terms of an exemplary walkthrough.\n\nThe approach follows an interesting direction towards explaining RL agents, but I am missing concrete design decisions and empirical evaluations for the proposed set of visualizations. While evaluating interpretability/explainability is difficult in general, it is still essential for such kind of contribution. I feel that the authors should explore some kind of user study (as conducted in cited works, such as contrastive RL explanations [1]), where end-users need to solve a challenging RL-related task and use the tools for actual debugging/search for improvements. I am aware that other explainable RL approaches based on counterfactuals or attentions might be easier to evaluate (and might not require end-users for acceptance at a conference), but I still feel a deeper evaluation is necessary here. To this end, an evaluation then should include mentioned related works on explainable RL in order to empirically prove the superiority for specific tasks / use cases. \n\nTo this end, I am wondering it which situations the tool would be beneficial over other explainable RL approaches. For example, does the approach work well for procedural, hard-exploration tasks? More specifically, which tools of the framework would I use and which potentially not? Again, I feel like such questions can only be answered by asking actual end-users. \n\nLastly, there seem to be numerous minor errors and in the references, as publication years are often missing. \n\n[1] van der Waa, J., van Diggelen, J., Bosch, K.V.D. and Neerincx, M., 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv preprint arXiv:1807.08706.\n\n--- Update after author response period ---\nThank you for the clarifications! After reading the other reviews and the paper updates, I still feel that the paper requires an additional, empirical evaluation to prove the value and contribution of the approach. Otherwise I find it hard to tell to what extent / in which situations / for which user group the tool is useful. While I appreciate that the authors made changes to their manuscript based on the reviewers' comments, I thus keep my recommendation for rejection for the current version of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fjKDdnP_oms", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a new interactive visualization tool for better debugging of RL algorithms. The tool provides two fundamental views, spatial and temporal views of the data generated by an RL algorithm. Various viewports under the two different views are described in details. \n\nThis tool is definitely useful to many RL researchers because debugging RL algorithms is known to be more difficult than traditional ML algorithms. The paper appears to have covered every component of the tool in details and a brief example of walkthrough. However, I feel that the paper has not yet sufficiently demonstrated the proposed tool’s full value. The tool provides a number of viewports, each covering a particular aspect of the RL algorithm under investigation. Is there any mechanism for the user to jointly operate several viewports (e.g., state, action, reward) together? What are the use cases that would be valuable to many users? The paper briefly describes one, but I think a better way is to offer multiple detailed case studies detailing the following points: 1. what is the problem being addressed (since we are debugging)? 2. How does the user operate through a combination of different viewports to find the source of the problem? This way, we will have a better idea on when the tool is most effective and how the different components contribute to the values created.\n\nMinor comment: The citation (noa, 2020a) under 4.2.1 seems incorrect.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a_ZoKmgLLj_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is introducing a tool intended for analysing RL processes based on visual information, in order to help the researchers to understand what is happening with the agent environment interaction. The system has varied options for plotting useful information that researchers/engineers normally have to analyze for debugging, but additionally it allows the user to have interaction data in the plots and visualizations of the environments.\n- This kind of works are promising and have some valuable contribution in order to unify and standardize the tools in the community, so that the reproducibility is more straightforward, as it has been with some RL libraries, sets of environments, or even the libraries for deep learning. However, the paper is too much focused on the side of showing what can be visualized by the users interactively, and it is missing more technical information about it. I assume there will be manuals, readmes, or so, but some technical information about the implementation of the tool would make the paper more useful for the readers, otherwise, going through this paper will not be necessary at all once the repo of the tool is released along with its documentation (making the published paper only useful for citing the use of the tool).\n- Does the tool bring a library of algorithms and environments? or is it built to be compatible with any implementation? If that is the case, what is the architecture of the system? which helps the users to understand how to connect their code with the tool. Without this information, it is not possible for the users to extend its use to other applications.\n- Figures need to be more connected with text in the paragraphs, some are not mentioned and some are mentioned very far from its place.\n- In temporal views it could be also possible to visualize value functions, advantage functions, returns, and also the cost functions computed for training the models (NNs).\n- In Section 4.1.2 it is mentioned \"This idea can easily be extended to agents with stochastic actions, where we could generate a viewport using histograms to visualize the change in action distribution over time\", this indeed might be a better example rather than the one given in Figure 2.\n- Also in Section 4.1.3  it is mentioned \"The viewports discussed so far can be combined to provide the user more insight into the correspondence between states (stateviewport), actions (action viewport), and the components of the reward function (reward viewport)that the agent is attempting to maximize.  Such a visualization could help alert the researcher toreward hacking (Amodei et al.) and thus design reward functions that are immune to this problem.\", a good example in a Figure would be very depicting.\n- Can this tool be used to interactively change settings while running learning processes? e.g. graphically modifying reward functions, selecting regions of points in the replay buffer to be sampled more often or to be ignored, etc.\n- In section 4.2.1 it is mentioned \"We can instead visualize the data samples by transforming the points (van der Maaten & Hinton, 2008) to a lower-dimensional representation. This technique helps visualize the distribution of samples in the replaybuffer, which is a visual representation of the replay buffer diversity\". It would be good to add the name of the technique, not only the reference.\n- What does  \"ask questions\" means in Section 4.2.2, in the sentence: \"The distribution viewport (Figure 4) complements the replay buffer viewport by allowing the user to select clusters of data samples and ask questions...\"\n- It would be good to add some comments about computational performance and limitations.\n- Is it useful for doing experiments with real physical systems?\n- Small detail, Figure 2 is not split in left and right.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4AW79rY48nQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\t1. Summary of Paper\n\t\ta. This article contributes a framework and tool for visualising data collected from a policy during RL training. It also contributes a set of views for visualising RL training data that could be used in general. Finally, the paper contributes a high-level workflow and a set of example use cases for how this tool might impact debugging a training session.\n\t2. Strengths\n\t\ta. Problem Motivation\n\t\t\ti. The visualisation process for debugging/understanding RL training is certainly different from supervised learning use case. There is a need for a general tool to perform debugging on RL-trained models. This could be partially satisfied using visualisation methods based on the samples collected during training.\n\t\tb. Clarity\n\t\t\ti. The article provides a supplementary link in the main manuscript to an anonymised demo of the tool on (potentially) simulated data. This really helps understand the use of the tool by interacting with it. This was particularly helpful to understand the descriptions in the static manuscript.\n\t\t\tii. The article also used figures effectively providing a clear understanding of each type of viewport described. \n\t\tc. Technical Approach\n\t\t\ti. The usage of simple visual effects to guide the user's attention and \"nudge\" them is useful.\n\t3. Weaknesses\n\t\ta. Anonymisation Failure in References\n\t\t\ti. A reference uncited in the manuscript body contains a non-anonymised set of author names to a paper with the same title as the system presented in this paper. This was not detected during initial review. \"Shuby Deshpande and Jeff Schneider. Vizarel: A System to Help Better Understand RL Agents.arXiv:2007.05577.\"\n\t\tb. Citations\n\t\t\ti. There is egregiously missing information in almost all citations. This is quite obvious in all the missing dates in the manuscript text.\n\t\tc. Clarity\n\t\t\ti. There are a few unclear or misleadingly worded statements made as below:\n\t\t\t\t1) \"However, there is no corresponding set of tools for the reinforcement learning setting.\" - This is false. See references below (also some in the submitted paper).\n\t\t\t\t2) \"stronger feedback loop between the researcher and the agent\" - This is at least confusing. In any learning setting, there is a strong interaction loop between experimentation by the researcher and resulting outcomes for the trained model.\n\t\t\t\t3) \"To the best of our knowledge, there do not exist visualization systems built for interpretable reinforcement learning that effectively address the broader goals we have identified\" - It isn't clear what these broader goals are that have been identified. Therefore it isn't possible to evaluate this claim.\n\t\t\t\t4) \"For multi-dimensional action spaces, the viewport could be repurposed to display the variance of the action distribution, plot different projections of the action distribution, or use more sophisticated techniques (Huber).\" - It would be clearer to actually state what the sophisticated techniques from Huber are here.\n\t\t\tii. The framework could be clearer that it applies most directly as described on single agent RL. The same approach could be used with multi-agent RL but the observation state and visualisations around that get more confusing when there are multiple potentially different sets of observations. Is this not the case, please clarify?\n\t\td. Experimental rigour\n\t\t\ti. The paper does include 3 extremely brief examples of how the tool might be used. However, it does not include any experiments to suggest that this tool would actually improve the debugging process for training RL in a real user study. Not every paper requires a user study, however, the contributions proposed by this particular manuscript require validation at some level from actual RL users (even a case study with some feedback from users would address this to some extent).\n\t\te. Novelty in Related Work\n\t\t\ti. The manuscript contains references to several relevant publications that can be compared to the current work. However, the paper is also missing references to many related and relevant works in the space of debugging reinforcement learning using visualisations, especially with an eye towards explainable reinforcement learning. See a sampling below.\n\t\t\t\t1) @inproceedings{ Rupprecht2020Finding, title={Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents}, author={Christian Rupprecht and Cyril Ibrahim and Christopher J. Pal}, booktitle={International Conference on Learning Representations}, year={2020}, url={https://openreview.net/forum?id=rylvYaNYDH} }\n\t\t\t\t2) @inproceedings{ Atrey2020Exploratory, title={Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning}, author={Akanksha Atrey and Kaleigh Clary and David Jensen}, booktitle={International Conference on Learning Representations}, year={2020}, url={https://openreview.net/forum?id=rkl3m1BFDB} }\n\t\t\t\t3) @inproceedings{ Puri2020Explain, title={Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution}, author={Nikaash Puri and Sukriti Verma and Piyush Gupta and Dhruv Kayastha and Shripad Deshmukh and Balaji Krishnamurthy and Sameer Singh}, booktitle={International Conference on Learning Representations}, year={2020}, url={https://openreview.net/forum?id=SJgzLkBKPB} }\n\t\t\t\t4) @article{reddy2019learning, title={Learning human objectives by evaluating hypothetical behavior}, author={Reddy, Siddharth and Dragan, Anca D and Levine, Sergey and Legg, Shane and Leike, Jan}, journal={arXiv preprint arXiv:1912.05652}, year={2019} }\n\t\t\t\t5) @inproceedings{mcgregor2015facilitating, title={Facilitating testing and debugging of Markov Decision Processes with interactive visualization}, author={McGregor, Sean and Buckingham, Hailey and Dietterich, Thomas G and Houtman, Rachel and Montgomery, Claire and Metoyer, Ronald}, booktitle={2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)}, pages={53--61}, year={2015}, organization={IEEE} }\n\t\t\t\t6) @article{puiutta2020explainable, title={Explainable Reinforcement Learning: A Survey}, author={Puiutta, Erika and Veith, Eric}, journal={arXiv preprint arXiv:2005.06247}, year={2020} }\n\t\t\t\t7) @book{calvaresi2019explainable, title={Explainable, Transparent Autonomous Agents and Multi-Agent Systems: First International Workshop, EXTRAAMAS 2019, Montreal, QC, Canada, May 13--14, 2019, Revised Selected Papers}, author={Calvaresi, Davide and Najjar, Amro and Schumacher, Michael and Fr{\\\"a}mling, Kary}, volume={11763}, year={2019}, publisher={Springer Nature} }\n\t\t\t\t8) @inproceedings{juozapaitis2019explainable, title={Explainable reinforcement learning via reward decomposition}, author={Juozapaitis, Zoe and Koul, Anurag and Fern, Alan and Erwig, Martin and Doshi-Velez, Finale}, booktitle={IJCAI/ECAI Workshop on Explainable Artificial Intelligence}, year={2019} }\n\t\t\t\t9) @misc{sundararajan2020shapley, title={The many Shapley values for model explanation}, author={Mukund Sundararajan and Amir Najmi}, year={2020}, eprint={1908.08474}, archivePrefix={arXiv}, primaryClass={cs.AI} }\n\t\t\t\t10) @misc{madumal2020distal, title={Distal Explanations for Model-free Explainable Reinforcement Learning}, author={Prashan Madumal and Tim Miller and Liz Sonenberg and Frank Vetere}, year={2020}, eprint={2001.10284}, archivePrefix={arXiv}, primaryClass={cs.AI} }\n\t\t\t\t11) @article{Sequeira_2020, title={Interestingness elements for explainable reinforcement learning: Understanding agents’ capabilities and limitations}, volume={288}, ISSN={0004-3702}, url={http://dx.doi.org/10.1016/j.artint.2020.103367}, DOI={10.1016/j.artint.2020.103367}, journal={Artificial Intelligence}, publisher={Elsevier BV}, author={Sequeira, Pedro and Gervasio, Melinda}, year={2020}, month={Nov}, pages={103367} }\n\t\t\t\t12) @article{Fukuchi_2017, title={Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agents}, ISBN={9781450351133}, url={http://dx.doi.org/10.1145/3125739.3125746}, DOI={10.1145/3125739.3125746}, journal={Proceedings of the 5th International Conference on Human Agent Interaction}, publisher={ACM}, author={Fukuchi, Yosuke and Osawa, Masahiko and Yamakawa, Hiroshi and Imai, Michita}, year={2017}, month={Oct} }\n\t4. Recommendation\n\t\ta. I recommend this paper for rejection as the degree of change needed to validate the contribution through a user study or other validation is likely not feasible in the time and space needed. However, the contribution is potentially valuable to RL, so with the inclusion of this missing evaluation and additions to related work/contextualisation of the contribution, I would consider increasing my score and changing my recommendation.\n\t5. Minor Comments/Suggestions\n\t\ta. It is recommended to use the TensorFlow whitepaper citation for TensorBoard (https://arxiv.org/abs/1603.04467). This is the official response (https://github.com/tensorflow/tensorboard/issues/3437).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We also had some discussions about the paper that are not visible to the authors. To summarize: the reviewers appreciated the efforts the authors put into the replies and updates. While those clarifies quite a few points, the paper unfortunately is still not publishable in its current form at ICLR.

Overall the paper tackles a very relevant and important question, proposing a tool that could be extremely useful for research on RL.
On the downside the paper is mainly descriptive, outlining WHAT the tool can do. Multiple reviewers pointed out that deeper, new insights are missing, e.g., WHY certain features were included and whether the tool actually is helpful for practitioners. A user study has been commenced, which is an excellent step in this direction.
Decision: Reject