State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
Authors: Devleena Das
Sonia Chernova
Been Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Concept-Based Explanations
Reinforcement Learning
Human-AI Interaction

Abstract: As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining ``concepts'' in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
Devleena Das
Sonia Chernova
School of Interactive Computing Georgia Institute of Technology ddas41 ddas410gatech. edu
School of Interactive Computing
Georgia Institute of Technology
chernova@gatech.c edu
Been Kim
Google Research
beenkim@google.coo com
Abstract
As more non-Al experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-Al experts. Towards this effort, leveraging higher-level concepts and producing concept- concept-based explanations have become a popular method. Most concept-based explanations have been developed for clas- sification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute desiderata for defining "concepts" in sequential decision making settings. Additionally, in- spired by the Protégé Effect which states explaining knowledge often reinforces one's seif-learning, we explore how concept-based explanations of an RL agent's s decision making in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's 's decision making. To this end, we contribute unified framework. State2Explanation (S2E), that involves learning joint em- bedding model between state-action pairs and concept- leveraging such learned model to both (1) inform reward shaping during agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully inform- ing reward shaping and improving agent learning rate, well significantly improving end user task performance at deployment time.
explanations, and
1 Introduction
Black-box Al systems are increasingly being deployed to help end-users with everyday tasks. Exam- ples include doctors leveraging decision support systems to aid in diagnosis [39. warehouse managers
relying on robots for goods transportation [8], and drivers using autonomous vehicles for assisted driving 18]. To increase the transparency of these black-box models, researchers have developed numerous techniques to provide explanations of agent decision making [45
23
51 14
30 15].
A popular method towards non-expert friendly explanations has been to attribute higher-level "con- cepts" to an agent's decision making, and these concepts have primarily been used to explain Nassification-based AI systems 30
33 54 19]. An example of a concept-based explanation for a classification model includes, "wing color" for a bird class label [19]. sequential decision making, concept-based explanations have been less widely explored, and existing works leverage preconditions of states and action costs [48) or logical formulas [2 as "concepts".
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we posit that concept-based explanations may not only benefit the end user but also benefit the AI agent. Our claim loosely motivated by the Psychological phenomenon. the Protégé Effect, that states explaining material to another student also helps the explainer learn and reinforce material 27 17 Thus the objective of our work is to demonstrate the dual-benefit of concept-based explanations to both the end-user for improved understanding, as well as the Al agent for improved learning rate. Note, prior works, typically in Reinforcement Learning (RL), have explored various methods to improve agent learning, common approach including the use of human feedback and natural language commands as a method of reward shaping to improve RL agent sample efficiency
and learning rate [8 50 20]. However, these methods have focused on leveraging language one-way benefit to the agent, not considering how language provided to the agent during training may also benefit end-user understanding at deployment. Therefore in work, explore the following research question: Can ancept-based explanations have a two-way benefit to both the user and agent, such as improve end-user understanding of a deployed agent's decision making, while also improving the agent's ' learning rate via reward shaping? ?" As a solution, we contribute a unified framework, State2Explanation (S2E), that learns single joint embedding model between agent states and associated concept-based explanations and leverages such learned model to (1) inform reward shaping for agent learning, and (2) improve end-user understanding of the agent's decision making at deployment.
Additionally, we contribute desiderata of what entails "concept" in concept- based explanation for sequential decision making systems. Given that sequential decision making agents engage in long- term interaction with their environment, posit that the scope of concept-based explanations should span beyond representing preconditions and action cost [48), and control logic (22). Specifically, believe concept-based baceeptt-aaddddteee explanations in sequential decision making should function much higher-level of abstraction, highlighting knowledge that are applicable across multiple states, and most importantly, expressing positive or negative influence towards the agent's goal. Our claim is motivated by the importance of the agent's goal in sequential decision making formulation [4]
Contributions. (1) We provide desiderata for what constitutes "concept" in concept-based ex- planations of sequential decision making. (2) We introduce a novel framework, State2Explanation (S2E) which learns a joint embedding model between agent states aaction-acionnno pairs and concept- based explanations to provide a benefit to both the agent, by informing reward shaping. as well as the end user by providing explanations that improve user task performance. We perform both model and user evaluations of our S2E framework in two complex RL. domains, Connect 4 and Lunar Lander.
Our model evaluations demonstrate that S2E can successfully inform reward shaping, comparable to expert-defined reward functions. Additionally, our user evaluations demonstrate that S2E can successfully provide meaningful concept-based explanations to end users such that exposure to our explanations significantly improve user task performance.
2 Related Work
In this work, we explore the utility of concept-based explanations in providing a two way benefit to both the AI agent during training, via reward shaping, as well as end user at deployment time, via improved user task performance. Below, we highlight related prior work.
Reward Shaping using Language Many prior works have focused on improving an agent's learning rate or sample efficiency [35] 21). Closely related to our work are methods providing reward shaping via language. Specifically, 38 propose ELLA, which leverages relevance classifier to associate a lower level instruction to a higher level task for providing a subtask shaping reward. In (20). the authors leverage natural language instruction to derive potential-base shaping rewards that encourage an RL agent to more frequently select relevant actions in the game of Montezuma's 's Revenge. Similarly, (50] learn a joint embedding model between subgoal language commands and agent states to provide shaping rewards for completing subtasks in StarCraft II. Specifically, [50] demonstrate that their model can generalize to unseen, semantically similar language commands, and improve agent learning rate. In our work, we take inspiration from [50], and learn joint embedding model between agent states and associated natural language commands to inform agent reward shaping. However, our work differs in that our natural language commands are concept-bused explanations of the associated state, and not subgoal descriptions required for task completion. By learning& common embedding space between state explanations and state representations, our work

---

provides mwv-way benefit to both the RL agent for learning as well as end-user for improved understanding.
Explanations for non-Al Experts Prior works have established that concept bansep explanations
for classification tasks provide explanations that are meaningful to end-users [30 33 54 [9]. Methods
for concept-based based explanations have included Concept Activation Vectors B0], Concept Bottleneck
Models [33], and Concept Embedding Models [34]. For classification tasks, "concepts" have primarily
features that model's prediction is sensitive (i.e. 'wing color' for image classification.) 33 Related to sequential decision making, (26) provide concept- explanations of 3D action recognition CovNets by clustering learned, human- iimaaninteerretabble features (i.e. "grass" or "hand"" from segmented videos. In 488 concept-based explanations for sequential decision making are formulated in relation to state preconditions (e.g. "the action move-left failed in the state as the precondition skull-not-on-left was false in the state") and action costs (e.g.. "executing the action attack in the presence of the concept skull-on-lege will cost at least 500"). Specifically, in [8). concepts have represented any factual statement user associates to given state, such as "skull-on-left" in the previous example. Similarly, in [22], concepts have represented logical formulas for summarizing RL policies. Additionally, (29 propose prototype wrapper network for
represented human-interpretabh
training interpretable RI. policies with user-defined prototypes. Similar to 48.. the authors in [29] define prototypes as factual observations about state, such as "turn right" or "turn left". Other explanation generation methods have demonstrated that end user understanding is improved when
explanations are presented in natural language rather than features or labels [14] 9 I0] Inspired by
these findings, our work contributes the first method for producing natural language concept-based explanations, where we define a general desiderata of concepts that goes beyond action costs and preconditions to capture goal- goal-driven decision making.
3 Concept Based Explanations for Sequential Decision Making
We posit that the objective of a concept-based explanation, in sequential decision making settings, is to communicate higher-level knowledge that aids the agent and/or user in reasoning about the utility of taking given action from a specific state. In this section, provide general definition of concept-based explanations and "concepts", in sequential decision making, that extends beyond prior usage of state preconditions and action-costs [48], and control logic [22]. Specifically, concept-based explanation, & as one that presents, in natural language form, the set of concepts, C. = (cy..mm }. that can describe a particular acteonction pair (st, az) in the context of goal G. This naturally poses the question of: "What is concept cj Below several desiderata for defining concept in sequential decision making settings:
define
1. A concept should be grounded in human domain knowledge. A concept, cj. should be contextualized in the overall task domain. For example, although an agent's goal may be to optimize its Q-function, concept should represent a higher-level abstraction of what the "higher Q-value" relates to in the domain. For example, in Chess, agent receiving "higher Q-value" for particular state-action pair may be translated by a domain expert to "capturing a queen".
posit that "higher Q-value" is not valid concept. Instead,
2. A concept should relate to the task goal. A concept, Cj, is an abstracted representation of (s) that encompasses how (st, a;) may lead to inhibit the agent's goal G. This claim is motivated by the general objective of a sequential decision making agent, that is to perform a sequence of actions (a1, a,...., a,), 0; E A, defined via a plan policy that transforms the agent's 's current state / to its goal state G 44 49 For example, we do not consider the description "Knight in A4" as valid concept, since, by itself, it does not provide relation to the game's objective. However, we would consider "king safety" as a valid concept, since it relates to game's objective of checkmate.
3. A concept should be generalizable. A concept, Cj. should generalize across multiple (si, pairs. This means that a concept is typically not unique description of a single (si, a.). For example, a valid concept in Chess is "checkmate" since the abstraction of "checkmate" can generalize to many different (s,,a,). An invalid concept involves detailing the entire game board by piece position.
Note, above we define a concept, Cy E C₁, the context of describing a single state-action pair, (st, a:). However, in many domains meaningful concept ej can only be defined in terms of
3

---

sequence of state-action pairs. This is especially true in domains in which state-action pairs are sampled at high- frequecy such as robotics and video games 440 [56]. For example, concepts such as "rotate wrist" or "bend arm" can only be defined by a series of state-action pairs, as opposed to gli,i+al
single state-action pair. As result, we define concept-based explanation in these domains as & which presents the set of concepts C, {m }. for a series of (f(,,it)) ([(s+n) pairs.
Finally, we highlight that some (s;, a4) or (8(,,i4m)) ([.i+n) pairs may not have a concept- basee explanation associated with them. In other words, C₁ = 0 for some (s;, 02) pair or (8],,i+m) ((cs+n)) pairs. This is especially true in long horizon tasks, in which a (8i, 0z) may not relate directly to G but is important for another, future (si+n: (i+n) pair which then leads G. For example, "Move the pawn to D5 so that two moves from now you may capture the rook". Our definition of concept-based explanations do not consider these cases, and an opportunity for future work.
Train Joint Embedding Model M
Leverage o/c
Leverage of, for User Understanding
"oeatec thone in now and provides center domiitance"
uee Whll good mone!
Action of
State of
M
LSTM
S2E
S2E
expleaad
Hendea
Samwedd
a'bensd Elbengged
Explaration
Abstraction
Tampenl Cou
Reword
Lusrred Joine Enteodiing Space
lesrent Enbeddine Sance
explenaed
Pindees
Explanetion Natriced
Explanation Retrical
Loarned
explebee 2E2
Space
(4)
Figure I: Our S2E framework involves (a) learning a joint embedding model M from which a Eg is extracted and utilized (b) during agent training to inform reward shaping and benefit agent learning, and (c) at deployment to provide end-users with E for agent actions.
4 State2Explanation (S2E) Framework
cruure[intoouuess our State2Explanation (S2E) framework which includes three-step process to
provide a benefit to both the RL agent and the end-user. Specifically, S2E learns a joint embedding
model M to align agent state-action pair (si, a;) concept- based explanations E (Fig. Ia). S2E
then leverages the learned M to inform reward shaping during agent training (Fig. (p. Finally, S2E
leverages the learned M to provide E about the trained agent's decision making end-users at deployment (Fig. (b). Below, we further detail S2E.
4.1 Joint Embedding Model M
A core component of S2E is learning a joint embedding model, M, to align an agent state-action pair, (s,, a;) with an associated concept-based explanation E The motivation for learning M is to explicitly map an agent's 's representation, st into a representation understandable by our non-At AI expert users, E Joint embedding models have been leveraged in many applications, such as in image captioning [55] 34), knowledge- grwppdgg-graphh generation (28 43 and RL [42 41 50 to improve task learning. In the context of S2E, we demonstrate that joint embedding M can provide the dual benefit to i) agent learning, and ii) improved end user understanding of agent actions. Our M is inspired by (50 who learn an embedding space in the context of hierarchical RI. tasks. However, unlike in (50), our M is not constrained to hierarchical RI. tasks, and we consider domains in which multiple states can be associated to multiple, non-unique concepts.
The input to our joint embedding model, M, is detined by ([st, s; Ec,, which includes an agent's ' current state s; previous state s;-1, other contextual game information (i.e., player in multiplayer game, whether game is over) 9, associated concept-based explanation & and which defines if s; and E are aligned misaligned. Note, including s, and s,-1 implicitly encodes the

---

action taken to transition from 8;-1 to 8;. Our contrastive loss function (Eq. 1) is adapted from [50]. in which 0 represents the model's parameters, and E N represents the 2thh training sample. Specifically, M minimizes the 1.2-norm of the difference between the embedding vectors when embbed and exPemhed are aligned (y = 0) and maximize the L2-norm of the difference when s embedd and expambed are misaligned (y/ = 1).
N
I(O)= = E(1ss Eembed - y)²
(1)
N
To train M, we leverage dataset D = (D=, [D", in which D and Dm denote the set of aligned and misaligned data samples, respectively. To collect D we leverage domain knowledge to annotate relevant 8; with its aligned concept set C. The concept set C; is then produced into a natural language explanation, E manually via fixed, expert-defined templates. That is, each C2 has one templated EC. The list of concepts for our evaluation domains are in Section 5.1 To collect D each 8; is paired with concepts not present in concept set C, Section 5.2 provides more details on D.
Explanation Retrieval Given M, for any (s,, o;), we extract the closest explanation embedding exPemtee for corresponding s and decode exPemted an Eg; similar to image-to-text retrieval 3I 6). Specifically, we rank the set of possible explanation embeddings, {expémbess "ereemmaa...epaaa.... expemtedl by their L2-norm distance sq. The decoding of the best ranked ezpembed to EC. then a vocabulary look-up. It is possible to utilize large language models (LLMs), and generate Ec, from expembed which we consider future work. Instead, in this work, we focus on validating the dual utility of embedding-based E retrieval to both the RL agent as well as end-user.
4.2 Reward Shaping in RL via Joint Embedding Model M
During agent learning, we utilize M to retrieve the closest Eg, associated with a (sx, a,) for informing reward shaping. Prior work has shown that reward shaping can improve agent learning rate and sample However, designing effective dense reward function is not trivial [16] 12). often requiring a two-stage trial-and-error procedure to 1) determine when a state-action pair deserves an intermediate reward, and 2) determine what reward value such state-action pair should be attributed. Many methods, such as inverse reinforcement learning, utilize learned, black box reward functions to minimize the trial-and-error process (37 1 [57]. However, with black-box reward functions, we lose transparency of why a (s,, a.) is rewarded. To balance the design-trane transparency trade-off of dense reward functions, we leverage M inform when a (st, a.) pair should be rewarded based on the retrieved E and as a result, remove the first step of the trial-and-erye process. Specifically, if M returns meaningful EC. for (8;, a;) ce.g..concept set C₁ + 0), then such (si, uu) represents concepts that influence the agent's 's goal, and should receive an intermediate reward.
efficiency [1] 25 322
Note, M does not remove the trial-and-erre process needed to determine what values each reward component should receive. Instead, we perform a hyperparameter sweep assign shaping values to each concept component when an expert-determined continuous shaping function is not derivable. module in S2E performs a look-up to assign a reward value based on the retrieved E In Section 5.3 we validate how leveraging M to inform reward shaping can improve an agent's 's learning rate comparable to expert-defined shaping functions.
Thus, the
4.3 Concept-Based Explanations End Users via Joint Embedding Model M
Once RL agent is trained and deployed, we leverage M to provide Eτ; about an agent's decision- making to non AI expert end-users. Recall from Section that Et, can include multiple concepts (i.e. [C,] > 1). We posit that in some cases, EG with too many concepts may become ineffective aiding end-user understanding. In fact, prior works discuss the challenge of presenting information at the "right" level of abstraction to end-users [46] (5). To the best of our knowledge, there are no explicit guidelines for what information abstractions are beneficial in sequential decision making settings. From analyzing prior end-user friendly explanation techniques [9] [4], we infer two important dimensions to consider for explanation abstraction in sequential decision making: Information Filtering (InF), and Temporal Grouping (TeG).
Information Filtering (InF) InF filters out concepts within E to only include those that imme- diately influence the goal. For example, consider a E in Lunar Lander, "Fire left engine because it brings lander closer to the center, decreases lander velocity to avoid crashing, and decreases tilt". Applying InF may produce a filtered explanation, "Fire left engine because it brings lander closer to the center and decreases tilt" given (st, as) when the lander needs to immediately correct its tilt and
5

---

Connact
Description
Blocks opponent
Provides center dominnance
3RR
Creates three-ino-row
3IR, BL
Crestes three in row but blocktdd from win
I
four-in-a-row win
"conogt: Play becsuse provides center dominance and createss three in 1DW.
(C,
noutral state that advances game
Lunar Lander
Description
PQS
Brings lander closer to center
VEL
Decresse lander speed avoid crashing
TILT
Decrease tit
RLEG
Encourage right leg contact with ground
Incourage left log contact ground
Foumogpt" Fire left engine because lander closer the decresses lander speed avoid creshing, decreases landar, and conserves main fusl usage.
fuel usage
side
Land without creshing
Figure 2: The tables present concepts for Connect 4, derived from from 44, with accompanying example E
3 and Lunar Lander, derived
position to avoid a trajectory path leading to crash. While the other concepts are important, they are filtered out since they do not effect the immediate chance of meeting G.
To perform InF, we leverage domain knowledge and expert-defined thresholding to determine which concepts in C₁ for (si, a:) critically contribute towards the agent's 's ability to reach goal G. Thresholds determined via qualitative analysis of an RL agent's policy. In continuous domains, these threshold values are extracted by finding the values in the agent's ' state space that determine turning points in the agent's s ability to reach goal G. For example, qualitative analysis may produce that an agent's pos, > 0.15 is a critical turning point for the agent's 's ability to land close to the center, and if pos, > 0.15, the agent's 's action crucially linked with decreasing its p0sg. Thus, Cewinn represents the concept set after thresholding is applied, in which concepts E C; that do not meet the determined thresholds are filtered out. As a result, an abstracted explanation, using InF, templates C.w/tnf into a natural language explanation Eognng In Appendix A we provide more details on how concepts are filtered via qualitative analysis Inf and include an ablation study analyzing the sensitivity of chosen thresholds. Note, while we qualitatively determine thresholds, the InF method can be automated in future work.
Temporal Grouping (TeG) TeG automatically groups explanations over a sequence of time that represents a consecutive pattern. For example, if in Lunar Lander, E and gi+ is "Fire left engine because it decreases tilt" and gi+1 and gi+3 is "Fire right engine because it decreases tilt", applying TeG produces "For the next steps, alternate firing left and right engine to decrease tilt".
To perform TeG, we analyze an agent's 's rollout at deployment to extract series of ($ (iitt)) with repeated pattern of concept sets Cj.i+=)- Therefore, grouped explanation provides a single E Commee that templates the repeated pattern within Cj,itn] across the state-action pairs. We posit that TeG is likely to be important in domains where actions are sampled at high frequency (e.g. Lunar Lander or Robotics), requiring an abstraction over actions provide meaningful explanations for consecutively alike state-action pairs.
Overall, the "Explanation Abstraction" component in S2E (see Fig. [Tp) determines whether InF, TeG or both should be applied to E% for providing an abstracted, concept-based explanation. In Section we demonstrate the effectiveness of InF and TeG in high frequency domains, such as Lunar Lander, for improved end user understanding.
5 Model Evaluations
In this section, we perform quantitative evaluations to validate the joint embedding model M in S2E by evaluating M's ss explanation retrieval accuracy (Sec. 5.2). and M's ability to inform reward shaping during agent training (Sec. 5.3).
5.1 Evaluation Domains
Connect is an adversarial board game in which the objective to achieve a four-in-a-row, in any direction. The state space is discrete, 2D array representation of token positions. Also, the action

---

space is discrete and each player selects the column to place token in. Lunar Lander is a trajectory optimization problem in which the lander must land on a landing pad. We utilize Landrran from OpenAl Gym [4] The state space is continuous ID array including the lander's 's linear position and velocity, angular velocity, angle, and leg contact. The game has four discrete actions: fire left. engine, fire right engine, do nothing, and fire main engine.
Concepts for a domain may be extracted from expert players or commentators. However for Lunar Lander and Connect 4, there are existing datasets for mining concepts. Thus, we leverage expert domain knowledge to attribute concepts to state-action pairs. Figure 2 lists the set of concepts we consider when attributing concepts to (si, a,) in Connect 4 and Lunar Lander, as well as provides example EC.- Concepts for Connect 4 are derived from the strategies outlined in [3] Note, "NULL" describes all other (s,, 0z) that are not attributed to a concept. Alternatively, concepts for Lunar Lander derived from the domain's existing reward function [4]. In this domain, the existing dense reward function includes the primary concepts important towards successful task completion. In Appendices B. and B.2 we provide an expanded list of example E spanning all concepts derived for Connect 4 and Lunar Lander.
Concept Collection
Note, for both domains, the concepts are defined via mathematical or logical representations of the state. For example, in Connect 4, the concept of "BW", blocking an opponent win, can explicitly be encoded by board representations where any three-in-a-row pattern is blocked with token from the opponent player. Similarly, in Lunar Lander, the "POS", position concept, modelled by mathematical representation of the agent's 's position logical representations allow us to automatically collect concepts from states, as well as use such mathematical logical rules to evaluate that concepts and states accurately paired. In many applications, it may be infeasible to derive mathematical or logical rules from a state representation. In these scenarios, concepts can be collected via crowd sourcing, (20] obtained via "think-aloud" procedures 14]
time. Deriving concepts via mathematical or
5.2 Evaluation of Joint Embedding Models
Avg. Recall Rate M
Below we demonstrate high recall rates of the joint embedding models in S2E in both the Connect 4 and Lunar Lander domain.
100
61.0
$3.
80
Datasets Recall from Section 4.1 that train M we utilize a dataset D = {Da, D For Connect 4, Dea includes approximately 3 million samples, and for Lunar Lander, DLL includes approximately 2 million samples. We randomly sample Do and DL from an
Rate
40
expert RL policy. We obtain D and D'L by selecting z incorrect concepts from Cca and CI.,,, as replacement for each sample in D2 and D2 (more detailed numbers in Appendix C).
20
Connect 4 Luner Lander
R
R@3
Model Architectures & Training Mci and MLL denote the joint embedding models for Connect 4 and Lunar Lander, respectively. The model architectures for Mc1 and MLL leverage LSTMs to learn explanation embedding ezp'mbed for a given E For Connect 4, to learn Sembed for (8;, oz). we leverage Convolutional Neural Networks to learn local and spatial relationships between tokens on a 2D game board. For Lunar Lander, we leverage Fully Connected Networks to learn s² embedd More details (split, learning rate, etc) are in Appendix C Results Similar to image-to-text retrieval 31 6 we evaluate Mc4 and M,.. via recall rate, at A: = {1, 2, 3). which evaluates whether the retrieved EC, is ranked in the top k. Figure 3a provides the average recall rates of Mo: and MLL. across random seeds. We observe that MLL performs with near 100% accuracy, whereas Mc1 has average recall rate of 88%. Given the lower recall rates by Mcc. in Figure 3b) we examine the false positive and false negative explanation retrievals. We see that Mcs has greatest challenge in correctly retrieving E%. with "BW" and "3IR BL". We posit that "BW" and "3IR BL" may occur in board states with greater variation in comparison to other concepts, leading to higher incorrect retrievals. Additional analyses of Mc4 and Mr.. are in Appendix C.4
(a)
Cies tele
68
30
07
04
1991
(80r)
roe ct))
(n)
Figure 3: (a) recall@k of Mcs and MLL, and (b) con- fusion matrix for Mcs 's Ec. retrievals.
Note, incorrect retrievals of E for a given (8i, (z) can have negative downstream impact within S2E. Specifically, when S2E is leveraged during the RL agent's training, incorrect retrievals of
7

---

Connect4
Lunar Lander
lunar Lander
1.0
1.0
80.0
80.8
Daward
0.6
0.6
100
0.4
-200
2.2
Nutoro 85
E-S
0.22
NuZero E-B
-300
Midero ERS
0.0
525-85
SIE-85
-400
525-85
200k
300k 400k
100k 200k
400k
Treining Steps
Training Steps
(a)
(b)
(c)
Figure 4: Mc4 improves agent learning rate by 200 training steps compared to SoTA agent without reward shaping (a), while ML maintains similar agent learning rate to SoTA agent with expert-defined reward shaping c).
Eg, can incorrectly providing shaping rewards to the agent and in return impacting learned agent policy. Similarly, when S2E is leveraged at deployment to provide end-user understanding, incorrect retrievals of Ec, can confuse end-users and hinder their understanding of the agent's behavior. However, in Section 5.3 our results demonstrate that the percentage of incorrect retrievals from the joint embedding models do not significantly impact the RL agent's learned policy. Similarly, in Section 6 we demonstrate that the percentage of incorrect explanation retrievals are not significantly detrimental to user performance. Nevertheless, is important that the joint embedding models within S2E have high recall rate, especially when applied in high-stakes or mission critical scenarios.
5.3 Evaluation of M for Reward Shaping in Agent Training
We validate that both joint embedding models in S2E,Mca and MLL, inform reward shaping comparable expert-defined dense, reward functions.
RL Agent & Shaping Rewards Our S2E framework is model agnostic and does not make any assumptions the type of RL model utilized. Given our domains are complex games, we leverage state-of-the-an art (SoTA) RL algorithm MuZero [47] to evaluate M s ability to inform reward shaping.
We use the open-source version of MuZero available in [52] (details in Appendix D.Ip.
Recall from Section that Mcs and MLL inform when to provide shaping rewards, but the shaping values are expert determined. For Lunar Lander, there exists a SoTA dense, reward function [4] In Connect 4, to our knowledge, there is no SoTA dense, reward function. Thus, we perform hyperparameter sweep to assign shaping values for each concept (values provided in Appendix (.2)
Results To evaluate the efficacy of Mc4 and Mr. in informing reward shaping, we measure the agents' Reward and Win%. In Lunar Lander, we evaluate My.,'s 's ability inform reward shaping (MuZero+SZE-RS. in comparison to baseline MuZero agent that is informed by existing expert-defined reward shaping (MuZero+E-RS)) Figure 4b and Figure 44c demonstrate that both the MuZero++S2-RR RSS and MuZero+E-RS agents achieve comparable Reward and Win%. While performance is comparable between both agents, in MuZZer++SEE-RS, MLL provides an automated method to determine when reward value should be presented, which otherwise has to be manually encoded, such as in MuZero+E-Ri via an expert-defined function.
In Connect 4, since there does not exist SoTA reward shaping function, we compare Mcs's ability to inform reward shaping (MuZero+S2E-RR) against a baseline MuZero agent with sparse rewards (MuZero++No-RS) Figure 4a shows that MuZero+SSF-RR has improved learning rate and requires 200k less training samples to achieve similar Win% in comparison to MuZero+No-RS. We also evaluate MuZero+S2E-RS against shaping (MuZero+E-RS) to analyze how incorrect retrievals from Mca may affect the agent's s learning rate. Although Mos retrieves incorrect explanations 13.9% of times (see Appendix C.4), we observe that MuZZer+SSEERR''s learning rate is not significantly affected (Fig. 4a).
upper bound in which we provide expert-defined reward
Overall, these results demonstrate the joint embedding models' ability to effectively inform reward shaping. Connect 4, we demonstrate that, even with an imperfect joint embedding model, S2E can inform reward shaping and improve the agent's 's learning rate compared to the SoTA agent trained on sparse rewards. Similarly, Lunar Lander, we demonstrate that our S2E informs reward shaping comparably to the existing SoTA agent that is trained with expert-defined dense, reward function.

---

User Evaluation
In this section, validate S2E with end-users, demonstrating that retrieved &c, significantly improve user task performance in both Connect 4 and Lunar Lander.
Study Procedure Participants performed an online study which they played several games from the domains in four stages. Specifically, 1) Practice included playing 2 practice games. 2) Pre-Test included playing 3 scored games, 3) Explanation included interacting with an expert player (well-trained RL agent) with exposure to explanations from an assigned study condition, if applicable and 4) Post-Test included playing 3 more scored games. Details about the the study procedures are in Appendix E.
Metries We measure the difference between participant Pre-Test and Post-Test Adjusted Task Score (AT') to analyze any task improvement with exposure to the study conditions. Participant ATS is defined by the expert-defined reward functions utilized by the RL agents during training (evaluated in Sec. 5.3 detailed in Appendix D.2). If R(s, a) denotes the reward associated with given (s, a) and defines the total actions taken by the user in a game, then ATS is defined as a normalized N
aggregation of user rewards received in game: ATS = Σn R(sn. anN/N.
6.1 Connect 4 Study Specifics
Connect ATS
Study Conditions We conducted five-way between-subjer study with the following conditions. Participants could receive: 1) None (Baseline): no information about the agent's action to played, 2) & (Baseline): action-based explanation that contains the action to be played, 3) v (Baseline): value-based explana- tion stating the action be played along with the action's 's value in comparison to the values of all other actions for a given state, 4) GT Ec.: concept-based explanation using expert- deeeeedd ground concepts for a state-action pair, and 5) S2E & Ec, : a concept-based explanation from S2E for a state-action pair. Note, given that Connect 4 has a discrete state space, and actions are sampled low frequency, the domain does not need further ab- stracted Ec, via InF and TeG. Additionally. yv is similar to the action-cost condition in [8). well the Q-values presented explanations [48)
0.4
S2E cc
GT &
Standardized
0.2
Ey
None
0.1
0.0
-0.1
-0.2
PreTest
PostTest
Figure 5: User adjusted task
scores (ATS) in Connect 4. Sta- tistical significance: * p 0.05, p 0,01, and p 0.001; Details in Appendix E.3
per action in (27) Note, precondition- require hierarchical domains, and causal- explanations in (36] require learned structured
causal model which prevent direct comparison with our framework and domains.
Results The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=75, details in Appendix F.2). Our data is analyzed with one-way ANOVA and Tukey HSD post-hoe test, given that the assumptions homoscedasticity (Levene's Test, p>0.3), and normality Shipiro-Wilke,, p>0.1) are met. Figure S shows significant effect of explanation type on ATS (F(4, 70)=8.56, 56, p<0.001). Specifically, we observe improvement in participant's s ATS with exposure to our S2E &c,, in comparison to None (1(70) = 3.08, p<0.05), EA (x(70)=-3.25, p<0.05) and εy (1(70)=4.03, p<0.01). Additionally, we observe similar ATS improvement when exposed S2E Ec, in comparison to GT Ec, These results indicate the benefit of our S2E produced Ec, in helping participants better understand Connect 4 and improve their ATS.
6.2 Lunar Lander Study Specifies
Lunar Lander ATS
The underlying physics needed for Lunar Lander results in agent (st, 0z) being sampled at high frequency and multiple, repetitive concepts being attributed to (st, a.) pairs. For example, "decrease velocity" is a valid concept for every (8i, 0z) until the end of a game. Recall from Section 4.3 that &c, may need be further ab- stracted in these scenarios to provide meaningful explanations to end-users. Therefore. evaluate the utility of abstracted concept- based explanations, we introduce 3 more study conditions that utilize our InF and TeG methods outlined in Section
0.25
fuwwe 1o0
Standardized ATS
0.20
S2E Eoww
S2E Cownres
0.15
P
Ev
None
0.00
-0.05
-0.10
-0.15
PreTest PostTest
Study Conditions We perform seven-way between-subjects user study. Similar to Connect 4, we include None, A εv, and S2E &c.- Unique to our Lunar Lander study participants
Figure 6: User ATS in Lunar Lan- der; & > 0.05, ** p < 0.01

---

could receive 1) S2E TeG: a sascept-based explanation from S2E with additional abstractions performed using the TeG method, 2) S2E C InF: a concept- based explanation from S2E with additional abstractions performed using the InF method, and 3) S2E C.cw/ InF, TeG: concept-based explanation from S2E with additional abstractions performed using both TeG and InF.
The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=105, details in Appendix E.2) Our data is analyzed with one-way ANOVA and Tukey HSD post-hoc test, given that the assumptions of homoscedasticity (Levene's s Test, p>0.7), and normality (Shapiro- Wilke, p>0.1) are met. Figure shows significant effect of explanation type on participant ATS (F(6,88) (666988)3.67; p<0.01). Specifically, we see significant improvement in participant's s ATS with exposure to S2E & InF TeG, in comparison to None (t(98)= 3.15, p <0.05)), ΣA =3.3,) p<0.05), Ey (1(98)=3.59, p<0.01) and &c, (1(98)=-3.19 19, p<0.05). These results demonstrate the need of our S2E abstraction methods when providing concept- based explanations to end-users, and that usage of both InF and TeG in high-frequency RL domains are crucial for ATS improvement.
Results
Discussion and Conclusions
Our work introduces a unified framework, S2F, that involves learning a joint embedding model between agent state-action pairs and concept-based explanations to provide a dual benefit to the agent and end-user. We additionally outline a desiderata for what may constitute sequential decision making problems, beyond the scope of prior concept-based explanations for sequential decision making. Our model evaluations demonstrate that the joint embedding model in S2E provides an automatic method for determining when reward shaping should be provided to improve agent learning rates. Our user evaluations demonstrate that concept- based explanations can significantly improve user task performance (Connect 4), but when considering high-frequency RL domains (Lunar Lander), the additional abstraction methods from S2E important for producing abstracted concept-basse explanations that significantly improve user task performance.
"concept" in
Limitations: We present several areas of future work that aims to improve the generalizability of S2E. For instance, the sancept-based explanations in our work are derived from mathematical representations and expert knowledge in each domain (see Sec. [.T). To expand the generalizability of the S2E method to other complex domains, such as Robotics or open- games, future work should explore how to collect and extract concepts in scenarios where mathematical representations of concepts may be infeasible. A future direction may include collecting expert commentary [20] or think-aloud [14] sessions from which concepts are automatically derived. Additionally, in many real-world applications, having access to large amounts of data for a domain may be infeasible. Thus, future work should also explore how to adapt the joint embedding model within S2E for few shot learning. Additionally, for application to high-stakes domain, future work should explore how to remediate incorrect explanation retrievals from the joint embedding model within S2E. While the small percentage of incorrect explanation retrievals do not significantly impact user task performance and agent learning in tested domains, inaccurate explanation retrievals may have significant effects in mission-critte crittial tasks. is also important to highlight that our current concept-basee explanations follow fixed template for each unique concept set (see Appendices B. B.2 However, there multiple ways of explaining the same concept, and future work includes learning a joint embedding model with greater language variety in the explanations combined with automatically generating such templates using language models. Furthermore, the InF method within S2E utilizes manually defined thresholds to produce abstracted concept-based explanations, and future work entails developing an automated InF method.
Broader Impacts: S2E shows promise in applying the Protégé Effect to Human-Al AI Interaction, and that explanations of agent behavior are beneficial to both agents and users. These insights may promote Al agent architects to consider the utility of self-explaining agents in accelerating learning as well as providing transparency. Additionally, our work shows promise in using concept- based explanations of well-trained AI agents as a teaching tool, helping users improve their task performance. potential risk of S2E is purposely training a joint embedding model to associate misleading concept-based explanations with state-action pairs. This could lead to a badly performing agent, and the agent providing deceitful explanations of its actions. Future work in explaining black-box AI systems should explore how to detect and prevent deceptive explanations.
10

---

References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1, 2004.
[2] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52160, 2018.
[3] Louis Victor Allis. A knowledge-based approach of connect-four. J. Int. Comput. Games Assoc., 11(4):165, 1988.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
[5] Gerard Canal, Senka Krivic, Paul Luff, and Andrew Coles. Task plan verbalizations with causal justifications. In ICAPS 2021 Workshop on Explainable Al Planning (XAIP), 2021.
[6] Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.14713, 2022.
[7] Peter A Cohen, James A Kulik, and Chen-Lin C Kulik. Educational outcomes of tutoring: A meta-analysis of findings. American educational research journal,, 19(2):3372248, 1982.
[8] Larissa Custodio and Ricardo Machado. Flexible automated warehouse: literature review and an innovative framework. The International Journal of Advanced Manufacturing Technology. 106:533-558, 2020.
[9] Devleena Das and Sonia Chernova. Semantic- based explainable ai: Leveraging semantic scene graphs and pairwise ranking explain robot failures. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3034-3041. IEEE, 2021.
[10] Devleena Das, Been Kim, and Sonia Chernova. Subgoal-based explanations for unreliable intelligent decision support systems. Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 240-250, 2023.
[11] Sam Devlin, Daniel Kudenko, and Marek Grześ. empirical study of potential-based reward shaping and advice in complex, multi-agent systems. Advances in Complex Systems, 14(02): 251-278, 2011.
[12] Daniel Dewey. Reinforcement learning and the reward engineering principle. In 2014 AAAI Spring Symposium Series, 2014.
[13] David Dunning. The Boaming-krugee effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology, volume 44, pages 247-296. Elsevier, 2011.
[14] Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. Automated rationale generation: technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263-274, 2019.
[15] Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Martina Mara, Mare Streit, Sandra Wachter, Andreas Riener, and Mark O Riedl. Operationalizing human-centered perspectives in explain- able ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-6, 2021.
[16] Jonas Eschmann. Reward function design reinforcement learning. Reinforcement Learning Algorithms: Analysis and Applications, pages 25-33, 2021.
[17] Logan Fiorella and Richard E Mayer. The relative benefits of learning by teaching and teaching expectancy. Contemporary Educational Psychology, 38(4):281-283 2013.
[18] Eva Fraedrich and Barbara Lenz. Taking a drive, hitching a ride: autonomous driving and car usage. Autonomous driving: Technical, legal and social aspects, pages 665-685, 2016.

---

[19] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[20] Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward
shaping in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2385-2391, 2019.
[21] Marek Grzes. Reward shaping in episodic reinforcement learning. 2017.
[22] Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 2017 ACM/IEEE international conference on human- robot interaction, pages 303-312, 2017.
[23] Alexandre Heuillet, Fabien Couthouis, and Natalia Dodz-Rodríg.ez. Explainability in deep reinforcement learning. Knowledge-Baad Based Systems, 214:106685, 2021.
[24] Vincent Hoogerheide, Lian Deijkers, Sofie MM Loyens, Anita Hcijltjes, and Tamara van Gog. Gaining from explaining: Learning improves from explaining to fictitious others on video, not from writing to them. Contemporary Educational Psychology, 44:95-106, 2016.
[25] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 13:15531-11591 2020.
[26] Ying Ji, Yu Wang, and Jien Kato. Spatial-temporal concept based explanation of 3d convnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 15444-15453, 2023.
[27] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
[28] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlic Huang. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1526-2538 2021.
[29] Eoin M Kenny, Mycal Tucker, and Julie Shah. Towards interpretable deep reinforcement learning with noman-friendee prototypes. In The Eleventh International Conference on Learning Representations, 2022.
[30] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages
PMLR, 2018.
[31] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.
[32] W Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, volume 1004, pages 475-482, Valencia, 2012.
[33] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference Machine Learning, pages 5338-5348. PMLR, 2020.
[34] Iro Laina, Christian Rupprecht, and Nassir Navab. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7414-7424, 2019.
[35] Adam Daniel Laud. Theory and application of reward shaping in reinforcement learning. University of Miinois at Urbana-Chane Chaama-Champainn, 2004.
[36] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2493-2500, 2020.
12

---

[37] Alberto Maria Metelli, Matteo Pirotta, and Marcello Restelli. Compatible reward inverse reinforcement learning. Advances in neural information processing systems, 30, 2017.
[38] Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 34:29529-29540, 2021.
[39] Stefania Montani and Manuel Striani. Artificial intelligence in clinical decision support: a focused literature survey. Yearbook of medical informatics, 28(01):120-127, 2019.
[40] Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE International Conference on Robotic Computing (IRC), pages 590-595, IEEE, 2019.
[41] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188-204. PMLR, 2021.
[42] Paul J Pritz, Liang Ma, and Kin K Leung. Jointly- state-action embedding for efficient re- inforcement learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1447-1456, 2021.
[43] Saed Rezayi, Handong Zhao, Sungchul Kim, Ryan A Rossi, Nedim Lipka, and Sheng Li. Edge: Enriching knowledge graph embeddings with external text. arXiv preprint 1xXiv:2104.09900 2021.
[44] Stuart Russell and Peter Norvig. Ai a modern approach. Learning, 2(3):4, 2005.
[45] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable AI: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
[46] Lindsay Sanneman and Julie A Shah. A situation awareness-based framework for design and evaluation of explainable ai. In Explainable, Transparent Autonomous Agents and Multi-Agent Systems: Second International Workshop, EXTRAAMAS 2020, Auckland, New Zealand, May 9-13, 2020, Revised Selected Papers 2. pages 94-110. Springer, 2020.
[47] Julian Schritwieserr, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
[48] Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, and Subbarao Kambham- pati. Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making making problems with inscrutable representations. arXiv preprint Sxki::2002.00888 2020.
[49] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[50] Nicholas Waytowich, Sean Barton, Vernon Lawhern, and Garrett Warnell. A narration- based reward shaping approach using grounded natural language commands. arXiv preprint arXiv:1911.0049) 2019.
[51] Lindsay Wells and Tomasz Bednarz. Explainable ai and reinforcement learning- systematic review of current approaches and trends. Frontiers in artificial intelligence, 4:550030, 2021.
[52] Aurèle Hainaut Werner Duvaud. Muzero general: Open reimplementation of muzero. 2019.
https
//itthub. ccmm://eenee-dddrrdddmarrrr-re-e
[53] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfister.
On concept-based explanations deep neural networks. 2019.
[54] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, al. Concept embedding models. In NewrIPS 2022-36th Conference on Neural Information Processing Systems, 2022.
13

---

[55] Luowei Zhou, Chenliang Xu, Parker Koch, and Jason J Corso. Watch what you just said: Image captioning with eott-oonitiinal attention. Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 305-313, 2017.
[56] Kai Zhu and Tao Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 161 2021.
[57] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433- 1438. Chicago, IL. USA, 2008.
14

---

A Information Filtering (InF) Details
Recall from Section 4.3 that we utilize expert-defined thresholding and domain knowledge to determine which concepts within C, for a (8,, a,) critically contribute towards the agent's ability to reach goal G. Specifically, these thresholds are expert-defined upper and lower bounds on the agent's state values that denote the agent's s ability to succeed or fail in its goal. In our InF method, these thresholds are not mathematically derived, but are derived from RL-expert analysis. For a given domain, an RL expert visualizes multiple policy rollouts and analyze the different state values over time to manually determine the upper and lower bounds (turning points) that influence the agent's s ability to reach G. Below we demonstrate how thresholds in InF for one of our evaluation domains, Lunar Lander.
The concepts relevant to Lunar Lander include, position, velocity, tilt, side and main fuel, right and left leg contacts, and landing, and these concepts are derived from the existing, expert-defined, dense reward function [4] (see Section 5.1). Within the dense, reward function, binary concept components are rewarded via constant shaping values, while the continuous concepts are rewarded through a continuous shaping function (see Appendix D.2) In our work, we consider applying thresholding to concepts defined via a continuous function since these scenarios typically result in multiple concepts being rewarded over multiple time steps, and are more likely warrant information filtering.
ssow In the context of Lunar Lander, Figure 7 and Figure X show the two concepts on which thresholding is applied when examining a well trained agent's 's policy, specifically the lander's s x-position and tilt. Recall that thresholds for InF can be determined by examining an agent's state values to identify turning points in the agent's is ability to reach G. In Figure we extract that a suitable threshold for the x-position concept is 0.15. We identify that such x-position value corresponds to the time step at which the agent starts firing its left engine to critically correct its increasing position value. Similarly, in Figure [ we extract that suitable thresholds for the tilt concept are 0.01 and -0.05. We find that such tilt values correspond with frequently alternating firing of left and right engine to critically correct an increasing tilt value. We do not perform a thresholding analysis on velocity. as with domain analysis we observe that decreasing the agent's 's velocity is directly associated with firing the main engine and is performed to prevent crashing. Additionally, we do not consider the agent's 's y-position for thresholding as the agent follows steady, downward linear trajectory, and visible turning points cannot be extracted by qualitative analysis. While we qualitatively determine thresholds, the InF method can be automated in future work.
Overall, in our InF method, if C. denotes the original concept set for a given (st, a.). then Cewitn represents the concept set after thresholding is applied, in which concepts E C; that do not meet the threshold are filtered out. As a result, an abstracted explanation, using InF, templates Ct2 into a natural language explanation EC
A.1 Thresholding Sensitivity Analysis
As stated above, the thresholds are manually derived from RL-expert analysis. Specifically, an RL expert visualizes multiple policy rollouts and analyzes the different state values over time to determine approximate upper and lower bounds (turning points) that influence the agent's ability to reach G. Given the thresholding approach in InF, different threshold values can result in filtering of different concepts and therefore produce different abstracted, concept- based explanations. To study the sensitivity of our chosen thresholds in Lunar Lander, we analyze how different threshold values for each concept affect the number of concepts filtered in policy rollout.
Figure/ demonstrate what fraction of concepts are filtered (y-axis) as the threshold values change (x-axis). When looking at Figure Dt. analyzing threshold values for the x-position concept, we see our chosen value is within the elbow of the curve, denoting that the rate of filtration of the concept slows down after 051 Similarly, when analyzing the threshold values for the tilt concept in Figure D and Figure De we see the lower and upper bound of the tilt thresholds are also within the "elbow" of each curve. Note, when analyzing the upper bound threshold for the tilt concept (Figure (p)) the lower bound tilt threshold is fixed. Similarly, when analyzing the lower bound threshold for the tilt concept (Figure the upper bound tilt threshold is fixed, Overall, our threshold values being within the elbow of each curve provide some validation on their soundness. However, to study the complete impact of varying thresholding values for each concept, additional user studies with abstracted concept-based explanations produced from varying threshold values are required. We
15

---

Threshold identifiction for
Agert Rostton
0.00
-335
-.20
Acton
Figure 7: The position (top) and corresponding actions (bottom) of a trajectory sampled from well-trained Lunar Lander agent policy.
Threshold identification for silt
6.02
Agant
0.00
0.86
4.08
Time
Figure 8: The tilt angle (top) and corresponding actions (bottom) of trajectory sampled from well-trained Lunar Lander agent policy.
X-Position Threshold Experimentation
0.25
0.20
poston
0.10
0.05
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Position Threshold Value
(a)
Upper Bound Titt Threshold Experimentation w/ -0.05 lower bound
Lower Bound Tit Threshold Experimentation 0.01 bound
Traction oftit concepla fitorre
0.20
Trection et bit concepts fiterre
0.175
0.100
0.10
0.075
0.050
0.05
0.025
0.0
0.1
0.2
0.3
0.4
0.0
-0.1
-0.2
-0.3
-0.4
Tit (Upper Bound) Threshold Value
Tilt (Lower Bound) Threshold Value
(b)
(c)
Figure 9: Sensitivity analysis how the rate of concept filtration changes across various threshold values for Lunar Lander concepts of x-position (a) and tilt (b and c).
consider such analyses beyond the scope our work, and one to further explore when considering future methods automating the information filtering submodule within our S2E framework.
16

---

B Concept-Based Explanations for Experimental Domains
Below we present an exhaustive list of the different E utilized for each domain, paired with an instance of (si, a;) pair for context. Note, the denote the possible concept sets C₁ and their corresponding fixed, templated explanations starting with "because". The action da is always appended to the beginning of the explanation.
B.1
Connect
(8i, (s))
C₁
Corresponding &
"Play column 4 because it creates three-in-a- row."
{3IR}
"Play column as a neutral move, it creates three- in-a-row that is blocked by the opponent from a win."
[3IR_BL
"Play column because it provides center domi- nance and creates a three-in-a-row..
{3IR, CD}
"Play column 6 because it blocks the opponent from a win"
[BW)
"Play column 5 because it creates three-in-a-row and blocks the opponent from a win."
(BW.
"Play column because provides center domi- nance and blocks the opponent from a win."
(BW, CD}
"Play column because provides center domi- nance"
{CD)
"Play column 7 because leads four-in-a-row win"
(W)
"Play column 6 as generic move not tied to particular strategy"
(NULL)
Table 1: Example state action pairs with associated concept lists and concept- based explanations for the Connect 4 domain.
17

---

B.2
Lunar Lander
(8i, as))
C
Corresponding &
"Fire main engine because moves lander closer to the center, decreases lander speed avoid crashing, decreases tilt of lander, and conserves side fuel usage."
(POS, VEL, TILT, SF]
"Fire side engine because it moves lander closer the center, decreases lander speed to avoid crashing, decreases tilt of lander, and conserves main fuel usage."
[POS, VEL, TILT, MF]
"Do nothing because moves lander closer to the center, decreases lander speed to avoid crashing, and decreases tilt of lander."
(POS, TILT)
VEL,
"Fire main engine because moves lander closer to the center, decreases lander speed to avoid crashing, decreases lander, encourages left leg contact, and conserves side fuel usage."
{POS, VEL, TILT, LLEG, SF}
"Fire side engine because it moves lander closer to the center, decreases lander speed avoid crashing, decreases tilt of lander, encourages left leg contact, and conserves main fuel usage."
{POS, VEL, TILT, LLEG, MF)
"Do nothing because moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt of lander and encourages left leg con- tact."
[POS, VEL. TILT, LLEG)
"Fire main engine because moves lander closer to the center, decreases lander speed avoid crashing, decreases tilt of lander and encourages right leg contact, and conserves side fuel.."
(POS, VEL, TILT, RLEG, SF}
"Fire side engine because it moves lander closer to the center, decreases lander speed to avoid crash- ing, decreases tilt of lander, encourages right leg contact, and conserves main fuel."
[POS, VEL. TILT, RLEG. MF)
"Do nothing because moves lander closer to the center, decreases lander speed to avoid crashing, decreases tilt lander, and encourages right leg contact."
[POS, VEL, TILT, RLEG)
"Fire main engine because moves lander closer to the center, decreases lander speed avoid crashing, decreases the tilt of the lander, encourages left and right leg contact and conserves side fuel."
(POS, VEL. TILT, LLEG, RLEG, SF]
"Fire side engine because moves lander closer to the center, decreases the lander speed to avoid crashing, decreases the tilt of the lander, encour- ages left and right leg contact, and conserves main fuel."
(POS, VEL, TILT, LLEG. RLEG, MF)
18

---

"Do nothing because moves lander closer to the center, decreases the lander speed to avoid crashing, decreases the tilt of the lander, encourages left and right leg contact."
{POS, VEL, TILT, LLEG, RLEG]
[L}
"Do nothing because results in land."
Table Example state action pairs with associated concept lists and concept- based explanations for the Connect domain.
C Joint Embedding Model Details
C.1 Dataset Details
ligned and Aligned Data Breakdown Recall from Section that De4 represents the dataset used for training and evaluating M in Connect Specifically, Dca includes 2, 948, 742 total samples, in which & = 327, 638 aligned samples and D = 2, 621, 104 misaligned samples. Similarly, in Lunar Lander. Specifically,
DLL represents the dataset used for training and evaluating DEL includes 1, 941, 042 total samples, which DiL 323, 507 aligned samples and DIL 1,617,53 535 misaligned samples. Recall that to create the misaligned dataset, D we utilize incorrect concept sets available in each domain as replacement for the existing correct concepts utilized in Ec, for each d₃ E D Specifically, for Connect 4, & and we leverage all possible non-aligned concept sets for perturbation. In Lunar Lander, non-aligned concept sets for perturbation. We randomly choose 5 of 13 possible concept sets to generate misaligned sample pairs per state avoid larger than necessary dataset. As refresher, the list of possible concept sets for each domain are presented in Figure
5 and we randomly choose five
Concept Breakdown Aligned Data Jad provide breakdown of the occurrences
of each concept set within the training data for the joint embedding models, Mcs (Connect 4) and MLL (Lunar Lander). We specifically show the breakdown in the aligned datasets, D&g and D given that such data is the raw data collected via rollout simulations in each domain. Recall that the additional misaligned data in the training set, D and D are synthetically generated via perturbations, and used for the contrastive learning. When the misaligned data is included in the training set, given the perturbations, the number of occurrences of each concept (aligned or misaligned) become equal.
Concept Set
# of Samples in Daligned
{3IR}
43,896
(3IR BL}
21,911
3IR, CD}
6,901
[BW]
25,352
BW, 3IR)
10,668
BW. CD)
4,940
[CD]
30,591
[
23,137
[NULL]
160,242
Table 3: Concept set occurrence analysis of the aligned data within the training set used to train Mcs-
C.2 Model Architectures
Figure 10 shows the model architecture Mcs. Recall that the inputs Mcs includes {[s,, 81-1, 96. E,,, v1). In the diagram, we don't explicitly show but it is used to denote whether
19

---

Concept Set
# of Samples in D Daligned
[POS, VEL, TILT, MF}
136,917
(POS, VEL. TILT, SF}
62,140
{POS. VEL, TILT)
53,944
(POS, VEL, TILT, LLEG, MF)
1,610
{POS. VEL, TILT, LLEG, SFY
2,981
(POS, VEL, TILT, LLEG]
4,219
[POS, VHL. TILT, RLEG, MF
1,523
(POS. VEL, TILT, RLEG, SF}
2,399
{POS, VEL, TILT, RLEG)
3,228
(POS. VEL, TILT, LLEG, RLEG, MF)
1,730
[POS. VEL, TILT, LLEG, RLEG, SF)
12,026
[POS, VEL. TILT, LLEG, RLEG]
39,981
(L)
809
Table 4: Concept set occurrence analysis of the aligned data within the training set used to train MLL-
si and Ec, are aligned or misaligned (sce Section 4 The game boards (s; and 8:-1) in Connect 4
are represented as 6x7 2D array which are passed into two consecutive CNNs after padding to 7x7 2D arrays. The first set of CNNs (CNN1 and CNN3) for 8; and 8 have the following parameters: input channels=1, output sraputtchannel=44.. kernel_size (3.3), stride=1. The second set of CNNs (CNN2 and CNN4) have the following parameters: input_ channels=4, output_ channels=6, kernel size= (3,3), stride=1. The embedding outputs of CNN2 and CNN4 are then concatenated with the game information ft. which includes boolean representations of the player/opponent as well whether the game is over. The concatenated embedding is then passed through three FCNs. Specifically, FCNT 's output dimension 64. FCN2's output dimension is 32. FCN3's output dimension is 16. The output of FCN3 represents our final state embedding, SEmbed The concept- basd explanation Eg, is first passed through an embedding layer and then passed to a LSTM network to produce our exPembed The input size to the LSTM is of dimension 32, and output dimension is 16.
Figure shows the model architecture for MLL. The inputs to MLL includes ([s,, Ec,, v.). In the diagram, we don't I explicitly show yı. but it is used to denote whether 8; and Eg, are aligned or misaligned (see Section 4 The game boards (s; and 8;-1) in Lunar Lander are represented as 1x10 arrays which are passed into two consecutive FCNs. Specifically, the game board representation includes the 1x8 game state representation from OpenAl Gym 44. along with an inclusion of whether the side fuel main fuel is in use. The first set of FCNs (FCNI and FCN3) for s; and 8;-1 have an output dimension of 64. The second set of FCNs (FCN2 and FCN4) have an output dimension of 32. The embedding outputs of FCN2 and FCN4 are then concatenated with the game information Jr. which includes a boolean representation of whether the game is over. The concatenated embedding is then passed through three more FCNs Specifically, FCN5 has output dimension of 16. FCN6 has an output dimension of 8. Similar to the architecture for Mc4, the concept- based explanation E is first passed through an embedding layer and then passed to a LSTM network to produce our exp expembed The input size to the L.TM is of dimension 32, and output dimension is 8.
C.3 Training Details
To train and evaluate Mcs and MLL. we utilize a 60%-20%-200 -0% train muin-valid-test split on Des and DL. (see total dataset size in Appendix C.T). The models are trained with learning rate of 0.001, batch size of 128, Adam Optimizer, and trained with 10 epochs. To train Mca and MLL. we utilize desktop computer with a NVIDIA GTX 1060 6GB GPU and an Intel i7 processor. The runtime for training and testing Mcs and Mr. takes approximately 30 minutes. See link to code in Appendix F
C.4 Additional Evaluations
Learned Embedding Spaces of Mcs and Mr. In Figure 12a and Figure 12b we provide a visualization of our learned joint embedding models, Mcs and MLL.. The TSN-E visualizations is created using the models' training dataset. We do not draw any conclusions via these visualizations, but present them as an intuitive way to interpret our high dimensional, learned embedding spaces.
20

---

RelU
RedU
CNN1
CNN2
FDN1
FCN2
Sembhed
Concar
CNNS
CNN
Embedding
exPemord
LSTM
info
Player [1,0]
Provides dominaamce and
createe Bree-ii----so::
Figure 10: Joint Embedding Model M Architecture for Connect 4.
RELu
RelU
FON1
FONZ
FCNS
FCNS
Concet
ReiU
Embedding
exp
exPumbed
LSTM
(1,0)
Movee der closer
center, deczesees speed
creshing. decreseesi lander, serves main usage"
Figure 11: Joint Embedding Model M Architecture for Lunar Lander.
MM
PSS VLL TET
(ROS TET. LEEL
(D)
CL)
(8)
(b)
Figure 12: (a) and (b) demonstrate visualizations of the learned embedding spaces of the joint embedding models learned for Connect 4,M and Lunar Lander, MY
In the "Wild" Evaluation of Mc1 and ML. We performed evaluation of how well Mcs and Mr. continue to perform when utilized during an RL agent's training to inform reward shaping. Specifically, analyzed the model's Recall@) examining how often Ec; was incorrectly retrieved for given (st, aj). For Lunar Lander, MLL's 's average Recall@ 1, across the 5 seeds, during the
21

---

(B9)
0.0
24.
(BW (SW.CD)
0.0
TBW 2033
7.2
0.0
21.
18.1
0.0
15.3
0.0
0.0
833
(3R)
0.1
0.0
40.4
0.0
1.4
0.0
0.0
7.0
(3IR.CD)
(200 BL.
0.4
0.0
233
877
0.0
80.0
0.0
8.6
(W1
0.0
0.0
00
0.0
0.0
0.1
36.9
0.0
0.1
(CD)
(RULL)
0.6
(Bw)
(BW.CD)
(BW 3IR)
(38)
(3R..D)
(3RR, BL)
(W)
(CD)
(HILL)
Figure 13: Evaluation of Mc4's 's ability to retrieve correct Ed, during the RL agent's training.
agent's training continued to be 100%. Figure 1 provides an analysis the Connect 4 joint embedding model's. Mcs, average Recall 1. We average, Mos has average Recall @ of 86.1 %, with 13.9% incorrect retrievals. As mentioned in Section $5. results, these incorrect retrievals do not significantly affect Mc4's 's ability to inform reward shaping and the agent's learning rate.
D
RL Agent Details
D.1 RL Model Training
We utilized the open-source version of MuZero from 52 and utilize the provided hyperparameters. Please see Muzero's 's Github Repo from 52 as well as our code linked in Appendix F for more details. To train and validate our MuZero agents we leverage an Intel i7 processor. The runtime for training a Connect 4 MuZero agent to around 400k training steps takes approximately 30 hours. The runtime for training Lunar Lander MuZero agent to around 400k training steps takes approximately hours.
NVIDIA GTX 1060 6GB GPU and
D.2 Shaping Rewards Per Concepts
For Lunar Lander, utilize the existing dense reward function to determine the relevant concepts as well as their shaping values. Therefore each c E CLL has an expert-defined shaping value. The existing shaping values be found in the LunarLanderpy provided by OpenAI Gym (4]. Table 5 summarizes the continuous functions outlined in [4] that we use to reward each concept E CL..- Alternatively, there existing SoTA reward shaping values for the game of Connect 4. Therefore we perform hyperparameter sweep to determine an optimal set of shaping values, and Table summarizes utilized shaping values for each concept Ccs-
Lunar Lander Concept
Shaping Values
POS (position)
100 / (pos,) )2 + (pos,)) )2)
VEL (velocity)
100( / (ved,,) + (vel,))
TILT (tilt)
100 (angletander)
RLEG (right leg contact)
10(bool
LLEG (left leg contact)
10 10(bool
MF (main fuel)
0.31 - (bool
SF (side fuel)
(888bboo.sco side
L (land)
100
Table 5: Corresponding shaping values from [4] for each Lunar Lander concept.
22

---

Connect Concept
Shaping Values
3IR (Three- in Row)
3IR BL (Three-in Row Blocked from Win)
CD (Center Dominance)
BW (Block Win)
5
W (Win)
10
NULL (Neutral State)
0
Table 6: Corresponding shaping values determined from hyperparameter sweep for each Connect 4
concept.
E
User Study Details
E.1 Procedure Details
Participants performed an online study in which they were asked to play several games from the domain. The user studies for each domain were broken into the following four stages: Practice, Pre-Test, Explanation, Post-Test. First, prior to the Practice stage, participants were introduced to the game including background information, game rules, and their task. Then in the Practice, participants played 2 practice games to get further acquainted with the specified domain. In Pre-Test, participants played 3 Explanation, participants interacted with an expert player (well-trained RI. agent) and were exposed explanations about the player's actions via their assigned study condition. Note, participants in the "None" study condition did not receive any accompanying explanations during the Explanation stage. Specific Connect 4, this stage consisted of observing the expert player (RL agent) play games. In Lunar Lander, the Explanation stage consisted of observing the expert player (RL agent) play I game. In the Post- Test, participants played more scored games, similar to the Pre- Pee-Test. After the Post-Test stage, participants completed a cosst-uustioonaire survey that collected demographics data as well as their perceived task performance and their perceived utility of their explanation condition.
games, this time being aware that their games were being scored. In
Some specifics unique to each domain include that for Connect 4, participants always played first, and well-trained MuZero agent. Additionally, for Lunar Lander, reduce the impact of learning effects, the starting position and initial force applied to the lander was randomized across all games; however, the random combinations remained fixed across participants. Also, participants played the version of Lunar Lander implemented by OpenAl Gym [4]
their opponent player
E.2 Participant Information
We recruited 84 participants from Amazon Mechanical Turk, for an IRB-approved study and participants were required to novice players of Connect Of the 84 participants, were filtered out for not finishing the study. The remaining 75 participants, 15 per study condition, included 40 males and 35 females, all over the age of 18 (M=31.57, SD=7.12). The study took on average 20 minutes, and participants were compensated $5.00.
Connect 4
Lunar Lander We recruited 118 participants from Amazon Mechanical Turk, for an IRB-approved study. Participants were required to be novice players of Lunar Lander. Of the 118 participants, 13 were filtered out for not finishing the study or demonstrating no effort. The remaining 105 participants, 15 per study condition, included 50 males and 55 females, all over the age of 18 (M=30.57, SD=6.75). The study took on average 20 minutes, and participants were compensated $5.00.
E.3 Additional Evaluation Details
Participant ATS Note, the adjusted task score (ATS) adjusts participant's 's Post-Test average task performance by their Pre-Test average task performance. Note, this adjustment does not change any of the results or statistical analyses and is purely for visualization purposes. Specifically, we perform the adjustment to visualize the relative changes among participant ATS using common starting point.
23

---

Perceived Performance
None
Ey
EA
EC
EG w/
Ec; w/ inF
EG w/ TeG,inF
120
Disagree
Strongly Disagree
Neutral
Agree
Strongly Agree
Figure 14: Analysis of perceived performance Perf shows participants across all conditions believed their performances improved after exposure to the expert player and assigned explanations.
Perceived Explanation Utility
Ey
EA
EC
c; w/ TeG
Ec w/ InF
Ec w/ TeF InF
-40
-20
20
40
60
80
100
120
Disagree
Strongly Disagree
Neutral
Agree
Strongly Agree
Figure 15: Analysis of perceived explanation utility EU shows participants across all explanation conditions believed their assigned study condition improved task understanding.
Additionally, to keep the ATS graphs legible, we present the mean and standard deviations for each study condition here, opposed to including error bar lines the graphs. The following the mean and standard deviations the Connect 4 user study conditions. Specifically, (1) None (Baseline):
M=-0.039, SD=0.222, (2) A (Baseline): M=-0.041, 1D40.114 (3) &y (Baseline): M=-0. 104, SD= 0.346 (4) GT &c,: M=0.256, SD=0.318 (5) S2E & M-0.215, 1D-0.194. The following are the mean and standard deviations in the Lunar Lander user study conditions. Specifically (1) None (Baseline): M=-0.051, SD=0.191 1911, (2) & A (Baseline): M=-0.049, SD=0.136 (3) E (Baseline): M=-0.067, SD=0.211 (4) S2E & c, M=-0.043, SD=0.192, (5) S2E & Tea: M=0. 100, SD=0.154 (6) S2E Cc,w M=0.034, SD=0.199 190, (7) S2E C.w// TeG: M=0. SD=0.182.
User Perception Analyses Below we present additional evaluations performed on user responses in our questionnaire. Specifically, we present participant's perceived performance Perf and participant's perceived explanation utility EU. Note, Perf is measured on a 5-point Likert scale to the following question "I believe my game performance improved after witnessing the expert play and seeing the expert's 's provided reasoning". The EU metric is measured on 5-point Likert scale to the following question "I believe the expert's provided reasoning helped improve my understanding of the game". and is only measured for the explanation conditions. From both Figure [4] and Figure 15 we observe that participants across study conditions had high agrees for Perf and EU, despite their being significant differences in their objective task performance. These qualitative results demonstrate the Dunning-Krunn Effect [I3), that users with low expertise an area often overestimate their own performance or knowledge.

---

play
prinaay
Tectire
Nat
(b)
Figure 16: Participants saw the following as introductions to the task for (a) Connect 4 and (b) Lunar Lander.
Hoall appecte Bhe
cosepppl autee rabo colegaaly warleeaeeeeeeee
Wetc Dample
Weach Goa
Teat Aufie
(a)
(b)
Figure 17: Participants saw the following as introductions to the game for (a) Connect 4 and (b)
Lunar Lander.
E.4 Visuals of User Study
In this section, we provide visuals of the user interface for each user study.
Introduction Stage In both user studies, after receiving consent of participation, participants
were given introduction to both the user study well as the specific domain of the study. Figure 16al a)and Figure [bb(b) show the introductions participants received for Connect while Figure 17a a) and Figure 17b(b) b) show the introductions participants received for Lunar Lander. Note the
25

---

Practice Game 2
Practice 2 of
Your Tum
Inveenes
Fir seer Fagle
(a)
(b)
Figure 18: Visualizations the game interface participants utilized during the Practice, Pre-Test and Post-Test stages for both Connect 4 (a) and Lunar Lander (b).
Interact with an expert playing a Slart Game scenario
an experi playing game
Explanation
Explanation
Lett braine seceuse
a
(b)
Figure 19; Visualizations of the Explanation stage for both (a) Connect 4 and (b) Lunar Lander.
introduction to the task was first presented to the users, and then the introduction to the domain was presented.
Game-Playing Stages During the study, participants played various games during the Practice, Pre-Test and Post-Test stages. These games in real-time; users clicked the action play via a button, and saw the game interface update accordingly. Figure T8a a) and Figure (8b(b) present visuals on how participants played games in each domain.
Explanation Stage During the Explanation stage, participants interacted with an expert player (expert RL agent) and stepped through the expert player's actions while being exposed explanations from a given study condition. The participants were not told that the expert player was an RL agent limit the confounding effect of user biases towards Al agents in study. Figure 19a (a) and Figure b) provide example of the explanation stage for Connect 4 and Lunar Lander. Specifically, in Figure [a a), a &c, is provided, and in Figure 19b(b), a C.w/ InF, TeG is provided.
Survey At the end of the user study, participants were asked to fill out short survey including demographic questions, as well as additional Likert questions to gauge user experiences with comput- ers, the domain, as well as their perceived performance and explanation utility. Figure 20 provides visual of what the survey looked liked in the domain of Connect The survey for Lunar Lander participants was identical, except making an reference to Lunar Lander as opposed to Connect 4.
F
Github Repo
26

---

Wrap Up Survey
Great job, are almost dane! Please answer few final questions to help us with our study.
Question 1. Please select your age range:
18-24
25-34
35-44 94-44
55 and older
Prefer not to
Question Please your gender:
Male
Other
Prefer not to
Question your computer usage frequency
Never
Rarely
Sometimes
Often
Always
Question Please indicate frequency which you play computer games
Never
Rarely
Sometimes
Often
Always
Question 5. Please indicate your level of exposure to Connect prior to this study
Very Familiar
Familar
Moderately Familiar
Slighty Familiar
Not Familiar
Question 6. belleve my game performance improved after witnessing expert play and seeing the expert's provided reasoning
Strongly
Agree
Undecided
Disagree
Strongly Disagree
Question 7. believe the expert's provided reasoning helped improve my understanding of the game
Strongly
Agree
Undecided
Disagree
Strongly Disagree
Figure 20: Visualization of the survey questions participants received at the end of the Connect Study; questions were identical for the Lunar Lander study.
27


Reviews:
[{"review_id": "xbyVvf87i2", "reviewer": "Reviewer_smxt", "summary": "The authors propose a unified framework called State2Explanation (S2E) that combines learning a joint embedding model between state-action pairs and concept-based explanations. The authors draw inspiration from the Protégé Effect, which suggests that explaining knowledge reinforces self-learning. They propose that concept-based explanations can benefit both the RL agent and the end-user by improving the agent's learning rate and the end-user's understanding of the agent's decision making. The S2E framework is designed to inform reward shaping during an agent's training and provide explanations to end-users at deployment for improved task performance.Results on Connect 4 and Lunar Lander demonstrate the success of S2E in providing a dual benefit.", "questions": "- To be specific, how concept candidates are selected when performing InF? \n- How thresholds are determined via qualitative methods? Is there any results showing that different concepts selected will influence the final performance? ", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel framework S2E. The authors suggest that explaining knowledge to the agent can improve its learning rate, while providing explanations to end-users can enhance their understanding of the agent's decision making. The framework is supposed to be useful in providing explanations understandable in various applications.", "weaknesses": "There are no significant weaknesses from my point of view, while there are some improvements can be made. Although the paper mentions experimental validations in the Connect 4 and Lunar Lander domains, the proposed method should be tested on more complex tasks to test its effectness. The authors should also consider how concepts can be better selected beyond expert-defined thresholding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer smxt, we thank you for your valuable feedback. Below we provide responses to weaknesses and questions:\n\n**W: “..the proposed method should be tested on more complex tasks to test its effectiveness. The authors should also consider how concepts can be better selected beyond expert-defined thresholding.”**\n\nWe agree that more investigation of the applicability of S2E should be performed in other complex domains.  However, we believe insights from Connect 4 and Lunar Lander are themselves valuable, as both domains are challenging and complex in their own ways. We chose Connect 4 given its large state space (~4 trillion unique states), and Lunar Lander for its complex continuous state space with actions sampled at high frequency, properties that make both domains challenging in RL as well as for providing explanations.\n\nWe believe S2E can be applicable in other settings, such as domains with more complex trajectory optimization (robotics) or complex concept representations (i,e. Go, Chess) with small modifications made to the joint embedding architecture to improve concept-explanation to state-action pair mappings. We agree that future work should investigate other automated methods for concept-filtering beyond expert-defined thresholding. We will add this to our limitations section. \n\nOverall, we would like to highlight that given, to our knowledge, that this is the first work that explores the notion of a dual-benefit of concept-explanations to both the agent and end-user, we believe that our proposed S2E framework and its success in providing a dual benefit in Lunar Lander and Connect 4 is an important contribution towards understanding how concept-based explanations can be utilized to provide a dual benefit in other RL domains.\n\n**Q1:To be specific, how concept candidates are selected when performing InF?**\n\nIn InF, the thresholds are defined through a qualitative analysis of the agent’s state values when rolling out the agent’s policy. Note, the state values that are analyzed are directly mapped to a concept(s). For example, in Lunar Lander we observe the agent’s “x-position” and “tilt” state values which correspond to concepts “position” and “tilt”. These details are provided in Appendix A.\nFor more details, we will add the following details to Appendix A: The derived thresholds signify the positive and negative turning points in the agent’s ability to reach G. In other words, these thresholds are expert-defined upper and lower bounds on the agent’s state values that denote the agent’s ability to succeed or fail in its goal. In our InF method, these thresholds are not mathematically derived, but are derived from RL-expert analysis. Specifically, an RL expert visualizes multiple policy rollouts while analyzing the different state values over time to manually determine upper and lower bounds (turning points) that influence the agent’s ability to reach G. In our discussion section we will include that future work involves automating the InF method.  \n\n\n**Q2:How thresholds are determined via qualitative methods? Is there any results showing that different concepts selected will influence the final performance?**\n\nAs mentioned in response to Q1, the thresholds are manually derived from RL-expert analysis. Specifically, an RL expert visualizes multiple policy rollouts while analyzing the different state values over time to determine approximate upper and lower bounds (turning points) that influence the agent’s ability to reach G. It is true that different thresholds can result in different abstracted, concept-based explanations. \n\nIn response to the rebuttal, we performed an additional study to analyze the sensitivity of our current chosen thresholds for the Lunar Lander domain. Check the PDF in the global review for the graphs. We will add these graphs to Appendix A as additional analyses. All graphs show what fraction of concepts are filtered (y-axis)  as the threshold values change (x-axis). When looking the first graph, “X-position: Threshold Experimentation”, we see our chosen value is within the elbow of the curve, denoting that the rate of filtration of the “position” concepts slows down after 0.15. Similarly, in the “tilt” experimentation graphs (2nd and 3rd graphs), we see the lower and upper bound of the tilt thresholds to also be within the “elbow” in each curve. Note, in the upper bound threshold experimentation, the lower bound value is fixed, and vice versa.  To study the impact of these threshold values, additional user studies need to be conducted on the utility of the different abstracted concept-based explanations that result from varying the experimental threshold values.  However, we consider such analyses beyond the scope of our work, and one to further explore when improving the information filtering submodule within our S2E framework.\n"}, {"review_id": "MdDExye7zO", "reviewer": "Reviewer_ppHL", "summary": "The paper proposes a framework to incorporate explanation concepts to sequential decision tasks. The framework can be applied both to the training of the agent by improving RL and to provide explanations to end-users during deployment. The framework is tested using two simple games, Connect 4 and Lunar Lander, where it shows it help improve training performance and user performance.", "questions": "1. How do you expect the framework to perform when it is applied to more complex tasks and domains, both in terms of describing the actual concepts and on the performance during training?\n2. How hard is to identify, create, store, and debug concepts?\n3. What kind of wrong explanations it produces? How often? How that impacts training and user performance?", "limitations": "The paper does not do a good job of discussing how it can scale to real-world applications. By not doing so, it failed in identifying an important limitation of the work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The main strengths of the paper are:\n\n1. It provides an unified framework for explanations which can be used both for training and deploying.\n2. The explanation framework is based on user-understandable concepts and terms.\n3. The paper provides a sound theoretical and empirical analysis of the proposed framework.\n4. The paper performs user testing and subsequent data analysis in a methodologically sound manner, rare in NeurIPs papers.\n", "weaknesses": "The main weaknesses of the paper are:\n\n1. The paper condenses two much information in very little space, including impossible to read figures. It is very hard to read.\n2. The paper fails to discuss how difficult will be to create frameworks for more complex scenarios where there are more and more complex concepts. For instance, consider a system making decisions about buying and selling stocks.\n3. Because of 2, the ideas presented in the paper may never be applicable to real-world problems.\n4. It provides a very brief description of how the concepts are actually created, stored, and verified. \n5. The paper does not discuss situations where the explanation is wrong or inadequate for a situation (for instance, there is a better move), and how that impacts training and user performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ppHL, we thank you for your valuable feedback. Below we provide responses to weaknesses and questions:\n\n**W1:** Thank you for pointing out--we will increase our image sizes.\n\n**W2 & W3:** \n\nWe believe that our S2E framework will be applicable to more complex scenarios; however, two components within the framework may need modifications for scalability. We will add the discussion of these two components and their importance in the discussion section.\n\nFirst, with an increased number of concepts (i.e. Chess) coupled with some concepts likely to occur more rarely (i.e. castling in Chess), we hypothesize that leveraging more complex joint-embedding model architectures that favor few-shot learning may be necessary. Second, with more complex domains there may be a greater need for providing abstracted concept-based explanations to end-users. S2E currently includes temporal grouping and information filtering, which we observed to be important for improved user task performance in Lunar Lander. We believe such abstraction methods will continue to be important in other complex domains. \n\nOverall, our evaluations in Lunar Lander and Connect 4 demonstrate the efficacy of the S2E framework. We believe its modular design (a joint embedding model learning decoupled from an explanation abstraction method) allows for implementing and validating extensions to both components important in providing a dual-benefit to agent and end-user.\n\n**Q1:**\n\nAs mentioned in response to the weakness above, we expect our S2E framework to be applicable in more complex tasks and domains. We expect some modifications needed to the joint-embedding model architecture to provide accurate retrievals of concepts to state-action pairs. If the joint-embedding model can retrieve concepts with high Recall@1, then based on our current findings, S2E can provide accurate concept-based explanations that both benefit end-user understanding at deployment as well as improve agent learning during training time. \n\n**W4:**\n\nWe will add the following to Section 5.1 as more details.In our work, each state is associated with relevant concepts by objective, mathematical representations of each concept (i.e. function for position over time, rule for existence of a three-in-a-row, etc). Therefore we verify that concepts are accurately paired by ensuring the mathematical rule for a concept to be paired to a state-action pair is met. \n\nIn the appendix we will also provide a more detailed guideline (with examples) on how we derive and verify concepts mathematically, and discuss how to derive and verify concepts that are collected via human annotation or commentary.\n\n**Q2:**\n\nIn our work, we leverage prior expert domain knowledge in Lunar Lander and Connect 4 to derive concepts important to the games via mathematical rules. In many real-world applications, it may be infeasible to leverage mathematical rules for concept derivation. Instead, concepts can be collected via crowdsourcing, such as in [20 from paper], or obtained via “think-aloud” procedures mentioned in [14 from paper]. Note, when concepts are collected via domain expert labeling, such labels should be verified via inter-rater reliability tests to ensure consistency among concept representations. These details will be added to the Appendix section.\n\n**W5:** + **Q3:\"..What kind of wrong explanations..How often?...”**\n\nFigure 3a and 3b show the accuracy of our trained joint embedding models in retrieving concept-based explanations within Lunar Lander and Connect 4. Specifically, Fig. 3b quantifies a breakdown of the incorrect concept-based explanation retrievals in Connect 4 on the joint embedding model’s test set. The learned joint embedding model for Lunar Lander has a near-perfect recall@1(99.9). In the Appendix, we will include example scenarios where the retrieved concept-based explanations are incorrect.\n\n**Q3:“..How [wrong explanations] impacts training and user performance?..”**\n\nIncorrect retrievals of concepts-based explanations for a given state-action pair can negatively impact agent training in S2E by incorrectly providing shaping rewards to the agent and in return impacting learned agent policy. However, our results in Fig. 4a and 4b show that the percentage of incorrect retrievals from our joint embedding models do not significantly impact the RL agent’s learning rate. In Fig. 4a when studying Connect 4, the reward shaping informed via S2E (MuZero + S2E-RS) results in a slightly lower learning rate than the upper bound expert-defined reward shaping (MuZero + E-RS). However, MuZero + S2E-RS still improves the agent’s learning rate by ~200 training steps compared to the SoTA baseline agent (MuZero + no-RS). Similarly, in Fig. 4b and 4c, we see that MuZero + S2E-RS performs similarly to the SoTA MuZero + E-RS, given that there exists expert-defined reward shaping for Lunar Lander. We will clarify this in Sec. 5.3.\n\nWith respect to user performance, concept-based explanations provided to the end-users during the “Explanation” stage are retrieved from our learned joint-embedding models. However, the number of incorrect retrievals was not significantly detrimental to user performance. As shown in Fig. 5, while the ground truth concept-based explanation condition had slightly greater ATS improvement, there is no significant difference in ATS improvements between the ground truth concept-based condition and its S2E counterpart. For Lunar Lander, we did not experiment with a ground truth concept-based condition given that our Lunar Lander joint-embedding model had near-perfect Recall@1 (99.9). We hypothesize that if we did perform such a comparison, we’d see no significant differences given the near-perfect Recall@1. We will clarify this in Sec. 6.1 and 6.2.\n"}, {"review_id": "Mxw4Mg8o4s", "reviewer": "Reviewer_g9Rv", "summary": "Inspired by the Protege effect, learning and developing explanations should provide a dual benefit, both to the readers of the explanations, and to the developers of the explanations. Based on this idea, the paper proposes State2Explanation, an algorithm to learn joint embeddings between state-action pairs and concept-based explanations. This allows for reward shaping, which benefits the explanation developer, while also providing explanations. These claims are validated for agents in two different reinforcement learning settings: Connect 4 and Lunar Lander.", "questions": "1. For reward shaping, how is the amount of reward determined/amount the reward function is changed by? \n2. In general, who would annotate the explanations, and what is the impact if there are multiple explanations for a single move/conflicting annotations? \n", "limitations": "The paper discusses limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Provides concrete desiderata for a concept is, including the need for generalizibility and its relationship to the task goal \n2. Reward shaping through explanations is potentially novel, and explores use cases of explanations beyond explainability\n3. Thorough evaluation investigates the impact of various additions to the model, making it clear what the impact of Information Filtering (InF) and Temporal Group (TeG) are\n4. User study demonstrates real-world viability model through a human explainability lense\n", "weaknesses": "1. Concepts for domains are dependent on expert knowledge for state-action pairs, making it unclear how easy it would be to generalize beyond well-studied games. This is especially the case when state-actions pairs tend to get very large. \n2. An additional baseline would make it clearer what the impact of each study condition was in Section 6. In particular, a baseline with no intervention/information would make it clear how much the rise in performance is due to the additional practice attained from the PreTest when completing the PostTest. \n3. If concepts for Lunar Lander are derived from the existing domain reward function, then claiming that $M_{LL}$ \"informs reward shaping comparable to expert-defined dense reward functions\" seems to follow from the definition of the concepts rather than an indicator of the performance of the algorithm. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer g9Rv, we thank you for your valuable feedback. Below we provide responses to weaknesses and questions:\n\n**W1: “Concepts for domains are dependent on expert knowledge for state-action pairs, making it unclear how easy it would be to generalize beyond well-studied games. This is especially the case when state-actions pairs tend to get very large.”**\n\nThank you for highlighting this point. It is true that concepts related to state-action pairs should be derived from domain experts to ensure accurate concept representations. To scale to large state-action pair scenarios where collecting large amounts of expert annotated data may be unfeasible, our joint embedding model in S2E can be optimized for few shot learning. Note, a strength of leveraging a joint embedding model in our S2E is that our trained joint embedding model can also be used to retrieve accurate concepts-based explanations for unannotated state-action pairs. In this manner, when considering scenarios with large state-action pairs, we can leverage our joint embedding model to provide annotations as well. We will include this in the discussion section.\n\n**W2: “An additional baseline would make it clearer what the impact of each study condition was in Section 6. ….a baseline with no intervention/information would make it clear how much the rise in performance is due to the additional practice attained from the PreTest when completing the PostTest.”**\n\nIt is true that an additional baseline where there is no information provided in the “Explanation stage” could have offered an additional layer of analyses when compared to our current study conditions. \n\nImportantly, we’d like to highlight that such additional baseline is **not necessary** to show “how much the rise in performance is due to the additional practice attained from the PreTest when completing the PostTest.” In our experimental analysis, we do currently take into consideration any learning effects obtained via the “PreTest” since our ATS metric measures the *difference* between the Pre-Test and Post-Test scores. That is, we account for participants’ initial performance on the “PreTest” and measure whether the intervention/explanations provided in the “Explanation stage” help to provide any *improvements* to their “Post Test” scores. \n\n**W3: “If concepts for Lunar Lander are derived from the existing domain reward function, then claiming that MLL \"informs reward shaping comparable to expert-defined dense reward functions\" seems to follow from the definition of the concepts rather than an indicator of the performance of the algorithm.”**\n\nWe disagree with this comment. Concepts derived from an existing expert-defined reward function allow us to directly compare how well a joint embedding model trained using these concepts (M_{LL}) can inform reward shaping when compared to expert-defined reward shaping. In other words, having an existing dense reward function for Lunar Lander allows us to make a direct comparison of whether M_{LL} has learned good mappings between concepts and state-action pairs to effectively inform reward shaping. \n\n**Q1: For reward shaping, how is the amount of reward determined/amount the reward function is changed by?**\n\nMentioned in Section 4.2 and 5.3, the actual amount of reward for the Lunar Lander domain is determined by the existing reward shaping function. Similarly for Connect 4, we perform a hyperparameter sweep to determine corresponding shaping values for each concept. In Appendix D.1 we have presented the shaping values for each concept for each domain. Note, we do not claim these are the most optimal shaping values, but the best that either currently exist (Lunar Lander) or were found via a hyperparameter sweep.\n\n**Q2: In general, who would annotate the explanations, and what is the impact if there are multiple explanations for a single move/conflicting annotations?**\n\nIn general, concepts for any domain should be collected from experts within a domain. We will emphasize this in Section 4.1 and Section 5.1. As mentioned in Sec. 5.1,  the concepts in our work are identified by access to expert-domain knowledge within the fields. \n\nIt is interesting to consider cases where “k” different concept-based explanations exist for a given state-action pair.  While in our work, we did not consider these scenarios, we believe there are at least two possible avenues to employ as solutions without drastically changing the S2E framework. One solution is to perform an inter-rater reliability to determine if there is agreement among experts on a single concept-based explanation out of the “k” explanations being more preferable or “agreed” upon, and using such single explanation when training the joint embedding model . A second solution is to allow the joint embedding model to see state-action pairs be associated with all “k” different concept-based explanations. During retrieval, the top ranked explanation can be retrieved, and given a well trained joint embedding model, the retrieved explanation will be one of the “k” possible correct explanations. These solutions are an opportunity for future work and we will include this discussion in an appendix that is related to Section 5.1.\n"}, {"review_id": "tkAIrAqPMN", "reviewer": "Reviewer_GSRV", "summary": "The paper proposes State2Explanation, a framework for training RL agents in such a way that both the human and agent benefit, the Protégé Effect as the authors state. The basic idea is to learn a joint embedding space with \"temporal\" concepts that actively helps the agent train better by shaping their rewards. At test time, these explanations are provided to humans to also help train them.", "questions": "1. During the training there's no feedback from the human right? If I understand, the human defines the concepts, and then these help the agent train better? (but Fig 4 seems to show this isn't the case?)\n2. Whilst you used a CNN in connect 4, does this scale to harder Deep RL problems?\n3. Is my understanding that Fig 3a shows the explanations are often \"wrong\" accurate?\n4. What is i, j, and k in Fig 1?\n5. How did you pick a sample size in the user study? Although it seems fairly rigorous, I don't see any power analysis, and you aren't clear what your attention checks are? It also seems strange that you had 66->60 and 98->90 in the two domains, a perfect balance for 15 in each group. \n\nI might recommend using Prolific instead, it seems to have less issues compared to MTurk which is full of bots and LLMs.\n\nI am happy to revise my review after the rebuttal.", "limitations": "This is ok", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "A strength of the paper is I feel the motivation. Explainability in RL is harder than supervised learning due to the temporal added component, so approaching this problem is well motivated.\n\nIn addition, the interplay between agent and human is also very exciting, I love the idea of the two helping and playing off each other. I feel this is a great way to impose human understandable concepts onto an RL agent's policy.\n\nI also appreciate the rigour in the user study (although I need some clarifications in the rebuttal), it's clear to me the authors tried here.", "weaknesses": "What I always look for in a paper is a single nice idea, and I am not certain this paper really contains one, let me explain by iterating your claimed contributions.\n\nRegarding the desiderata: (1) the idea that a 'concept should be grounded in human domain knowledge' is not a new idea I feel. The original T-CAV paper noted this by e.g. using zebra stripes to explain zebra classifications, rather than e.g. logit values. Simple saying the same thing is true in a sequential setting doesn't seem that original to me. I'm not aware of a great many people suggesting q-values are a good explanation. (2) The idea that a concept should relate to the task goal is an interesting one, and could be useful in certain context, but I feel it doesn't generalize well. For example, if I'm in a self-driving car going from a => b, and halfway through the journey the car brakes to avoid hitting a pedestrian, your framework seems to posit the explanation should be \"I avoiding killing the human so I could still get to point b\". I feel that's a questionable belief system here we are teaching the agent. (3) is a nice idea, but I don't feel it's particularly surprising, the idea that explanations should be robust and generalisable is well agreed upon.\n\nThe framework you propose does seem quite interesting to me, but reward feedback loops are not my area of expertise, so I defer to my colleges to evaluate the novelty of such a framework.\n\nSection 5.2: I am slightly concerned that the explanations are not always matching in Figure 3, does that mean that this method will inevitably give \"wrong\" explanations? Also, I don't know what i, j, and k are in the figure? Now I personally believe it matters more that it just helps people in a user study, which you show it does, so that's great and I feel makes up for this.\n\nSection 5.3: If I understand this correctly, your method doesn't seem to show much difference here at all, and in fact Fig 4a seems to show the baseline outperforming it?\n\nUser Evaluation: I appreciate the attention to detail here, but a lot of details are missing or (I feel) badly explained. I don't really understand how people's performance could get worse in group $E_{A}$, all they are being shown is the action as an explanation, so the idea their ATS would get worse seems extremely odd to me, and makes me feel I am misunderstanding something here. I tried to explore the appendix to understand what was happening, but didn't have much luck.\n\nI think there's some citations that you might also like to be aware of regarding human friendly concepts in sequential settings.\n\nJi, Ying, Yu Wang, and Jien Kato. \"Spatial-temporal Concept based Explanation of 3D ConvNets.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nKenny, Eoin M., Mycal Tucker, and Julie Shah. \"Towards interpretable deep reinforcement learning with human-friendly prototypes.\" The Eleventh International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GSRV, we thank you for your valuable feedback. Below we provide responses to weaknesses and questions:\n\n**W: single idea missing..**\n\nWe disagree with this comment. To our knowledge, our framework S2E is the first unified framework that considers how concept-based explanations can provide a dual benefit to the RL agent and user, inspired by the Protege Effect. Specifically, S2E leverages a joint embedding model to retrieve concept-based explanations that both inform reward shaping to aid agent learning during training time, as well as improve user task performance at deployment time. Our work is important since S2E goes beyond showing the utility of explaining an agent’s behavior for user understanding and provides a joint benefit to the agent as well, all within a single framework.\n\n**W: desiderata..**\n\nPrior work in concept-based explanations for sequential decision making have had limited definitions of concepts using state preconditions, action costs or control logic [22,46 in paper]. However, our work posits that concepts in sequential decision making should be defined by higher-level properties as outlined by our desiderata. Our desiderata, inspired by concept-based explanations in classification tasks (like T-CAV), is a first step to providing a generalized definition of concepts in sequential decision making. We don’t claim that our desiderata is a complete set and will clarify this in Sec. 3. Future work includes extending our desiderata, including your valuable point about considering environmental uncertainties for defining concepts.\n\n**W: Sec. 5.2** + **Q3:**\n\nFig. 3 shows the accuracy of the joint embedding models’ ability to correctly retrieve concept based explanations related to state-action pairs. Given the recall rates are not 100%, incorrect explanations are retrieved a fraction of the times as shown in Fig 3. However even with imperfect joint embedding models in S2E, results in Sec. 5 and Sec. 6 show S2E’s utility in a dual-benefit to agent learning rate and user task performance (see weakness response).\n\n**W: Sec. 5.3**\n\nIn Fig. 4a, the current SoTA baseline agent that exists for Connect 4 is an agent not trained with reward shaping–referred to as “MuZero + No-RS”. Fig 4a shows that even with an imperfect joint embedding model, S2E is able to inform reward shaping and improve the agent’s learning rate compared to “MuZero + No-RS” in Connect 4. As mentioned in Sec. 5.3, our S2E informed reward shaping (“MuZero + S2E-RS”) improves the agent’s learning rate by 200 training steps compared to “MuZero + No-RS”. The “MuZero + E-RS” curve in Fig 4a shows the upper bound performance of S2E if our Connect 4’s joint embedding model was perfectly trained. We expect the small gap seen between “MuZero + E-RS”  and “MuZero + S2E-RS”, given from Fig. 3a that Connect 4’s joint embedding model is not 100% accurate. Similarly, in Fig. 4b and 4c we see that in Lunar Lander, our MuZero + S2E-RS can inform reward shaping compared to the existing soTA dense reward shaping (“MuZero + E-RS”). We will clarify this in Sec. 5.3. \n\n**W: User Eval..**\n\nThis was an interesting finding; however, not a weakness of our method. The downward trend with “Action-Based” Explanations demonstrates possible negative effects of having to learn from only expert actions as feedback. We hypothesize that users provided with only expert actions caused users to project an incorrect reasoning to the experts’ actions which in return confused their understanding and led to worsened performance. We will discuss this finding in the appendix. \n\n**W: Citations** -- We appreciate you pointing out these; we will include in our related works.\n\n**Q1:**\n\nThat is correct, during the RL agent’s training time there is no feedback from the human. To clarify, concepts are defined by domain experts and S2E learns a joint embedding model that maps concept based explanations to state-action pairs. The learned joint embedding model is then used to inform reward shaping during RL agent training and produce concept-based explanations at deployment to improve user task performance. See our response related to the weakness around Sec. 5.3 that our S2E can inform reward shaping and thereby improve agent learning rates compared to no-reward shaping (Connect 4), and provide comparable learning rates to existing dense-reward functions (Lunar Lander).\n\n**Q2:**\n \nCNN’s have been leveraged in existing Deep RL architectures, including the SoTA MuZero RL model architecture. We do not claim our exact CNN architecture will apply to all Deep RL problems, and modifications (such as additional layers in the joint embedding model) may be necessary for harder RL problems. Developing a single joint embedding model architecture generalizable across multiple complex RL domains was beyond the scope of this work and is mentioned in our limitations as future work.\n\n**Q4:**:\n\nThe “i”, “j” and “k” in Fig. 1 describe three different concept-based explanation embeddings that exist in the learned joint embedding space. As noted in Sec. 4.3, the best ranked explanation embedding is retrieved from the learned joint embedding space similar to image-to-text-retrieval. We will qualify “i”, “j” and “k” in the Fig. 1 caption.\n\n**Q5:** \n\nIt is good practice to have an equal number of study participants per study conditions to conduct fair statistical analysis comparisons. After filtering participants, we continued to recruit participants to ensure we had 15 participants per condition. We filtered participants by analyzing visualizations of the participants’ games. We determined a participant to show no effort if they only played a single action during the entire course of any of their pretest or posttest games. We did not conduct a power analysis in this particular study given that power analyses require making estimations about each study conditions’ mean or variance which we did not deem appropriate to assume. We will add these details to Appendix E.3. "}]

Author rebuttal:
We thank all our reviewers for their detailed comments. Firstly, we are encouraged that reviewers saw the importance of our framework S2E in providing a dual benefit to the end-user as well as RL agent. Reviewer ppHL found our work provided “sound theoretical and empirical analysis”, and reviewer g9Rv saw strength in our “concrete desiderata for a concept”. Reviewer smxt believed in the novelty of our work, and reviewer GSRV agreed with the important need for a framework like S2E, and loved the “idea of the two [agent and user] playing off each other”. We are also glad that reviewers saw value in our analyses, in particular that our user studies demonstrated “real-world viability” (g9Rv), was “methodologically sound” (ppHL) and its “rigor” was appreciated (GSRV). 

We found all feedback to be constructive and informative. We respond to each reviewer’s questions and weaknesses below. We believe that our responses should clarify and address reviewer concerns. If additional details or clarifications are needed, we are happy to provide them.

**Reviewer smxt**: please see attached graphs that support our response to your Q2.

Meta-review: The work introduces a framework for using explanations to guide representation learning. They demonstrate the effectiveness of their approach on two simple domains (Connect 4 and Lunar Lander) which have descriptions that correspond to actions, states, and basic strategy.  Authors were happy with the work and its analysis.  The primary shared concern is generality to more complex and realistic settings.

The final draft can be strengthened by addressing several natural concerns raised during the reviewing.  In particular, the amount of human effort, how this will change with more complex domains, and how the approach handles incorrect explanations.
Decision: Accept (poster)