Title: State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Concept-Based Explanations

Reinforcement Learning

Human-AI Interaction
Abstract: As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining ``concepts'' in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding

Abstract

As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining 'concepts' in sequential decision making settings. Additionally, inspired by the Protégé Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

1 Introduction

Black-box AI systems are increasingly being deployed to help end-users with everyday tasks. Examples include doctors leveraging decision support systems to aid in diagnosis [39], warehouse managers relying on robots for goods transportation [8], and drivers using autonomous vehicles for assisted driving [18]. To increase the transparency of these black-box models, researchers have developed numerous techniques to provide explanations of agent decision making [45, 2, 23, 51, 14, 9, 30, 15].

A popular method towards non-expert friendly explanations has been to attribute higher-level 'concepts' to an agent's decision making, and these concepts have primarily been used to explain classification-based AI systems [30, 33, 54, 19]. An example of a concept-based explanation for a classification model includes, 'wing color' for a bird class label [19]. In sequential decision making, concept-based explanations have been less widely explored, and existing works leverage preconditions of states and action costs [48] or logical formulas [22] as 'concepts'.

In this work, we posit that concept-based explanations may not only benefit the end-user but also benefit the AI agent. Our claim is loosely motivated by the Psychological phenomenon, the Protégé Effect, that states explaining material to another student also helps the explainer learn and reinforce material [7, 24, 17]. Thus the objective of our work is to demonstrate the dual-benefit of concept-based explanations to both the end-user for improved understanding, as well as the AI agent for improved learning rate. Note, prior works, typically in Reinforcement Learning (RL), have explored various methods to improve agent learning, a common approach including the use of human feedback and natural language commands as a method of reward shaping to improve RL agent sample efficiency and learning rate [38, 50, 20]. However, these methods have focused on leveraging language as a one-way benefit to the agent, not considering how language provided to the agent during training may also benefit end-user understanding at deployment. Therefore in our work, we explore the following research question: Can concept-based explanations have a two-way benefit to both the user and agent, such as to improve end-user understanding of a deployed agent's decision making, while also improving the agent's learning rate via reward shaping?" As a solution, we contribute a unified framework, State2Explanation (S2E) , that learns a single joint embedding model between agent states and associated concept-based explanations and leverages such learned model to (1) inform reward shaping for agent learning, and (2) improve end-user understanding of the agent's decision making at deployment.

Additionally, we contribute a desiderata of what entails a 'concept' in a concept-based explanation for sequential decision making systems. Given that sequential decision making agents engage in longterm interaction with their environment, we posit that the scope of concept-based explanations should span beyond representing preconditions and action cost [48], and control logic [22]. Specifically, we believe concept-based explanations in sequential decision making should function at a much higher-level of abstraction, highlighting knowledge that are applicable across multiple states, and most importantly, expressing a positive or negative influence towards the agent's goal. Our claim is motivated by the importance of the agent's goal in a sequential decision making formulation [44, 49].

Contributions . (1) We provide desiderata for what constitutes a 'concept' in concept-based explanations of sequential decision making. (2) We introduce a novel framework, State2Explanation (S2E) which learns a joint embedding model between agent states-action pairs and concept-based explanations to provide a benefit to both the agent, by informing reward shaping, as well as the end user by providing explanations that improve user task performance. We perform both model and user evaluations of our S2E framework in two complex RL domains, Connect 4 and Lunar Lander.

Our model evaluations demonstrate that S2E can successfully inform reward shaping, comparable to expert-defined reward functions. Additionally, our user evaluations demonstrate that S2E can successfully provide meaningful concept-based explanations to end users such that exposure to our explanations significantly improve user task performance.

2 Related Work

In this work, we explore the utility of concept-based explanations in providing a two way benefit to both the AI agent during training, via reward shaping, as well as end user at deployment time, via improved user task performance. Below, we highlight related prior work.

Reward Shaping using Language Many prior works have focused on improving an agent's learning rate or sample efficiency [35, 21]. Closely related to our work are methods providing reward shaping via language. Specifically, [38] propose ELLA, which leverages a relevance classifier to associate a lower level instruction to a higher level task for providing a subtask shaping reward. In [20], the authors leverage natural language instruction to derive potential-based shaping rewards that encourage an RL agent to more frequently select relevant actions in the game of Montezuma's Revenge. Similarly, [50] learn a joint embedding model between subgoal language commands and agent states to provide shaping rewards for completing subtasks in StarCraft II. Specifically, [50] demonstrate that their model can generalize to unseen, semantically similar language commands, and improve agent learning rate. In our work, we take inspiration from [50], and learn a joint embedding model between agent states and associated natural language commands to inform agent reward shaping. However, our work differs in that our natural language commands are concept-based explanations of the associated state, and not subgoal descriptions required for task completion. By learning a common embedding space between state explanations and state representations, our work

provides a two-way benefit to both the RL agent for learning as well as end-user for improved understanding.

Explanations for non-AI Experts Prior works have established that concept-based explanations for classification tasks provide explanations that are meaningful to end-users [30, 33, 54, 19]. Methods for concept-based explanations have included Concept Activation Vectors [30], Concept Bottleneck Models [33], and Concept Embedding Models [54]. For classification tasks, 'concepts" have primarily represented human-interpretable features that a model's prediction is sensitive to (i.e. 'wing color' for image classification.) [33]. Related to sequential decision making, [26] provide concept-based explanations of 3D action recognition CovNets by clustering learned, human-interpretable features (i.e. 'grass' or 'hand') from segmented videos. In [48], concept-based explanations for sequential decision making are formulated in relation to state preconditions (e.g. 'the action move-left failed in the state as the precondition skull-not-on-left was false in the state') and action costs (e.g., 'executing the action attack in the presence of the concept skull-on-left , will cost at least 500 '). Specifically, in [48], concepts have represented any factual statement a user associates to a given state, such as "skull-on-left" in the previous example. Similarly, in [22], concepts have represented logical formulas for summarizing RL policies. Additionally, [29] propose a prototype wrapper network for training interpretable RL policies with user-defined prototypes. Similar to [48], the authors in [29] define prototypes as factual observations about a state, such as 'turn right' or 'turn left'. Other explanation generation methods have demonstrated that end user understanding is improved when explanations are presented in natural language rather than features or labels [14, 5, 9, 10]. Inspired by these findings, our work contributes the first method for producing natural language concept-based explanations, where we define a general desiderata of concepts that goes beyond action costs and preconditions to capture goal-driven decision making.

3 Concept Based Explanations for Sequential Decision Making

We posit that the objective of a concept-based explanation, in sequential decision making settings, is to communicate higher-level knowledge that aids the agent and/or user in reasoning about the utility of taking a given action from a specific state. In this section, we provide a general definition of concept-based explanations and 'concepts", in sequential decision making, that extends beyond prior usage of state preconditions and action-costs [48], and control logic [22]. Specifically, we define a concept-based explanation, E i C i , as one that presents, in natural language form, the set of concepts, C i = { c j ..c m } , that can describe a particular state-action pair ( s i , a i ) in the context of goal G . This naturally poses the question of: "What is a concept c j ?" Below are several desiderata for defining a concept in sequential decision making settings:

A concept should be grounded in human domain knowledge . A concept, c j , should be contextualized in the overall task domain. For example, although an agent's goal may be to optimize its Q-function, we posit that "higher Q-value" is not a valid concept. Instead, a concept should represent a higher-level abstraction of what the 'higher Q-value' relates to in the domain. For example, in Chess, an agent receiving a 'higher Q-value' for a particular state-action pair may be translated by a domain expert to 'capturing a queen'.

A concept should relate to the task goal . A concept, c j , is an abstracted representation of ( s i , a i ) that encompasses how ( s i , a i ) may lead to or inhibit the agent's goal G . This claim is motivated by the general objective of a sequential decision making agent, that is to perform a sequence of actions 〈 a 1 , a 2 , ..., a n 〉 , a i ∈ A , defined via a plan or policy π , that transforms the agent's current state I to its goal state G [44, 49]. For example, we do not consider the description 'Knight in A4' as a valid concept, since, by itself, it does not provide relation to the game's objective. However, we would consider 'king safety' as a valid concept, since it relates to game's objective of checkmate.

A concept should be generalizable. A concept, c j , should generalize across multiple ( s i , a i ) pairs. This means that a concept is typically not a unique description of a single ( s i , a i ) . For example, a valid concept in Chess is "checkmate" since the abstraction of "checkmate" can generalize to many different ( s i , a i ) . An invalid concept involves detailing the entire game board by piece position.

Note, above we define a concept, c j ∈ C i , in the context of describing a single state-action pair, ( s i , a i ) . However, in many domains a meaningful concept c j can only be defined in terms of a

sequence of state-action pairs. This is especially true in domains in which state-action pairs are sampled at high-frequency, such as robotics and video games [40, 56]. For example, concepts such as 'rotate wrist' or 'bend arm' can only be defined by a series of state-action pairs, as opposed to a single state-action pair. As a result, we define concept-based explanation in these domains as E [ i,i + n ] C i , which presents the set of concepts C i = { c j ..c m } , for a series of ( s [ i,i + n ] , a [ i,i + n ] ) pairs.

Finally, we highlight that some ( s i , a i ) or ( s [ i,i + n ] , a [ i,i + n ] ) pairs may not have a concept-based explanation associated with them. In other words, C i = ∅ for some ( s i , a i ) pair or ( s [ i,i + n ] , a [ i,i + n ] ) pairs. This is especially true in long horizon tasks, in which a ( s i , a i ) may not relate directly to G but is important for another, future ( s i + n , a i + n ) pair which then leads to G . For example, 'Move the pawn to D5 so that two moves from now you may capture the rook'. Our definition of concept-based explanations do not consider these cases, and is an opportunity for future work.

TrainJointEmbeddingModel M

Leverage ec, in RL

Leverage &c, for User Understanding

"Creates a three in

Isee why C6 is

a row and provides

agoodmove!

center dominance"

[0]

Actionai

Actiona

ECi

[s, si-1]

16

Statesi

State s

iM

expembed↓

LSTM

NN

Rewardr

S2E

Information

ExplanationAbstraction

Eabstracted

concept

Temporal

S2E

L(0)=ZN(lexpembed - sembed 1- y)

Reward Shaping

Explanationto

Filtering (InF)

Grouping(TeG)

expembed

Sembed

Learned

ExplanationRetrieval

Learned

ExplanationRetrieval

Learned

expembed

expembed

expembed

expembed

Joint EmbeddingSpace

[56 238 7 ... 4]

[56 2 38 7 ...4]

expemhed

expembed

[“because it leads to

[“because it leads to

exp

dxa

expembed

k

[Um mo-e-Uj-Jnoj e

exp

expembed

k

a four-in-a-row win°]

expembed

expgbed

expembed

(b)

(c)

(a)

4 State2Explanation (S2E) Framework

Figure 1 introduces our State2Explanation (S2E) framework which includes a three-step process to provide a benefit to both the RL agent and the end-user. Specifically, S2E learns a joint embedding model M to align agent state-action pair ( s i , a i ) to concept-based explanations E i C i (Fig. 1a). S2E then leverages the learned M to inform reward shaping during agent training (Fig. 1b). Finally, S2E leverages the learned M to provide E i C i about the trained agent's decision making to end-users at deployment (Fig. 1c). Below, we further detail S2E.

4.1 Joint Embedding Model M

A core component of S2E is learning a joint embedding model, M , to align an agent state-action pair, ( s i , a i ) with an associated concept-based explanation E i C i . The motivation for learning M is to explicitly map an agent's representation, s i , into a representation understandable by our non-AI expert users, E i C i . Joint embedding models have been leveraged in many applications, such as in image captioning [55, 34], knowledge-graph generation [28, 43], and RL [42, 41, 50] to improve task learning. In the context of S2E, we demonstrate that joint embedding M can provide the dual benefit to i) agent learning, and ii) improved end user understanding of agent actions. Our M is inspired by [50] who learn an embedding space in the context of hierarchical RL tasks. However, unlike in [50], our M is not constrained to hierarchical RL tasks, and we consider domains in which multiple states can be associated to multiple, non-unique concepts.

The input to our joint embedding model, M , is defined by 〈 [ s i , s i -1 ] , g i , E i C i , y i 〉 , which includes an agent's current state s i , previous state s i -1 , other contextual game information (i.e., player in a multiplayer game, whether game is over) g i , associated concept-based explanation E i C i , and y i which defines if s i and E i C i are aligned or misaligned. Note, including s i and s i -1 implicitly encodes the

action a i taken to transition from s i -1 to s i . Our contrastive loss function (Eq. 1) is adapted from [50], in which θ represents the model's parameters, and i ∈ N represents the i th training sample. Specifically, M minimizes the L2-norm of the difference between the embedding vectors when s i embed and exp i embed are aligned ( y = 0 ) and maximize the L2-norm of the difference when s i embed and exp i embed are misaligned ( y = 1 ).

To train M , we leverage a dataset D = { D a , D m } , in which D a and D m denote the set of aligned and misaligned data samples, respectively. To collect D a , we leverage domain knowledge to annotate relevant s i with its aligned concept set C i . The concept set C i is then produced into a natural language explanation, E i C i , manually via fixed, expert-defined templates. That is, each C i has one templated E i C i . The list of concepts for our evaluation domains are in Section 5.1. To collect D m , each s i is paired with z concepts not present in concept set C i . Section 5.2 provides more details on D .

Explanation Retrieval Given M , for any ( s i , a i ) , we extract the closest explanation embedding exp i embed for corresponding s i embed , and decode exp i embed to an E i C i similar to image-to-text retrieval [31, 6]. Specifically, we rank the set of possible explanation embeddings, { exp i embed ..exp k embed } , by their L2-norm distance to s i . The decoding of the best ranked exp i embed to E i C i is then a vocabulary look-up. It is possible to utilize large language models (LLMs), and generate E i C i from exp i embed which we consider future work. Instead, in this work, we focus on validating the dual utility of embedding-based E i C i retrieval to both the RL agent as well as end-user.

4.2 Reward Shaping in RL via Joint Embedding Model M

During agent learning, we utilize M to retrieve the closest E i C i associated with a ( s i , a i ) for informing reward shaping. Prior work has shown that reward shaping can improve agent learning rate and sample efficiency [11, 25, 32]. However, designing an effective dense reward function is not trivial [16, 12], often requiring a two-stage trial-and-error procedure to 1) determine when a state-action pair deserves an intermediate reward, and 2) determine what reward value such state-action pair should be attributed. Many methods, such as inverse reinforcement learning, utilize learned, black box reward functions to minimize the trial-and-error process [37, 1, 57]. However, with black-box reward functions, we lose transparency of why a ( s i , a i ) is rewarded. To balance the design-transparency trade-off of dense reward functions, we leverage M to inform when a ( s i , a i ) pair should be rewarded based on the retrieved E i C i , and as a result, remove the first step of the trial-and-error process. Specifically, if M returns a meaningful E i C i for a ( s i , a i ) (e.g.,concept set C i = ∅ ), then such ( s i , a i ) represents concepts that influence the agent's goal, and should receive an intermediate reward.

glyph[negationslash]

Note, M does not remove the trial-and-error process needed to determine what values each reward component should receive. Instead, we perform a hyperparameter sweep to assign shaping values to each concept component when an expert-determined, continuous shaping function is not derivable. Thus, the 'Explanation-to-reward-shaping' module in S2E performs a look-up to assign a reward value based on the retrieved E i C i . In Section 5.3 we validate how leveraging M to inform reward shaping can improve an agent's learning rate comparable to expert-defined shaping functions.

4.3 Concept-Based Explanations to End Users via Joint Embedding Model M

Once an RL agent is trained and deployed, we leverage M to provide E i C i about an agent's decisionmaking to non AI expert end-users. Recall from Section 3 that E i C i can include multiple concepts (i.e. | C i | > 1 ). We posit that in some cases, E i C i with too many concepts may become ineffective in aiding end-user understanding. In fact, prior works discuss the challenge of presenting information at the 'right' level of abstraction to end-users [46, 53]. To the best of our knowledge, there are no explicit guidelines for what information abstractions are beneficial in sequential decision making settings. From analyzing prior end-user friendly explanation techniques [9, 14], we infer two important dimensions to consider for explanation abstraction in sequential decision making: Information Filtering (InF), and Temporal Grouping (TeG).

Information Filtering (InF) InF filters out concepts within E i C i to only include those that immediately influence the goal. For example, consider a E i C i in Lunar Lander, 'Fire left engine because it brings lander closer to the center, decreases lander velocity to avoid crashing, and decreases tilt'. Applying InF may produce a filtered explanation, 'Fire left engine because it brings lander closer to the center and decreases tilt' given a ( s i , a i ) when the lander needs to immediately correct its tilt and

position to avoid a trajectory path leading to a crash. While the other concepts are important, they are filtered out since they do not effect the immediate chance of meeting G .

To perform InF, we leverage domain knowledge and expert-defined thresholding to determine which concepts in C i for ( s i , a i ) critically contribute towards the agent's ability to reach goal G . Thresholds are determined via qualitative analysis of an RL agent's policy. In continuous domains, these threshold values are extracted by finding the values in the agent's state space that determine turning points in the agent's ability to reach goal G . For example, qualitative analysis may produce that an agent's pos x > 0 . 15 is a critical turning point for the agent's ability to land close to the center, and if pos x > 0 . 15 , the agent's action is crucially linked with decreasing its pos x . Thus, C i w/InF represents the concept set after thresholding is applied, in which concepts c ∈ C i that do not meet the determined thresholds are filtered out. As a result, an abstracted explanation, using InF, templates C i w/InF into a natural language explanation E i C i w/InF . In Appendix A, we provide more details on how concepts are filtered via qualitative analysis in InF and include an ablation study analyzing the sensitivity of chosen thresholds. Note, while we qualitatively determine thresholds, the InF method can be automated in future work.

Temporal Grouping (TeG) TeG automatically groups explanations over a sequence of time that represents a consecutive pattern. For example, if in Lunar Lander, E i C i and E i +2 C i is 'Fire left engine because it decreases tilt' and E i +1 C i and E i +3 C i is 'Fire right engine because it decreases tilt', applying TeG produces 'For the next 4 steps, alternate firing left and right engine to decrease tilt'.

To perform TeG, we analyze an agent's rollout at deployment to extract series of ( s [ i,i + n ] , a [ i,i + n ] ) with a repeated pattern of concept sets C [ i,i + n ] . Therefore, a grouped explanation provides a single E i C i w/TeG , that templates the repeated pattern within C [ i,i + n ] across the n state-action pairs. We posit that TeG is likely to be important in domains where actions are sampled at high frequency (e.g. Lunar Lander or Robotics), requiring an abstraction over actions to provide meaningful explanations for consecutively alike state-action pairs.

Overall, the 'Explanation Abstraction' component in S2E (see Fig. 1c) determines whether InF, TeG or both should be applied to E i C i for providing an abstracted, concept-based explanation. In Section 6 we demonstrate the effectiveness of InF and TeG in high frequency domains, such as Lunar Lander, for improved end user understanding.

5 Model Evaluations

In this section, we perform quantitative evaluations to validate the joint embedding model M in S2E by evaluating M 's explanation retrieval accuracy (Sec. 5.2), and M 's ability to inform reward shaping during agent training (Sec. 5.3).

5.1 Evaluation Domains

Connect 4 is an adversarial board game in which the objective is to achieve a four-in-a-row, in any direction. The state space is a discrete, 2D array representation of token positions. Also, the action

space is discrete and each player selects the column to place a token in. Lunar Lander is a trajectory optimization problem in which the lander must land on a landing pad. We utilize LunarLander-v2 from OpenAI Gym [4]. The state space is a continuous 1D array including the lander's linear position and velocity, angular velocity, angle, and leg contact. The game has four discrete actions: fire left engine, fire right engine, do nothing, and fire main engine.

Concept Collection Concepts for a domain may be extracted from expert players or commentators. However for Lunar Lander and Connect 4, there are no existing datasets for mining concepts. Thus, we leverage expert domain knowledge to attribute concepts to state-action pairs. Figure 2 lists the set of concepts we consider when attributing concepts to a ( s i , a i ) in Connect 4 and Lunar Lander, as well as provides example E i C i . Concepts for Connect 4 are derived from the strategies outlined in [3]. Note, "NULL" describes all other ( s i , a i ) that are not attributed to a concept. Alternatively, concepts for Lunar Lander are derived from the domain's existing reward function [4]. In this domain, the existing dense reward function includes the primary concepts important towards successful task completion. In Appendices B.1 and B.2, we provide an an expanded list of example E i C i , spanning all concepts derived for Connect 4 and Lunar Lander.

Note, for both domains, the concepts are defined via mathematical or logical representations of the state. For example, in Connect 4, the concept of 'BW', blocking an opponent win, can explicitly be encoded by board representations where any three-in-a-row pattern is blocked with a token from the opponent player. Similarly, in Lunar Lander, the 'POS', position concept, is modelled by mathematical representation of the agent's position over time. Deriving concepts via mathematical or logical representations allow us to automatically collect concepts from states, as well as use such mathematical or logical rules to evaluate that concepts and states are accurately paired. In many applications, it may be infeasible to derive mathematical or logical rules from a state representation. In these scenarios, concepts can be collected via crowd sourcing, [20], or obtained via 'think-aloud' procedures [14].

5.2 Evaluation of Joint Embedding Models

Below we demonstrate high recall rates of the joint embedding models in S2E in both the Connect 4 and Lunar Lander domain.

Datasets Recall from Section 4.1 that to train M we utilize a dataset D = { D a , D m } . For Connect 4, D C 4 includes approximately 3 million samples, and for Lunar Lander, D LL includes approximately 2 million samples. We randomly sample D a C 4 and D a LL from an expert RL policy. We obtain D m C 4 and D m LL , by selecting z incorrect concepts from C C 4 and C LL , as replacement for each sample in D a C 4 and D a LL (more detailed numbers in Appendix C).

Model Architectures & Training M C 4 and M LL denote the joint embedding models for Connect 4 and Lunar Lander, respectively. The model architectures for M C 4 and M LL leverage LSTMs to learn explanation embedding exp i embed for a given E i C i . For Connect 4, to learn s i embed for ( s i , a i ) , we leverage Convolutional Neural Networks to learn local and spatial relationships between tokens on a 2D game board. For Lunar Lander, we leverage Fully Connected Networks to learn s i embed . More details (split, learning rate, etc) are in Appendix C.

Results Similar to image-to-text retrieval [31, 6], we evaluate M C 4 and M LL via recall rate, at k = { 1 , 2 , 3 } , which evaluates whether the retrieved E i C i is ranked in the top k . Figure 3a provides the average recall rates of M C 4 and M LL , across 5 random seeds. We observe that M LL performs with near 100% accuracy, whereas M C 4 has an average recall rate of 88%. Given the lower recall rates by M C 4 , in Figure 3b we examine the false positive and false negative explanation retrievals. We see that M C 4 has greatest challenge in correctly retrieving E i C i

Avg.RecallRateM

100

99.9

95.4

100

97.7100

88.1

80

Rate

60

Recall

40

20

Connect4

0

R@1

R@2

R@3

(a)

67.5

0.1 0.0 0.7 0.7 1.1 0.0

0.029.9

{BW}

0.0100.00.00.00.00.0

0.0

0.00.0

{BW,CD}

0.1

0.0

98.50.00.0

0.0

0.7

0.40.3

{BW,3IR}

0.3

0.0

0.081.8

2.71.1

0.0

0.014.0

{3IR}

0.5

0.00.0 11.869.4

1.7

0.0

0.016.5

{3IR,CD}

9.6

0.0

0.031.8 16.8 24.70.00.017.1

{3IR_BL}

0.0

0.015.60.20.00.0

82.2

1.40.5

0.5

0.010.60.00.00.09.5

78.7

0.7

(CD}

1.4

0.0

0.01.4 0.7 0.1

0.0

0.0

96.4

{NULL}

{BW}

{BW.CD}

(BW,3IR})

[3IR}

[3IR,CD}

{3IR_BL}

{NULL}

(b)

with 'BW' and '3IR_BL'. We posit that 'BW' and '3IR_BL' may occur in board states with greater variation in comparison to other concepts, leading to higher incorrect retrievals. Additional analyses of M C 4 and M LL are in Appendix C.4.

Note, incorrect retrievals of E i C i for a given ( s i , a i ) can have a negative downstream impact within S2E. Specifically, when S2E is leveraged during the RL agent's training, incorrect retrievals of

(a)

(b)

(c)

Connect4

1.0

ercentage

0.8

0.6

Per

0.4

Win

0.2

MuZero + No-RS

MuZero + E-RS

0.0

MuZero + S2E-RS

0

100k

200k

300k

400k

Training Steps

1.0

Percentage

%0.8

M

0.6

0.4

Win

0.2

MuZero + E-RS

0.0

MuZero + S2E-RS

0

100k

200k

300k

400k

Training Steps

LunarLander

200

100

0

-100

Rev

-200

-300

MuZero + E-RS

-400

MuZero + S2E-RS

0

100k200k300k

400k

Training Steps

E i C i can incorrectly providing shaping rewards to the agent and in return impacting learned agent policy. Similarly, when S2E is leveraged at deployment to provide end-user understanding, incorrect retrievals of E i C i can confuse end-users and hinder their understanding of the agent's behavior. However, in Section 5.3 our results demonstrate that the percentage of incorrect retrievals from the joint embedding models do not significantly impact the RL agent's learned policy. Similarly, in Section 6, we demonstrate that the percentage of incorrect explanation retrievals are not significantly detrimental to user performance. Nevertheless, it is important that the joint embedding models within S2E have high recall rate, especially when applied in high-stakes or mission critical scenarios.

5.3 Evaluation of M for Reward Shaping in Agent Training

We validate that both joint embedding models in S2E, M C 4 and M LL , inform reward shaping comparable to expert-defined dense, reward functions.

RL Agent & Shaping Rewards Our S2E framework is model agnostic and M does not make any assumptions on the type of RL model utilized. Given our domains are complex games, we leverage a state-of-the-art (SoTA) RL algorithm MuZero [47] to evaluate M 's ability to inform reward shaping. We use the open-source version of MuZero available in [52] (details in Appendix D.1).

Recall from Section 4 that M C 4 and M LL inform when to provide shaping rewards, but the shaping values are expert determined. For Lunar Lander, there exists a SoTA dense, reward function [4]. In Connect 4, to our knowledge, there is no SoTA dense, reward function. Thus, we perform a hyperparameter sweep to assign shaping values for each concept (values provided in Appendix D.2).

Results To evaluate the efficacy of M C 4 and M LL in informing reward shaping, we measure the agents' Reward and Win% . In Lunar Lander, we evaluate M LL 's ability to inform reward shaping (MuZero+S2E-RS) in comparison to a baseline MuZero agent that is informed by an existing expert-defined reward shaping (MuZero+E-RS) . Figure 4b and Figure 4c demonstrate that both the MuZero+S2E-RS and MuZero+E-RS agents achieve comparable Reward and Win% . While performance is comparable between both agents, in MuZero+S2E-RS , M LL provides an automated method to determine when a reward value should be presented, which otherwise has to be manually encoded, such as in MuZero+E-RS via an expert-defined function.

In Connect 4, since there does not exist a SoTA reward shaping function, we compare M C 4 's ability to inform reward shaping (MuZero+S2E-RS) against a baseline MuZero agent with sparse rewards (MuZero+No-RS) . Figure 4a shows that MuZero+S2E-RS has improved learning rate and requires ∼ 200k less training samples to achieve similar Win% in comparison to MuZero+No-RS . We also evaluate MuZero+S2E-RS against an upper bound in which we provide expert-defined reward shaping (MuZero+E-RS) to analyze how incorrect retrievals from M C 4 may affect the agent's learning rate. Although M C 4 retrieves incorrect explanations 13.9% of times (see Appendix C.4), we observe that MuZero+S2E-RS 's learning rate is not significantly affected (Fig. 4a).

Overall, these results demonstrate the joint embedding models' ability to effectively inform reward shaping. In Connect 4, we demonstrate that, even with an imperfect joint embedding model, S2E can inform reward shaping and improve the agent's learning rate compared to the SoTA agent trained on sparse rewards. Similarly, in Lunar Lander, we demonstrate that our S2E informs reward shaping comparably to the existing SoTA agent that is trained with an expert-defined dense, reward function.

6 User Evaluation

In this section, we validate S2E with end-users, demonstrating that our retrieved E C i significantly improve user task performance in both Connect 4 and Lunar Lander.

Study Procedure Participants performed an online study in which they played several games from the domains in four stages. Specifically, 1) Practice included playing 2 practice games, 2) Pre-Test included playing 3 scored games, 3) Explanation included interacting with an expert player (well-trained RL agent) with exposure to explanations from an assigned study condition, if applicable and 4) Post-Test included playing 3 more scored games. Details about the the study procedures are in Appendix E.1.

Metrics We measure the difference between participant Pre-Test and Post-Test Adjusted Task Score (ATS) to analyze any task improvement with exposure to the study conditions. Participant ATS is defined by the expert-defined reward functions utilized by the RL agents during training (evaluated in Sec. 5.3, detailed in Appendix D.2). If R ( s, a ) denotes the reward associated with a given ( s, a ) and N defines the total actions taken by the user in a game, then ATS is defined as a normalized aggregation of user rewards received in a game: ATS = ∑ N n R ( s n , a n ) /N .

6.1 Connect 4 Study Specifics

Study Conditions We conducted a five-way between-subjects study with the following conditions. Participants could receive: 1) None (Baseline) : no information about the agent's action to be played, 2) E A (Baseline) : action-based explanation that contains the action to be played, 3) E V (Baseline) : value-based explanation stating the action to be played along with the action's value in comparison to the values of all other actions for a given state, 4) GT E C i : concept-based explanation using expert-defined, ground-truth concepts for a state-action pair, and 5) S2E E C i : a concept-based explanation from S2E for a state-action pair. Note, given that Connect 4 has a discrete state space, and actions are sampled at low frequency, the domain does not need further abstracted E C i via InF and TeG. Additionally, E V is similar to the action-cost condition in [48], as well as the Q-values presented per action in [27]. Note, precondition-based explanations in [48]

Connect4ATS

0.4

S2E8C

ATS

0.3

GT&Ci

EA

zed

0.2

&v

N

0.1

None

anda

0.0

*

**

**

-0.1

-0.2

PreTest

PostTest

require hierarchical domains, and causal-based explanations in [36] require a learned structured causal model which prevent direct comparison with our framework and domains.

Results The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=75, details in Appendix E.2). Our data is analyzed with a one-way ANOVA and a Tukey HSD post-hoc test, given that the assumptions of homoscedasticity (Levene's Test, p>0.3), and normality (Shapiro-Wilke, p>0.1) are met. Figure 5 shows a significant effect of explanation type on ATS (F(4, 70)=8.56, p<0.001). Specifically, we observe improvement in participant's ATS with exposure to our S2E E C i , in comparison to None (t(70) = 3.08, p<0.05), E A (t(70)=-3.25, p<0.05) and E V (t(70)=4.03, p<0.01). Additionally, we observe similar ATS improvement when exposed to S2E E C i in comparison to GT E C i . These results indicate the benefit of our S2E produced E C i in helping participants better understand Connect 4 and improve their ATS .

6.2 Lunar Lander Study Specifics

The underlying physics needed for Lunar Lander results in agent ( s i , a i ) being sampled at high frequency and multiple, repetitive concepts being attributed to ( s i , a i ) pairs. For example, 'decrease velocity" is a valid concept for every ( s i , a i ) until the end of a game. Recall from Section 4.3 that E C i may need to be further abstracted in these scenarios to provide meaningful explanations to end-users. Therefore, to evaluate the utility of abstracted conceptbased explanations, we introduce 3 more study conditions that utilize our InF and TeG methods outlined in Section 4.3.

Study Conditions We perform a seven-way between-subjects user study. Similar to Connect 4, we include None , E A , E V , and S2E E C i . Unique to our Lunar Lander study participants

LunarLanderATS

0.30

S2E &C;w/InF, TeG

0.25

A

0.20

S2E &C;w/InF

S2E EC;w/TeG

zed

0.15

S2E8C

N

0.10

EA

0.05

Ev

**

*

*

n

0.00

None

记

-0.05

S

-0.10

-0.15

PreTest

PostTest

could receive 1) S2E E C i w/ TeG : a concept-based explanation from S2E with additional abstractions performed using the TeG method, 2) S2E E C i w/ InF : a concept-based explanation from S2E with additional abstractions performed using the InF method, and 3) S2E E C i w/ InF, TeG : a concept-based explanation from S2E with additional abstractions performed using both TeG and InF.

Results The following results are from an IRB-approved study with participants recruited via Amazon Mechanical Turk (n=105, details in Appendix E.2). Our data is analyzed with a one-way ANOVA and a Tukey HSD post-hoc test, given that the assumptions of homoscedasticity (Levene's Test, p>0.7), and normality (Shapiro-Wilke, p>0.1) are met. Figure 6 shows a significant effect of explanation type on participant ATS (F(6,98)=3.67; p<0.01). Specifically, we see significant improvement in participant's ATS with exposure to S2E E C i w/ InF, TeG, in comparison to None (t(98) = 3.15, p <0.05)), E A (t(98)=-3.35, p<0.05), E V (t(98)=3.59, p<0.01) and E C i (t(98)=-3.19, p<0.05). These results demonstrate the need of our S2E abstraction methods when providing concept-based explanations to end-users, and that usage of both InF and TeG in high-frequency RL domains are crucial for ATS improvement.

7 Discussion and Conclusions

Our work introduces a unified framework, S2E, that involves learning a joint embedding model between agent state-action pairs and concept-based explanations to provide a dual benefit to the agent and end-user. We additionally outline a desiderata for what may constitute as a 'concept' in sequential decision making problems, beyond the scope of prior concept-based explanations for sequential decision making. Our model evaluations demonstrate that the joint embedding model in S2E provides an automatic method for determining when reward shaping should be provided to improve agent learning rates. Our user evaluations demonstrate that concept-based explanations can significantly improve user task performance (Connect 4), but when considering high-frequency RL domains (Lunar Lander), the additional abstraction methods from S2E are important for producing abstracted concept-based explanations that significantly improve user task performance.

Limitations: We present several areas of future work that aims to improve the generalizability of S2E. For instance, the concept-based explanations in our work are derived from mathematical representations and expert knowledge in each domain (see Sec. 5.1). To expand the generalizability of the S2E method to other complex domains, such as Robotics or open-world games, future work should explore how to collect and extract concepts in scenarios where mathematical representations of concepts may be infeasible. A future direction may include collecting expert commentary [20] or think-aloud [14] sessions from which concepts are automatically derived. Additionally, in many real-world applications, having access to large amounts of data for a domain may be infeasible. Thus, future work should also explore how to adapt the joint embedding model within S2E for few shot learning. Additionally, for application to high-stakes domain, future work should explore how to remediate incorrect explanation retrievals from the joint embedding model within S2E. While the small percentage of incorrect explanation retrievals do not significantly impact user task performance and agent learning in our tested domains, inaccurate explanation retrievals may have significant effects in mission-critical tasks. It is also important to highlight that our current concept-based explanations follow a fixed template for each unique concept set (see Appendices B.1, B.2). However, there are multiple ways of explaining the same concept, and future work includes learning a joint embedding model with greater language variety in the explanations combined with automatically generating such templates using language models. Furthermore, the InF method within S2E utilizes manually defined thresholds to produce abstracted concept-based explanations, and future work entails developing an automated InF method.

Broader Impacts: S2E shows promise in applying the Protégé Effect to Human-AI Interaction, and that explanations of agent behavior are beneficial to both agents and users. These insights may promote AI agent architects to consider the utility of self-explaining agents in accelerating learning as well as providing transparency. Additionally, our work shows promise in using conceptbased explanations of well-trained AI agents as a teaching tool, helping users improve their task performance. A potential risk of S2E is purposely training a joint embedding model to associate misleading concept-based explanations with state-action pairs. This could lead to a badly performing agent, and the agent providing deceitful explanations of its actions. Future work in explaining black-box AI systems should explore how to detect and prevent deceptive explanations.

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access , 6:52138-52160, 2018.

Louis Victor Allis. A knowledge-based approach of connect-four. J. Int. Comput. Games Assoc. , 11(4):165, 1988.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Gerard Canal, Senka Krivic, Paul Luff, and Andrew Coles. Task plan verbalizations with causal justifications. In ICAPS 2021 Workshop on Explainable AI Planning (XAIP) , 2021.

Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.14713 , 2022.

Peter A Cohen, James A Kulik, and Chen-Lin C Kulik. Educational outcomes of tutoring: A meta-analysis of findings. American educational research journal , 19(2):237-248, 1982.

Larissa Custodio and Ricardo Machado. Flexible automated warehouse: a literature review and an innovative framework. The International Journal of Advanced Manufacturing Technology , 106:533-558, 2020.

Devleena Das and Sonia Chernova. Semantic-based explainable ai: Leveraging semantic scene graphs and pairwise ranking to explain robot failures. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3034-3041. IEEE, 2021.

Devleena Das, Been Kim, and Sonia Chernova. Subgoal-based explanations for unreliable intelligent decision support systems. In Proceedings of the 28th International Conference on Intelligent User Interfaces , pages 240-250, 2023.

Sam Devlin, Daniel Kudenko, and Marek Grze´ s. An empirical study of potential-based reward shaping and advice in complex, multi-agent systems. Advances in Complex Systems , 14(02): 251-278, 2011.

Daniel Dewey. Reinforcement learning and the reward engineering principle. In 2014 AAAI Spring Symposium Series , 2014.

David Dunning. The dunning-kruger effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology , volume 44, pages 247-296. Elsevier, 2011.

Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. Automated rationale generation: a technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces , pages 263-274, 2019.

Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O Riedl. Operationalizing human-centered perspectives in explainable ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-6, 2021.

Jonas Eschmann. Reward function design in reinforcement learning. Reinforcement Learning Algorithms: Analysis and Applications , pages 25-33, 2021.

Logan Fiorella and Richard E Mayer. The relative benefits of learning by teaching and teaching expectancy. Contemporary Educational Psychology , 38(4):281-288, 2013.

Eva Fraedrich and Barbara Lenz. Taking a drive, hitching a ride: autonomous driving and car usage. Autonomous driving: Technical, legal and social aspects , pages 665-685, 2016.

Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic conceptbased explanations. Advances in Neural Information Processing Systems , 32, 2019.

Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 2385-2391, 2019.

Marek Grzes. Reward shaping in episodic reinforcement learning. 2017.

Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 2017 ACM/IEEE international conference on humanrobot interaction , pages 303-312, 2017.

Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. Explainability in deep reinforcement learning. Knowledge-Based Systems , 214:106685, 2021.

Vincent Hoogerheide, Lian Deijkers, Sofie MM Loyens, Anita Heijltjes, and Tamara van Gog. Gaining from explaining: Learning improves from explaining to fictitious others on video, not from writing to them. Contemporary Educational Psychology , 44:95-106, 2016.

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems , 33:15931-15941, 2020.

Ying Ji, Yu Wang, and Jien Kato. Spatial-temporal concept based explanation of 3d convnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15444-15453, 2023.

Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence , 2019.

Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 2526-2538, 2021.

Eoin M Kenny, Mycal Tucker, and Julie Shah. Towards interpretable deep reinforcement learning with human-friendly prototypes. In The Eleventh International Conference on Learning Representations , 2022.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 , 2014.

WBradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS , volume 1004, pages 475-482. Valencia, 2012.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

Iro Laina, Christian Rupprecht, and Nassir Navab. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7414-7424, 2019.

Adam Daniel Laud. Theory and application of reward shaping in reinforcement learning . University of Illinois at Urbana-Champaign, 2004.

Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 2493-2500, 2020.

Alberto Maria Metelli, Matteo Pirotta, and Marcello Restelli. Compatible reward inverse reinforcement learning. Advances in neural information processing systems , 30, 2017.

Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems , 34:29529-29540, 2021.

Stefania Montani and Manuel Striani. Artificial intelligence in clinical decision support: a focused literature survey. Yearbook of medical informatics , 28(01):120-127, 2019.

Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE International Conference on Robotic Computing (IRC) , pages 590-595. IEEE, 2019.

Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning , pages 188-204. PMLR, 2021.

Paul J Pritz, Liang Ma, and Kin K Leung. Jointly-learned state-action embedding for efficient reinforcement learning. In Proceedings of the 30th ACM International Conference on Information &Knowledge Management , pages 1447-1456, 2021.

Saed Rezayi, Handong Zhao, Sungchul Kim, Ryan A Rossi, Nedim Lipka, and Sheng Li. Edge: Enriching knowledge graph embeddings with external text. arXiv preprint arXiv:2104.04909 , 2021.

Stuart Russell and Peter Norvig. Ai a modern approach. Learning , 2(3):4, 2005.

Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable AI: interpreting, explaining and visualizing deep learning , volume 11700. Springer Nature, 2019.

Lindsay Sanneman and Julie A Shah. A situation awareness-based framework for design and evaluation of explainable ai. In Explainable, Transparent Autonomous Agents and Multi-Agent Systems: Second International Workshop, EXTRAAMAS 2020, Auckland, New Zealand, May 9-13, 2020, Revised Selected Papers 2 , pages 94-110. Springer, 2020.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, and Subbarao Kambhampati. Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations. arXiv preprint arXiv:2002.01080 , 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Nicholas Waytowich, Sean L Barton, Vernon Lawhern, and Garrett Warnell. A narrationbased reward shaping approach using grounded natural language commands. arXiv preprint arXiv:1911.00497 , 2019.

Lindsay Wells and Tomasz Bednarz. Explainable ai and reinforcement learning-a systematic review of current approaches and trends. Frontiers in artificial intelligence , 4:550030, 2021.

Aurèle Hainaut Werner Duvaud. Muzero general: Open reimplementation of muzero. https: //github.com/werner-duvaud/muzero-general , 2019.

Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfister. On concept-based explanations in deep neural networks. 2019.

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, et al. Concept embedding models. In NeurIPS 2022-36th Conference on Neural Information Processing Systems , 2022.

Luowei Zhou, Chenliang Xu, Parker Koch, and Jason J Corso. Watch what you just said: Image captioning with text-conditional attention. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017 , pages 305-313, 2017.

Kai Zhu and Tao Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology , 26(5):674-691, 2021.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Information Filtering (InF) Details

Recall from Section 4.3 that we utilize expert-defined thresholding and domain knowledge to determine which concepts within C i for a ( s i , a i ) critically contribute towards the agent's ability to reach goal G . Specifically, these thresholds are expert-defined upper and lower bounds on the agent's state values that denote the agent's ability to succeed or fail in its goal. In our InF method, these thresholds are not mathematically derived, but are derived from RL-expert analysis. For a given domain, an RL expert visualizes multiple policy rollouts and analyze the different state values over time to manually determine the upper and lower bounds (turning points) that influence the agent's ability to reach G. Below we demonstrate how thresholds in InF for one of our evaluation domains, Lunar Lander.

The concepts relevant to Lunar Lander include, position, velocity, tilt, side and main fuel, right and left leg contacts, and landing, and these concepts are derived from the existing, expert-defined, dense reward function [4] (see Section 5.1). Within the dense, reward function, binary concept components are rewarded via constant shaping values, while the continuous concepts are rewarded through a continuous shaping function (see Appendix D.2). In our work, we consider applying thresholding to concepts defined via a continuous function since these scenarios typically result in multiple concepts being rewarded over multiple time steps, and are more likely to warrant information filtering.

In the context of Lunar Lander, Figure 7 and Figure 8 show the two concepts on which thresholding is applied when examining a well trained agent's policy, specifically the lander's x-position and tilt. Recall that thresholds for InF can be determined by examining an agent's state values to identify turning points in the agent's ability to reach G . In Figure 7, we extract that a suitable threshold for the x-position concept is 0.15. We identify that such x-position value corresponds to the time step at which the agent starts firing its left engine to critically correct its increasing position value. Similarly, in Figure 8, we extract that suitable thresholds for the tilt concept are 0.01 and -0.05. We find that such tilt values correspond with frequently alternating firing of left and right engine to critically correct an increasing tilt value. We do not perform a thresholding analysis on velocity, as with domain analysis we observe that decreasing the agent's velocity is directly associated with firing the main engine and is performed to prevent crashing. Additionally, we do not consider the agent's y-position for thresholding as the agent follows a steady, downward linear trajectory, and visible turning points cannot be extracted by qualitative analysis. While we qualitatively determine thresholds, the InF method can be automated in future work.

Overall, in our InF method, if C i denotes the original concept set for a given ( s i , a i ) , then C i w/InF represents the concept set after thresholding is applied, in which concepts c ∈ C i that do not meet the threshold are filtered out. As a result, an abstracted explanation, using InF, templates Ct i into a natural language explanation E i C i w/InF .

A.1 Thresholding Sensitivity Analysis

As stated above, the thresholds are manually derived from RL-expert analysis. Specifically, an RL expert visualizes multiple policy rollouts and analyzes the different state values over time to determine approximate upper and lower bounds (turning points) that influence the agent's ability to reach G . Given the thresholding approach in InF, different threshold values can result in filtering of different concepts and therefore produce different abstracted, concept-based explanations. To study the sensitivity of our chosen thresholds in Lunar Lander, we analyze how different threshold values for each concept affect the number of concepts filtered in a policy rollout.

Figure 9 demonstrate what fraction of concepts are filtered (y-axis) as the threshold values change (x-axis). When looking at Figure 9a, analyzing threshold values for the x-position concept, we see our chosen value is within the elbow of the curve, denoting that the rate of filtration of the concept slows down after 0.15. Similarly, when analyzing the threshold values for the tilt concept in Figure 9b and Figure 9c, we see the lower and upper bound of the tilt thresholds are also within the 'elbow' of each curve. Note, when analyzing the upper bound threshold for the tilt concept (Figure 9b), the lower bound tilt threshold is fixed. Similarly, when analyzing the lower bound threshold for the tilt concept (Figure 9c), the upper bound tilt threshold is fixed. Overall, our threshold values being within the elbow of each curve provide some validation on their soundness. However, to study the complete impact of varying thresholding values for each concept, additional user studies with abstracted concept-based explanations produced from varying threshold values are required. We

Thresholdldentificationforx-position

Agent Position

0.05

0.00

-0.05

-0.10

-0.15

position

-0.20

threshold

3.0

Action

2.0

0 = Do Nothing

1.0

1 = Fire Left

2 = Fire Main

0.0

3 = Fire Right

0

100

200

300

400

Time

Thresholdldentificationfortilt

0.04

0.02

0.00

-0.02

-0.04

tilt angle

-0.06

threshold

-0.08

3.0

Action

2.0

0 = Do Nothing

1.0

1 = Fire Left

2 = Fire Main

0.0

3 = Fire Right

0

100

200

300

400

Time

(a)

(b)

(c)

X-PositionThresholdExperimentation

O.25

0.0

0.1

0.2

0.3

0.4

0.5

0.6

X-PositionThresholdValue

UpperBoundTilt Threshold Experimentationw/-0.05lowerbound

LowerBound Tilt Threshold Experimentationw/ 0.01 upper bound

0.175

filtered

0.150

0.100

0.10

0.075-

0.050

0.05

0.025

0.00

0.000

0.0

0.1

0.2

0.3

0.4

0.0

-0.1

0.2

0.3

-0.4

Tilt(UpperBound)Threshold Value

Tilt(LowerBound)Threshold Value

consider such analyses beyond the scope of our work, and one to further explore when considering future methods on automating the information filtering submodule within our S2E framework.

B Concept-Based Explanations for Experimental Domains

Below we present an exhaustive list of the different E i C i , utilized for each domain, paired with an instance of a ( s i , a i ) pair for context. Note, the rows denote the possible concept sets C i , and their corresponding fixed, templated explanations starting with 'because'. The action a i is always appended to the beginning of the explanation.

B.1 Connect 4

(

s

i

, a

i

)

C

i

Corresponding

E

i

C

i

{

3IR

}

"Play column 4 because it creates a three-in-a-

row."

{

3IR_BL

}

"Play column 7 as a neutral move, it creates a three-

in-a-row that is blocked by the opponent from a

win."

{

3IR, CD

}

"Play column 4 because it provides center domi-

nance and creates a three-in-a-row."

{

BW

}

"Play column 6 because it blocks the opponent

from a win"

{

BW, 3IR

}

"Play column 5 because it creates a three-in-a-row

and blocks the opponent from a win."

{

BW, CD

}

"Play column 4 because it provides center domi-

nance and blocks the opponent from a win."

{

CD

}

"Play column 4 because it provides center domi-

nance"

{

W

}

"Play column 7 because it leads to a four-in-a-row

win"

{

NULL

}

"Play column 6 as a generic move not tied to a

particular strategy"

.·

B.2 Lunar Lander

{

POS,

VEL,

TILT,

LLEG,

RLEG

}

"Do nothing because it moves lander closer to the center, decreases the lander speed to avoid crashing, decreases the tilt of the lander, encourages left and right leg contact."

{

L

}

"Do nothing because it results in a land."

C Joint Embedding Model Details

C.1 Dataset Details

MisAligned and Aligned Data Breakdown Recall from Section 5 that D C 4 represents the dataset used for training and evaluating M in Connect 4. Specifically, D C 4 includes 2 , 948 , 742 total samples, in which D a C 4 = 327 , 638 aligned samples and D m C 4 = 2 , 621 , 104 misaligned samples. Similarly, D LL represents the dataset used for training and evaluating M in Lunar Lander. Specifically, D LL includes 1 , 941 , 042 total samples, in which D a LL = 323 , 507 aligned samples and D m LL = 1 , 617 , 535 misaligned samples. Recall that to create the misaligned dataset, D m , we utilize z incorrect concept sets available in each domain as replacement for the existing correct concepts utilized in E i C i for each d i ∈ D a . Specifically, for Connect 4, z = 8 and we leverage all possible non-aligned concept sets for perturbation. In Lunar Lander, z = 5 and we randomly choose five non-aligned concept sets for perturbation. We randomly choose 5 of 13 possible concept sets to generate misaligned sample pairs per state to avoid a larger than necessary dataset. As a refresher, the list of possible concept sets for each domain are presented in Figure 2.

Concept Breakdown in Aligned Data Table 3 and Table 4 provide a breakdown of the occurrences of each concept set within the training data for the joint embedding models, M C 4 (Connect 4) and M LL (Lunar Lander). We specifically show the breakdown in the aligned datasets, D a C 4 and D a LL , given that such data is the raw data collected via rollout simulations in each domain. Recall that the additional misaligned data in the training set, D m C 4 and D m LL , are synthetically generated via perturbations, and used for the contrastive learning. When the misaligned data is included in the training set, given the perturbations, the number of occurrences of each concept (aligned or misaligned) become equal.

C.2 Model Architectures

Figure 10 shows the model architecture for M C 4 . Recall that the inputs to M C 4 includes 〈 [ s i , s i -1 ] , g i , E i C i , y i 〉 . In the diagram, we don't explicitly show y i but it is used to denote whether

s i and E i C i are aligned or misaligned (see Section 4). The game boards ( s i and s i -1 ) in Connect 4 are represented as 6x7 2D array which are passed into two consecutive CNNs after padding to 7x7 2D arrays. The first set of CNNs (CNN1 and CNN3) for s i and s i -1 have the following parameters: input_channels=1, output_channels=4, kernel_size= (3,3), stride=1. The second set of CNNs (CNN2 and CNN4) have the following parameters: input_channels=4, output_channels=6, kernel_size= (3,3), stride=1. The embedding outputs of CNN2 and CNN4 are then concatenated with the game information g i , which includes boolean representations of the player/opponent as well as whether the game is over. The concatenated embedding is then passed through three FCNs. Specifically, FCN1's output dimension is 64. FCN2's output dimension is 32. FCN3's output dimension is 16. The output of FCN3 represents our final state embedding, s i embed . The concept-based explanation E i C i is first passed through an embedding layer and then passed to a LSTM network to produce our exp i embed . The input size to the LSTM is of dimension 32, and output dimension is 16.

Figure 11 shows the model architecture for M LL . The inputs to M LL includes 〈 [ s i , s i -1 ] , g i , E i C i , y i 〉 . In the diagram, we don't explicitly show y i , but it is used to denote whether s i and E i C i are aligned or misaligned (see Section 4). The game boards ( s i and s i -1 ) in Lunar Lander are represented as 1x10 arrays which are passed into two consecutive FCNs. Specifically, the game board representation includes the 1x8 game state representation from OpenAI Gym [4], along with an inclusion of whether the side fuel or main fuel is in use. The first set of FCNs (FCN1 and FCN3) for s i and s i -1 have an output dimension of 64. The second set of FCNs (FCN2 and FCN4) have an output dimension of 32. The embedding outputs of FCN2 and FCN4 are then concatenated with the game information g i , which includes a boolean representation of whether the game is over. The concatenated embedding is then passed through three more FCNs Specifically, FCN5 has an output dimension of 16. FCN6 has an output dimension of 8. Similar to the architecture for M C 4 , the concept-based explanation E i C i is first passed through an embedding layer and then passed to a LSTM network to produce our exp i embed . The input size to the LSTM is of dimension 32, and output dimension is 8.

C.3 Training Details

To train and evaluate M C 4 and M LL , we utilize a 60%-20%-20% train-valid-test split on D C 4 and D LL (see total dataset size in Appendix C.1). The models are trained with learning rate of 0.001, batch size of 128, Adam Optimizer, and trained with 10 epochs. To train M C 4 and M LL , we utilize a desktop computer with a NVIDIA GTX 1060 6GB GPU and an Intel i7 processor. The runtime for training and testing M C 4 and M LL takes approximately 30 minutes. See link to code in Appendix F.

C.4 Additional Evaluations

Learned Embedding Spaces of M C 4 and M LL In Figure 12a and Figure 12b, we provide a visualization of our learned joint embedding models, M C 4 and M LL . The TSN-E visualizations is created using the models' training dataset. We do not draw any conclusions via these visualizations, but present them as an intuitive way to interpret our high dimensional, learned embedding spaces.

ReLU

ReLU

ReLU

ReLU

Si-1

CNN1

CNN2

FCN1

FCN2

FCN3

Concat

ReLU

ReLU

!s

CNN4

Embedding

expembed

LSTM

GameInfo

Player [1,0]

!6

GameOver[1,0]

"Provides center dominance and

createsa three-in-a-row."

ReLU

ReLU

ReLU

Si-1

FCN1

FCN2

FCN6

Concat

ReLU

ReLU

!s

FCN3

FCN4

Embedding

expembed

LSTM

Game Info

Game Over[1,0]

!6

"Moves lander closerto the

center,decreases landerspeed

to avoid crashing, decreases tilt

of lander, and conservesmain

fuel usage"

(a)

(b)

{ NULL }

{ POS, VEL, TILT, MF }

{ CD}

{ POS, VEL, TILT, SF }

{ POS, VEL, TILT }

3IR }

{ POS, VEL, TILT, LLEG, SF}

{BW}

{ POS, VEL, TILT, LLEG}

{ 3IR_BL }

{ POS, VEL, TILT, RLEG, SF}

{ 3IR, CD }

{ POS, VEL, TILT, LLEG, RLEG, SF }

{BW, CD}

{ POS, VEL, TILT, LLEG, MF }

-20

{BW, 3IR }

BC-

{ POS, VEL, TILT, RLEG }

{ POS, VEL, TILT, RLEG, MF }

-40

{ POS, VEL, TILT, LLEG, RLEG, MF }

{ POS, VEL, TILT, LLEG, RLEG}

{ L}

49

29

In the 'Wild' Evaluation of M C 4 and M LL We performed an evaluation of how well M C 4 and M LL continue to perform when utilized during an RL agent's training to inform reward shaping. Specifically, we analyzed the model's Recall@1, examining how often E i C i was incorrectly retrieved for a given ( s i , a i ) . For Lunar Lander, M LL 's average Recall@1, across the 5 seeds, during the

73.2

0.01.50.60.00.40.00.024.2

{BW}

0.0

88.6

0.00.04.6

50.00.06.60.1

80

{BW,CD}

7.2

0.051.118.10.015.30.00.08.3

{BW,3IR}

0.1

0.0

01.090.40.01.40.00.0

7.0

60

{3IR}

0.0

1.0

0.00.193.1

0.00.05.5

50.2

{3IR,CD}

0.4

0.0

2.3

88.70.080.00.00.0

8.6

40

{3IR_BL}

0.0

0.0

0.0

0.0

00.00.199.90.00.1

0.0

0.1

0.0

0.0

0.1

0.00.099.7

0.0

20

{CD}

0.6

0.0

0.1

0.7

0.0

0.2

0.0

0.098.5

{NULL}

{BW}

{3IR}

{}

{BW.CD}

{BW,3IR}

{3IR_BL}

{NULL}

agent's training continued to be 100%. Figure 13 provides an analysis of the Connect 4 joint embedding model's, M C 4 , average Recall@1. We see on average, M C 4 has an average Recall@1 of 86.1%, with 13.9% incorrect retrievals. As mentioned in Section 5.3 results, these incorrect retrievals do not significantly affect M C 4 's ability to inform reward shaping and the agent's learning rate.

D RL Agent Details

D.1 RL Model Training

We utilized the open-source version of MuZero from [52], and utilize the provided hyperparameters. Please see Muzero's Github Repo from [52] 1 , as well as our code linked in Appendix F, for more details. To train and validate our MuZero agents we leverage an NVIDIA GTX 1060 6GB GPU and an Intel i7 processor. The runtime for training a Connect 4 MuZero agent to around 400k training steps takes approximately 30 hours. The runtime for training a Lunar Lander MuZero agent to around 400k training steps takes approximately 9 hours.

D.2 Shaping Rewards Per Concepts

For Lunar Lander, we utilize the existing dense reward function to determine the relevant concepts as well as their shaping values. Therefore each c ∈ C LL has an expert-defined shaping value. The existing shaping values can be found in the LunarLander.py provided by OpenAI Gym [4]. Table 5 summarizes the continuous functions outlined in [4] that we use to reward each concept c ∈ C LL . Alternatively, there are no existing SoTA reward shaping values for the game of Connect 4. Therefore we perform a hyperparameter sweep to determine an optimal set of shaping values, and Table 6 summarizes our utilized shaping values for each concept c ∈ C C 4 .

E User Study Details

E.1 Procedure Details

Participants performed an online study in which they were asked to play several games from the domain. The user studies for each domain were was broken into the following four stages: Practice , Pre-Test , Explanation , Post-Test . First, prior to the Practice stage, participants were introduced to the game including background information, game rules, and their task. Then in the Practice , participants played 2 practice games to get further acquainted with the specified domain. In Pre-Test , participants played 3 more games, this time being aware that their games were being scored. In Explanation , participants interacted with an expert player (well-trained RL agent) and were exposed to explanations about the player's actions via their assigned study condition. Note, participants in the 'None' study condition did not receive any accompanying explanations during the Explanation stage. Specific to Connect 4, this stage consisted of observing the expert player (RL agent) play 3 games. In Lunar Lander, the Explanation stage consisted of observing the expert player (RL agent) play 1 game. In the Post-Test , participants played 3 more scored games, similar to the Pre-Test . After the Post-Test stage, participants completed a post-questionnaire survey that collected demographics data as well as their perceived task performance and their perceived utility of their explanation condition.

Some specifics unique to each domain include that for Connect 4, participants always played first, and their opponent player was a well-trained MuZero agent. Additionally, for Lunar Lander, to reduce the impact of learning effects, the starting position and initial force applied to the lander was randomized across all 8 games; however, the random combinations remained fixed across participants. Also, participants played the version of Lunar Lander implemented by OpenAI Gym [4].

E.2 Participant Information

Connect 4 We recruited 84 participants from Amazon Mechanical Turk, for an IRB-approved study and participants were required to be novice players of Connect 4. Of the 84 participants, 9 were filtered out for not finishing the study. The remaining 75 participants, 15 per study condition, included 40 males and 35 females, all over the age of 18 (M=31.57, SD=7.12). The study took on average 20 minutes, and participants were compensated $5.00.

Lunar Lander We recruited 118 participants from Amazon Mechanical Turk, for an IRB-approved study. Participants were required to be novice players of Lunar Lander. Of the 118 participants, 13 were filtered out for not finishing the study or demonstrating no effort. The remaining 105 participants, 15 per study condition, included 50 males and 55 females, all over the age of 18 (M=30.57, SD=6.75). The study took on average 20 minutes, and participants were compensated $5.00.

E.3 Additional Evaluation Details

Participant ATS Note, the adjusted task score (ATS) adjusts participant's Post-Test average task performance by their Pre-Test average task performance. Note, this adjustment does not change any of the results or statistical analyses and is purely for visualization purposes. Specifically, we perform the adjustment to visualize the relative changes among participant ATS using a common starting point.

Perceived Performance

None

Ev

EA

ECi

ECi w/ TeG

ECi w/ InF

ECi w/ TeG,InF

-40

-20

0

20

40

60

80

100

120

■Disagree

■StronglyDisagree

Neutral

Agree

■StronglyAgree

Perceived Explanation Utility

Ev

EA

ECi

ECi w/ TeG

ECi w/ InF

ECi w/ TeG,InF

-40

-20

0

20

40

60

80

100

120

Disagree

■StronglyDisagree

Neutral

Agree

StronglyAgree

Additionally, to keep the ATS graphs legible, we present the mean and standard deviations for each study condition here, as opposed to including error bar lines on the graphs. The following are the mean and standard deviations in the Connect 4 user study conditions. Specifically, (1) None (Baseline) : M=-0.039, SD=0.222, (2) E A (Baseline) : M=-0.041, SD=0.114 (3) E V (Baseline) : M=-0.104, SD= 0.346 (4) GT E C i : M=0.256, SD=0.318 (5) S2E E C i : M=0.215, SD=0.194. The following are the mean and standard deviations in the Lunar Lander user study conditions. Specifically (1) None (Baseline) : M=-0.051, SD=0.191, (2) E A (Baseline) : M=-0.049, SD=0.136 (3) E V (Baseline) : M=-0.067, SD=0.211 (4) S2E E C i : M=-0.043, SD=0.192, (5) S2E E C i w/ TeG : M=0.100, SD=0.154 (6) S2E E C i w/ InF : M=0.034, SD=0.190, (7) S2E E C i w/ InF, TeG : M=0.156, SD=0.182.

User Perception Analyses Below we present additional evaluations performed on user responses in our questionnaire. Specifically, we present participant's perceived performance Perf and participant's perceived explanation utility EU . Note, Perf is measured on a 5-point Likert scale to the following question 'I believe my game performance improved after witnessing the expert play and seeing the expert's provided reasoning'. The EU metric is measured on a 5-point Likert scale to the following question 'I believe the expert's provided reasoning helped improve my understanding of the game', and is only measured for the explanation conditions. From both Figure 14 and Figure 15 we observe that participants across all study conditions had high agrees for Perf and EU , despite their being significant differences in their objective task performance. These qualitative results demonstrate the Dunning-Krunger Effect [13], in that users with low expertise in an area often overestimate their own performance or knowledge.

(a)

(b)

Introduction to Connect 4

In this study. you wil be playing the game of Connect 4. The image below shows what the game environment looks like. Please use a desktop or

laptop computer for this study.

In this study youwl be play the game oflarlander The mgebelwshws what th gme envroment lks kehe landerhas diffeent

Xpns sjun soj Janduos doydel o dopysap e asn aseald soufua apis ow pue *aujfiua upw e saufua

play.

The primary objective of the game is to achieve a four in a row with your colored tokens before the opponent. You willalways be the first player to

The primary objective o the game is to get the lander to land between the fag poles, as cose to the center as possble, without crashing. Additional

objetivse alty to land thlanewthut aanet an consee asmhflspssileh two tesf engins sidema e

their own fuel reserves.

To place a token in a particular column, specifcy the column number as depicted in the picture above. For example, the fourth column from the right is

Actions:

denoted as “C4".

page. The four possible actions are:

Actions: to move the lander, you can play one

- Fire Main Engine

· Fire Right Engine

· Do Nothing

Your Task

We will be examining how different forms of infomation help in improving your understanding of the game as well as your task performance. Your task

will consist of the following:

• watch 1 expert game to get acquainted to the game

Your Task

• play 3 assessment games that will measure your ability to achieve 4-in-a-row before your opponent

· play 2 practice games to get acquainted to the game

We wll be examining how diferent foms of infle

will consist of the following:

· play 2 practice games to get acquainted to the game

watch 1 expert game to get acquainted to the game

• play 3 more assessment games that wil measure your ability to achieve 4-in-a-row before your opponent

•play 3 assessment games that will measure your ablity to land the lander

· short survey related to your experience and background

• 3 interactive sesslons watching an expert player playing the game

• play 3 more assessment games that will measure your ability to land the lander

Continue Below to Enter TURK ID

short survey related to your experience and background

Continue Bclow to Enter TURK ID

(a)

(b)

Click Through an Example Game

Click the Next button to observe an example game Connect 4.

Recall,the objective of the game is to place your colored tke

some examples of how to achieve a four-in-a-row.

Watch an Example Game

Click the play video button to watch an expert play the game of lunar lander.

Recall, the objective of the gar

Continue to watch example

ueto e

Watch Example Game

Click Through Example Game

Prev Action

Next Action

E.4 Visuals of User Study

In this section, we provide visuals of the user interface for each user study.

Introduction Stage In both user studies, after receiving consent of participation, all participants were given an introduction to both the user study as well as the specific domain of the study. Figure 16a(a)and Figure 16b(b) show the introductions participants received for Connect 4, while Figure 17a(a) and Figure 17b(b) show the introductions participants received for Lunar Lander. Note the

(a)

(b)

PracticeGame1of2

PracticeGame2of2

C1C2C3C4C5C6C7

C1

C2

CS

90

Your Turn

Do Nothing

Fire Main Engine

Fire Left Engine

Fire Right Engine

(a)

(b)

InteractwithanexpertplayingaStartGamescenario

Interactwithanexpertplayinggame1

37/107

Explanation

Prev Action

Next Action

Explanation

Prev Action

Next Action

Fire Left Engine because it

Play C3 because it creates a three-in-a-row

· reduces agent's tilt while bringing agent closer to the center

introduction to the task was first presented to the users, and then the introduction to the domain was presented.

Game-Playing Stages During the study, participants played various games during the Practice , Pre-Test and Post-Test stages. These games were in real-time; users clicked the action to play via a button, and saw the game interface update accordingly. Figure 18a(a) and Figure 18b(b) present visuals on how participants played games in each domain.

Explanation Stage During the Explanation stage, participants interacted with an expert player (expert RL agent) and stepped through the expert player's actions while being exposed to explanations from a given study condition. The participants were not told that the expert player was an RL agent to limit the confounding effect of user biases towards AI agents in study. Figure 19a(a) and Figure 19b(b) provide an example of the explanation stage for Connect 4 and Lunar Lander. Specifically, in Figure 19a(a), a E C i is provided, and in Figure 19b(b), a E C i w/ InF, TeG is provided.

Survey At the end of the user study, participants were asked to fill out a short survey including demographic questions, as well as additional Likert questions to gauge user experiences with computers, the domain, as well as their perceived performance and explanation utility. Figure 20 provides a visual of what the survey looked liked in the domain of Connect 4. The survey for Lunar Lander participants was identical, except making an reference to Lunar Lander as opposed to Connect 4.

F Github Repo

https://anonymous.4open.science/r/S2E/README.md

Wrap Up Survey

Great job, you are almost done! Please answer a few final questions to help us with our study.

Question 1. Please select your age range:

O 18-24

○ 25-34

○ 35-44

O 45-54

O55 and older

O Prefer not to say

Question2.Pleaseselectyourgender:

O Male

O Female

Other

O Prefer not to say

Question3.Please indicateyourcomputer usage frequency

Never

Rarely

Sometimes

Often

Always

Never

Rarely

Sometimes

Often

Always

Question 5. Please indicate your level of exposure to Connect 4 prior to this study

Very Familiar

Famliar

ModeratelyFamiliar

Slightly Familiar

Not Familiar

uoseai pdsada  s pe id adxa a usa e padu auadaeaaaigsan

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

Question7.1believe the expert'sprovidedreasoning helped improve myunderstandingof the game

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

Continue to Finish

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes State2Explanation, a framework for training RL agents in such a way that both the human and agent benefit, the Protégé Effect as the authors state. The basic idea is to learn a joint embedding space with \"temporal\" concepts that actively helps the agent train better by shaping their rewards. At test time, these explanations are provided to humans to also help train them.\n\nSTRENGTHS:\nA strength of the paper is I feel the motivation. Explainability in RL is harder than supervised learning due to the temporal added component, so approaching this problem is well motivated.\n\nIn addition, the interplay between agent and human is also very exciting, I love the idea of the two helping and playing off each other. I feel this is a great way to impose human understandable concepts onto an RL agent's policy.\n\nI also appreciate the rigour in the user study (although I need some clarifications in the rebuttal), it's clear to me the authors tried here.\n\nWEAKNESSES:\nWhat I always look for in a paper is a single nice idea, and I am not certain this paper really contains one, let me explain by iterating your claimed contributions.\n\nRegarding the desiderata: (1) the idea that a 'concept should be grounded in human domain knowledge' is not a new idea I feel. The original T-CAV paper noted this by e.g. using zebra stripes to explain zebra classifications, rather than e.g. logit values. Simple saying the same thing is true in a sequential setting doesn't seem that original to me. I'm not aware of a great many people suggesting q-values are a good explanation. (2) The idea that a concept should relate to the task goal is an interesting one, and could be useful in certain context, but I feel it doesn't generalize well. For example, if I'm in a self-driving car going from a => b, and halfway through the journey the car brakes to avoid hitting a pedestrian, your framework seems to posit the explanation should be \"I avoiding killing the human so I could still get to point b\". I feel that's a questionable belief system here we are teaching the agent. (3) is a nice idea, but I don't feel it's particularly surprising, the idea that explanations should be robust and generalisable is well agreed upon.\n\nThe framework you propose does seem quite interesting to me, but reward feedback loops are not my area of expertise, so I defer to my colleges to evaluate the novelty of such a framework.\n\nSection 5.2: I am slightly concerned that the explanations are not always matching in Figure 3, does that mean that this method will inevitably give \"wrong\" explanations? Also, I don't know what i, j, and k are in the figure? Now I personally believe it matters more that it just helps people in a user study, which you show it does, so that's great and I feel makes up for this.\n\nSection 5.3: If I understand this correctly, your method doesn't seem to show much difference here at all, and in fact Fig 4a seems to show the baseline outperforming it?\n\nUser Evaluation: I appreciate the attention to detail here, but a lot of details are missing or (I feel) badly explained. I don't really understand how people's performance could get worse in group $E_{A}$, all they are being shown is the action as an explanation, so the idea their ATS would get worse seems extremely odd to me, and makes me feel I am misunderstanding something here. I tried to explore the appendix to understand what was happening, but didn't have much luck.\n\nI think there's some citations that you might also like to be aware of regarding human friendly concepts in sequential settings.\n\nJi, Ying, Yu Wang, and Jien Kato. \"Spatial-temporal Concept based Explanation of 3D ConvNets.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nKenny, Eoin M., Mycal Tucker, and Julie Shah. \"Towards interpretable deep reinforcement learning with human-friendly prototypes.\" The Eleventh International Conference on Learning Representations. 2023.\n\nQUESTIONS:\n1. During the training there's no feedback from the human right? If I understand, the human defines the concepts, and then these help the agent train better? (but Fig 4 seems to show this isn't the case?)\n2. Whilst you used a CNN in connect 4, does this scale to harder Deep RL problems?\n3. Is my understanding that Fig 3a shows the explanations are often \"wrong\" accurate?\n4. What is i, j, and k in Fig 1?\n5. How did you pick a sample size in the user study? Although it seems fairly rigorous, I don't see any power analysis, and you aren't clear what your attention checks are? It also seems strange that you had 66->60 and 98->90 in the two domains, a perfect balance for 15 in each group. \n\nI might recommend using Prolific instead, it seems to have less issues compared to MTurk which is full of bots and LLMs.\n\nI am happy to revise my review after the rebuttal.\n\nLIMITATIONS:\nThis is ok", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by the Protege effect, learning and developing explanations should provide a dual benefit, both to the readers of the explanations, and to the developers of the explanations. Based on this idea, the paper proposes State2Explanation, an algorithm to learn joint embeddings between state-action pairs and concept-based explanations. This allows for reward shaping, which benefits the explanation developer, while also providing explanations. These claims are validated for agents in two different reinforcement learning settings: Connect 4 and Lunar Lander.\n\nSTRENGTHS:\n1. Provides concrete desiderata for a concept is, including the need for generalizibility and its relationship to the task goal \n2. Reward shaping through explanations is potentially novel, and explores use cases of explanations beyond explainability\n3. Thorough evaluation investigates the impact of various additions to the model, making it clear what the impact of Information Filtering (InF) and Temporal Group (TeG) are\n4. User study demonstrates real-world viability model through a human explainability lense\n\nWEAKNESSES:\n1. Concepts for domains are dependent on expert knowledge for state-action pairs, making it unclear how easy it would be to generalize beyond well-studied games. This is especially the case when state-actions pairs tend to get very large. \n2. An additional baseline would make it clearer what the impact of each study condition was in Section 6. In particular, a baseline with no intervention/information would make it clear how much the rise in performance is due to the additional practice attained from the PreTest when completing the PostTest. \n3. If concepts for Lunar Lander are derived from the existing domain reward function, then claiming that $M_{LL}$ \"informs reward shaping comparable to expert-defined dense reward functions\" seems to follow from the definition of the concepts rather than an indicator of the performance of the algorithm.\n\nQUESTIONS:\n1. For reward shaping, how is the amount of reward determined/amount the reward function is changed by? \n2. In general, who would annotate the explanations, and what is the impact if there are multiple explanations for a single move/conflicting annotations?\n\nLIMITATIONS:\nThe paper discusses limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework to incorporate explanation concepts to sequential decision tasks. The framework can be applied both to the training of the agent by improving RL and to provide explanations to end-users during deployment. The framework is tested using two simple games, Connect 4 and Lunar Lander, where it shows it help improve training performance and user performance.\n\nSTRENGTHS:\nThe main strengths of the paper are:\n\n1. It provides an unified framework for explanations which can be used both for training and deploying.\n2. The explanation framework is based on user-understandable concepts and terms.\n3. The paper provides a sound theoretical and empirical analysis of the proposed framework.\n4. The paper performs user testing and subsequent data analysis in a methodologically sound manner, rare in NeurIPs papers.\n\nWEAKNESSES:\nThe main weaknesses of the paper are:\n\n1. The paper condenses two much information in very little space, including impossible to read figures. It is very hard to read.\n2. The paper fails to discuss how difficult will be to create frameworks for more complex scenarios where there are more and more complex concepts. For instance, consider a system making decisions about buying and selling stocks.\n3. Because of 2, the ideas presented in the paper may never be applicable to real-world problems.\n4. It provides a very brief description of how the concepts are actually created, stored, and verified. \n5. The paper does not discuss situations where the explanation is wrong or inadequate for a situation (for instance, there is a better move), and how that impacts training and user performance.\n\nQUESTIONS:\n1. How do you expect the framework to perform when it is applied to more complex tasks and domains, both in terms of describing the actual concepts and on the performance during training?\n2. How hard is to identify, create, store, and debug concepts?\n3. What kind of wrong explanations it produces? How often? How that impacts training and user performance?\n\nLIMITATIONS:\nThe paper does not do a good job of discussing how it can scale to real-world applications. By not doing so, it failed in identifying an important limitation of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

comments: SUMMARY:
The authors propose a unified framework called State2Explanation (S2E) that combines learning a joint embedding model between state-action pairs and concept-based explanations. The authors draw inspiration from the Protégé Effect, which suggests that explaining knowledge reinforces self-learning. They propose that concept-based explanations can benefit both the RL agent and the end-user by improving the agent's learning rate and the end-user's understanding of the agent's decision making. The S2E framework is designed to inform reward shaping during an agent's training and provide explanations to end-users at deployment for improved task performance.Results on Connect 4 and Lunar Lander demonstrate the success of S2E in providing a dual benefit.

STRENGTHS:
The paper introduces a novel framework S2E. The authors suggest that explaining knowledge to the agent can improve its learning rate, while providing explanations to end-users can enhance their understanding of the agent's decision making. The framework is supposed to be useful in providing explanations understandable in various applications.

WEAKNESSES:
There are no significant weaknesses from my point of view, while there are some improvements can be made. Although the paper mentions experimental validations in the Connect 4 and Lunar Lander domains, the proposed method should be tested on more complex tasks to test its effectness. The authors should also consider how concepts can be better selected beyond expert-defined thresholding.

QUESTIONS:
- To be specific, how concept candidates are selected when performing InF? 
- How thresholds are determined via qualitative methods? Is there any results showing that different concepts selected will influence the final performance?

LIMITATIONS:
See weaknesses and questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04