Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents
Authors: Quentin Delfosse
Sebastian Sztwiertnia
Mark Rothermel
Wolfgang Stammer
Kristian Kersting
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainable AI (XAI)
Reinforcement Learning
Concept Bottlenecks
Primary area: interpretability_and_explainability
TL;DR: We introduce Successive Concept Bottleneck agents for competitive and aligned reinforcement learning.

Abstract: Goal misalignment, reward sparsity and difficult credit assignment are only a few of the many issues that make it difficult for deep reinforcement learning (RL) agents to learn optimal policies. 
Unfortunately, the black-box nature of deep neural networks impedes the inclusion of domain experts for inspecting the model and revising suboptimal policies.

To this end, we introduce Successive Concept Bottleneck Agents (SCoBots), that integrate consecutive concept bottleneck (CB) layers. 
In contrast to current CB models, SCoBots do not just represent concepts as properties of individual objects, but also as relations between objects which is crucial for many RL tasks. 

Our experimental results provide evidence of SCoBots' competitive performances, but also of their potential for domain experts to understand and regularize their behavior. Among other things, SCoBots enabled us to identify a previously unknown misalignment problem in the iconic video game, Pong, and resolve it. Overall, SCoBots thus result in more human-aligned RL agents.

Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents
Quentin Delfosse* Sebastian Sztwiertnia".' Mark Rothermel'
Wolfgang Stammer Kristian Kersting
Coomutter Science Department, TU Darmstadt, Germany
Hessian Center for Artificial Intelligence (hessian.AD), Darmstadt, Germany
*Centre for Cognitive Science, TU Darmstadt, Germany
German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany {firstname, lastname) &cc. hu-darmstadt.de
Abstract
Goal misalignment, reward sparsity and difficult credit assignment are only a few of the many issues that make it difficult for deep reinforcement learning (RL) agents to learn optimal policies. Unfortunately, the black-box nature of deep neural networks impedes the inclusion domain experts for inspecting the model and revising suboptimal policies. To this end, introduce Successive Concept Bottleneck Agents (SCoBots), that integrate consecutive concept bottleneck (CB) layers. In contrast to current CB models, SCoBots do not not just represent concepts as properties of individual objects, but also as relations between objects which crucial for many RL tasks. Our experimental results) provide evidence of SCoBots' competitive performances, but also of their potential for domain experts to understand and regularize their behavior. Among other things, SCoBots enabled previously unknown misalignment problem in the iconic video game, Pong, and resolve Overall, SCoBots thus result more human-aligned RL agents.
identify a
Introduction
Deep Reinforcement learning (RL) agents are prone to suffer from variety of issues that hinder them from learning optimal or generalizable policies. Prominent examples are reward sparsity [Andrychow] iez et al. 2017] and difficult credit assignment [Raposo al. 2021 Wu et al. 2024 A more pressing issue is the goal misalignment problem. It occurs when an agent optimizes a different iie-gool,, aligned with the original target goal during training (Koch al. [22TT) but not test time. Such misalignments can be difficult to identify (di Langosco et al.) 2022]. For instance, in this work discover that such a misalignment (ef. Fig. T.. In Pong, the agent's 's target goal is to catch a ball with its own paddle and to return passed the enemy's s one. The enemy programmed constantly follow the ball, thus, agents can learn to focus on the position of the enemy paddle for placing their own, rather than the position of the ball itself. If left unchecked, such shorteut learning [Geirhos et al. (2200] may lead to a lack of model generalization and unintuitive failures e.g. at deployment time IZhang et al. 2021].
occur in the oldest and most iconic video game, Pong
Recently eXplainable AI (XAI) methods have emerged to detect such shorteut behavior, by identifying the reasons behind a model's decisions [Schramowski et al.] 2020) Roy et al.) 2022 Saeed and Omlin 2023 However, many XAI methods' explanations do faithfully present the model's
'Equal contribution
"Code available at https:/gitthb..omm///////SoOBB
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Deep RL agents inspection falls short
Training
Evaluation
Explanation
Action
SCoBots' inspection and revision mitigate misalignment
State
Symbolic Abstraction
Relational Abstraction
ball: (x.y.8.6,8)
dist(player, bell).x
player: (K.y.R
distiplayar, onamy----
distiball, player)
enexy: (K,
coloribakis
distiball, player)
scorel: (x.Y.R.G,B] 8.8)
speedrbal11 1..
score2: (x.2. C,
speed (bel).
disttenern, plateeneww-ppayerrry--->
distiball, player)
Figure Successive Concept Bottlenecks Agents (SCoBots) allow for easy inspection and revision. Top: Deep RL agents trained on Pong produce high playing scores with importance map explanations that suggest sensible underlying reasons for taking an action (o). However, when enemy is hidden, the deep RL agent fails to even catch the ball without clear reasons (0). Bottom: SCoBots, on the other hand, allow for multi-level inspection of the reasoning behind the action selection, e.g., at relational concept, but also action level. Moreover, they allow users easily intervene on them (/) to prevent the agents from focusing on potentially misleading concepts. this way, SCoBots can mitigate RL specific caveats like goal misalignment.
underlying decision process [Chan et al.) 2022 For example, importance map explanations indicate the importance of an input element without indicating why this element is important (Kambhampati et al. 2021 Stammer et al., 2021 Teso et al.) 2023). A recent branch of research therefore focuses on models that provide inherent concept-based explanations. Prominent examples are concept bottlenecks models (CBMs), which provide predictive performances on par with standard deep learning approaches for supervised image classification [Koh et al.) 2020). More importantly, CBMs allow identify and revise incorrect model behavior on concept level Stammer et al. 2021).
Contrary exising CBMs' fields of applications, RL requires relational reasoning Kaiser al. 2019]. In this work, we therefore introduce Successive Concept Bottleneck Agents (SCoBots, of Fig. bottom) bringing the concept bottleneck approach to RL. SCoBots integrate successive concept bottlenecks into their decision processes, where each bottleneck layer provides concept representations that integrate the concept representations the previous bottleneck layer. Specifically, provided set of predefined concept functions, SCoBots automatically extract relational concept representations based on objects and their properties of the initial bottleneck layers. Finally, the set of relational and object concept representations thus represent inherently explainable RL agents that, in comparison to deep RL agents, allow for inspecting and revising their learned decision policies at multiple levels of their reasoning processes: from single object properties, through relational concepts to the action selection.
used for optimal action selection. SCoBots
Our evaluations on the iconic Atari Learning Environments (ALE, Mnih et al. (2013) provides experimental evidence that SCoBots perform on par with deep RL agents. More importantly, we showcase SCoBots' ability to provide valuable explanations and the potential of mitigating variety of RL specific issues, from reward sparsity to misalignment problems, via simple guidance from domain experts. We identify previously unknown shortcut behavior of deep agents, even on the simple Pong game. By utilizing the interaction capabilities of SCoBots, this behavior is casily corrected. Ultimately, our work illustrates the severity of goal misalignment issues in RL and the importance of being able to mitigate these and other RL specific issues, via relational concept based models. In summary,
contributions are:
(i) We introduce Successive Concept Bottleneck agents (SCoBots).
(ii) We show that SCoBots allow inspect their internal decision processes.
(iii) We show that their inherent inspectable nature can helps identifying unknown misalignments.
(iv) We show that they allow for human interactions for mitigating various RL specific issues.
We proceed as follows. We introduce SCoBots and discuss their specific properties. We continue with experimental evaluations and analysis. Before concluding, touch upon related work.

---

Environment reward
Expert reward
Environment
Expert
LLM
ball: (x,),8,6,8)
dist(player, ball).x distiplayer,- enemytry color(ball)- sbaed(
player: (K,y,87668)
eneny: (x,
scorel: (8,
P0s
score2t (629,,,688)
Object Representations
Relational
Concepts
A
Action
Figure 2: An overview of Successive Concept Bottlenecks Agents (SCoBots). SCoBots decompose the policy into consecutive interpretable concept bottlenecks (ICB). Objects and their properties are first extracted from the raw input, human-underssandaal functions are then employed to derive relational concepts, used to select an action. The understandable concepts enable interactivity. Each bottleneck allows expert users to, egg., prune or utilize concepts to define additional reward signals.
Successive Concept Bottleneck Agents
In this work, we represent RL problems through the framework of a Markov Decision Process, M =<< = S, P2.a; Rs... >, with S as the state space, the set of available actions, P(s, a) the transition probability, and R[s,, a) the immediate reward function, obtained from the environment, and ry the didscount factor. Classic deep RL policies, TØ(s) = P(A = alS = 8) parameterized by 6. are usually black-box models that process raw input states, e.g. set of frames, to provide distribution over the action space [Mnih et al. 2015 van Hasselt et al. 2016].
Concept bottleneck models (CBMs) initiate a learning process by extracting relevant concepts from raw input (e.g. image data). These concepts can represent the color or position of a depicted object. Subsequently, these extracted concepts are used for downstream tasks such as image classification. Formally, a bottleneck model g x + transforms an input x RDD with D dimensions into concept representation E I (a vector of & concepts). Next, the predictor network f + y uses this representation to generate the final target output (e.g. y E I for classification problems).
The main body of research on CBMs focuses on image classification tasks, where the extracted concepts represent attributes of objects. In contrast, RL agents must learn not just from static data, but also through interaction with dynamically evolving environments. RL tasks thus often require relational reasoning, as they involve understanding the relationships between instances that evolve through time and interact with another. This is crucial for learning effective policies in complex, dynamic environments. Note that, in the following descriptions, we use specific fonts to distinguish objects' properties from their relations.
2.1 Building inspectable ScoBots
An underlying assumption SCoBots is that their processing steps should be inspectable and understandable by human user. This stands in contrast to e.g. unsupervised CBM approaches Jabrr [2019] (2019) Zhou et al. (Sinias aligned concepts. SCoBots rather take a different approach by dividing the concept learning and RL-specific action selection into several inspectable steps that are grounded
[2020). where there is no guarantee for learning human-
uamaa-understandablle
concept representations. These steps are described hereafter and depicted in Fig.
Similar to other RL agents, SCoBots process the last n observed frames from sequence of images, H For each frame, SCoBots need an initial concept extraction method to extract objects and their properties, as previous works on CBMs [Stammer et al., 2021 Koh al.) 2020). Specifically, we start from an initial bottleneck model, wo, (-), that is parameterized by parameter set, 0,. Given state, 8t, this model provides a set of c; object representations per frame, wi, (+t) = Ω2 {(o) (=) where each object representation corresponds to tensor of different extracted properties of that object, (e.g. its category, position (i.e. x, y coordinates), etc.). done in previous works on CBMs, the bottleneck model our SCoBots, wP, can correspond to model that was supervisedly pretrained for extracting the objects and their properties from images.

---

One of the major differences to previous work on CBMs is that SCoBots further extract and utilize relational concepts that are based on the previously extracted objects and their properties. Formally, SCoBots utilize consecutive bottleneck model, (*)('), called the relation extractor. This model, μ., parameterized by a predetermined set of transparent relational functions, J, used to extract a set of d, relational concepts. These relations are based on each individual object (and its properties) for unary relations or on combinations of objects for n-ary relations, and are denoted as (u,((2;) = T; {gt f11d 1==1- Without strong prior knowledge. F can initially correspond to universal object relations such distance and speed. However, this set can easily be updated with e.g. additional, novel relational functions. Note that F can include the identity, to let the relation extractor pass through initial object concepts from 22 the relational concepts T-
the fly by human user
Finally, SCoBots employ an action selector, pe., parameterized by 0z. on the relational concepts to select the best action, dz, given the initial state, s; (i.e. the set of frames). Up to now, all extracted concepts in SCoBots, both Q2 and Ir represent human-understandabbe concept representations. To guarantee understandability also within the action selection step of SCoBots, we need to embed an interpretable action selector. While neural networks, a standard choice in RL, literature, are performative and easy to train using differentiable optimization methods, they lack this required interpretability feature. However, Çaglar Aytekin [022 have recently shown the equivalence between ReI.U-based neural networks and decision trees, which in contrast represent inherently interpretable models. To trade-off these issues of flexibility and performance vs interpretability. SCoBots thus break down the action selection process by initially training small ReL.Ubaase neural network action selector, Per, via gradient-based RL optimization. After this, Pe; is finally distilled into a decision tree paz. Lastly, note that one can add residual link from the initial object concepts, Sz to the relational concepts, r such that the action selector can, if necessary, also make decisions based on basic object properties, e.g., the height of an object.
Overall, our approach preserves the MDP formulation used by classic deep approaches. but decom-
poses the classic deep policy 8t _ Or into successive concept bottleneck one 8t -01, Qt M r, P02 az, where O = (0), 02 F)) constitutes the set of policy parameters. For simplicity, we will discard the parameter notations in the rest of the manuscript. Further explanations of the input and output space of each module, as well as the properties and relational functions used in this work are provided in the appendix (ef. A.p and
Instead of jointly learning object detection, concept extraction, and policy search, SCoBots enable independent optimization of each policy component. Separating the training procedure of different components reduces the complexity of the overall optimization problem [Koh et al. 2020)
2.2 Guiding SCoBots
The inspectable nature of SCoBots not only brings the benefit of improved human--ndeerstndaabiltyy but importantly allows for targeted human-machine interactions. In the following, we describe two guidance approaches for interacting with the latent representations of SCoBots: concept pruning and object-centric rewarding. We refer to the revised SCoBot agents as guided SCoBots in the following.
Concept pruning. The type and amount of concepts that are required for a successful policy may vary across tasks. For instance, the objects colors are irrelevant in Pong but required to distinguish vulnerable ghosts from dangerous ones in MsPacman. However, overloading the action selector with irrelevant concepts, whether these are object properties (22) or relational concepts (T,), can lead to difficult optimization (e.g. the agent focusing on noise in unimportant states) as well as difficult inspection of the decision tree (p). Moreover, for a single RL task, the need for specific concepts might even change during training, in e.g. progressive environments (where agents need to master early stages before being provided with additional, more complex tasks [Delfosse et al. 2024c)) ))
SCoBot's compreenimppheesiiible concepts therefore allow domain experts to prune unnecessary concepts. In this way, expert users can guide the learning process towards relevant concepts. Formally, users can (i) select subset of the object property concepts, Ω. Additionally, by (ii) selecting a subset of the relational functions, F, or (iii) specifying which objects specific functions should be applied on, experts can implicitly define the relational concepts subset, r' Guided SCoBots thus formally
refining the policy extraction to 8 _ Ωr F4 HJ, T, P02, at- Furthermore, users can (iv) prune out redundant actions resulting in a new action space A.

---

To give brief examples of these four pruming possibilities we refer to Fig. focusing here on the blue subparts. Particularly, in Pong a user can remove objects (e.g. scores) or specific object properties (e.g. R.G.B values) to obtain S2,, Second, the color relation, color (-), is irrelevant for successfully playing Pong and can therefore be removed from F. Third, the vertical distance function (i,st(;)).) can be prevented from being applied to the (player, enemy) input couple. These pruning actions provide SCoBots with r, Lastly, the only playable actions in Pong are UP and DOWN. To ease the policy search, the available FIRE, LEFT and RIGHT from the base Atari action space might be discarded to obtain A. as they are equivalent to NOOP (i.e. no action).
Importantly, being able to prune concepts can help mitigate misalignement issues such as those of agents playing Pong (ef. [... where an agent can base its action selection on the enemy's position instead of the ball's one. Specifically, pruning the enemy position from the distance relation concept enforces SCoBot agents to rely on relevant features for their decisions such as the ball's position, rather than the spurious correlation with the enemy's 's paddle.
Object-centric feedback reward. Reward shaping [Touzet 1997 Ng al. 1999 is standard RL technique that is used to facilitate the agent's learning process by introducing intermediate reward signals, aggregated to the original reward signal. The object-centric and importantly concept-based approach of SCoBots allows to easily craft additional reward signals. Formally, one can use the extracted relational concepts to express new expert reward signal:
R (T,) == Σ Ogg 9t.
grEl',
where R : T - R and r, - ({)((s)) ((=)((()))) is the relational state, extracted by the relation extractor. The coefficient Oig, E IK is used to penalize or reward the agent proportionally to relational concepts. Our expert reward only relies on the state, as we make use of the concepts extracted from it. However, incorporating the action into the reward computation is straightforward. In practice, this expert reward signal can lead to guiding the agent towards focusing on relevant concepts, but also help smoothing sparse reward signals, which we will discuss further our evaluations.
For example, one can impose penalties, based on the distance between the agent and objects (cf. "ex- pert reward" arrow in Fig. 2), with the intention of incentivizing the agent to maintain close proximity with the ball. As shown in our experimental evaluation, we can use this concept pruning and concept based reward shaping to easily address many RL specific caveats such as reward sparsity, ill-defined objectives, difficult credit assignment, and misalignment (cf. App. A.8 for details on each problem).
3 Experimental Evaluations
In our evaluations, we investigate several properties and potential benefits of the transparent SCoBot
agents. We specifically aim to answer the following research questions:
(Q1) Are concept based agents able to learn competitive policies on different RL environments?
(Q2) Does the inspectable nature of SCoBots allow to detect issues in their decision processes?
(Q3) Can concept-based guidance help mitigate common RL caveats, such policy misalignments?
(Q4) Can SCoBots learn with imperfect object extraction methods?
(Q5) How crucial is the the relation extractor for SCoBots performances?
We evaluate SCoBots on Atari games (cf. Fig. 3 from the Atari Learning Enviromments [Bellemare et al.) 2012) (by far the most used RL framework (cf. App. A.I. as well as the HackAtari modified [Defoosee et al. 2024a] Pong environments, where the enemy is not visible yet active (NoEnemy), and where the enemy stops moving after returning the ball (LazyEnemy). We provide human normalized scores (following eq.3) that are averaged over 3 seeds for each agent configuration. We compare our SCoBot agents to deep agents with the classic convolutional network introduced by Mnnh et al. (2055), that process a stack of 4 black and white frames and denote these as deep agents in the following. Note, that evaluate all agents
Experimental setup:
the latest v5 version of the environment to prevent overfitting. All agents are trained for 20M frames under the Proximal Policy Optimization algorithm (PPO, [Schulman et al. 2017)) specifically the table-baselines implementation [Raffin et al., 2021 and its default hyperparameters (cf. Tab. 2 in App. A.5).
@3)
'This leads to deep agents overall slightly worse than in Schulman et al. 2017 however we obtain similar results when evaluating on the old v4 versions (cf. Tab
5

---

Performances
Pang Misallgnment
Ill-defined
reward
Level completion (9)
HN Scores (Log, %)
100
100
100
75
10-
50
10
Deep
NN-SC NN-SCooBots
25
SCoBots
NN-SCoBatt (NG)
Asterix Bowling
Boxing
FreewayKangaroo
Pong Seaquest Skiing
Tennis
Original
NoEnemy
LazyEnemy
Kangaroo
Figure 3: Object-centric agents can master different Atari environments and interactive SCoBots allow for corrections. Hlmann-orrmllized scores of different agents trained using PPO on ALE environments, including deep agents (i.e. using CNNs), guided decision tree policy (SCoBots), their neural object-centre baseline (NN-), and these baselines without guidance (NG). SCoBots obtain similar or better scores than the deep agents, showing that object-centric agents can also solve RL tasks while making use of human- human-understarn numan-understandable understandable concepts (left). Guiding SCoBots allow to correct misalignment in Pong (center) and to obtain the originally intended agents, depicted by level completion score of 100% on the intended goal's evaluation Kangaroo (right).
We focus SCoBot evaluations on the more interesting aspects of the agent's reasoning process underlying the action selection, rather than the basic object identification step (which has been thoroughly investigated in previous works e.g. (Koh et al. 2020 Stammer et al. 2021 Wu et al.) 2024)). We thus assume access to a pretrained object extractor and provide our agents with object- centric descriptions of these states (52)) based on the information from OCAtari [Delfosse al., 2024b). Specifically, in our evaluations, the set of object properties consists of object class (e.g. enemy paddle, ball etc.), (x*, y²) coordinates, coordinates the previous position (x²-1,y²²1)) height and width, the most common RGB values (i.e. the most representative color of an object), and object orientation. The specific set of functions, F, that are used extract object relations is composed of: euclidean distance, directed distances (on x and y axes), speed, velocity, plan intersection, the center (of the shortest line between two objects), and color name (cf. App. A.5.I for implementation details). To distill SCoBots' learned policies, we use the decision-tree extraction algorithm VIPER [Bastani al, 2018]. For the human-guided SCoBot evaluations (denoted as (guided) SCoBots our evaluations), we illustrate human guidance and prune out the concepts that we consider unimportant to master the game (cf. Tab. 4 Furthermore, to mitigate RL specific problems, we also provide SCoBot agents with simulated human-expert feedback signals, the details of which we describe at the relevant sections below. More details about the setup and the hyperparameters are in App. A.5 In our evaluation, SCoBots' training is slightly faster than deep agents' one (cf. Appendix A.9
SCoBots learn competitive policies (Q1).
We present human-normalin normalized (HN) scores of both SCoBot and deep agents trained on each inves- tigated game individually in Fig. 3 (left). Numerical values are provided in Tab. (cf. App. A.7). We observe that SCoBot agents perform at least on par with deep agents on all games, even slightly outperform these on 5 out of (namely, Asterix, Boxing, Kangaroo, Seaquest and Tennis). Our results suggest that SCoBots provide competitive performances on every tested game despite the constraints of multiple bottlenecks within their architectures. Overall, our experimental evaluations show that RL policies based on interpretable concepts extraction decoupled from the action selection process can, in principle, lead to competitive agents.
Inspectable SCoBots' to detect misalignments (Q2).
The main target of developing ScoBots is to obtain competitive, yet transparent agents that allow for human users to identify their underlying reasoning process. Particularly, for specific state, the inspectable bottleneck structure of SCoBots allows to pinpoint not just the object properties, but importantly the relational concepts being used for the final action decision. This is exemplified in Fig. 4 on Skiing, Pong and Kangaroo (cf. App. A.4 for explanations of SCoBots on the remaining games). Here we highlight the decision-malee at specific states of each game. For example, for the game state extracted from Skiing, the SCoBot agent selects RIGHT as the best action, because the signed distance from its character to the left flag larger than a specific value (+15 pixels). Given the nature of the game this inherent explanation suggests that the agent is indeed basing its selection process on relevant relational concepts.
path (from SCoBots' decision tree based policies),

---

00504
Horbaylba
FIRE
RyPayerr nele
RIGHT
aasull
FraiDue-42
Hoyo)
DaPayer Powee
Figure 4: Interpretable SCoBots allow to follow their decision process, thanks to their interpretable concepts. The states and associated decision processes of SCoBots (extracted from the decision trees) on Skiing (left), and from unguided SCoBots on Pong (middle) and Kangaroo (right). For example, in this Skiing state, our SCoBot selects RIGHT, as the signed distance between Player and the (left) Flag/ (on the axis) is bigger than 15. This agent selects the correct action for the right reasons.
A more striking display of the benefits of the inherent explanations of SCoBot agents is depicted by the Pong agent in Fig 4 The provided explanation suggests that the agent is basing its decision on the vertical positions of the enemy and of its own paddle (distance between the two paddles on the y axis). In fact, this suggests that the agent is largely ignoring the ball's position. Interestingly, upon closer inspection of the enemy movements, we observed that, previously unknown, the enemy is programmed to follow the ball's ss vertical position (with a small delay). Thus, the vertical positions of these two objects are highly correlated (Pearson's 's and Spearman's correlations coefficient above Moreover, the ball's rendering contains flickering, explaining why SCoBots base
99%, cf their decision on this potentially more reliable feature, the enemy paddle's vertical position.
To validate these findings further, we perform evaluations on 2 modified Pong environments in which (i) the enemy is invisible, yet playing (NoEnemy, of. Fig. []. and on one environment where the enemy is not moving after returning the ball (LazyEnemy). We reevaluate the deep and SCoBot agents that were trained on the original Pong environment on NoEnemy and observe catastrophic performance drops in HN scores (cf. Fig. 3) at a level of random policies for both types of agents. Evaluations with different DQN agents lead to similar performance drops (cf. App. A.7). These drops are particularly striking as initial importance maps produced by PPO and DQN agents highlight all 3 moving objects (i.e. the player, ball, and enemy) as relevant for their decisions (cf. Fip left and App.(..) aligned with findings on importance maps of WWitkamp et al. (0018). These maps suggest that the deep agents had based its decisions on right reasons. Our novel results on the NoEnemy and LazyEnemy environments, however, greatly calls to question the faithfulness and granularity of such post-hoc explainability methods. They highlight the importance of inherently transparent RL models on the level of concepts, provided via SCoBots, to identify such potential issues. In the following, we will investigate how to mitigate the discovered issues via SCoBot's bottlenecks.
Guiding SCoBots to mitigate RL-specific caveats (Q3).
We here make use of the interactive properties of SCoBots to address several famous RL specific problems: goal misalignment, ill-defined rewards, difficult credit assignment and reward sparsity.
Realigning SCoBots: To mitigate the goal misalignment issues of the SCoBots trained on Pong, we simply remove (prune) the enemy from the set of considered objects (92). Thus, the enemy cannot be used for the action selection process. This leads to SCoBots that are able to play Pong and its NoEnemy version (cf. SCoBot in Fig 3 (center). Furthermore, this shows that playing Pong without observing the enemy is achievable, by simply returning vertical shots, difficult for the enemy to catch.
Ill-effind reward: Defining reward upfront that incentivizes agents to learn an expected behavior a difficult problem. An example of ill-defined reward (i.e. a reward that will lead to an unaligned behavior if the agent maximizes the return) is present in Kangaroo. According to the documentation of the game") "the mother kangaroo on the bottom floor tries to reach the top floor where her joey is being held captive by some monkeys". However, punching the monkey enemies gives higher cumulative reward than climbing up to save the baby. RL agents thus tend to learn to kick monkeys on the bottom floor rather than reaching the joey (ef. Fig. 4 For revising such agents, we provide an additional reward signal, based on the distance from the mother kangaroo to the joey (detailed in App. A.8). As can be seen in Fig, 3 (right), this reward allows the guided SCoBots to achieve the originally intended goal by indeed completing the level. In contrast, deep agents and particularly unguided SCoBots achieve relatively high HN scores, but do not complete the levels.
www www.zetrogames.c/pp.e. A9t---tar26000 phpp
7

---

Ablation studies
HN Scores (Log. %)
100
10
NN-SCoBots
NN-SCaBots (No Rel.)
SCoBots
NN-SCoBots(+ noise)
SCoBots (No Rel.)
Asterix
Bowling
Boxing
Freeway
Kangaroo
Seaquest
Skling
Tennis
Garne
Figure 5: SCoBots can learn with noisy object detectors, transparent SCoBots rely on relations. Final human normalized scores (with stds) comparing SCoBots and the object-centric neural baselines (NN-SCoBots), with and without relations. We also provide the scores of NN-SCoBots that learned on noisy environments. The noise only noticeably affects the agents on Kanguroo. Ablating the relations is harmless on NN-SCobots, as neural networks can recompute them, but impacts SCoBots performances on games. (oFor better visualization, we used a human score of 100 in Boxing.)
Difficult Credit Assignment Problem:
The difficult credit assignment problem is the challenge of correctly attributing credit or blame to past actions when the agent's actions have delayed or non-obvious effects on its overall performance. We illustrate this in the context of Skiing, where in standard configurations, agents receive at each step negative reward that corresponds to the number of seconds elapsed since the last steps (i.e. varying between -3 and -7). This reward aims at punishing agents for going down the slope slowly. Additionally, the game keeps track of the number of flag pairs that the agent has correctly traversed (displayed at the top of the screen, of Fig. 4). and provide large reward, proportional to this number, at the end of the episode. Associating this reward signal with the number of passed flags is extremely difficult, without prior knowledge on human skiing competitions. In the next evaluations, timestep, proportional to the agent's progression to the flags' posit iisitionin to incentivize the agent to pass in between them, as well signal rewarding for higher speed:
provide SCoBots with another reward each
Rexp = EE 0)* D(Player, 0)--1 + (Player)).)
oE (eFFFFa,,Faag
These rewards are further detailed in App. A.8 They allow guided SCoBots to reach human scores, whereas the deep and unguided SCoBots perform worse than random (cf. Fig. 3 (left)). Note that providing additional reward signals, as done above, obviously also allows to mitigate reward sparsity, we illustrate Pong (ef. App.A.8.3) for detailed explanation).
A.8.3
Object-centric agents can learn with imperfect object extractors (Q4).
To test object-centric agents' ability to work in more realistic environments, we have tested if they can learn viable policies with imperfect object extraction methods. Based on Delfosse al. (2023b) detection results, added 5% misdetection probability and Gaussian noise (of 0 mean and 3 pixels of standard deviation on each axis). As depicted in Figure 5 NN-SCoBots (i.e. neural network based object-centric agents, using relations) learn comparable policies all games but Kangaroo, demonstrating their ability to mitigate suboptimal object extraction methods. Implementing robustness techniques, such as Kalman filters, would help further push the performances of such agents.
Transparent SCoBots benefit from explicit relations (Q5).
While the ablation of the relation extractor only has significant impact the neural-based object- centric agents on out of 9 games (Seuquest). deteriorates the decision tree based SCoBots on 6 environments. This is due to the fact that relations can be implicitly recomputed within neural network, but not within decision trees. As shown by Kohler et al. (2024), the use of the distance relation (on specific axis) allows for performing agents with compact decision tree-based policies. Furthermore, even if some relations can be rediscovered and implicitly encoded within the decision trees, the lack of explicit relational representations can reduce the interpretability of the agents, and will prevent the experts from using these within their guidance.
Overall, our experimental evaluations not only show that SCoBots are on par with deep agents in terms of performances, but that their concept-based and inspectable nature allows to identify and revise several important RL specific caveats. Importantly, in the course of our evaluations, we identified a previously unknown and critical misalignment of existing RI. agents on the simple and iconic Pong environment, via the previously mentioned properties of SCoBots.
8

---

4 Limitations
On the use of OCAtari. To limit our resource consumption, we made use of the quasi-perfect object extractor (w) of OCAtari, which efficiently extract objects from the RAM. We added an ablation to simulate potential imperfect detection capabilities of Atari object extraction methods. Such extractors can be optimized using supervised [Redmon et al., 2016 Locatello et al. 2020] or self-supervised [Lin et al.) 2020 Delfosse et al. 2023b] object detection methods. This last work showcase that unsupervised object extraction methods can replace OCAtari at test time, however leading to performance drops. Grandien al. (2024) have further showcase training RL agents with such pretrained object extractors further improve the object-centric RL agents performances.
The limit of object-centricity. Other environments require additional information extraction pro- cesses. E.g. in MsPucman, an agent must navigate a maze. Extending the concept representations to cover such concepts in maze or platform environments is an important step for future work. Other representations could allow for the integration of e.g. path finding methods such as the A* algorithm.
Training time of SCoBots. Thanks to the efficient OCAtari object extraction, SCoBots required in average 7.5 hours of training time, while deep agents needed 10.8 hours (cf. Appendix (A.9). All SCoBots variations require less training time compared to deep agents on environments with few objects (e.g. Boxing, Pong, Tennis). Currently, at every step, the concept bottleneck values are calculated sequentially in single process on the CPU, leaving significant room for training time improvement by optimizing these computations (bringing them to GPUs). Thus, in environments with many objects, e.g. Kangarvo, the training time of unguided SCoBots exceeded that of its deep agent counterpart. Attention on relevant objects could be used to further save computational resources.
Related Work
The basic idea of Concept Bottleneck Models (CBMs) can be found in work as early as Lampert et al. (2009) and|Kumr al. [2009. A first systematic study of CBMs was delivered by Koh et al. 2020). followed by Stammer et al. [202]]. who described CBMs as two-stage models that first computes intermediate representations used for the final task output. Where learning valuable initial object
concept representations without strong supervision is still a tricky and open issue for concept-
models [Lage and Doshi Velez 2020 Stammme al. 2022, (Sawada and Nakamura 2022 Marconato
et al. |022 Steinmann et al. 2023) Stammer et al. 2024)), receiving relational concept representations
in SCoBots is performed automatically via the function set F. Since the initial works on CBMs they
(202T) e.g. apply CBMs to text sentiment classification, and Kraus et al. [2024) to time series' analysis. However, these works consider single object concept representations and focus supervised learning. The ability of users to revise concepts and decisions has also been parallelly shown by Friedrich et al. [2023).
have found utilization in several applications. Antognimi and Faltings
In fact, CBMs have found their way into RL only to a limited extent. Zabounidis et al. 2023 and Grupen et al. [2022) utilized CBMs in a multi-agent setting, where both identify improved inter- pretability through concept representations while maintaining competitive performance. Zabounidis al. 2023) further report better training stability and reduced sample complexity. Their extension includes an optional "residual" layer which passes additional, latent information to the action selector part. SCoBots omit such a residual component for the sake of human- understandability, yet offer the flexibility to the user to modify the concept layer via updating FF for improving the model's s representations. Similar to how SCoBots invite the user to reuse the high-level concepts to shape the reward signal. Guan et al. [2023) allow the user to design reward function based on higher-level properties that occur over a period of time. Lastly, SCoBots not only separate state representation learning from policy search, as done by Cuccu et al. [2020). but also enforce object-centricity. putting interpretability requirements on the consecutive feature spaces.
Explainable RL (XRL) is an extensively surveyed XAI field [Dazeley et al. 2022 Krajna et al. 2022
Milani et al. 2023] with a wide range of unsolved issues [Vouros] [022]. J. Milani et al. [2023] introduce a taxonomy for XRL methods, with: (1) feature importance methods that generate explanations that point out decision-relevant input features, (2) learning process & MDP methods which present which past experience or MDP components affect the policy, and (3) policy-level methods, describing the agent's long-term behavior. Based on this, SCoBots extract relations from low-level features, making
high-level information available to explanations and thereby support feature importance methods.

---

According to Qiin et al. [2022]* 's categorization of XRL frameworks, SCoBots are "mooellexpllaningg
(in contrast to reward-, state-, and task-explaining). Other XRL methods rely on LLM to explain the
policy [Luo et al.] decision trees [Fuhrr et al. 2024 Marton et al. 2024). logic [Jiang and Luo, 2019
Kimura et al.) 2021 Delfosse et al. [2.3,|223a Sha et al. [024] or programs [Verma et al. 2018 Trivedi
et 2 Cao et al., 2022] Kohler et al., 2024 Wüst et al. 2024 to encode transparent policies.
To overcome the potentially unavailable concepts necessary to learn symbolic policies, Shindo et al.
[2024] learn mixture of symbolic and neural policies. Finally, concepts have further been used
t0 derive reward from context using LLMs, as we did for Kangaroo and Skinng [Kwon et al. 2023
Kaufmann et al. 2024 Wu 2024
Shen et al., 2024].
The misalignment problem is an RL instantiation of the shortcut learning problem, frequently studied failure mode that has been identified in models and datasets across the spectrum of Al from
deep networks [Lapuschkin et al.) [0199] Schramowskii et al. 2020] to neuro-symbolic (Stammer et al., 2021 Marconato et al., 2023], prototype-based models [Bontempelli et al. [023] and RL approaches (di Langosco et al. 2022]. A misaligned RI. agent, first empirically studied by [Koch et al.)
2021] represents serious issue, especially when it is misaligned to recognized ethical values [Arnold and Kasenberg 2017] or if the agent has broad capabilities [go 2022]. Nahian et al. [2021] ethically align agents by introducing a second reward signal. In comparison, SCoBots aid to resolve RL specific issues such as the misalignment problem through inherent interpretability.
6 Conclusion
In this work, we have provided evidence for the benefits of concept-based models in RL tasks, specifically for identifying issues such as goal misalignment. Among other things our proposed Successive Concept Bottleneck agents integrate relational concepts into their decision processes. With this, we have exposed previously unknown misalignment problems of deep RL. agents a game as simple as Pong. SCoBot agents allowed us to revise this issue, as well as different RL specific caveats with minimal additional feedback. Our work thus represents an important step in developing aligned RL agents, i.e. agents that are not just aligned with the underlying task goals, but also with human user's understanding and knowledge. Achieving this is particularly valuable for applying RL agents in real-world settings where ethical and safety considerations are paramount.
Avenues for future research are incorporating a high level action bottleneck [Bacon al. 2017 One can also incorporate attention mechanisms into RL agents, as discussed in [Itaya et al. 2021 or use language models (and e.g., the game manuals) to generate the additional reward signal. as done by Wu et al. 2024). Additionally, we are considering the use of shallower decision trees [Broelemann and Kasneci 2019]. - An interesting research question is how far the task reward signal can aid in learning games object-centric representations [Delfosse et al. (023bb in the first place.
Impact statement
Our work aims at developing transparent RL agents, whose decision can be understood and revised to be aligned with the beliefs and values of a human user. We believe that such algorithms are critical to uncover and mitigate potential misalignments of AI systems. A malicious user can, however, utilize such approaches for aligning agents in a harmful way. thereby potentially leading to a negative impact on further users or society as a whole. Even so, the inspectable nature of transparent approaches will allow to identify such potentially harmful misuses, or hidden misalignment.
Acknowledgments
This work has benefited from the HMWK projects "The Third Wave of Artificial Intelligence 3AI", their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the "SenPai: XReLeaS" project. has also received support of Hessian.Al, as well as the Hessian research priority program LOEWE within the project WhiteBox, and the EU-funded "TANGO" project (EU Horizon 2023, GA No 57100431). The authors also would like to thank Stefan Lichtenstein, Raban Emunds for their help on refactoring the code, and Dwarak Vittal for his initial contributions.
10

---

References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, Advances in neural information processing systems, 2017.
Diego Antognini and Boi Faltings. Rationalization through concepts. ArXiv, 2021.
Thomas Arnold and Daniel Kasenberg. Value alignment or misalignment - what will keep systems accountable? In AAAI Workshop on AI, Ethics, and Society, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, 2017.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurlPS, 2018
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents (extended abstract). In International Joins Conference Artificial Intelligence, 2012.
Andrea Bontempelli. Stefano Teso, Katya Tentori, Fausto Giunchiglia, and Andrea Passerini. Concept- level debugging of part-prototype networks. In International Conference on Learning Representa- tions (ICLR), 2023.
Klaus Broelemann and Gjergji Kasneci. gradient-based split criterion for highly accurate and transparent model trees. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10- 10-16, 2019, 2019.
Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Li, Jianye Hao, and Yang Liu. Galois: boosting deep reinforcement learning via generalizable logic synthesis. Advances in Neural Information Processing Systems, 2022.
Chun Sik Chan, Huanqi Kong, and Guanqing Liang. comparative study of faithfulness metrics for model interpretability methods. In Conference of the Association for Computational Linguistics (ACL), 2022.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 2019.
Giuseppe Cuccu, Julian Togelius, and Philippe Cudré-Mauroux. Playing atari with six neurons (extended abstract). In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, 2020.
Richard Dazeley, Peter Vamplew, and Francisco Cruz. Explainable reinforcement learning for broad-xai: conceptual framework and survey. Neural Computing and Applications, 2022.
Quentin Delfosse, Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Interpretable and explainable logical policies via neurally guided symbolic abstraction. In Advances in Neural Information Processing (NeurIPS), 2023a.
Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting. Boosting object representation learning via motion and object continuity. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi, editors, European Confer- ence on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML), 2023b.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, and Kristian Kersting. Hackatari: Atari learning environments for robust and continual reinforcement learning. arXiv, 2024a.

---

Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sziwiertnia, and Kristian Kersting. OCAtari: Object-centric Atari 2600 reinforcement learning environments. Reinforcement Learning Journal, 2024b.
Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. International Conference on Learning Representations (ICLR), 2024c.
Lauro Langosco di Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning ICML, 2022.
Felix Friedrich, David Steinmann, and Kristian Kersting. One explanation does not fit xil. arXiv, 2023.
Benjamin Fuhrer, Chen Tessler, and Gal Dalal. Gradient boosting reinforcement learning. arXiv, 2024.
Robert Geirhos, Jörn- Höm--eerrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2020.
Nils Grandien, Quentin Delfosse, and Kristian Kersting. Interpretable end-to-end neurosymbolic reinforcement learning agents. arXiv, 2024.
Niko Grupen, Natasha Jaques, Been Kim, and Shayegan Omidshafiei. Concept- understanding of emergent multi-agent behavior. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Lin Guan, Karthik Valmeekam, and Subbarao Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. In International Conference on Learning Representations (ICLR), 2023.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2017.
Geoffrey Irving, Paul Francis Christiano, and Dario Amodei. safety via debate. ArXiv, 2018.
Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, and Komei Sugiura. Visual explanation using attention mechanism in actor-critic-based deep reinforcement learning. 2021 International Joint Conference on Neural Networks (IJCNN,, 2021.
A. Jabri, Kyle Hsu, Benjamin Eysenbach, Abhishek Gupta, Alexei A. Efros, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinnfrrcement learning. ArXiv, 2019.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In International conference on machine learning, 2019.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, K. Czechowski, D. Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, G. Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. ArXiv, 2019.
Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and L. Guan. Symbols as lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In AAAI Conference on Artificial Intelligence, 2021.
Timo Kaufmann, Jannis Blüml, Antonia Wüst, Quentin Delfosse, Kristian Kersting, and Eyke Hüllermeier. Ocalm: Object-centric assessment with language models. arXiv, 2024.
Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agra- vante, Michiaki Tatsubori, Asim Munawar, and Alexander Gray. Neuro-symbolic reinforcement learning with first-order logic. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
12

---

Jack Koch, Lauro Langosco, Jacob Pfau, James Le, and Lee Sharkey, Objective robustness in deep reinforcement learning, 2021.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, 2020.
Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Interpretable and editable programmatic tree policies for reinforeement learning. In Workshop on Interpretable Policies in Reinforcement Learning@ RLC-2024, 2024.
Agneza Krajna, Mario Brčič, Tomislav Lipić, and Juraj Doncevic. Explainability in reinforcement learning: perspective and position. ArXiv, 2022.
Maurice Kraus, David Steinmann, Antonia Wüst, Andre Kokozinski, and Kristian Kersting. Right on time: Revising time series models by constraining their explanations. arXiv, 2024.
Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Attribute and simile classifiers for face verification. 2009 IEEE 12th International Conference on Computer Vision, 2009.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv, 2023.
Isaac Lage and Finale Doshi Velez. Learning interpretable concept-based models with human feedback. ArXiv, 2020.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling, Learning detect unseen object classes by between-class attribute transfer. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 2019.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 2020.
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro-symbolic reinforcement learning with textual explanations. In Forty-first International Conference on Machine Learning.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents (extended abstract). In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI, 2018.
Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. In Advances in Neural Information Processing (NeurIPS), 2022.
Emanuele Marconato, Stefano Teso, and Andrea Passerini. Neuro-symbolic reasoning shortcuts: Mitigation strategies and their limitations. In International Workshop Neural-Symbolic Learning and Reasoning, 2023.
Sascha Marton, Tim Grams, Florian Vogt, Stefan Lüdtkee Christian Bartelt, and Heiner Stucken- schmidt. Sympol: Symbolic tree-based on-policy reinforcement learning. arXiv, 2024.
13

---

Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney. Thomas S. Stepleton, Nicolas Heess, Arthur Guez, Eric Moulines. Marcus Hutter, Lars Buesing, and Rémi Munos. Counterfactual credit assignment in model-free reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning (ICML., 2021.
Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review. ACM Computing Surveys, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. ArXiv, 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Md Sultan Nahian, Spencer Frazier, Brent Harrison, and Mark O. Riedl. Training value-aligned reinforcement learning agents using normative prior. ArXiv, 2021.
A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. International Conference on Machine Learning, 1999.
Richard Ngo. The alignment problem from a deep learning perspective. ArXiv, 2022.
Yunpeng Qing, Shunyu I. Liu, Jie Song, and Mingli Song. A survey on explainable reinforcement learning: Concepts, algorithms, challenges. ArXiv, 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. : Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
David Raposo, Samuel Ritter, Adam Santoro, Greg Wayne, Théophane Weber, Matthew M. Botvinick, H. V. Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. ArXiv, 2021.
Joseph Redmon, Santosh Kumar Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Conference on Computer Vision and Pattern Recognition, CVPR 2016, 2016.
Nicholas A. Roy, Junkyung Kim, and Neil C. Rabinowitz. Explainability via causal self-talk. 2022.
Waddah Saeed and Christian W. Omlin. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledze-Based Systems, 2023.
Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. JEEE Access, 2022.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representa- fions, ICLR, 2016.
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 2017.
Jingyuan Sha, Hikaru Shindo, Quentin Delfosse, Kristian Kersting, and Devendra Singh Dhami. Expil: Explanatory predicate invention for learning in games. arXiv preprint, 2024.
Zichao Shen, Tianchen Zhu, Qingyun Sun, Shiqi Gao, and Jianxin Li. Beyond human preferences: Exploring reinforcement learning trajectory evaluation and improvement through llss. arXiv, 2024.
14

---

Hikaru Shindo, Quentin Delfosse. Devendra Singh Dhami, and Kristian Kersting. Blendrl: A framework for merging symbolic and neural policy learning. arXiv, 2024.
A. Srinivas, Michael Laskin, and Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. ArXiv, 2020.
Wolfgang Stammer, Patrick Schramosski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
Wolfgang Stammer, Marius Memmel, Patrick Schramowski. and Kristian Kersting, Interactive disen- tanglement: Learning concepts by interacting with their prototype representations. Conference Computer Vision and Pattern Recognition, (CVPR), 2022.
Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder. Advances in Neural Information Processing (NeurIPS), 2024.
David Steinmann, Wolfgang Stammer, Felix Friedrich, and Kristian Kersting. Learning to intervene on concept bottlenecks. ArXiv, 2023.
Stefano Teso, Öznur Alkan, Wolfgang Stammer, and Elizabeth Daly. Leveraging explanations in interactive machine learning: An overview. Frontiers in Artificial Intelligence, 2023.
Claude F. Touzet. Neural reinforcement learning for behaviour synthesis. Robotics Auton. Syst., 1997.
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph Lim. Learning synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 2021.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, 2016.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. International Conference on Machine Learning. PMLR, 2018.
George A. Vouros. Explainable deep reinforcement learning: State of the art and challenges. ACM Computing Surveys, 2022.
Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep reinforcement learning for atari games. In BNCAI, 2018.
Xiefeng Wu. From reward shaping to q-shaping: Achieving unbiased learning with Ilm Im-guided knowledge. arXiv, 2024.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. Advances in Neural Information Processing Systems, 2024.
Antonia Wäst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Pix2code: Learning compose neural visual concepts as programs. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024.
Renos Zabounidis, Joseph Campbell, Simon Stepputtis. Dana Hughes, and Katia P. Sycara. Concept learning for interpretable multi-agent reinforeement learning. ArXiv, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 2021.
Xiaomao Zhou, Tao Bai, Yanbin Gao, and Yuntao Han. Vision-based robot navigation through com- bining unsupervised learning and hierarchical reinforcement learning. Sensors (Basel, Switzerland). 2019.
Çağlar Aytekin. Neural networks are decision trees. ArXiv, 2022.
15

---

A Appendix
As mentioned in the main body, the appendix contains additional materials and supporting information for the following aspects: further information on the fact that Atari is the most common set of games ). details on the reward sparsity Pong (A.8.3). detailed numerical results (.2).. learning curves of our RL agents (A.3). the hyperparmeters used in this work (A.5), formal definitions on the SCoBots policies (A.6) and further details on the misalignment problem in Pong.
A.1 Atari games are most common set of environments
We here show that the Atari games from the Ar- cade Learning Environment (Bellemare al., 2012) is the most used benchmark to test rein-
ATARI
CARLA
Benchmark
forcement learning agents.
MUJOCO
DMCS
A.2 Detailed numerical results
AirSIM
TORCS
For completeness, we here provide the numer-
AI2THOR
ical results of the performances obtained by
each agent type. For fair comparison, we reim- plemeneted the deep PPO agents, and used the default hyperparameters for Atari from stable- baselines3 for the deep RL agents. detailed overview of hyperparameters and their corre- sponding values can be found in AA.5
D4RL
RLBench
0
5K
10K
15K
20K
25K
Number of publications
Figure 6: The Atari Learning Environments is more used in scientific research than the next 8 other benchmarks together. Graph borrowed from [Deffosse et al. 20246)
All agents are trained on gymnasium's ALE using 8 actors and 3 training seeds ([0, 16, rankk, Each training seed's s perfor- mance evaluated every 500k frames on differently seeded training seed) environments for episodes each. After training, the best performing checkpoint is then ultimately evaluated on 4 seeded (123,456, 789, 1011) test environments. The final return is determined by averaging the return over 5 episodes per training-tee -test seed and every training seed for the respective environment. We use deterministic actions for both evaluation and testing stages.
SCoBots-v5 NoGuidance
PPO-v5 ours
PPO-v4 ours
PPO-v4 Schulman et al.
Game
SCoBots-vS
Random
Human
Asterix
5080=614.8
5043.3+729.9
2126.7+1484
10433=2778
4532
210
8503
Bowling
106.6141.8
137 1.241
102.2_
51.4118.0
40.1
23.1
160.7
Boxing
97. U7.1=2.2
69.0=10.9
90.3133
99.5=11.4
94.6
0.10
4.3
Freeway
32.820.1
32, 12.9.10.7
33.610.2
33. 620.3
32.5
0.00
29.6
Kangaroo
1776.6.11122.
4050.0.121..8
790 0..290.5
13296. 71111111
9929
52.0
3035
Pong
17.2=1.9
17.5+1..
16.4+1.5
20.9=0.2
20.7
-00.7
14.6
Seaquest
1055.3-2724
2411.31377.0
337.3
082.0.246..1
1204.5
68.4
20182
Skiing
20004
653013332
23004_110333
22983110338 10333
13901
17098
4336
Tennis
2.4+3.0
0.0+0.0
-0.9=0.1
-1,2+0.3
-14.8
23.8
-8.3
Table I: ScoBots and guided ScoBots obtain similar results than deep agents averaged over 3 seeded
runs. Neural refers to the agent that use CNN instead of our ICB layers. We added the results reported in the original paper Schulman et al. (2017) (original), which are on par with ours, as well as from Random baseline and Humans (from van Hasselt al. (0166). ). Our agents have been trained with only 20M frames.
Normalisation techniques. To compute human normalised scores, used the following equation:
score agent
score random
ScOrenormalised
100
(3)
score human score
16

---

A.3
Learning curves
Adter'
Soal
Decing
170
4043
2003
1048
Rames
Promes
15
6000
22
25
2010
22
22
3000
-10
5
26
10
35
26
Saeing
1543
1009
-30
-19
Tramsa
Frama
Figure 7: Training Return over frames (x (x10%) seen for SCoBots, SCoBots w/o guidance and neural agents.
17

---

A.4 Agent reasoning through the decision trees
In this section, we provide more decision trees from SCoBots on states from Freeway and Bowling.
#ap UP
OOO
Figure 8: Decision trees from Freeway and Bowling.
A.5 Hyperparameters and experimental details
All Experiments were run AMD Ryzen proces- sor, 64GB RAM and one NVIDIA GeForce RTX 2080 Ti GPU. Training SCoBot on 20M frames with 8 actors takes approximately hours. We use the same PPO hyperparameters
Actors N
Minibatch size
32
Horizon T
2048
Num. epochs K
the (Schulman et al. [2017] agents that learned to master the games. For the Adam optimizer, SCoBots start with slightly increased learn- ing rate of x x00- (compared to 2.5x 10-4). The PPO implementation used and the respective MLP hyperpa- rameters are based on waabl-bbaeeling [002]). SCoBots have the same PPO hyperparameter values as deep agents but use MLPs (2 x 64) with ReLU activation functions as policy and value networks. Deep agents use CnnPolicy in stable- balele-bssinee3 as their pol- icy value network architecture, which aligns with the initial baseline implementation of PPO. Further, we normalize the advantages in our experiments, since it showed only beneficial effects on learning. This is the default setting in the stable-baseline3 implementation.
Adam stepsize
2.5 * 10-4
*
Discount y
0.99
GAE parameter À
0.95
Clipping parameter e
0.1 * a
VF coefficient C1 Entropy coefficient C2
Raffin et al.)
0.01
Table 2: PPO Hyperparameter Values. a linearly decreases from to 0 over the course of training.
The Atari environment version used in gymnasium is NoFrameskip-v4 for agents reproducing the reported PPO results, and v5 for SCoBots and neural. v5 defines a deterministic skipping of 5 frames per action taken and sets the probability repeat the last action taken to 0.25. This is aligned with recommended best practices by Machado et al. (2018). The experiments using NoFrameskip-vé utilize the same environment wrappers as OpenAl's 's initial baselines implementation" This includes frame skipping of 4 and reward clipping. Frame stacking is not used. SCoBots are not trained on clipped reward signal. For comparability, the neural agents we compare SCoBots to, are not well. A list of all hyperparameter values used is provided in Table
A.5.1 The properties and features used for SCoBots
In this paper, we used different properties and functions (to create features). They are listed hereafter. Further, Table shows a detailed overview of the concepts and actions pruned for every environment evaluated.
A.6 SCoBots policies: formal definitions
The set S denotes the problem-specific set of raw environment states as defined by the RL problem, e.g., the space of RGB images [0, 255]512x512xx Similarly, AA represents the action space, e.g., "move right" or "jump."
github. ccw./oppenna/bbaslleeee
18

---

Name
Definition
Description
class
NAME
object class (e.g. "Agent". "Ball", "Ghost") position on the screen
Properties
position
position history
Jt, Vt, 1t-1, VC-1
position and past position on the sereen object's orientation if available
orientation
RGB
R.G.B
RGB values
distance
Dz(01:02) 02): Dy(01,00 02)
distance the and y axis
cuclidean distance
De(01, 02)
cuclidean distance orthogonal projection of onto trajectory center of the shortest line between or and speed of object (pixel per step)
Relations
trajectory
landing pointr 02)
center
co1 02)
speed
s(r,, 1t-1- M6--1)
velocity
v(It, 2t-1 11-1)
velocity object (vector)
color
color(o1)
the CSS2. color category (e.g. Red)
top k
only ls closest (De to player) objects visible Table 3: Descriptions of properties and relations used by SCoBots.
Topu((O1,-.) ..., 0n), ki class)
Features
Bowling
Boxing
Pong
Freeway
Tennis
Skiing
Kangaroo
Asterix
Seaquest
class
no enemy
Properties
position
position history
V
V
orientation
X
X
X
RGB
X
X
X
X
X
X
X
X
trajectory
X
X
X
X
X
X
X
distance
V
V
NLNKAA
V
V
X
V
NAKIKA
XX
euclidean distance
X
Relations
center
AAKAKA
speed
velocity
color
2
closest objects
NOOP
FIRE
J
UP
J
RIGHT
J
LEFT
DOWN
UPRIGHT
UPLEFT
Actions
DOWNRIGHT
X
DOWNLEFT
UPFIRE
RIGHTFIRE
X
LEFTFIRE
X
DOWNFIRE
UPRIGHTFIRE
UPLEFTFIRE
X
DOWNRIGHTFIRE
X
X
DOWNLEFTFIRE
X
X
Table 4: Feature selection and pruning for guided SCoBots. / denotes the included features, whereas the features that are pruned out.
Let Og be the set of all possible objects, identified by their and characterized by their properties like position, size, color, etc. Then, O = {O e P(Oo) id(or) * id(o)) 02 E O} is the object
19

---

S
1
R
F
Figure 9: Functional summary of the SCoBots model architecture: maps states to sets of objects, j maps sets of objects to relation vectors by applying relational functions, and maps relation vectors to actions.
detection space, defined as the family of object sets in which each object (identified by its ID) occurs once most.
F is the relation function space. It the family of user-defined relation function sets F. Each relation function f E F is of the form f O IR" for k, n E IN, that is, maps fixed number of objects to real-valued vector. As an instance, the function that maps two objects to the Euclidean distance between each other is relation function.
The set R S Rd is the relation (or feature) space. The dimension m is implied by F. More specifically, dimension.
Σ5e= dim(co(f)), i.e., is the of each relation function's codomain
Definition A.1. The overall model policy : S - A given relation function set F is defined as = oo pa=, F²) w, where
1. w S O is the object detector, defined as the function that maps a raw state s E S to set of detected objects E O.
2. μ is the feature selector
p. 0 F F- + R
(4)
(0, r-:= (z)) SEP where 01,..., Ok E O.
(5)
That is, a applies each relation function f E F to the respective detected object(s) in O E O resulting in a real-valued relation vector.
3. p: R A is the action selector that assigns actions to relation vectors.
See also Figure 9 for summary. The policy parameters 0 == (01, 02 0s) split up into the parameters of the object detector, the feature selector, and the action selector, accordingly. They are left out for brevity.
The architecture's bottleneck is induced by the object detector and the feature selector μr. From this function perspective, the bottleneck consists of two stages: an object-centric bottleneck stage at O and semantic bottleneck stage at R.
A.7 Pong misalignment problem
The misalignment problem had previously been identified other games such as the Coinrun platform game [Cobbe et al. 2019). in which an agent's target-goal is to reach coin which is always placed at the end of level at training time. When the coin is repositioned at test time di Langosco et al. observed that trained deep RI. agents avoid the coin and simply target the end of the level, indicating that agents in fact learn to follow simpler side-goal rather than the underlying strategy.
For the correlation between the enemy's and the ball's y positions, the game for 100000 frames and collect the positions of the enemy and ball every 10 frames. We then compute different correlation coefficients, and obtain 99.6% Pearson coefficient, 96.4% as Kendall coefficient and 99.5% as Spearman coefficient.
let an random agent play
We also collected importance maps of deep DQN RL agents playing Pong using ReLU and Rational activation functions (borrowed from Delfosse et 2024c
20

---

Figure 10: More importance maps of DQN agents playing Pong with ReLU (left) and rational
activation functions (right).
Heatmaps of deep DQN agents
Original environment
with hidden enemy
Importance maps of trained deep DQN agents playing Pong (left) show that all 3 moving objects are important for the agent's 's decision, suggesting that the agent takes aims at sending the ball past the enemy. The same trained agent completely outperform its enemy in the original environment while cannot return the ball a modified version the game where the enemy is hidden.
Figure 11:
A.8
RL specific caveats
In this section, we give more details on the different RL specific caveats, as well
A.8.1 Ill-defined Objectives
Defining an accurate reward signal when creating an RL task is challenging, as it requires the designer specify precise and balanced reward function that effectively guides the agent towards the desired behavior while avoiding unintended consequences [Henderson et al., 2017 Irving et al. 2018). Shaping the reward after having observed (and understood) the non-intended behavior of the agent very common [Andrychowicz et al. (017]].
To realign our SCoBots agent, we provide them with reward signal proportional the progression to the joey:
player gget__aae_ohjictts._y_.ceeeee (gwneeoyggaaa_aeeeeeebee ["Player"])
Get current platform platform up. ceell xy [1] player. 16) 48) 0: topmost,
3: lovest platform
Encourage moving the child
if not episode_starts and not last bast_crashedi
if platform 2 == 0: even platform, encourage left movement reward = dlayer.dx
else: encourage right movement reward player. dx
11
13
13
Encourage upward movement reward = player / 5
else:
16
reward 0
21

---

03764
20
10
02244
10
18
Figure 12: iSCoBots learn to land in between the Flag to maximize its reward, when only RTT is provided.
A.8.2 Difficult Credit Assignment
When an outcome is delayed or uncertain, it is challenging to identify the action that is relevant for that outcome [Mesnard al. 2021]. This problem arises in games like chess, where reward is passed to the agent only when the game is over. The problem can be addressed by more instant informative feedback, given directly after successful actions or after positive state is reached.
In the main part of this manuscript, we address this problem on the game Skiing To encourage the agent go in between the flag, add reward that corresponds to the distance to both of the next flags (on the x axis):
R D(Plager, Flag1).x + D(Player, Flag2).x.
This resulted in an agent that learned to stop itself between the flags (depicted in Fig. 12). showing once again, how difficult is create reward that would favor specific behavior. To correct therefore adjoined another signal to reward our SCoBot agents proportionally to the speed of the agent:
R V(Player)
A.8.3 The reward sparsity of Pong
Let us move on to the issue of sparse reward in the context of RL. In Pong, to score point, the player needs to return the ball and have it go past the enemy. To do so, the player has to perform spiky shot, obtained by touching the ball on one of its paddle's sides (otherwise, the shot is flat and the enemy is easily catching it). Its vertical position may vary between 34 and 190. Its paddle height is 16, but needs to shoot from the side of the paddle (the pixel of each border) to get spiky shot that is likely to go beyond the opponent. If the balls arrives not close to the top or bottom borders, the enemy will still catch it, which has the probability of 22% in experiments. The probability of getting a successful shot is thus of ~ 6/156 * 0.78 = 3%. The average number of corresponding tryouts for the agent to get rewarded is Σx (n x 0.03 x 0.97" = 33.3. The agent usually need 60 steps (in average) from the initialization of the point to the reward attribution, hence, will need ~ 2200 steps to be rewarded. This is consistent with the experiment depicted in In this figure, random agent (original) needs in average 2230 steps to observe reward.
Figure B
Reward discovery In Pong
Original
60K
Assisted
Steps
40K
20K
0K
1
3
5
10
15
20
25
30
Observed rewards
Figure 13: Expert user feedback allow for faster discovering of the reward signal in the sparse reward Pong environment.
Providing a SCoBot agent with an additional reward that inversely proportional the distance between its paddle and the ball incentivizes the agent to keep a vertical position close to the ball's
22

---

one:
Rap D(Player, Ball).y
(8)
This extra reward signal lets starting agent winning shot every ~ 820, multiplying by 2.7 their occurrences. Thus, the reward signal in Pong relatively sparse, but our example illustrates how the interpretable concepts allow to easily guide SCoBots, making up for the reward sparsity. Interestingly, this also giving a clearer incentive to the agent to concentrate on the ball's position, and not on the enemy's 's one. We observe that such agent do not rely on the enemy to master the game. Alternative techniques such as prioritized experience replay (Schaul et al. 2016 allow offline methods to learn in such environments, but providing smoother reward signal is another elegant way to address sparsity. Note that sparser environments exists (such as robotics ones).
A.8.4 Misalignment
In our experimental evaluations, used the concept based reward shaping to realign the agents the true (target) task objectives, preventing undesirable behaviors resulting from misaligned optimization. Shortcut learning can be mitigated through the implementation of such reward signals that necessitate the genuine objective's consideration.
A.9 Computational load
We here provide the overall computational walltime for training each agent type, each Atari environment. When focusing on limited number of objects, their computational time is nearly halved, particularly in environments with many objects.
SCoBots
Deep
SCoBots (NG)
Asterix
07 08
08 07
12 12
Bowling
07 17
13
08
Boxing
07 26
11 20
07
Freeway
07 34
11 13
09: 44
Kangaroo
06 01
10 43
21 21
Pong
05:42
10: 55
07 41
Skiing
07 02
10 37
08
Seaquest
08 : 53
10 51
25 12
Tennis
11 19
12 08
11 25
Mean
07 35
10 47
12 27
Max
11 19
12 08
25 12
Min
05 42
08 07
07 :22
Table SCoBots (with OCAtari) train faster than deep agents, particularly in environments with limited number of objects. Computational training time of each method on each used environment (format HH:MM).
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, claim that object-centric RL agents that successive concept bottlenecks can challenge deep RL ones, and that they allow for easy detection and corrections of diverse RL-specific caveats. This is shown in Figure 3 both for the performances, as well as for the interpretability and correction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation, as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses SCoBots' limitations in the Limitation section included in the main paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
24

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper presents the experimental setup and evaluation process in detail, both in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
25

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The paper includes all information necessary reproduce the experimental results and provides open access to a git code repository (using anonymous- github until publication, then the code will be publicly available on GitHub).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mtiiss.cc/public//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup is described in the main paper, all hyperparameter values are listed in detail in the appendix and the code available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are reported in the main paper, detailed numerical results included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources and time required are listed in detail in Appendix A5. The total compute time for our experiments was approximately 1200 CPU+GPU hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///aaaaaeccccooo//////cccaa
Answer: [Yes]
Justification: We discuss the Potential Harms and negative societal impacts that interpretable algorithms provide, as misintended person with discriminative behavior could increase the misalignment of such agents. We hope that overall, the transparency of agents' decisions will result in positive societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses societal impacts in an impact statement included in the main paper.
27

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We simply train transparent agents Atari environments, do not societal negative impact that these agents could bring.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite every creator of the used environments and RL training repositories, all which are publicly available and freely distributed.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.paperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our code is commented, but this does not constitute an asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "IHi5umBI8A", "reviewer": "Reviewer_KTbS", "summary": "This paper proposes Successive Concept Bottleneck Agents that takes an object oriented view on the environment and construct interpretable policies based on objects and associated properties.", "questions": "- How would you handle more complex relationships like those cooperative behaviors in multi-agent games?\n- In POMDP setting, would you allow the relational functions reveal information on previously unobserved properties? If not, how would you plan to handle them?", "limitations": "See weakness.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed method is well motivated and has strong empirical performance.\n- The object oriented idea has been long existed in the RL literature but not until this paper, with slight help of LLM, the idea of object oriented RL unleashed its real power.", "weaknesses": "- Would like to see without the help of OCAtari, how the proposed pipeline would be affected by inaccurate object detectors and how this method could work in the wild, e.g., real life robotics domain.\n- Instead of distilling neural networks, to achieve better faithfulness and interpretability against the policy network, it would be better to also show results with Calgary Aytekin's NN to DT conversion algo directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their high appreciation of our paper and valuable feedback. We are glad that they found our method *well motivated* and it's *strong empirical performances*. We address the reviewers' raised points hereafter.\n\n**Add an experiment with inaccurate object detectors:**\n\nWe first want to stress that the object detectors included in OCAtari are imperfect (in terms of both F-scores and IoU). Nevertheless, we have conducted a novel experiment, with additional noise (of 5% probability of misdetection and gaussian noise on the position of standard deviation of 3 pixels).\n\nThe results are in the PDF, together with another ablation study, using agents with no relational concept bottleneck. \\\n\n\nWhile the results show that the policy is still mainly able to maintain correct performances besides the added noise, Kalman Filters could further help stabilize the policies. We are interested in robotic domains, we plan on expanding our work to integrate detection methods such as Segment Anything methods (maybe with Kalman Filters). We think that this will require adaptations like the use of control policies as options (i.e. high level actions), as we have added in our future work paragraph.\n\n \\\nThis suggestion experiment indeed helped us to show that concept bottleneck agents can be used in more complicated environments. \n\n**Use Calgary Aytekin's NN to DT conversion algorithm:**\n\nWe have not been able to reproduce this algorithm to directly extract decision trees. We, however, believe that the decision tree that it will provide will be larger than those of VIPER, which reduces the sampling space. As shown by the authors, the decision tree obtained on the simple two moons dataset already contains many nodes. If the reviewer knows of any other implementation (than [the one on this github page](https://github.com/CaglarAytekin/)), we would be happy to try another algorithm again. \n\nWe have also tried the REMIX algorithm for rule extraction, but they both led to bad performing agents. \n\nWe can add this comment in the paper. \n\n\n### Questions:\n\n\n\n1. How would you handle more complex relationships like those of cooperative behaviors in multi-agent games? \\\nWe have asked an LLM to provide such functions, simply prompting it with: \\\n\"_What about more complex relationships like those of cooperative behaviors in multi-agent games ? Can you think of any function that might be required for these cases ?_\"  \\\nIts answer provided python functions to evaluate and perform:  \n* _Communication Signal_,  \n* _Collective Goal Achievement_, \n* _Formation Maintenance_,  \n* _Role Assignment_,  \n* _Tactical Retreat or Advance_,  \n* _Resource Allocation Among Agents_,  \n* _Cooperative Targeting_,  \nwhich showcase abilities to use LLM in Multi Agents RL as well. \nWe believe that when provided with more context about the task's objectives, the relational functions will be even more accurate. We can as well add this to the appendix.\n2. In POMDP setting, would you allow the relational functions to reveal information on previously unobserved properties? If not, how would you plan to handle them?\n\n    Many approaches can be taken. In POMDP, one could indeed maintain a state of beliefs at the relational level as well, from which one could deduce partially occluded properties. For example, an occlusion relation could be utilized, allowing to deduce if an object is likely to be situated behind another one. We are now discussing this advantage in our manuscript. Thank you for the hint!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsVP9zuAgy", "reviewer": "Reviewer_5yYA", "summary": "The paper introduces Successive Concept Bottleneck Agents (ScoBots) that utilizes concept bottleneck models to enhance interpretability and decision-making. SCoBots incorporate successive concept bottleneck layers to integrate relational and object-based concepts for action selection. Unlike traditional deep RL agents, SCoBots allow for the inspection and revision of decision policies at various levels, from individual object properties to relational concepts and ultimately to action choices. The authors provide experimental evidence suggesting that SCoBots perform comparably to deep RL agents while offering significant advantages. Notably, SCoBots can justify their actions and address several RL-specific issues such as reward sparsity and goal misalignment through straightforward guidance. During evaluations, SCoBots identified and corrected a previously unknown shortcut behavior in deep RL agents playing Pong, highlighting the potential of this approach to resolve significant challenges in RL through relational concept-based models.", "questions": "1. Why not study more realistic tasks like vision-based manipulation?\n2. What are the model structure and objective functions for learning the extractors (relation and object ones)? \n3. How to deal with the state space that can not be represented by objects (e.g., windy grid world, 3D robot hand)?\n4. How to resolve the fidelity issues coming from the added rewards?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper is generally straightforward and well-organized. The structure is clear, and most concepts are clearly defined. The major contribution is easy to understand and follow.\n2.\tThe interpretability offered by SCoBots is user-friendly and easy to comprehend. Leveraging object-oriented conceptions, domain experts can conveniently identify and address inherent errors and issues.\n3.\tThe experimental results show the advantages of SCoBots from multiple perspectives (Q1 to Q3). The issues addressed in these experiments are crucial for developing effective decision-making policies in RL. The results are presented clearly and effectively.", "weaknesses": "1.\tMy major concern is about the **environments that the paper studies**. This paper primarily focuses on the application of models to Atari games. This kind of evaluation was popular in explainable RL a few years ago. However, regardless of how promisely the model can interpret Atari games, its practical relevance remains questionable. People gradually realize one cannot depend on an Atari solver to execute any realistic tasks. In today's context, to ensure that reinforcement learning (RL) has a real impact, the RL community has begun to explore tasks related to Embodied AI, such as vision-based manipulation and autonomous driving. These areas are where RL can truly be profitable, offering more than just conceptual benefits. I strongly recommend that the authors investigate these tasks, even within simulators like RoboGen and RoboCasa, which are publicly available. Pursuing research in these environments can ensure that Explainable Artificial Intelligence (XAI) in RL achieves significant, practical impact beyond mere conceptional advantages.\n2.\tOne of the major innovations, **the relation extractor, is not adequately defined in the paper**. It is unclear whether this component is based on a specific model structure, what its objective function for learning is, or if it involves manually designed functions. I briefly scanned the appendix but was unable to find these details. Please clarify these aspects and consider including them in the main body of the paper for better accessibility and understanding.\n3.\tTo write a self-contained paper, the author should **clarify which object extractor is employed by the authors**. Specifically, details the objective of this extractor and describes how the parameter $\\omega_{\\theta_1}$ is learned from image inputs, whether through self-supervised or unsupervised methods. This information will help readers better understand the methodology and its implementation.\n4.\tLine 69 contains an incomplete definition of the Markov Decision Process (MDP). It **omits essential components** such as the initial state distribution and the discount factor. Please include these elements to provide a comprehensive description of the MDP.\n5.\t**Not all state spaces in sequential decision processes can be accurately represented using object-level information**. Such transformations may result in the loss of critical information. Typically, assumptions of an object-oriented MDP are made before conducting these transformations. I recommend that the author explore the literature on object-oriented MDPs, particularly focusing on how objects and their relationships are modeled using graph neural networks. For instance, consider the work by Carlos Diuk, Andre Cohen, and Michael L. Littman titled \"An object-oriented representation for efficient reinforcement learning,\" presented at the Twenty-Fifth International Conference on Machine Learning (ICML 2008). This paper provides valuable insights into object-oriented approaches in reinforcement learning and could greatly enhance the theoretical foundation of your research.\n6.\t**The set of relations is fixed** before the learning phase, based either on human knowledge or AI-derived insights (as referenced in the Python code on line 100 and limitations). However, this set is not adaptable to new or previously unobserved relations in new or partially observable environments. For instance, both the set of relations and the corresponding action selectors are tailored to specific tasks, such as Pong, and may not generalize well to other contexts.\n7.\tThe introduction of **additional reward signals** (line 169) can significantly **affect the optimal policy in an MDP**. There is no guarantee that the optimal policies before and after the addition of these signals will be identical. These signals can alter the behavior of policies, which in turn changes the interpretations associated with them. Consequently, the **fidelity of the resulting interpretation becomes a critical issue to consider.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their dedicated effort to help us improve our manuscript. Let us address the raised concerns.\n\n**1. Limited evaluation (Atari)**: ALE environments are light, incorporate a diverse set of challenges, and are still heavily used by RL practitioners. E.g., searching \"reinforcement learning\" together with \"robogen\" on Scholar, since 2023 leads to 51 results, 6 with \"robocasa\", and 9330 with \"atari\". \nMoreover, our evaluations highlight that even these simple tasks already lead to agents that learn spurious correlations, as acknowledged by reviewer Zj4A (\"*the experiments do a good job of demonstrating advantages of this approach*\"). Specifically SCoBots interpretability allowed us to detect and correct these, as well as other RL specific caveats, (e.g. sparse reward, often encountered in robotic environments).\nAdditionally, we evaluate relational reasoning, and ALE games overall include many interacting objects, thus making it a valid choice for evaluations. Robotic environments are indeed more applicable, but they are typically used for learning accurate controls (e.g. the tasks of RoboGen, and Pick and Place, Turning levers, Twisting knobs of RoboCasa). \nControl policies are complementary to our work, and can be used as high level actions together with our SCoBots higher level reasoning system.\nWe are indeed planning to extend SCoBots to robotic environments for future work, e.g., for tasks involving relational thinking such as the Composite Tasks from RoboCasa. But this will require much adaptive work. We are adding this discussion to the paper and thank the reviewer for the hint.\n\n**2. Improve the relation extractor definition.**\nWe are now extending the definitions of Diuk et al.:\nUsing their definitions of classes, $C_i$, explicitly reintroduced in our manuscript.\nThe relation extractor augments the object centric space, by applying interpretable expert/LLM provided python functions to every detected object. We extend Diuk et al.'s relation definition from boolean relations (e.g. touch) to real ones (e.g. distance). \nFormally, $r: C_i \\times C_j \\rightarrow \\mathbb{R}$ is a function, defined over the attributes of objects of classes $C_i$ and $C_j$. \nThe relation extractor is meant to provide richer interpretable states, from which an action selection is easy to derive (allowing for compact interpretable policies). The functions are here not optimized through an explicit loss, but are pruned at tree extraction time. \nWe added these precisions to our manuscript. Thank you helping us improving the clarity of our method.\n\n**3. Which object extractor is employed ?**\nAs said in the evaluation (l.205): \"We focus our evaluations on the agent’s reasoning process, rather than the object identification\". We thus used the imperfect light-weight object extraction of OCAtari. For a fairer comparison to deep RL agents, we have conducted additional evaluations, with noisy object extraction (detailed in the PDF and global response).\n\n**4. Incomplete MDP definition.**\nThank you for pointing this out, we have completed this definition (l. 69).\n\n**5. Object-centric states are limited.**\nWe agree with this remark. We discuss it in the limitation section (cf. l. 320). To address these cases, we can either find suitable neuro-symbolic representations, or combine the symbolic module with a deep one for cases where symbolic reasoning falls short. We updated our limitation section, thank you.\n\n**6. Fixed set of relations.**\nIndeed, we have not touched upon continual RL. However, our setup is interactive (human the loop RL) so this set can be updated by the expert/LLM if the current one is insufficient. E.g. if an interpretable agent performs worse than the deep baseline, or new object classes are observed, the expert/LLM can be asked to extend the set of available functions. This represents an interesting future work opportunity. We have added this discussion.\n\n**7. Additional reward signals can affect the optimal policy.**\nModifying the MDP may indeed change the optimal policy, but we report the scores using the original reward signal, thus show that guidance allows for better performing policies in the original MDP. Concerning misinterpretation, our SCoBots policies are completely transparent: one can follow the decision tree (that contains relations and properties of each detected uniquely identified object). Contrary to e.g. importance maps, our method does not imply interpretation hypotheses, but each decision at every time step is fully transparent, independently of the training MDP. We are completing our manuscript to avoid confusion.\n\n**Questions**\n1. Study more realistic tasks like vision-based manipulation?\nSee our detailed answer above (1).\n\n2. What are the model structure and objective functions?\nSee our detailed answer above (2).\n\n3. How to deal with the state space that can not be represented by objects? \nWe actually have a broader definition of objects, which could also be referred to as entity. In this manner, the wind can be represented as an entity (with the characteristic direction and speed). Now if the wind is explicitly measurable, e.g., displayed on the rendered state, the object/concept extractor could extract it as well. For interactive properties that are not explicitly rendered (e.g., wind or invisible wall), we can extend our object detector to deduce properties based on the difference between the expected state and the observed state. We have added this discussion to our limitations.\n\n4. How to resolve the fidelity issues coming from the added rewards?\nSCoBots have a transparent pipeline, from the detected objects, to each evaluated condition in the decision tree. For example, in the Skiing state depicted in Fig. 4, the agent decides to select RIGHT because the distance between the player and the flag (on the x-axis) is bigger than 15. The interpretable decision is thus readable, independent of the MDP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mi86LK8IT6", "reviewer": "Reviewer_Zj4A", "summary": "The authors introduce SCoBots, an explainable RL approach that first extracts symbols from images, extracts relations between the objects, and then uses them to train a distilled decision tree policy. They demonstrate better performance than naive CNN-based deep RL agents in Atari games.\n\nFor full disclosure, I have reviewed this paper before.", "questions": "See weaknesses", "limitations": "Adequately addressed", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Strengths\n\n**Clarity:** Overall the paper is clear and very well written. \n\n**Method:** This method is pretty intuitive and seemingly novel in the RL context.\n\n- It’s also quite an interesting approach to enable explainable RL.\n- generating relational features with LLMs is a smart idea to reduce human burden and per-environment tuning\n\n**Performance:** Performance of SCoBots matches or surpasses that of image-based deep RL agents in Atari.\n\n**Experiments:** While I think there’s some issues with the experiments (see weaknesses below), overall the experiments at a high level do a good job of demonstrating advantages of this approach.", "weaknesses": "### Weaknesses\n\n**Missing Baselines/Ablations:** \n\n- A baseline/ablation that are missing that actually were addressed in the last time I reviewed this paper but somehow left out of even the appendix of this version:\n    - A missing comparison is that of deep rl agents trained with the presumed object detector that SCoBots assumes access to (not the relational extractor, as that’s a component introduced in this method for this setting). This would be an important comparison because object detectors are pretty standard and very easy to give deep RL agents access to (essentially, give it the same state space as SCoBots). **The authors presented an ablation with noisy object detectors last time in the rebuttal, this should probably be highlighted in the paper.**\n\n**Minor issues:**\n\n- Another prior work that also demonstrates being able to have humans intervene on (program) policies given explicitly extracted concepts: Learning to Synthesize Programs as Interpretable and Generalizable Policies, Trivedi et al. 2021. **The authors claimed to have added both works i suggested in the rebuttal last time, they did add one of them (VIPER).**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer, again, for their valuable time and feedback. We are glad that they found our paper _very well written_, our method _quite interesting_ and that our experiments _do a good job of demonstrating advantages of [our] approach_.\nLet us now address the reviewer's concerns.\n\n**Add missing ablation studies.**\n\nBefore adding these ablations, we wanted to assess their soundness by training agents with the CNN/VAE based object extractors from [1]. We conducted prechecking experiments training neural SCoBots on Pong and Boxing using their object extraction methods and obtained comparable results to the one with our simulated noise. \n\nDue to time constraints and limited writing space, we excluded these experiments from the initial submission. \nWe have summarized our results in the one-page PDF attachment. \nWe are of course incorporating these ablation studies into the main body of the manuscript, along with the following new questions, which are addressed in greater details in our manuscript:\n\n**Q4) Can object-centric agents learn with imperfect object extractors ?** \n\nYes, SCoBots policies can compensate for noisy object extraction, as the added noise only leads to slight performance drops or to no performance issue in our experimental evaluation. We believe that the use of robustness techniques such as Kalman filters can help to stabilize further the performances of such agents.\n\n**Q5) Can SCoBots learn without the relational concept bottleneck ?**\n\nYes, the neural SCoBots are mostly able to recompute the relations provided to the SCoBots, and obtain comparable performances to the neural SCoBots that use relations. However, the fully interpretable decision tree based SCoBots already suffer some performance loss, and most importantly, agents that do not use relations are more difficult to interpret, thus leading to more difficult misalignment detection and corrections.\n\n[1] Delfosse, et al. \"Boosting object representation learning via motion and object continuity.\" _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Cham: Springer Nature Switzerland, 2023.\n\n\n**Missing related work.**\n\nWe apologize for this. We apparently have forgotten this related work, in the submitted version of the manuscript. We added it to another paragraph on interpretable/transparent RL methods, placed in our related work section together with further recent publications about interpretable RL. \nDo not hesitate to provide us with any other reference that we might have missed, we will gladly add them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohpIlCNSKt", "reviewer": "Reviewer_udPa", "summary": "This paper provides evidence for the benefits of concept-based models in RL tasks, especially for identifying issues like goal misalignment. The proposed SCoBot integrates relational concepts into decision processes with minimal feedback, which not only brings the advantage of enhanced human understanding (Object Representations) but importantly enables targeted human-machine interactions (Relational Concepts). Experimental results show SCoBots' competitive performances and the potential for domain experts to understand and regularize their behavior.", "questions": "See above.", "limitations": "If the user does not understand the task and cannot define the concepts and the relation functions well.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The method is novel. It not only applies the concept bottleneck to RL tasks, but also further transforms it into an understandable and editable function through LLMs and returns a reward signal.\n* The expert reward signal is a reasonable intrinsic reward to solve sparse reward issues.\n* The visualization of the experiment makes it easy to understand.", "weaknesses": "* Will the space of Object Representations and the types of Relational Concepts change along with the change in the environment? If so, how to solve it?\n* Is the concept learned through supervised learning? What if there is no concept label?\n* What if the generated relation function keeps having errors by LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their work and their appreciation of our manuscript, notably for highlighting the novelty of our method and the clarity of our visualizations. We are also adding to our manuscript new ablation studies. The reviewer can find them in the general answer and in the attached PDF. \nLet us address the opportunities for improvement provided by the reviewer:\n\n**How to adapt to new objects types that could require new relations to consider?**\n\nTo extend our method to continual RL settings, i.e. if novel objects are detected, the object detector would provide a new object instance. We could incorporate a mechanism to automatically reprompt the LLM/expert to ask them if this novel object type might necessitate novel relations, and train our SCoBots with these novel objects and relations.\n\n**Is the concept learned through supervised learning? What if there is no concept label?**\n\nOur work indeed assumes access to the objects' properties and labels, and/or to the task context. Interpreting the policies would be difficult without human understandable concepts. However, there exists work in the field of predicate discovery (or predicate invention) [1,2,3,4,5]. Such methods can unsupervisely identify useful relations without access to any label. This represents a valuable future work avenue, so we are adding this discussion to our manuscript.\n\n**What if the LLMs generated relation function keeps having errors?**\n\nIn our experimental settings, this has not been observed. A simple first solution here would be to use code checkers, and reprompt the LLM if the code checker returns an error. There also exists research works on more complex approaches to ensure that LLMs provide working code [6,7,8]. We believe that these could also be incorporated in SCoBots.\n\n[1] Clarke, Edmund, et al. \"Counterexample-guided abstraction refinement.\" _Computer Aided Verification: 12th International Conference_, 2000.\n\n[2] Athakravi, Duangtida, Krysia Broda, and Alessandra Russo. \"Predicate invention in inductive logic programming.\" _2012 Imperial College Computing Student Workshop_.\n\n[3] Hocquette, Céline, and Stephen H. Muggleton. \"Complete bottom-up predicate invention in meta-interpretive learning.\" _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_. 2021.\n\n[4] Silver, Tom, et al. \"Predicate invention for bilevel planning.\" _Proceedings of the AAAI Conference on Artificial Intelligence_. 2023.\n\n[5] Sha, Jingyuan, et al. \"EXPIL: Explanatory Predicate Invention for Learning in Games.\" _arXiv preprint_ (2024).\n\n[6] Liu, Jiawei, et al. \"Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.\" _Advances in Neural Information Processing Systems_ (2024).\n\n[7] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" _International Conference on Machine Learning_. PMLR, 2023. \n\n[8] Kwon, Minae, and Sang Michael. \"Reward Design with Language Models.\" _International Conference on Learning Representations (ICLR)_. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We first thank all the reviewers for their time and valuable feedback. 

The reviews are highly appreciated as they raise very interesting points, which ultimately enable us to further improve our manuscript.

Reviewers **KTbS**, **Zj4A** and **udPa** point out the beneficial and interesting role of LLMs for handling relational functions. 

This is great feedback for us as we are very excited about this LLM integration.

Reviewer **5yYA** raised several points concerning formal and methodological clarity, which was very helpful as we strive to make our contribution as clear and as coherent as possible. 

In addition, we appreciate this reviewer's feedback of extending the context ScoBots are applied to with the highly interesting context of robotics and its implications.

The clear presentation and comprehensive evaluation are very important to us, therefore we appreciate the positive feedback expressed by all four reviewers in this regard.

As requested, we are providing two additional experiments that are summarized in Figure 1.

The first experiment measures SCoBots' robustness to learn with noisy object extraction methods.

The second one ablates SCoBots' relational concepts bottleneck. These allow us to answer the following two additional questions in our manuscript, here concisely answered:

**Q4**: Can object-centric agents learn with imperfect object extractors?

Yes, SCoBots agents are able to learn using imperfect object extractors. Using extra noise (5% probability of object misdetection and Gaussian noise, with standard deviation of 3px on the x, y position) on the imperfect OCAtari detectors, SCoBots are still able to learn Robust policies only leads to small performance reduction in 4 environments, and no noticeable one in the 5 others. We believe that the incorporation of stabilization techniques (such as Kalman filters) can further help to reduce the performance reduction.

**Q5**: Can SCoBots learn without the relational concept bottleneck?

The neural versions of SCoBots (i.e. SCoBots before decision tree extraction) can implicitly learn to reconstruct most relations within their neural network, we thus observe similar performances to the baseline. However their interpretable versions (that use decision) show performance drops on 7 out of 9 environments. Moreover, this ablation reduces the interpretability and ease of correcting misbehavior.

Meta-review: Reviewers generally agreed that this paper is clear, well written and novel. After some improvements by the authors, the primary weakness identified by any of the reviewers is the difficulty of the benchmark involved (Atari). While I agree that striving for more realistic benchmarks should always be encouraged, considering the novelty of this paper, I believe more realistic benchmarks can be left for future work.
Decision: Accept (poster)