Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents

Authors: quentin  delfosse
sebastian  sztwiertnia
mark  rothermel
wolfgang  stammer
kristian  kersting
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:16:42

Abstract: Goal misalignment, reward sparsity and difficult credit assignment are only a few of the many issues that make it difficult for deep reinforcement learning (RL) agents to learn optimal policies. 
Unfortunately, the black-box nature of deep neural networks impedes the inclusion of domain experts for inspecting the model and revising suboptimal policies.

To this end, we introduce Successive Concept Bottleneck Agents (SCoBots), that integrate consecutive concept bottleneck (CB) layers. 
In contrast to current CB models, SCoBots do not just represent concepts as properties of individual objects, but also as relations between objects which is crucial for many RL tasks. 

Our experimental results provide evidence of SCoBots' competitive performances, but also of their potential for domain experts to understand and regularize their behavior. Among other things, SCoBots enabled us to identify a previously unknown misalignment problem in the iconic video game, Pong, and resolve it. Overall, SCoBots thus result in more human-aligned RL agents.

Main paper: 
 introduction: 
 Deep Reinforcement learning (RL) agents are prone to suffer from a variety of issues that hinder them from learning optimal or generalizable policies. Prominent examples are reward sparsity [Andrychowicz et al., 2017] and difficult credit assignment [Raposo et al., 2021 , Wu et al., 2024] . A more pressing issue is the goal misalignment problem. It occurs when an agent optimizes a different side-goal, aligned with the original target goal during training [Koch et al., 2021] , but not at test time. Such misalignments can be difficult to identify [di Langosco et al., 2022] . For instance, in this work we discover that such a misalignment can occur in the oldest and most iconic video game, Pong (cf. Fig. 1 ). In Pong, the agent's target goal is to catch a ball with its own paddle and to return it passed the enemy's one. The enemy is programmed to constantly follow the ball, thus, agents can learn to focus on the position of the enemy paddle for placing their own, rather than the position of the ball itself. If left unchecked, such shortcut learning [Geirhos et al., 2020] may lead to a lack of model generalization and unintuitive failures e.g. at deployment time [Zhang et al., 2021] .Recently eXplainable AI (XAI) methods have emerged to detect such shortcut behavior, by identifying the reasons behind a model's decisions [Schramowski et al., 2020 , Roy et al., 2022 , Saeed and Omlin, 2023] . However, many XAI methods' explanations do not faithfully present the model's Figure 1 : Successive Concept Bottlenecks Agents (SCoBots) allow for easy inspection and revision. Top: Deep RL agents trained on Pong produce high playing scores with importance map explanations that suggest sensible underlying reasons for taking an action ( ). However, when the enemy is hidden, the deep RL agent fails to even catch the ball without clear reasons ( ). Bottom: SCoBots, on the other hand, allow for multi-level inspection of the reasoning behind the action selection, e.g., at a relational concept, but also action level. Moreover, they allow users to easily intervene on them ( ) to prevent the agents from focusing on potentially misleading concepts. In this way, SCoBots can mitigate RL specific caveats like goal misalignment.underlying decision process [Chan et al., 2022] . For example, importance-map explanations indicate the importance of an input element without indicating why this element is important [Kambhampati et al., 2021 , Stammer et al., 2021 , Teso et al., 2023] . A recent branch of research therefore focuses on models that provide inherent concept-based explanations. Prominent examples are concept bottlenecks models (CBMs), which provide predictive performances on par with standard deep learning approaches for supervised image classification [Koh et al., 2020] . More importantly, CBMs allow to identify and revise incorrect model behavior on a concept level [Stammer et al., 2021] .Contrary to exising CBMs' fields of applications, RL requires relational reasoning [Kaiser et al., 2019] . In this work, we therefore introduce Successive Concept Bottleneck Agents (SCoBots, cf. Fig. 1 , bottom) bringing the concept bottleneck approach to RL. SCoBots integrate successive concept bottlenecks into their decision processes, where each bottleneck layer provides concept representations that integrate the concept representations of the previous bottleneck layer. Specifically, provided a set of predefined concept functions, SCoBots automatically extract relational concept representations based on objects and their properties of the initial bottleneck layers. Finally, the set of relational and object concept representations are used for optimal action selection. SCoBots thus represent inherently explainable RL agents that, in comparison to deep RL agents, allow for inspecting and revising their learned decision policies at multiple levels of their reasoning processes: from single object properties, through relational concepts to the action selection.Our evaluations on the iconic Atari Learning Environments (ALE, Mnih et al. [2013] ) provides experimental evidence that SCoBots perform on par with deep RL agents. More importantly, we showcase SCoBots' ability to provide valuable explanations and the potential of mitigating a variety of RL specific issues, from reward sparsity to misalignment problems, via simple guidance from domain experts. We identify previously unknown shortcut behavior of deep agents, even on the simple Pong game. By utilizing the interaction capabilities of SCoBots, this behavior is easily corrected. Ultimately, our work illustrates the severity of goal misalignment issues in RL and the importance of being able to mitigate these and other RL specific issues, via relational concept based models. In summary, our contributions are: (i) We introduce Successive Concept Bottleneck agents (SCoBots).(ii) We show that SCoBots allow to inspect their internal decision processes. (iii) We show that their inherent inspectable nature can helps identifying unknown misalignments. (iv) We show that they allow for human interactions for mitigating various RL specific issues.We proceed as follows. We introduce SCoBots and discuss their specific properties. We continue with experimental evaluations and analysis. Before concluding, we touch upon related work. (SCoBots) . SCoBots decompose the policy into consecutive interpretable concept bottlenecks (ICB). Objects and their properties are first extracted from the raw input, human-understandable functions are then employed to derive relational concepts, used to select an action. The understandable concepts enable interactivity. Each bottleneck allows expert users to, e.g., prune or utilize concepts to define additional reward signals. 
successive concept bottleneck agents: 
 In this work, we represent RL problems through the framework of a Markov Decision Process, M =< S, A, P s,a , R s,a , γ >, with S as the state space, A the set of available actions, P (s, a) the transition probability, and R(s, a) the immediate reward function, obtained from the environment, and γ the didscount factor. Classic deep RL policies, π θ (s) = P (A = a|S = s) parameterized by θ, are usually black-box models that process raw input states, e.g. a set of frames, to provide a distribution over the action space [Mnih et al., 2015 , van Hasselt et al., 2016] .Concept bottleneck models (CBMs) initiate a learning process by extracting relevant concepts from raw input (e.g. image data). These concepts can represent the color or position of a depicted object. Subsequently, these extracted concepts are used for downstream tasks such as image classification. Formally, a bottleneck model g : x → c transforms an input x ∈ R D with D dimensions into a concept representation c ∈ R k (a vector of k concepts). Next, the predictor network f : c → y uses this representation to generate the final target output (e.g. y ∈ R for classification problems).The main body of research on CBMs focuses on image classification tasks, where the extracted concepts represent attributes of objects. In contrast, RL agents must learn not just from static data, but also through interaction with dynamically evolving environments. RL tasks thus often require relational reasoning, as they involve understanding the relationships between instances that evolve through time and interact with another. This is crucial for learning effective policies in complex, dynamic environments. Note that, in the following descriptions, we use specific fonts to distinguish objects' properties from their relations .An underlying assumption of SCoBots is that their processing steps should be inspectable and understandable by a human user. This stands in contrast to e.g. unsupervised CBM approaches Jabri et al. [2019] , Zhou et al. [2019] , Srinivas et al. [2020] , where there is no guarantee for learning humanaligned concepts. SCoBots rather take a different approach by dividing the concept learning and RL-specific action selection into several inspectable steps that are grounded in human-understandable concept representations. These steps are described hereafter and depicted in Fig. 2 .Similar to other RL agents, SCoBots process the last n observed frames from a sequence of images, s t = {x i } t i=t-n . For each frame, SCoBots need an initial concept extraction method to extract objects and their properties, as in previous works on CBMs [Stammer et al., 2021 , Koh et al., 2020] . Specifically, we start from an initial bottleneck model, ω θ1 (•), that is parameterized by parameter set, θ 1 . Given a state, s t , this model provides a set of c i object representations per frame, ω θ1 (s t ) = Ω t = {{o j i } ci j=1 } t i=t-n , where each object representation corresponds to a tensor of different extracted properties of that object, (e.g. its category, position (i.e. x, y coordinates), etc.). As done in previous works on CBMs, the bottleneck model of our SCoBots, ω θ1 , can correspond to a model that was supervisedly pretrained for extracting the objects and their properties from images.One of the major differences to previous work on CBMs is that SCoBots further extract and utilize relational concepts that are based on the previously extracted objects and their properties. Formally, SCoBots utilize a consecutive bottleneck model, µ F (•), called the relation extractor. This model, µ, is parameterized by a predetermined set of transparent relational functions, F, used to extract a set of d t relational concepts. These relations are based on each individual object (and its properties) for unary relations or on combinations of objects for n-ary relations, and are denoted as µ F (Ω t ) = Γ t = {g k t } dt k=1 . Without strong prior knowledge, F can initially correspond to universal object relations such as distance and speed . However, this set can easily be updated on the fly by a human user with e.g. additional, novel relational functions. Note that F can include the identity, to let the relation extractor pass through initial object concepts from Ω t to the relational concepts Γ t .Finally, SCoBots employ an action selector, ρ θ2 , parameterized by θ 2 , on the relational concepts to select the best action, a t , given the initial state, s t (i.e. the set of frames). Up to now, all extracted concepts in SCoBots, both Ω t and Γ t , represent human-understandable concept representations. To guarantee understandability also within the action selection step of SCoBots, we need to embed an interpretable action selector. While neural networks, a standard choice in RL literature, are performative and easy to train using differentiable optimization methods, they lack this required interpretability feature. However, Çaglar Aytekin [2022] have recently shown the equivalence between ReLU-based neural networks and decision trees, which in contrast represent inherently interpretable models. To trade-off these issues of flexibility and performance vs interpretability, SCoBots thus break down the action selection process by initially training a small ReLU-based neural network action selector, ρ θ ′ 2 , via gradient-based RL optimization. After this, ρ θ ′ 2 is finally distilled into a decision tree ρ θ2 . Lastly, note that one can add a residual link from the initial object concepts, Ω t , to the relational concepts, Γ t such that the action selector can, if necessary, also make decisions based on basic object properties, e.g., the height of an object.Overall, our approach preserves the MDP formulation used by classic deep approaches, but decomposes the classic deep policy st π θ -→ a t into a successive concept bottleneck one s t ω θ 1 --→ Ω t µ F --→ Γ t ρ θ 2--→ a t , where θ = (θ 1 , θ 2 , F) constitutes the set of policy parameters. For simplicity, we will discard the parameter notations in the rest of the manuscript. Further explanations of the input and output space of each module, as well as the properties and relational functions used in this work are provided in the appendix (cf. App.A.5 and App. A.6).Instead of jointly learning object detection, concept extraction, and policy search, SCoBots enable independent optimization of each policy component. Separating the training procedure of different components reduces the complexity of the overall optimization problem [Koh et al., 2020] .The inspectable nature of SCoBots not only brings the benefit of improved human-understandability, but importantly allows for targeted human-machine interactions. In the following, we describe two guidance approaches for interacting with the latent representations of SCoBots: concept pruning and object-centric rewarding. We refer to the revised SCoBot agents as guided SCoBots in the following.Concept pruning. The type and amount of concepts that are required for a successful policy may vary across tasks. For instance, the objects colors are irrelevant in Pong but required to distinguish vulnerable ghosts from dangerous ones in MsPacman. However, overloading the action selector with irrelevant concepts, whether these are object properties (Ω t ) or relational concepts (Γ t ), can lead to difficult optimization (e.g. the agent focusing on noise in unimportant states) as well as difficult inspection of the decision tree (ρ). Moreover, for a single RL task, the need for specific concepts might even change during training, in e.g. progressive environments (where agents need to master early stages before being provided with additional, more complex tasks [Delfosse et al., 2024c] ).SCoBot's human-comprehensible concepts therefore allow domain experts to prune unnecessary concepts. In this way, expert users can guide the learning process towards relevant concepts. Formally, users can (i) select a subset of the object property concepts, Ω. Additionally, by (ii) selecting a subset of the relational functions, F, or (iii) specifying which objects specific functions should be applied on, experts can implicitly define the relational concepts subset, Γ. Guided SCoBots thus formally refining the policy extraction to s tω θ 1 --→ Ω t - → Ω t µ F --→ Γ t ρ θ 2--→ a t . Furthermore, users can (iv) prune out redundant actions resulting in a new action space A.To give brief examples of these four pruning possibilities we refer to Fig. 2 , focusing here on the blue subparts. Particularly, in Pong a user can remove objects (e.g. scores) or specific object properties (e.g. R,G,B values) to obtain Ω t . Second, the color relation, color(•), is irrelevant for successfully playing Pong and can therefore be removed from F. Third, the vertical distance function (dist(•, •).y) can be prevented from being applied to the (player, enemy) input couple. These pruning actions provide SCoBots with Γ t . Lastly, the only playable actions in Pong are UP and DOWN.To ease the policy search, the available FIRE, LEFT and RIGHT from the base Atari action space might be discarded to obtain A, as they are equivalent to NOOP (i.e. no action).Importantly, being able to prune concepts can help mitigate misalignement issues such as those of agents playing Pong (cf. Fig. 1 ), where an agent can base its action selection on the enemy's position instead of the ball's one. Specifically, pruning the enemy position from the distance relation concept enforces SCoBot agents to rely on relevant features for their decisions such as the ball's position, rather than the spurious correlation with the enemy's paddle.Object-centric feedback reward. Reward shaping [Touzet, 1997 , Ng et al., 1999] EQUATIONwhere R exp : Γ -→ R and Γ t = µ(ω(s t )) is the relational state, extracted by the relation extractor.The coefficient α gt ∈ R is used to penalize or reward the agent proportionally to relational concepts.Our expert reward only relies on the state, as we make use of the concepts extracted from it. However, incorporating the action into the reward computation is straightforward. In practice, this expert reward signal can lead to guiding the agent towards focusing on relevant concepts, but also help smoothing sparse reward signals, which we will discuss further in our evaluations.For example, one can impose penalties, based on the distance between the agent and objects (cf. "expert reward" arrow in Fig. 2 ), with the intention of incentivizing the agent to maintain close proximity with the ball. As shown in our experimental evaluation, we can use this concept pruning and concept based reward shaping to easily address many RL specific caveats such as reward sparsity, ill-defined objectives, difficult credit assignment, and misalignment (cf. App. A.8 for details on each problem).Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discusses SCoBots' limitations in the Limitation section included in the main paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental evaluations: 
 In our evaluations, we investigate several properties and potential benefits of the transparent SCoBot agents. We specifically aim to answer the following research questions: (Q1) Are concept based agents able to learn competitive policies on different RL environments? (Q2) Does the inspectable nature of SCoBots allow to detect issues in their decision processes? (Q3) Can concept-based guidance help mitigate common RL caveats, such as policy misalignments? (Q4) Can SCoBots learn with imperfect object extraction methods? (Q5) How crucial is the the relation extractor for SCoBots performances?Experimental setup: We evaluate SCoBots on 9 Atari games (cf. Fig. 3 from the Atari Learning Environments [Bellemare et al., 2012] (by far the most used RL framework (cf. App. A.1), as well as the HackAtari modified [Delfosse et al., 2024a] Pong environments, where the enemy is not visible yet active (NoEnemy), and where the enemy stops moving after returning the ball (LazyEnemy). We provide human normalized scores (following eq. 3) that are averaged over 3 seeds for each agent configuration. We compare our SCoBot agents to deep agents with the classic convolutional network introduced by Mnih et al. [2015] , that process a stack of 4 black and white frames and denote these as deep agents in the following. Note, that we evaluate all agents on the latest v5 version of the environments3 to prevent overfitting. All agents are trained for 20M frames under the Proximal Policy Optimization algorithm (PPO, [Schulman et al., 2017] ), specifically the stable-baseline3 implementation [Raffin et al., 2021] and its default hyperparameters (cf. Tab. 2 in App. A.5). Human-normalized scores of different agents trained using PPO on 9 ALE environments, including deep agents (i.e. using CNNs), guided decision tree policy (SCoBots), their neural object-centric baseline (NN-), and these baselines without guidance (NG). SCoBots obtain similar or better scores than the deep agents, showing that object-centric agents can also solve RL tasks while making use of human-understandable concepts (left). Guiding SCoBots allow to correct misalignment in Pong (center) and to obtain the originally intended agents, depicted by a level completion score of 100% on the intended goal's evaluation in Kangaroo (right).We focus our SCoBot evaluations on the more interesting aspects of the agent's reasoning process underlying the action selection, rather than the basic object identification step (which has been thoroughly investigated in previous works e.g. [Koh et al., 2020 , Stammer et al., 2021 , Wu et al., 2024] ). We thus assume access to a pretrained object extractor and provide our agents with objectcentric descriptions of these states (Ω t ) based on the information from OCAtari [Delfosse et al., 2024b] . Specifically, in our evaluations, the set of object properties consists of object class (e.g. enemy paddle, ball etc.), (x t , y t ) coordinates, coordinates at the previous position (x t-1 , y t-1 ), height and width, the most common RGB values (i.e. the most representative color of an object), and object orientation. The specific set of functions, F, that are used to extract object relations is composed of: euclidean distance, directed distances (on x and y axes), speed, velocity, plan intersection, the center (of the shortest line between two objects), and color name (cf. App. A.5.1 for implementation details).To distill SCoBots' learned policies, we use the decision-tree extraction algorithm VIPER [Bastani et al., 2018] . For the human-guided SCoBot evaluations (denoted as (guided) SCoBots in our evaluations), we illustrate human guidance and prune out the concepts that we consider unimportant to master the game (cf. Tab. 4). Furthermore, to mitigate RL specific problems, we also provide SCoBot agents with simulated human-expert feedback signals, the details of which we describe at the relevant sections below. More details about the setup and the hyperparameters are in App. A.5. In our evaluation, SCoBots' training is slightly faster than deep agents' one (cf. Appendix A.9).We present human-normalized (HN) scores of both SCoBot and deep agents trained on each investigated game individually in Fig. 3 (left) . Numerical values are provided in Tab. 1 (cf. App. A.7).We observe that SCoBot agents perform at least on par with deep agents on all games, even slightly outperform these on 5 out of 9 (namely, Asterix, Boxing, Kangaroo, Seaquest and Tennis). Our results suggest that SCoBots provide competitive performances on every tested game despite the constraints of multiple bottlenecks within their architectures. Overall, our experimental evaluations show that RL policies based on interpretable concepts extraction decoupled from the action selection process can, in principle, lead to competitive agents.Inspectable SCoBots' to detect misalignments (Q2).The main target of developing ScoBots is to obtain competitive, yet transparent agents that allow for human users to identify their underlying reasoning process. Particularly, for a specific state, the inspectable bottleneck structure of SCoBots allows to pinpoint not just the object properties, but importantly the relational concepts being used for the final action decision. This is exemplified in Fig. 4 on Skiing, Pong and Kangaroo (cf. App. A.4 for explanations of SCoBots on the remaining games). Here we highlight the decision-making path (from SCoBots' decision tree based policies), at specific states of each game. For example, for the game state extracted from Skiing, the SCoBot agent selects RIGHT as the best action, because the signed distance from its character to the left flag is larger than a specific value (+15 pixels). Given the nature of the game this inherent explanation suggests that the agent is indeed basing its selection process on relevant relational concepts. The provided explanation suggests that the agent is basing its decision on the vertical positions of the enemy and of its own paddle (distance between the two paddles on the y axis). In fact, this suggests that the agent is largely ignoring the ball's position. Interestingly, upon closer inspection of the enemy movements, we observed that, previously unknown, the enemy is programmed to follow the ball's vertical position (with a small delay). Thus, the vertical positions of these two objects are highly correlated (Pearson's and Spearman's correlations coefficient above 99%, cf. App. A.7). Moreover, the ball's rendering contains flickering, explaining why SCoBots base their decision on this potentially more reliable feature, the enemy paddle's vertical position.To validate these findings further, we perform evaluations on 2 modified Pong environments in which (i) the enemy is invisible, yet playing (NoEnemy, cf. Fig. 1 ), and on one environment where the enemy is not moving after returning the ball (LazyEnemy). We reevaluate the deep and SCoBot agents that were trained on the original Pong environment on NoEnemy and observe catastrophic performance drops in HN scores (cf. Fig. 3 ) at a level of random policies for both types of agents.Evaluations with different DQN agents lead to similar performance drops (cf. App. A.7). These drops are particularly striking as initial importance maps produced by PPO and DQN agents highlight all 3 moving objects (i.e. the player, ball, and enemy) as relevant for their decisions (cf. Fig. 1 top left and App. A.7), aligned with findings on importance maps of Weitkamp et al. [2018] . These maps suggest that the deep agents had based its decisions on right reasons. Our novel results on the NoEnemy and LazyEnemy environments, however, greatly calls to question the faithfulness and granularity of such post-hoc explainability methods. They highlight the importance of inherently transparent RL models on the level of concepts, as provided via SCoBots, to identify such potential issues. In the following, we will investigate how to mitigate the discovered issues via SCoBot's bottlenecks.Guiding SCoBots to mitigate RL-specific caveats (Q3).We here make use of the interactive properties of SCoBots to address several famous RL specific problems: goal misalignment, ill-defined rewards, difficult credit assignment and reward sparsity.Realigning SCoBots: To mitigate the goal misalignment issues of the SCoBots trained on Pong, we simply remove (prune) the enemy from the set of considered objects (Ω). Thus, the enemy cannot be used for the action selection process. This leads to SCoBots that are able to play Pong and its NoEnemy version (cf. SCoBot in Fig 3 (center) ). Furthermore, this shows that playing Pong without observing the enemy is achievable, by simply returning vertical shots, difficult for the enemy to catch.Ill-defined reward: Defining a reward upfront that incentivizes agents to learn an expected behavior is a difficult problem. An example of ill-defined reward (i.e. a reward that will lead to an unaligned behavior if the agent maximizes the return) is present in Kangaroo. According to the documentation of the game4 , "the mother kangaroo on the bottom floor tries to reach the top floor where her joey is being held captive by some monkeys". However, punching the monkey enemies gives a higher cumulative reward than climbing up to save the baby. RL agents thus tend to learn to kick monkeys on the bottom floor rather than reaching the joey (cf. Fig. 4 ). For revising such agents, we provide an additional reward signal, based on the distance from the mother kangaroo to the joey (detailed in App. A.8). As can be seen in Fig. 3 (right), this reward allows the guided SCoBots to achieve the originally intended goal by indeed completing the level. In contrast, deep agents and particularly unguided SCoBots achieve relatively high HN scores, but do not complete the levels. Final human normalized scores (with stds) comparing SCoBots and the object-centric neural baselines (NN-SCoBots), with and without relations. We also provide the scores of NN-SCoBots that learned on noisy environments. The noise only noticeably affects the agents on Kangaroo. Ablating the relations is harmless on NN-SCobots, as neural networks can recompute them, but impacts SCoBots performances on 6 games. ( * For better visualization, we used a human score of 100 in Boxing.)Difficult Credit Assignment Problem: The difficult credit assignment problem is the challenge of correctly attributing credit or blame to past actions when the agent's actions have delayed or non-obvious effects on its overall performance. We illustrate this in the context of Skiing, where in standard configurations, agents receive at each step a negative reward that corresponds to the number of seconds elapsed since the last steps (i.e. varying between -3 and -7). This reward aims at punishing agents for going down the slope slowly. Additionally, the game keeps track of the number of flag pairs that the agent has correctly traversed (displayed at the top of the screen, cf. Fig. 4 ), and provide a large reward, proportional to this number, at the end of the episode. Associating this reward signal with the number of passed flags is extremely difficult, without prior knowledge on human skiing competitions. In the next evaluations, we provide SCoBots with another reward at each timestep, proportional to the agent's progression to the flags' position to incentivize the agent to pass in between them, as well as a signal rewarding for higher speed:R exp = o∈{F lag1,F lag2} D(P layer, o) t -D(P layer, o) t-1 + V (P layer).y.(2)These rewards are further detailed in App. A.8. They allow guided SCoBots to reach human scores, whereas the deep and unguided SCoBots perform worse than random (cf. Fig. 3 (left)). Note that providing additional reward signals, as done above, obviously also allows to mitigate reward sparsity, as we illustrate on Pong (cf. App. A.8.3 for a detailed explanation).Object-centric agents can learn with imperfect object extractors (Q4).To test object-centric agents' ability to work in more realistic environments, we have tested if they can learn viable policies with imperfect object extraction methods. Based on Delfosse et al. [2023b] 's detection results, we added a 5% misdetection probability and a Gaussian noise (of 0 mean and 3 pixels of standard deviation on each axis). As depicted in Figure 5 , NN-SCoBots (i.e. neural network based object-centric agents, using relations) learn comparable policies on all games but Kangaroo, demonstrating their ability to mitigate suboptimal object extraction methods. Implementing robustness techniques, such as Kalman filters, would help to further push the performances of such agents.Transparent SCoBots benefit from explicit relations (Q5).While the ablation of the relation extractor only has a significant impact on the neural-based objectcentric agents on 1 out of 9 games (Seaquest), it deteriorates the decision tree based SCoBots on 6 environments. This is due to the fact that relations can be implicitly recomputed within a neural network, but not within decision trees. As shown by Kohler et al. [2024] , the use of the distance relation (on a specific axis) allows for performing agents with compact decision tree-based policies. Furthermore, even if some relations can be rediscovered and implicitly encoded within the decision trees, the lack of explicit relational representations can reduce the interpretability of the agents, and will prevent the experts from using these within their guidance.Overall, our experimental evaluations not only show that SCoBots are on par with deep agents in terms of performances, but that their concept-based and inspectable nature allows to identify and revise several important RL specific caveats. Importantly, in the course of our evaluations, we identified a previously unknown and critical misalignment of existing RL agents on the simple and iconic Pong environment, via the previously mentioned properties of SCoBots. 
limitations: 
 On the use of OCAtari. To limit our resource consumption, we made use of the quasi-perfect object extractor (ω) of OCAtari, which efficiently extract objects from the RAM. We added an ablation to simulate potential imperfect detection capabilities of Atari object extraction methods. Such extractors can be optimized using supervised [Redmon et al., 2016 , Locatello et al., 2020] or self-supervised [Lin et al., 2020 , Delfosse et al., 2023b]  
related work: 
 The basic idea of Concept Bottleneck Models (CBMs) can be found in work as early as Lampert et al.[2009] and Kumar et al. [2009] . A first systematic study of CBMs was delivered by Koh et al.[2020], followed by Stammer et al. [2021] , who described CBMs as two-stage models that first computes intermediate representations used for the final task output. Where learning valuable initial object concept representations without strong supervision is still a tricky and open issue for concept-based models [Lage and Doshi-Velez, 2020 , Stammer et al., 2022 , Sawada and Nakamura, 2022 , Marconato et al., 2022 , Steinmann et al., 2023 , Stammer et al., 2024] , receiving relational concept representations in SCoBots is performed automatically via the function set F. Since the initial works on CBMs they have found utilization in several applications. Antognini and Faltings [2021] e.g. apply CBMs to text sentiment classification, and Kraus et al. [2024] to time series' analysis. However, these works consider single object concept representations and focus on supervised learning. The ability of users to revise concepts and decisions has also been parallelly shown by Friedrich et al. [2023] .In fact, CBMs have found their way into RL only to a limited extent. Zabounidis et al. [2023] and Grupen et al. [2022] utilized CBMs in a multi-agent setting, where both identify improved interpretability through concept representations while maintaining competitive performance. Zabounidis et al. [2023] further report better training stability and reduced sample complexity. Their extension includes an optional "residual" layer which passes additional, latent information to the action selector part. SCoBots omit such a residual component for the sake of human-understandability, yet offer the flexibility to the user to modify the concept layer via updating F for improving the model's representations. Similar to how SCoBots invite the user to reuse the high-level concepts to shape the reward signal, Guan et al. [2023] allow the user to design a reward function based on higher-level properties that occur over a period of time. Lastly, SCoBots not only separate state representation learning from policy search, as done by Cuccu et al. [2020] , but also enforce object-centricity, putting interpretability requirements on the consecutive feature spaces.Explainable RL (XRL) is an extensively surveyed XAI field [Dazeley et al., 2022 , Krajna et al., 2022 , Milani et al., 2023] with a wide range of unsolved issues [Vouros, 2022] . Milani et al. [2023] introduce a taxonomy for XRL methods, with: (1) feature importance methods that generate explanations that point out decision-relevant input features, (2) learning process & MDP methods which present which past experience or MDP components affect the policy, and (3) policy-level methods, describing the agent's long-term behavior. Based on this, SCoBots extract relations from low-level features, making high-level information available to explanations and thereby support feature importance methods.According to Qing et al. [2022] 's categorization of XRL frameworks, SCoBots are "model-explaining" (in contrast to reward-, state-, and task-explaining). Other XRL methods rely on LLM to explain the policy [Luo et al.] decision trees [Fuhrer et al., 2024 , Marton et al., 2024] , logic [Jiang and Luo, 2019 , Kimura et al., 2021 , Delfosse et al., 2023a , Sha et al., 2024] or programs [Verma et al., 2018 , Trivedi et al., 2021 , Cao et al., 2022 , Kohler et al., 2024 , Wüst et al., 2024] to encode transparent policies.To overcome the potentially unavailable concepts necessary to learn symbolic policies, Shindo et al.[2024] learn a mixture of symbolic and neural policies. Finally, concepts have further been used to derive reward from context using LLMs, as we did for Kangaroo and Skiing [Kwon et al., 2023 , Kaufmann et al., 2024 , Wu, 2024 , Shen et al., 2024] .The misalignment problem is an RL instantiation of the shortcut learning problem, a frequently studied failure mode that has been identified in models and datasets across the spectrum of AI from deep networks [Lapuschkin et al., 2019 , Schramowski et al., 2020] to neuro-symbolic [Stammer et al., 2021 , Marconato et al., 2023] , prototype-based models [Bontempelli et al., 2023] and RL approaches [di Langosco et al., 2022] . A misaligned RL agent, first empirically studied by [Koch et al., 2021] represents a serious issue, especially when it is misaligned to recognized ethical values [Arnold and Kasenberg, 2017] or if the agent has broad capabilities [Ngo, 2022] . Nahian et al. [2021] ethically align agents by introducing a second reward signal. In comparison, SCoBots aid to resolve RL specific issues such as the misalignment problem through inherent interpretability.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The paper includes all information necessary to reproduce the experimental results and provides open access to a git code repository (using anonymous-github until publication, then the code will be publicly available on GitHub). Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this work, we have provided evidence for the benefits of concept-based models in RL tasks, specifically for identifying issues such as goal misalignment. Among other things our proposed Successive Concept Bottleneck agents integrate relational concepts into their decision processes.With this, we have exposed previously unknown misalignment problems of deep RL agents in a game as simple as Pong. SCoBot agents allowed us to revise this issue, as well as different RL specific caveats with minimal additional feedback. Our work thus represents an important step in developing aligned RL agents, i.e. agents that are not just aligned with the underlying task goals, but also with human user's understanding and knowledge. Achieving this is particularly valuable for applying RL agents in real-world settings where ethical and safety considerations are paramount.Avenues for future research are incorporating a high level action bottleneck [Bacon et al., 2017] . One can also incorporate attention mechanisms into RL agents, as discussed in [Itaya et al., 2021] , or use language models (and e.g., the game manuals) to generate the additional reward signal, as done by Wu et al. [2024] . Additionally, we are considering the use of shallower decision trees [Broelemann and Kasneci, 2019 ]. An interesting research question is how far the task reward signal can aid in learning games object-centric representations [Delfosse et al., 2023b] in the first place. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We simply train transparent agents on Atari environments, we do not see societal negative impact that these agents could bring. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite every creator of the used environments and RL training repositories, all of which are publicly available and freely distributed. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our code is commented, but this does not constitute an asset.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our work aims at developing transparent RL agents, whose decision can be understood and revised to be aligned with the beliefs and values of a human user. We believe that such algorithms are critical to uncover and mitigate potential misalignments of AI systems. A malicious user can, however, utilize such approaches for aligning agents in a harmful way, thereby potentially leading to a negative impact on further users or society as a whole. Even so, the inspectable nature of transparent approaches will allow to identify such potentially harmful misuses, or hidden misalignment.As mentioned in the main body, the appendix contains additional materials and supporting information for the following aspects: further information on the fact that Atari is the most common set of games (A.1), details on the reward sparsity in Pong (A.8.3), detailed numerical results (A.2), learning curves of our RL agents (A.3), the hyperparmeters used in this work (A.5), formal definitions on the SCoBots policies (A.6) and further details on the misalignment problem in Pong.A.1 Atari games are most common set of environments We here show that the Atari games from the Arcade Learning Environment [Bellemare et al., 2012] is the most used benchmark to test reinforcement learning agents.For completeness, we here provide the numerical results of the performances obtained by each agent type. For fair comparison, we reimplemeneted the deep PPO agents, and used the default hyperparameters for Atari from stable-baselines3 for the deep RL agents. A detailed overview of hyperparameters and their corresponding values can be found in A.5. (3) In this section, we provide more decision trees from SCoBots on states from Freeway and Bowling. The Atari environment version used in gymnasium is NoFrameskip-v4 for agents reproducing the reported PPO results, and v5 for SCoBots and neural. v5 defines a deterministic skipping of 5 frames per action taken and sets the probability to repeat the last action taken to 0.25. This is aligned with recommended best practices by Machado et al. [2018] . The experiments using NoFrameskip-v4 utilize the same environment wrappers as OpenAI's initial baselines implementation5 . This includes frame skipping of 4 and reward clipping. Frame stacking is not used. SCoBots are not trained on a clipped reward signal. For comparability, the neural agents we compare SCoBots to, are not as well.A list of all hyperparameter values used is provided in Table 2 .In this paper, we used different properties and functions (to create features). They are listed hereafter. Further, Table 4 shows a detailed overview of the concepts and actions pruned for every environment evaluated.The set S denotes the problem-specific set of raw environment states as defined by the RL problem, e.g., the space of RGB images [0, 255] 512×512×3 . Similarly, A represents the action space, e.g., "move right" or "jump." detection space, defined as the family of object sets in which each object (identified by its ID) occurs once at most.✓ ✓ ✓ ✓ ✓ ✓ position ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ position history ✓ X ✓ ✓ ✓ ✓ ✓ ✓ ✓ orientation X X X X S O F R A ω µ ρF is the relation function space. It is the family of user-defined relation function setsF . Each relation function f ∈ F is of the form f : O k → R n for k, n ∈ N,that is, it maps a fixed number of objects to a real-valued vector. As an instance, the function that maps two objects to the Euclidean distance between each other is a relation function.The set R ⊆ R m is the relation (or feature) space. The dimension m is implied by 2. µ is the feature selectorF. More specifically, m = f ∈F dim(co(f )), i.e.,EQUATIONThat is, µ applies each relation function f ∈ F to the respective detected object(s) in O ∈ O, resulting in a real-valued relation vector.3. ρ : R → A is the action selector that assigns actions to relation vectors.See also Figure 9 for a summary. The policy parameters θ := (θ 1 , θ 2 , θ 3 ) split up into the parameters of the object detector, the feature selector, and the action selector, accordingly. They are left out for brevity.The architecture's bottleneck is induced by the object detector ω and the feature selector µ. From this function perspective, the bottleneck consists of two stages: an object-centric bottleneck stage at O and a semantic bottleneck stage at R.The misalignment problem had previously been identified in other games such as the Coinrun platform game [Cobbe et al., 2019] , in which an agent's target-goal is to reach a coin which is always placed at the end of a level at training time. When the coin is repositioned at test time di Langosco et al. observed that trained deep RL agents avoid the coin and simply target the end of the level, indicating that agents in fact learn to follow a simpler side-goal rather than the underlying strategy.For the correlation between the enemy's and the ball's y positions, we let an random agent play the game for 100000 frames and collect the positions of the enemy and ball every 10 frames. We then compute different correlation coefficients, and obtain 99.6% as Pearson coefficient, 96.4% as Kendall coefficient and 99.5% as Spearman coefficient.We also collected importance maps of deep DQN RL agents playing Pong using ReLU and Rational activation functions (borrowed from Delfosse et al. [2024c] . In this section, we give more details on the different RL specific caveats, as wellDefining an accurate reward signal when creating an RL task is challenging, as it requires the designer to specify a precise and balanced reward function that effectively guides the agent towards the desired behavior while avoiding unintended consequences [Henderson et al., 2017 , Irving et al., 2018] . Shaping the reward after having observed (and understood) the non-intended behavior of the agent is very common [Andrychowicz et al., 2017] .To realign our SCoBots agent, we provide them with a reward signal proportional to the progression to the joey: 1 player = _ g e t _When an outcome is delayed or uncertain, it is challenging to identify the action that is relevant for that outcome [Mesnard et al., 2021] . This problem arises in games like chess, where reward is passed to the agent only when the game is over. The problem can be addressed by more instant informative feedback, given directly after successful actions or after a positive state is reached.In the main part of this manuscript, we address this problem on the game Skiing. To encourage the agent to go in between the flag, we add a reward that corresponds to the distance to both of the next flags (on the x axis):R exp 1 = D(P layer, F lag1).x + D(P layer, F lag2).x.This resulted in an agent that learned to stop itself in between the flags (depicted in Fig. 12 ), showing once again, how difficult it is to create a reward that would favor a specific behavior. To correct it, we therefore adjoined another signal to reward our SCoBot agents proportionally to the speed of the agent: R exp 2 = V (P layer).y.A.8.3 The reward sparsity of PongLet us move on to the issue of sparse reward in the context of RL. In Pong, to score a point, the player needs to return the ball and have it go past the enemy. To do so, the player has to perform a spiky shot, obtained by touching the ball on one of its paddle's sides (otherwise, the shot is flat and the enemy is easily catching it). Its vertical position may vary between 34 and 190. Its paddle height is 16, but it needs to shoot from the side of the paddle (the 3 pixel of each border) to get a spiky shot that is likely to go beyond the opponent. If the balls arrives not close to the top or bottom borders, the enemy will still catch it, which has the probability of ∼ 22% in our experiments. The probability of getting a successful shot is thus of ∼ 6/156 * 0.78 = 3%. The average number of corresponding tryouts for the agent to get rewarded is ∞ n=1 n × 0.03 × 0.97 n-1 = 33.3. The agent usually need 60 steps (in average) from the initialization of the point to the reward attribution, hence, it will need ∼ 2200 steps to be rewarded. This is consistent with the experiment depicted in Figure 13 . In this figure, a random agent (original) needs in average 2230 steps to observe reward. Justification: In this paper, we claim that object-centric RL agents that use successive concept bottlenecks can challenge deep RL ones, and that they allow for easy detection and corrections of diverse RL-specific caveats. This is shown in Figure 3 , both for the performances, as well as for the interpretability and correction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation, as long as it is clear that these goals are not attained by the paper.This leads to deep agents overall slightly worse than inSchulman et al. [2017], however we obtain similar results when evaluating on the old v4 versions (cf. Tab. 1).www.retrogames.cz/play_195-Atari2600.phpgithub.com/openai/baselinesX ✓ X X ✓ RGB X X X X X X X X X Relations trajectory X X X X ✓ X X X X distance ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ X euclidean distance X X X X X X X X X center X X X X X ✓ X X X speed X X X ✓ ✓ X X X X velocity X X ✓ X X ✓ ✓ ✓ X color X X X X X X Xreward = 0 


Decision: Reviewers generally agreed that this paper is clear, well written and novel. After some improvements by the authors, the primary weakness identified by any of the reviewers is the difficulty of the benchmark involved (Atari). While I agree that striving for more realistic benchmarks should always be encouraged, considering the novelty of this paper, I believe more realistic benchmarks can be left for future work.

Reviews: review: paper_summary: paper_summary: This paper proposes Successive Concept Bottleneck Agents that takes an object oriented view on the environment and construct interpretable policies based on objects and associated properties.
strength_weakness: strength_weakness: - Would like to see without the help of OCAtari, how the proposed pipeline would be affected by inaccurate object detectors and how this method could work in the wild, e.g., real life robotics domain.
- Instead of distilling neural networks, to achieve better faithfulness and interpretability against the policy network, it would be better to also show results with Calgary Aytekin's NN to DT conversion algo directly.
questions: questions: - How would you handle more complex relationships like those cooperative behaviors in multi-agent games?
- In POMDP setting, would you allow the relational functions reveal information on previously unobserved properties? If not, how would you plan to handle them?
limitations: limitations: See weakness.
paper_summary: paper_summary: The paper introduces Successive Concept Bottleneck Agents (ScoBots) that utilizes concept bottleneck models to enhance interpretability and decision-making. SCoBots incorporate successive concept bottleneck layers to integrate relational and object-based concepts for action selection. Unlike traditional deep RL agents, SCoBots allow for the inspection and revision of decision policies at various levels, from individual object properties to relational concepts and ultimately to action choices. The authors provide experimental evidence suggesting that SCoBots perform comparably to deep RL agents while offering significant advantages. Notably, SCoBots can justify their actions and address several RL-specific issues such as reward sparsity and goal misalignment through straightforward guidance. During evaluations, SCoBots identified and corrected a previously unknown shortcut behavior in deep RL agents playing Pong, highlighting the potential of this approach to resolve significant challenges in RL through relational concept-based models.
strength_weakness: strength_weakness: 1.	My major concern is about the **environments that the paper studies**. This paper primarily focuses on the application of models to Atari games. This kind of evaluation was popular in explainable RL a few years ago. However, regardless of how promisely the model can interpret Atari games, its practical relevance remains questionable. People gradually realize one cannot depend on an Atari solver to execute any realistic tasks. In today's context, to ensure that reinforcement learning (RL) has a real impact, the RL community has begun to explore tasks related to Embodied AI, such as vision-based manipulation and autonomous driving. These areas are where RL can truly be profitable, offering more than just conceptual benefits. I strongly recommend that the authors investigate these tasks, even within simulators like RoboGen and RoboCasa, which are publicly available. Pursuing research in these environments can ensure that Explainable Artificial Intelligence (XAI) in RL achieves significant, practical impact beyond mere conceptional advantages.
2.	One of the major innovations, **the relation extractor, is not adequately defined in the paper**. It is unclear whether this component is based on a specific model structure, what its objective function for learning is, or if it involves manually designed functions. I briefly scanned the appendix but was unable to find these details. Please clarify these aspects and consider including them in the main body of the paper for better accessibility and understanding.
3.	To write a self-contained paper, the author should **clarify which object extractor is employed by the authors**. Specifically, details the objective of this extractor and describes how the parameter $\omega_{\theta_1}$ is learned from image inputs, whether through self-supervised or unsupervised methods. This information will help readers better understand the methodology and its implementation.
4.	Line 69 contains an incomplete definition of the Markov Decision Process (MDP). It **omits essential components** such as the initial state distribution and the discount factor. Please include these elements to provide a comprehensive description of the MDP.
5.	**Not all state spaces in sequential decision processes can be accurately represented using object-level information**. Such transformations may result in the loss of critical information. Typically, assumptions of an object-oriented MDP are made before conducting these transformations. I recommend that the author explore the literature on object-oriented MDPs, particularly focusing on how objects and their relationships are modeled using graph neural networks. For instance, consider the work by Carlos Diuk, Andre Cohen, and Michael L. Littman titled "An object-oriented representation for efficient reinforcement learning," presented at the Twenty-Fifth International Conference on Machine Learning (ICML 2008). This paper provides valuable insights into object-oriented approaches in reinforcement learning and could greatly enhance the theoretical foundation of your research.
6.	**The set of relations is fixed** before the learning phase, based either on human knowledge or AI-derived insights (as referenced in the Python code on line 100 and limitations). However, this set is not adaptable to new or previously unobserved relations in new or partially observable environments. For instance, both the set of relations and the corresponding action selectors are tailored to specific tasks, such as Pong, and may not generalize well to other contexts.
7.	The introduction of **additional reward signals** (line 169) can significantly **affect the optimal policy in an MDP**. There is no guarantee that the optimal policies before and after the addition of these signals will be identical. These signals can alter the behavior of policies, which in turn changes the interpretations associated with them. Consequently, the **fidelity of the resulting interpretation becomes a critical issue to consider.**
questions: questions: 1. Why not study more realistic tasks like vision-based manipulation?
2. What are the model structure and objective functions for learning the extractors (relation and object ones)? 
3. How to deal with the state space that can not be represented by objects (e.g., windy grid world, 3D robot hand)?
4. How to resolve the fidelity issues coming from the added rewards?
limitations: limitations: Yes.
paper_summary: paper_summary: The authors introduce SCoBots, an explainable RL approach that first extracts symbols from images, extracts relations between the objects, and then uses them to train a distilled decision tree policy. They demonstrate better performance than naive CNN-based deep RL agents in Atari games.

For full disclosure, I have reviewed this paper before.
strength_weakness: strength_weakness: ### Weaknesses

**Missing Baselines/Ablations:** 

- A baseline/ablation that are missing that actually were addressed in the last time I reviewed this paper but somehow left out of even the appendix of this version:
    - A missing comparison is that of deep rl agents trained with the presumed object detector that SCoBots assumes access to (not the relational extractor, as that’s a component introduced in this method for this setting). This would be an important comparison because object detectors are pretty standard and very easy to give deep RL agents access to (essentially, give it the same state space as SCoBots). **The authors presented an ablation with noisy object detectors last time in the rebuttal, this should probably be highlighted in the paper.**

**Minor issues:**

- Another prior work that also demonstrates being able to have humans intervene on (program) policies given explicitly extracted concepts: Learning to Synthesize Programs as Interpretable and Generalizable Policies, Trivedi et al. 2021. **The authors claimed to have added both works i suggested in the rebuttal last time, they did add one of them (VIPER).**
questions: questions: See weaknesses
limitations: limitations: Adequately addressed
paper_summary: paper_summary: This paper provides evidence for the benefits of concept-based models in RL tasks, especially for identifying issues like goal misalignment. The proposed SCoBot integrates relational concepts into decision processes with minimal feedback, which not only brings the advantage of enhanced human understanding (Object Representations) but importantly enables targeted human-machine interactions (Relational Concepts). Experimental results show SCoBots' competitive performances and the potential for domain experts to understand and regularize their behavior.
strength_weakness: strength_weakness: * Will the space of Object Representations and the types of Relational Concepts change along with the change in the environment? If so, how to solve it?
* Is the concept learned through supervised learning? What if there is no concept label?
* What if the generated relation function keeps having errors by LLMs?
questions: questions: See above.
limitations: limitations: If the user does not understand the task and cannot define the concepts and the relation functions well.

Comments: title: Most interpretable RL methods are not evaluate on more complicated benchmarks
Discussion Period almost over
Reminder: The reviewer-author discussion period ends in a couple of days
comment: Dear reviewer, thank you for getting back at us. We are happy that we were able to address your concerns and that you are satisfied with our contribution. Concerning your questions:

1. We have looked at the NN to DT github repo at this link (https://github.com/CaglarAytekin/NN_DT), which contains only one script file (_yx2reproduce.py_), without any extraction method, but with only one hard coded tree, probably extracted from a TensorFlow based neural network. We were thus not able to adapt any tree extraction algorithm, as none was provided.

2. By default, in POMDP settings, we would of course not allow for revealing hidden attributes per se, as this would violate the POMDP assumption. However, we believe that SCoBots could be extended such that the agent can learn at training time that e.g. objects can be occluded. If the agent observes that the object was present in the environments (with low probability of suddenly disappearing) in the previous states, it could make use of the relational functions to update its belief on the raw property of the object to place it behind a potentially occluding object. This would of course be an extension to SCoBots that maintain e.g. a belief, and where this belief about objects properties can be influenced (backward) by the belief about the relational state.

We hope that this helps clarify your question. 

Thank you again for your insightful comment, that allowed us to consider further potentially interested lines of research.
We appreciate the reviewer's feedback and agree that the practical application of RL methods in industrial settings is an important direction for future research, and for research in general. However, it is important to recognize that the scope of our paper is to introduce and validate a novel interpretable reinforcement learning method using the most commonly used ALE RL benchmark, that is already enough to support our claims, to demonstrate that deep agents face many caveats, notably learn misalignments, that are easily detectable and
 fixable using our concept bottlenecks. Contrary to our work, most transparent RL methods are evaluated on even simpler environments (e.g. Cartpole, GridWorld) [1,2,3,4,5,6,7] and are thus not comparable with most RL research and many post hoc explanation methods use Atari or simpler environments [8,9,10,11]. While our method is indeed applicable to other environments, including industrial ones, demonstrating this would require a separate and more extensive study, as we discussed above, with the development and inclusion of options for e.g. robotic tasks. We believe it is somewhat unfair to critic our work based on this, as the gap between research on benchmark datasets and real-world application is a broader challenge facing the entire RL community. Our aim here is to contribute to the foundational science that will ultimately support such real-world applications. 

-----

[1] Verma, A., Murali, V., Singh, R., Kohli, P., & Chaudhuri, S. (2018). Programmatically interpretable reinforcement learning. In International Conference on Machine Learning ICML.

[2] Bastani, O., Inala, J. P., & Solar-Lezama, A. (2020). Interpretable, verifiable, and robust reinforcement learning via program synthesis. In International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers.

[3] Shi, W., Huang, G., Song, S., Wang, Z., Lin, T., & Wu, C. (2020). Self-supervised discovering of interpretable features for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4] Hein, D., Udluft, S., & Runkler, T. A. (2018). Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence.

[5] Silva, A., Gombolay, M., Killian, T., Jimenez, I., & Son, S. H. (2020). Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statistics.

[6] Roth, A. M., Topin, N., Jamshidi, P., & Veloso, M. (2019). Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy.

[7] Jiang, Z., & Luo, S. (2019, May). Neural logic reinforcement learning. In International conference on machine learning (pp. 3110-3119). PMLR.

[8] Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., & Jimenez Rezende, D. (2019). Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems.

[9] Shi, W., Huang, G., Song, S., Wang, Z., Lin, T., & Wu, C. (2020). Self-supervised discovering of interpretable features for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10] Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019). SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence.

[11] Liu, G., Schulte, O., Zhu, W., & Li, Q. (2019). Toward interpretable deep reinforcement learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD.
Thanks for the reply! Most of my concerns are solved. Two quick follow ups regarding the response:  
1. "We have not been able to reproduce this algorithm to directly extract decision trees." Could you be more specific on why the author's code (as in the link you posted) cannot work?

2. My concern is more about revealing "forbidden" attributes to the agents violating the env assumptions of **PO**MDP. If you use the relation function "occlusion" and it tells agent that B is behind A, this feels a little bit cheating to me as it's not sth available in direct image observations.

Overall I am satisfied with the paper's contributions and will keep the score. But I am keen to know the detailed answers to my above questions.
The discussion period is almost over, so both authors and reviewers please respond to any unaddressed questions. Reviewers, be sure that you have all of the information you need from the authors, since after the 13th, they won't be able to respond.
Thank you for your detailed response, which has deepened my understanding of this work. Having spent years researching interpretable RL and publishing several well-cited papers, I recently engaged more closely with the real industry and recognized a significant gap between XAI, such as XRL, and their practical applications. While the story of XRL focuses on enhancing user experience and trust, it remains unclear which companies or groups are actively utilizing these techniques. The gap between methods used in settings like Atari and their application in real-world production is substantial. Although this work is ok from a scientific exploration standpoint, the practical necessity and application of these findings remain ambiguous.

Based on the above points, I am inclined to maintain my rating, but I am not arguing for any final decision.
We would like to express our sincere gratitude to the reviewer for the thorough and constructive feedback. We are confident that our responses adequately address the concerns raised by the reviewer, including the following points:

* our experimental evaluation supports our claims, namely that relational concept bottlenecks can be used in competitive agents and help detect and correct many RL specific caveats,
* we now discuss the use of SCoBots in robotic environments (notably the one proposed by the reviewers), in our future work section,
* we improved the definitions of the object and the relation extractor, expanding the ones of [1] (and thank the reviewer for the reference),
* we highlight that our framework can easily be extended to search for new relations when the initial set is not sufficient, notably in continual RL setting,
* our agents are fully transparent and thus do not require humans to derive interpretation about the policy and when guided with a relation-based extra reward, we report their scores on the original MDPs.

Please kindly let us know if the reviewer has any additional concerns. We are fully committed to resolving any potential issues, should time permit. Again, we thank the reviewer for all the detailed review and the time the reviewer put into helping us to improve our submission.

------

[1] Diuk, Carlos, Andre Cohen, and Michael L. Littman. "An object-oriented representation for efficient reinforcement learning." _Proceedings of the 25th international conference on Machine learning_. 2008.
Thanks for the response. I have carefully reviewed the authors' rebuttal and the reviews from other reviewers. It answered my remaining questions sufficiently and I have no further concerns.
Thanks for the rebuttal! I am already voting accept so I will not change my score.