LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning
Authors: Zhuorui Ye
Stephanie Milani
Geoffrey J. Gordon
Fei Fang
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Explainable Reinforcement Learning
Concept Bottleneck Models
Concept-based Explainability
Interpretability
XRL
Interpretable Reinforcement Learning
Interpretable Agent
Primary area: interpretability and explainable AI
TL;DR: We algorithmically reduce the concept annotation effort during training for interpretable reinforcement learning. We also explore the use of VLMs as automated concept annotators.

Abstract: Recent advances in reinforcement learning (RL) have predominantly leveraged neural network policies for decision-making, yet these models often lack interpretability, posing challenges for stakeholder comprehension and trust. Concept bottleneck models offer an interpretable alternative by integrating human-understandable concepts into policies. However, prior work assumes that concept annotations are readily available during training. For RL, this requirement poses a significant limitation: it necessitates continuous real-time concept annotation, which either places an impractical burden on human annotators or incurs substantial costs in API queries and inference time when employing automated labeling methods. To overcome this limitation, we introduce a novel training scheme that enables RL agents to efficiently learn a concept-based policy by only querying annotators to label a small set of data. Our algorithm, LICORICE, involves three main contributions: interleaving concept learning and RL training, using an ensemble to actively select informative data points for labeling, and decorrelating the concept data. We show how LICORICE reduces human labeling efforts to 500 or fewer concept labels in three environments, and 5000 or fewer in two more complex environments, all at no cost to performance. We also explore the use of VLMs as automated concept annotators, finding them effective in some cases but imperfect in others. Our work significantly reduces the annotation burden for interpretable RL, making it more practical for real-world applications that necessitate transparency. Our code is released.

Published as a conference paper at ICLR 2025
LICORICE: LABEL-EFFICIENT CONSEPT-BASED INTERPRETABLE REINFORCEMENT LEARNING
Zhuorui Ye*
Stephanie Milani"
Institute for Interdisciplinary Information Sciences
Machine Learning Department
Tsinghua University
Camegie Mellon University Pittsburgh, PA 15213 amilani@es. cmu edu
Beijing, China
cuizhayefei (atfeee@mmaai.. com
Geoffrey Gordon
Fei Fang
Machine I Learning Department Carnegie Mellon University Pittsburgh, PA 15213 ggordonêcs, cmu. edu
Software and Societal Systems Department Carnegie Mellon University Pittsburgh, PA 15213 fe faiignguuuu. edu edu
ABSTRACT
Recent advances in reinforcement learning (RL. have predominantly leveraged neural network policies for decision-making, yet these models often lack inter- pretability, posing challenges for stakeholder comprehension and trust. Con- cept bottleneck models offer an interpretable alternative by integrating human- understandable concepts into policies. However, prior work assumes that concept annotations are readily available during training. For RL, this requirement poses a significant limitation: it necessitates continuous real-time concept annotation, which either places an impractical burden on human annotators or incurs substan- tial costs in API queries and inference time when employing automated labeling methods. To overcome this limitation, we introduce a novel training scheme that enables RL agents to efficiently learn concept- based policy by only querying annotators to label small set of data. Our algorithm, LICORICE, involves three main contributions: interleaving concept learning and RL training, using semble to actively select informative data points for labeling, and decorrelating the concept data. We show how LICORICE reduces human labeling efforts 500 or fewer concept labels in three environments, and 5000 or fewer in two more complex environments, all at no cost performance. We also explore the use of VLMs as automated concept annotators. finding them effective in some cases but imperfect others. Our work significantly reduces the annotation burden for interpretable RL, making it more practical for real-world applications that neces- sitate transparency. Our code released
INTRODUCTION
In reinforcement learning (RL), agents learn policy, which is strategy for making sequential decisions in complex environments. Agents typically represent the policy as a neural network, as this representation tends to yield high performance (Mirhoseini et al.) 2021I. However, this choice can come at a cost: such policies are challenging for stakeholders to interpret - particularly when also complex, such high-dimensional sensor data. This opacity can pose
the network inputs a significant hurdle, especially in applications where understanding the rationale behind decisions critical, such as healtheare (Yu al. (02I) finance (Liu et al. (022). In such applications, decisions can have significant consequences, so it is essential for stakeholders to fully grasp the reasoning behind actions confidently adopt or collaborate on a policy.
* "Equal Contribution.
This work was done when Ye was visiting intern at CMU. https: // thub. com/ cui /

---

Published as a conference paper at ICLR 2025
Data Decorrelation
g
"go up
opes
esart Jeft
Apent
Concept Learning
Behavior Learning
Iterative Training
Active Learning
Figure 1: LICORICE overview. In concept-based RL, the policy first maps from states to the concepts in a bottleneck layer with g. and then maps from concepts to (distributions over) actions with f. During training, LICORICE addresses concept label efficiency concerns with three key components: i) iterative training, ii) data decorrelation, and iii) active learning.
To address interpretability concerns in supervised learning, recent works have integrated human- understandable concepts into neural networks through concept bottleneck models (Koh et al. 2020 Espinosa Zarlenga et al.) 2022). These models insert bottleneck layer whose units correspond to terpretable concepts, ensuring that the final decisions depend on these concepts instead of on opaque raw inputs. By training the model both to have high task accuracy and to accurately match experts' concept labels, these models learn high-level concept-based based representation that is simultaneously meaningful to humans and useful for machine learning tasks. As an example, a concept-based explanation for a bird classification task might include unit that encodes the bird's wing color. Concepts also offer a path to compositional generalization by enabling subtask decomposition into meaningful, reusable components (Mao al. 2022 Wang et al. 2023
More recently, these techniques have been applied to RL by incorporating a concept bottleneck in the policy (Grupen et al. 2022 Zabounidis et al. 2023). so that the chosen actions a function of the human-understandable concepts. However, a significant challenge emerges in this method's practical application: past work assumes that concept annotations training. To learn a mapping from states to concepts, an RL agent requires concept information for every state encountered during training, which is often millions billions of state-action pairs. Because many domains do not automatically provide these concept labels (e.g., autonomous driv- ing), we require means of obtaining them. Relying on human labelers is impractical, risking errors due to fatigue (Marshall & Shipman 2013). Using vision language models (VLMs) for automated concept extraction (Oikarinen et al. 2023). while alleviating the human bottleneck, introduces sig- nificant API or inference costs that can be prohibitive.
readily available during RL
In this work, we address this challenge with LICORICE (Label-efffcient Interpretable COncept- based ReInforCEment learning), novel training paradigm designed to minimize the number of concept annotation queries while maintaining high task performance. Figure illlstrraess our algo- rithm. LICORICE tackles three key challenges. First, it addresses the problem of concept learning on off-policy or outdated data. concepts are learned from data collected by a random policy, the concept distribution may not reflect the data from an optimal policy. To ensure that concept learning occurs on more recent and on-policy data, LICORICE interleaves concept learning and RL training through iterative training: it alternately freezes the network layers corresponding either the con- cept learning part or the decision-making part. Second, LICORICE addresses the problem of limited training data diversity that occurs when an agent interacts with the environment, thereby generat- ing sequences of highly similar, temporally correlated data points. To tackle this issue, LICORICE implements a data decorrelation strategy to produce a more diverse set of training samples. Third, LICORICE addresses the inefficient use of annotation effort, where labeled samples may provide redundant information. To resolve this problem, we employ disagreement-uusel active learning using a concept ensemble select the most informative data points for labeling.
To evaluate the effectiveness LICORICE, we conduct experiments two scenarios-perle human annotation and VLM annotation-on on five environments with image input: image-based version of CartPole, two Minigrid environments, and two Atari environments. First, with assumption of perfect human annotation, we show that LICORICE yields both higher concept accuracy and higher reward while requiring fewer annotation queries compared to baseline methods. Second, we find that VLMs indeed serve concept annotators for some, but not all, of the environments.
2

---

Published as a conference paper at ICLR 2025
Our contributions are summarized follows:
To the best of our knowledge, we are the first to investigate the problem limited concept annotation budget for interpretable RL. We introduce LICORICE, novel training scheme that enables label efficient learning of concept-base RL policies.
We conduct extensive experiments to show the effectiveness of LICORICE across five environ- ments with varying budget constraints.
We study the use of VLMs as automated concept annotators, demonstrating their effectiveness in certain environments while highlighting challenges in others.
2 PRELIMINARIES
Reinforcement Learning. In RL, an agent learns to make decisions by interacting with envi- commonly modeled as a Markov decision process (Puterman 2014), consisting of the following components: set of states S, a set of actions A, a state transition function T A x S [0, 1] that indicates the probability of transitioning from one state to another given action, reward function R 8 A x S -> IR that assigns ward for each state-action-state transition, and discount factor E [0. 1] that determines the present value of future rewards. The agent learns policy т tributions over actions with the aim of maximizing the expected cumulative discounted reward. We evaluate policy via its value function, which is defined (*)(s E 1E==0 * & = s). The ultimate aim in RL is to determine the optimal policy, n², through iterative refinement based on environmental feedback.
ronment (Sutton & Barto 2018). The environment
A + [0, 1j. which maps states dis-
Concept Policy Models. Concept-based explanations have emerged a common paradigm in explainable Al (Poeta et al. all..2222)) (023). They explain model's decision-makinn process through human- understandable attributes and abstractions. In supervised learning, concept bottleneck models (Koh) et al. 2020) implement this approach using two key functions: a concept predictor g X + C, mapping inputs interpretable concepts, and output predictor f C - Y, mapping the concept predictions to a downstream task space, such as labels for classification. As a result, the prediction takes the form ò f((())), that the input z influences the output solely through the bottleneck layer = g In RI., since we want the policy to be interpretable, we insert a concept bottleneck layer into the policy network:
x (s) = f((()),
such that π maps from states s E to concepts e E C actions a A (Grupen et al., 2022. Zabounidis et al. 2202). This setup allows the policy to base its decisions understandable and meaningful features. As a result, we can use any RL algorithm that can be modificd to include an additional loss function for concept prediction.
Training Concept Policy Models. Training these models requires a dataset of state State-concept-actio triplets (s, a) E S x C A. The functions f and g typically implemented as neural networks, with their parameters collectively defined as 0. Previous work (Zabounidis et al., 2023) simply combines the concept prediction loss I. (0) and RI. loss T.RL(0):
L(0) (R) + (O),
where the exact definitions of 7, (0) and T depend on the choice of RL algorithm and concept learning task. The objective is to find the optimal parameters 0" that minimize this combined loss function with the coefficient λç. However, this approach requires continuous access to ground-truth concepts for training f. which may not always feasible or desirable in practical RL scenarios.
LICORICE
As we have mentioned, the standard way of training concept-based RL assumes continuous access to oracle to provide concept labels. However, this assumption is problematic due to the large annotation cost incurred by human or automated labeling efforts. To reduce the number of concept labels required for concept-based RL, propose LICORICE, novel algorithm for interpretable RL consisting of three main components: iterative training, data decorrelation, and disagreement- based active learning. The full pseudocode is in Algorithm
3

---

Published as a conference paper at ICLR 2025
Algorithm LICORICE (Label-efficient Interpretable COncept-based ReInforCEment learning)
1: Input: Total budget B, number of iterations M, sample acceptance threshold p, ratio for active learning, batch size for querying b, number of concept models N to ensemble
2: Initialize: training set Dimin f 0, and validation set Dval f- 0
3: for m = 1 to M do
4: Allocate budget for iteration m: Bm
B
5: while [L < T Bm, do
6: Execute policy 7m to collect unlabeled data Um using acceptance rate p end while
7:
8:
Initialize iteration-specifee datasets for collecting labeled data: Pirain + 0, D
0
9:
while Bn > 0 do
Train N concept models { on Drainn Di Pirain using Drat U D for early stopping
11:
Calculate acquisition function value a(s) for all s E Um \\ (D train U'
Choose b,, min(b. Bm) unlabeled points from Um according
arg max, (())
13:
Query for concept labels to obtain new dataset Dm <- {(s,c) )m
14:
Split Dm,. into train and validation splits and add to D D'rain D
15:
Decrement budget: Bm f- Bm bm
16:
end while
17:
Aggregate datasets: Dusin < Durin D Deal Dval U D
18: Continue training the concept network 9 on D urainn using Dval for early stopping
19:
Freeze g and continue training / using standard RL training to obtain um++1
20:
end for
Iterative Training. As the agent's policy improves, the distribution of the visited states and their associated concepts changes. Consider an MDP where states are indexed. With a random (initial) policy, the agent tends to visit small-index states near the initial state, encountering only the concepts relevant to those states. However, as the policy improves, the agent explores higher-index states, leading to shift in both the state and concept visitation distribution. If we exhaust all queries the beginning of training, we risk training the model only on the concepts associated with small-index states from the random policy, potentially missing important concepts that emerge the agent explores. We therefore propose iterative training to enable LICORICE to progressively refine its understanding of concepts as the policy improves. Iterative training consists of two parts: concept learning and behavior learning. Assume that we have access to an annotated concept dataset Irain generated by our current policy Ty- Then, concept learning focuses on training the concept portion of the network g on this dataset Dirain (line 18). Behavior learning involves freezing g and training the behavior portion of the network f with any standard RL algorithm with its associated loss RE (line 19). Upon completion of behavior learning, we obtain an updated policy Tm+11 which serves to collect more unlabeled concept data to help train g in the next iteration.
Data Decorrelation. We do not obtain ground truth concept labels for all states when rolling out the current policy km. Instead, rollouts produce a dataset Um of unlabeled states as candidates for querying for ground-truth concepts from an oracle. Simply collecting and labeling all encountered states would give us long chains of nearly-idention effective learning. To resolve this challenge, we introduce data decorrelation with two key compo- nents: a budget multiplier 7 that sets [m = T Bn. and a per-state acceptance probability p. This random acceptancerrejection mechanism leverages fundamental property of Markov processes- states become increasingly independent as their temporal distance grows according to the mixing time allowing us build more diverse datasets from our policy rollouts.
samples, undermining the diversity we need for
Disagreement-Based Active Learning. Equipped with Um, we are now prepared to select data points for querying for concept labels. The purpose of this stage is make use of our limited la- beling budget B,, to collect labeled dataset Dn,, for training g. We propose to train concept ensemble-cconiisting of N independent concept models-frie from scratch on the dataset of training points Drain that has been aggregated over all iterations (line TO). We use this ensemble to calculate the disagreement-bee based acquisition function ax-), which determines whether each candidate in Um ought to receive ground-truth concept label (line IT). This function targets samples where pre- diction disagreement highest among ensemble members, as these points often represent areas of uncertainty decision boundaries where additional labeled data would be most informative. For
4

---

Published as a conference paper at ICLR 2025
a()), two different formulations, depending the concept learning task. For concept classi- fication, we use a with high proportion of predicted class labels that not the modal class prediction (also called the variation ratio (Beluch et al., 2018):
approach (Seung et al., 1992), in which we prioritize points
a(s) max cEC N
Σ (d:(s)
1b:(())= =
Here, (8) the concept prediction of the i-th model on state 8. For concept regression, we directly the estimate of variance across the concept models measure of disagreement:
a(s) = oz)))
Σ (b:(s) - u((s)²
where uuss) = Σ (s) is the mean prediction of the N concept models. After querying for B,n ground-truth concept labels (line 13) using a(-), we are prepared to continue training g during the concept learning stage.
Implementation Details. For training g (line 18) we employ two types of loss functions depending on the nature of the concepts: MSE for continuous concepts and eross-entropy loss for categorical concepts. If the problem requires mixed-type concepts, we either discretize continuous attributes or simply use mixed loss. To train / (line 19). freeze g and use PPO to continue training / from the previous iterations, resulting in the final policy +1. For complex environments that require many iterations, we observe that RL-related parameters may get stuck becoming hard to optimize later. To mitigate this issue, in the last iteration, we additionally train new RL network f' from scratch when fixing 9 with #M+1 as the anchoring policy to get the final T "M The anchoring policy ensures "M41 has similar distribution to the previous one and the annotated observations are still highly relevant. Specifically, we initialize another policy та with the same frozen y parameters and randomly initialized f. then train it with the standard PPO loss function and an additional KL-divergence penalty between the current policy and aMs+1:
local region in early iterations,
I(0)= T.PP0 (0) B- [L {), no(. [s]]-
4 EXPERIMENTS
In our experiments, we investigate the following questions:
RQ Under limited concept annotation budget, does LICORICE enable both high concept accuracy and high environment reward?
RQ 2
How effective are VL.Ms as automated concept annotators when used with LICORICE?
RQ 3
How label efficient is LICORICE compared with other methods?
RQ 4 Does LICORICE support test-time concept interventions?
4.1 EXPERIMENT SETUP
In each experiment, we run each algorithm 5 times, each with random seed. All algorithms use PPO (Schulman al. 2017 Raflin et 202T) with a concept bottleneck. More implementation details and hyperparameters are in Appendix
Environments. We investigate these questions across five environments: PixelCartPole (Yang al.
2021 DoorKey (Chevalier-Borge
et al.] (023). DynamicObstacles (Chevalier-Boiswa)) et al. 2023 Boxing (Bellemare et al., 2013). and Pong (Bellemare et al. 2013). Each environment
cludes distinct challenge and features a set of interpretable concepts describing key objects and properties. We summarize the concepts in Table []] with more details Appendix A.I These environments are characterized by their dual representation: a complex image-based input and symbolic concept-based representation. PixelCartPole, DoorKey, and DynamicObstacles are sim- pler because
extract noiscless ground-truth concept labels from their source code. In contrast, Boxing, as implemented in OCAtari (Delfosse et al. 2023), uses reverse engineering to extract po- sitions of important objects from the game's RAM state. This extraction process introduces small
5

---

Published as a conference paper at ICLR 2025
Environment
# Concepts
Concept Type
Binarized Concepts
PixelCartPole
Continuous
N/A
DoorKey
12
Discrete
46
DynamicObstacles
Discrete
30
Boxing
Discrete
1480
Pong
Discrete
1848
Table 1: Summary of environments, including their associated concepts. Counts for the binarized
version of the concepts provided where applicable to illustrate the problem size.
amount of noise. The concepts in Pong are derived from the VIPER paper (Bastani et al. 2018 that also uses reverse engineering.
Baselines. To our knowledge, there exist no previous algorithms that seek to minimize the number of concept labels for interpretable RL, so we implement three: Sequential-Q, Disagreement-O. and Random-Q. In Sequential-Q. the agent spends ß queries on the first B states encountered during the initial policy rollout. In Disagreement-Q Q. the agent also spends its budget at the beginning of strategically choose
training; however, it uses active learning with the same a(-) as LICORICE the training data for annotation. In Random-Q, the agent receives B concept labels at random points using probability decide whether query in each state. As budget-unconstrainel baseline, we implement CPM from previous work in multi-agent RI. (Zabounidis et al. 2023) for the single- agent setting. As mentioned in $ CPM jointly trains g and assuming unlimited access concept labels, so it also represents an upper bound on concept accuracy.
GPT-4o (gpt). During training, we query GPT-4o each time LICORICE requires a concept label. As an example, for the concept related to the position of an object, we prompt GPT-4o with instructions to report the coordinates (x y). More details about our prompts can be found in Appendix A.3
VLM Details. For VLM experiments,
Performance Metrics. We compare our reward to an upper bound calculated by training PPO with ground-truth concept labels (PixelCartPole: 500, DoorKey: 0.97, DynamicObstacles: 0.91, Boxing: 86.05, Pong: 21). the first four environments, percentages (or ratios) make sense since the minimum reasonable reward is 0 In Pong. random policy has -21 reward so we first get the difference between rewards and -21 and then calculate the ratio. We additionally report the concept error (MSE for regression; 1 accuracy for classification). In Boxing, following the practice of OCAtari, we consider a concept prediction correct it is within 5 pixels of the ground truth label. In Pong, simply use classification error. All reported numbers calculated using 100 evaluation episodes.
4.2 RESULTS
RQ 1: REWARD AND CONCEPT ACCURACY UNDER A LIMITED ANNOTATION BUDGET
LICORICE achieves similarly high reward and concept performance to the state-of-the-art budget-uncone unconstrained baseline. We first seek understand how LICORICE performs compared to the state-of-the-ard in concept-based RL, CPM, which is not constrained by concept label budgets. Surprisingly, LICORICE and CPM achieve nearly 100% of the maximum reward in all environments in this unfair comparison (Figure 2 They also achieve a similar concept eттоr rate, only seeing a clear difference in error for the most complex environment, Pong. We emphasize that CPM given an unlimited budget; in practice, it uses 1M-15M concept labels for all environments, which is around 2000 to x3000x the budget used by LICORICE. contrast, LICORICE operates under strict budget constraint, with 3000 labels for Boxing and 5000 for Pong. and at most 500 concept labels for the simpler environments (PixelCartPole: 500, DoorKey: 300, Dynamiobsstaceles:::: 300).
LICORICE generally achieves higher reward and lower concept error than budget- constrained baselines. We study all budget-constrained algorithms (LICORICE plus those $4.1
In PixeICarPPole and DoorKey, all rewards are nonnegative; in DynamicObstacles, an agent can ensure nonnegative reward by simply staying in place; in Boxing, random policy can get around ( reward and all of the trained policies have positive reward.

---

Published as a conference paper at ICLR 2025
Reward
PirtPCartPole
Doorkey
DynamicObstacies
Boxing
Pong
Concept Eiror (4)
0.6
0.4
0.2
I
CartPPeee
Doorkey
DynamicObstacles
Boxing
Pong
Sequential-Q
Disagreement-O
Random-Q
UCCORCE
LICOICE+GPT-40
Figure 2: LICORICE generally achieves higher reward and lower concept error than all other budget- constrained algorithms (Sequential-Q. Disagreement-O Q. Random- ().ioomm--nnt The budgets for each environ- ment are PixelCartpole: 500, DoorKey: 300, DynamicObstacles: 300, Boxing: 3000, Pong: 5000. CPM has unlimited budget (in practice, it 4M, 4M, 1M, 15M, 10M labels, respectively). Despite this extreme budget difference, LICORICE achieves comparable reward and concept error.
Full results in Table Appendix
under the same fixed concept labeling budget as above. Figure Tiguree shows that LICORICE outperforms all baselines in terms of reward and concept error on all but arguably the easiest environment, Dy- namicObstacles. In that environment, LICORICE performs similarly to the baselines. The greatest performance differences occur in PixelCartPole and DoorKey, where LICORICE achieves 100% and 99% maximum reward, while the second-best algorithm achieves 36% and 89%, respectively. We therefore answer RQ affirmatively: not only does LICORICE achieve the same high concept accuracy and high reward as the state-of-the-ar in concept-basee RL, it also performs on on-par to budget-constrained basclines in one environment and outperforms them in the other four.
RQ 2: VLMS AS CONCEPT ANNOTATORS
We now seek to answer the question of whether VLMMs can successfully provide concept labels in lieu of a human annotator within our LICORICE framework. Because using VLMs incurs costs, we operate within the same budget-constrained setting as above. We present these results Figure
VLMs can serve concept annotators for some environments. In DoorKey and Dy- namicObstacles, LICORICE with GPT-4o la- bels achieves 83% and 88% the maximum reward, respectively. To assess the quality of VL.M-generatdd labels, we compare the con- cept error rate of our trained model against GPT-4o's ' labeling error, both evaluated on the same rollout observations, while GPT-4o evalu- ation is on random subset of 50 observations, due API cost constraints. Table shows that the concept error of LICORICE is comparable that of GPT-4o when GPT-4o labeling error is not high, which suggests that LICORICE can effectively utilize these concept labels. VLMs, particularly GPT-40, could serve as concept annotators for certain concepts certain environments.
Error
Environment
LICORICE+GPT-4
GPT-4o
PixelCarPPoe DoorKey
0.25
0.24
0.31
0.30
DynamicObstacles Boxing Pong
0.13
0.13
0,98
0.82
0.73
0.87
Table 2: Concept error comparison. The con- cept error of LICORICE+GPTT45 matches that of GPT-44 alone in all environments except Boxing.
VLMs struggle to provide accurate concepts in more complex environments. In PixelCanPPle, Boxing, and Pong environments, however, GPT-4o struggles to provide accurate labels. In these environments, not only does LICORICE+GPT-40 achieve less than 25% of the maximum reward, also incurs a large concept emror. The challenge with PixelCartPole is the continuous nature of the concepts and the lack of necessary knowledge of physical rules and quantities in this specific

---

Published as a conference paper at ICLR 2025
PixeiCartPole
Doorkey
ynnaiccbbstacles
Boxing
Pong
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
Reward
0.50
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
300
400
500
0.00
100
200
300
0.0%
100
200
300
0.09 000
2000
3000
0.00
1000
3000
5000
0.3
0.6
0.3
1.0
Concept Entr (U)
0.8
0.8
0.2
0.6
0.6
0.4
0.4
0.1
0.2
0.2
0.0
0.9
0.000
"fo00
300
400
500
100
200
300
200 Budget
300
2000 Budget
3000
3000
5000
Budget
Budget
Budget et
Sequential-Q
D
Random-Q
LICORICE
LICORICE+GPT-40
GPT-4o
Figure 3: Reward (top) and concept error (bottom) of all algorithms for different budgets. LICORICE more efficiently makes use of the varying budgets, achieving higher reward and lower concept error. Full results with standard deviation are Tables Appendix B
environment. Boxing is particularly challenging due to the large number of possible concept values that each of the 8 discrete concepts can take (160 or 210 possible values). Similarly, concepts in Pong have hundreds of possible values and some physics quantities require multi-frame inference. We therefore answer RQ with cautious optimism: there indeed cases in which LICORICE could be used alongside VLMs, though not all yet. Generally speaking, less complex environment more likely work well with VLMs by enabling clear and detailed labeling instructions in the prompt. However, VLM capabilities need improvement before they can fully alleviate the human annotation burden, especially in safety-critics settings.
RQ INVESTIGATION BUDGET REQUIREMENTS FOR PERFORMANCE
Given these results, we now investigate the minimum budget required for all environments to achieve 99% of the reward upper bound. To do so, incrementally increase the budget for each environ- ment starting from a reasonably small budget until LICORICE reaches 99% of the reward upper bound. In Pong, further increase the budget until the concept is below 0.25. In addi- tion to the baselines, we study LICORICE under perfect (human) annotation and noisy VLM labels Figure
LICORICE more rapidly achieves high reward compared to baselines. Across all environ- ments, LICORICE the most query-efficient. In three of the environments, consistently achieves higher reward than baselines across all budget levels. In DynamicObstacles and Pong, LICORICE outperforms baselines the smallest query budget, after which some baseline method achieves com- parable reward. LICORICE is similarly performant with respect to concept enror. Except for one budget setting for one environment, LICORICE consistently achieves lower concept error than the baselines.
For LICORICE+GPT-40, more concept labels is not always better. In DoorKey, LICORICE+GPT-4 exhibits predictable behavior in terms of concept error: as the budget increases, the reward increases and the concept error decreases. Counterintuitively. for some environments, the concept error and reward fluctuate with more budget, likely due to the additional labeling noise in- troduced. We therefore provide a more nuanced answer to RQ 3: LICORICE is indeed label efficient across varying budgets, but the benefit is amnotator-dependent.
8

---

Published as a conference paper at ICLR 2025
PixelCartPole
Doorkey
Dynamicobstacles
%: Oppttiall Reward
1.00
2.8
0.55
0.6
0.50
0.80
0.2
2 3 8.9 101112
112 3 4 6
1011
# of Concepts Intervened
of Concepts intervenied
of Concepts Intervened
Concepts Intervened
Figure 4: Concept intervention results:
LICORICE enables test-time concept intervention.
Door spen
Door open
wrong
Forward
.... 0.00,
1.00]
Intervention
Intervention
correct
comeect
|3.0, 0.00
1.00]
Toggla
[3.0, 1.00, 1.00)
Forward
Figure 5: Test-time intervention examples, where intervening on a single concept corrects the action.
RQ 4: INTERPRETABILITE ANALYSIS: TEST-TIME CONCEPT INTERVENTIONS
A great property of concept- based networks is the ability for people to successfully intervene on the concepts to correct them. In RL, this intervention enables the inspection how different concepts influence the immediate decisions of the agent.
LICORICE enables test-time concept intervention. To validate that LICORICE supports test- time concept intervention, simulate using noisy concept model with f to study the impact of concept interventions on reward. For PixelCartPole, we add Gaussian noise to each predicted concept with standard deviation of 0.2. For the other three environments with discrete concepts, each concept label is randomly changed with probability 0.2. Following previous work (Koh et al.) 2020), we first intervene individually on concepts, setting them to ground truth, and sort the con- cepts in descending order of reward improvement. We then sequentially intervene on the concepts following this ordering, such that any previously intervened concepts remain intervened. According to Figure 4 using a noisy concept model significantly degrades reward. However, the performance increases with more interventions, meaning LICORICE supports test-time concept intervention, firmatively answering RQ 4.
Concept interventions show how the policy decisions change. We now show how domain experts could interact with the model. Specifically, simulate counterfactual question: What if concept value is incorrectly predicted? Figure 5 depicts two examples in DoorKey. Just intervening on the concept of the door being open closed is sufficient to change the agent's 's behavior, both highlight- ing the importance of this specific concept and the ability user to intervene on the concepts to interrogate and understand agent behavior.
ADDITIONAL EXPERIMENTS
Ablation study: all components of LICORICE contribute positively to its performance. We now conduct ablation study of our three main contributions: iterative training, decorrelation, and disagreement-bassee active learning. LICORICE-IT uses only one iteration, LICORICE-DE does perform data decorrelation, and LICORICE-AC uses the entire unlabeled dataset for querying instead of a subset chosen by active learning. All components are critical to achieving both high reward and low concept error (Table g learning curves in Appendix B). However, the component that most contributes to the performance differs depending the environment. For example, com- pared with LICORICE, LICORICE-IT exhibits the largest reward gap for PixelCartPole; however, LICORICE-AC yields the largest reward gap for DynamicObstacles, and LICORICE-DE yields the largest gap for DoorKey. This difference may occur because the concepts in DynamicObstacles are simple enough that one iteration is sufficient, the largest gains can be made with active learn ing. In contrast, PixelCartPole requires further policy refinement to better estimate on-distribution concept values, the largest gains be made by leveraging multiple iterations.

---

Published as a conference paper at ICLR 2025
PixelCartPole
DoorKey
DynamicObstacles
Boxing
Algorithm
Rt
MSE І
R 1
c Error 4
1
Error
f
c Error 1
LICORICE-IT
0.53
0.08
0.99
0.09
1.00
0.00
0.98
0.14
LICORICE-DE
0.97
0.03
0.78
0.20
0,98
0.00
1.00
0.04
LICORICE-AC
0.91
0.02
0.82
0.16
0.58
0.00
0.99
0.09
LICORICE
1.00
0.02
0.99
0.05
0.99
0.00
1.00
0.05
Table 3: Ablation study results for LICORICE. All components generally help achieve better reward and lower concept error. Full results with standard deviation are Table Appendix
LICORICE is robust to various hyperparameter values. LICORICE involves few key hyperpa- rameters, described in and summarized here for convenience. In data decorrelation, 7 governs the size of the unlabeled dataset collected by the current policy and p controls the rate of acceptance of data points. Active learning involves a concept ensemble with N models. We study the effect of the values of these pyperaarametess on DynamicObstacless, finding that LICORICE achieves 96 99%
of the maximum reward across 2 values for N, 3 values for and 3 values for (Appendix B.5
5 RELATED WORK
Interpretable RL. Interpretable RL has gained significant attention in recent years (Glanois et al.
2024]. One prominent area uses rule-based methods -ucc as decision trees (Silva al.) 2020 Topin et al. 2021). logic (Delfosse et al. (004)), and programs (Verma et al., [018]] Penkov & Ramamoorthy| 2019 to represent policies. These works either assume that the state is already interpretable or that the policy is pre-specified. Unlike prior work, our method involves learning the interpretable representation (through concept training) for policy learning.
Concept Learning for RL. Due to successes in the supervised setting (Collins et al.) 2023 Sheth & Ebrahimi Kahou, 2023] Zarlenga et al.) (003), concept models have recently been used in RL. Das al. (023) trains a joint embedding model between state-action pairs and concept-based ex- planations to expedite learning via reward shaping. Unlike LICORICE, their policy is not a strict function of the concepts, allowing our techniques to be combined to provide concept-based expla- nations alongside an interpretable policy. Another example, CPM (Zabounidis et al.) 2023), is a multi-agent RL approach that assumes that concept labels are continuously available during train- ing. As we have shown, this approach uses over 1M concept labels, whereas LICORICE achieves similar performance while using 2000x fewer labels.
Learning Concepts with Human Feedback. Prior research has explored leveraging human concept labels, but not for RL and without focusing on reducing the labeling burden. For instance, Lage & Doshi- Velez (020) has users label additional information about the relevance of certain feature di- mensions to the concept label to facilitate concept learning for high-dimensional data. In a different Chauhan et al. (2023) develop test-time intervention policy for querying for concept labels to improve the final prediction. These methods do not directly tackle the issue reducing the overall labeling burden during training and could be used alongside our method. A concurrent work (Shao) ct al. 2024) also studies sample efficiency but mainly focus on human task demonstrations.
6 DISCUSSION AND CONCLUSION
In this work, we proposed LICORICE, a novel RL algorithm that addresses the critical issue of model interpretability while minimizing reliance continuous human annotation. We introduced training scheme that enables RL algorithms to learn concepts efficiently from little labeled concept data. We demonstrated how this approach reduces manual labeling effort. We conducted initial experiments to demonstrate how we can leverage powerful VLMs to infer concepts from raw visual inputs without explicit labels some environments. There are broader societal impacts of our work that must be considered, including both the impacts of using VLMs in
applications and more general considerations around interpretability. For a more detailed discussion of limitations and future work, please refer to Appendix C
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by NSF grant IIS-2046640 (CAREER). Co-author Fang is sup- ported in part by Sloan Research Fellowship. We thank NVIDIA for providing computing re- sources. We thank Zhicheng Zhang for some helpful discussions, Naveen Raman for pointing to relevant literature, and Rex Chen and Zhiyu (Zoey) Chen for their constructive comments on a draft of this work.
REFERENCES
Hello gpt-4o.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.086 088774 2023.
O Bastani, Y Pu, and A Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems, 2018
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
William H. Beluch, Tim Genewein, A. Nürnberger, and Jan M. Köhler. for active learning in image classification. 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9368 9377, 2018. URL htRp:t/::// /api www.etttttcetoooooo.o org/
The power of ensembles
CorpusID: 52838058
Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 37, pp. 5948-5955, 2023.
Maxime Chevalier-Boisvert Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, 13s/2336.13881, 2023.
Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 869-889, 2023.
Devleena Das, Sonia Chernova, and Been Kim. State2explanation Concept-basee explanations to benefit agent learning and user understanding. Advances in Neural Information Processing Systems, 36, 2023.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sehastian Sztwiertnia, and Kristian Kersting- Ocatari: Object-centric atari 2600 reinforcement learning environments. 2023.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and ex- plainable logical policies via neurally guided symbolic abstraction. Advances in Neural Informa- fion Processing Systems, 36, 2024.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et Concept embedding models: Beyond the eccuracy-explainabltte trade-off. Advances in Neural Information Processing Systems, 35:21400-2144 2022.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey interpretable reinforcement learning. Machine Learning, pp. 1-44, 2024.

---

Published as a conference paper at ICLR 2025
Niko Grupen, Natasha Jaques, Been Kim, and Shayegan Omidshafiei. Concept-based understanding of emergent multi-agent behavior. In Deep Reinforcement Learning Workshop NeurIPS 2022,
2022.
Harmampreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wort- Vaughan. Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems, pp. 1-14, 2020.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference Machine Learning, pp- 5338- PMLR, 2020.
Isaac Lage and Finale Doshi-Veleds Learning interpretable concept-based models with human feed- back. International Conference on Machine Learning: Workshop on Human Interpretability in Machine Learning, 2020.
Xiao-Yar Yang Liu, Ziyi Xia, Jingyang Rui, Jiechao Gao, Hongyang Yang. Ming Zhu, Christina Wang, Zhaoran Wang, and Jian Guo. Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning. Advances in Neural Information Processing Systems, 35:1835. 1849, 2022.
Jiayuan Mao, Tomás Lozano-Pérez, Josh Tenenbaum, and Leslie Kaelbling. Pdsketch: Integrated domain programming, learning, and planning. Advances Neural Information Processing Sys- lems, 35:36972-36984, 2022.
Catherine C Marshall and Frank M Shipman. Experiences surveying the crowd: methods, participation, and reliability. In Proceedings of the 5th Annual ACM Web Science Con- ference, pp. 234-243, 2013.
Reflections on
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodol- ogy for fast chip design. Nature, 594(7862):207-211 2021.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer humans. In AISTATS, 2023.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Teu Weng. Label-free concept bottle- neck models. arXiv preprint (iXx:2330400122 2023.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 381(6883):007-6099 1996.
Svetlin Penkov and Subramanian Ramamoorthy. Learning programmatically structured represen- tations with perceptor gradients. In Proceedings of the International Conference on Learning Representations, 2019.
Elconora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept- based explainable artificial intelligence: A survey, arXiv preprint arXiv:2312.129333 2023.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:; Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22((68):1 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (007:067770347 2017.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294, 1992.
12

---

Published as a conference paper at ICLR 2025
Jie-Jing Jing Shao, Hao-Ran Hao, Xiao Wen Yang. and Yu-Feng Li. Learning for long-horizon planning via neuro-symbolic abductive imitation, 2024. URL http://xxxiv.rrro http:///aaaxv...ooob////////
18201
Ivaxi Sheth and Samira Ebrahimi Kahou, Auxiliary losses for learning generalizable concept-based models. Advances in Neural Information Processing Systems, 36, 2023.
Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung-Hyun Son. Optimiza- tion methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference artificial intelligence and statistics, pp. 1855-1865. PMLR, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Alaa Tharwat and Wolfram Schenck. A survey on active learning: state-of-the-art, practical chal- lenges and research directions. Mathematics, (1(4):820, 2023.
Nicholay Topin. Stephanie Milani, Fei Fang, and Manuela Veloso. Iterative bounding mdps: Learn- ing interpretable policies via non-interpretable methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9923-9931, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 10, 2017.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning. pp. 1045-5054 PMLR, 2018.
Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. Learning to defer to multiple experts: Consis- tent surrogate losses, confidence calibration, and conformal ensembles. In International Confer- ence on Artificial Intelligence and Statistics, 2023.
Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao Jiajun Wu, and Yang Gao. Programmati- cally grounded, compositionally generalizable robotic manipulation. International Conference on Learning Representations, 2023.
Chao- Han Huck Yang, I Danny, Te Hung Yi Ouyang, and Pin- Yu Chen. Causal inference q- network: Toward resilient reinforcement learning. In Self-Supervision for Reinforcement Learn- ing Workshop-ICLR 2021, 2021.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Renos Zabounidis, Joseph Campbell, Simon Stepputtis, Dana Hughes, and Katia Sycara. Concept learning for interpretable multi-agent reinforcement learning. In Conference on Robot Learning, pp. 1828-1837. PMLR, 2023.
Mateo Espinosa Zarlenga, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. Learning to receive help: Intervention-awaree concept embedding models. Advances in Neural Information Processing Systems, 2023.
13

---

Published as a conference paper at ICLR 2025
g
Agent position: (2, 3)
Agent position: (2, 2)
Agent position: (0, 1)
Agent direction: down
Agent direction: up
Agent direction: up
Key position: (3, 1)
Key position: (2, 1)
Key position: (1, 5)
Door position: (4, 2)
Door position: (3, 4)
Door position: (2,
Door open: no
Door open: no
Door open: no
Can move right: yes
Can move right: no
Can move right: no
Can move down: yes
Can move down: yes
Can move down:
Can move left: yes
Can move left: yes
Can move left: no
Can move up: yes
Can move up: yes
Can move up: yes
Figure 6: Example start configurations of the DoorKey environment, along with the associated concepts for each configuration. This shows that concepts must be general enough to apply to many different configurations. As a result, the agent cannot memorize the exact position of the door and key.
A EXPERIMENTAL RESULT REPRODUCIBILITY
In this section, provide descriptions of concept definitions, prompts, and other experimental details toward the goal of reproducibility.
A.1 DEFINITIONS CONCEPTS
Table 4 provides more details regarding the concepts used in each environment, categorizing them by their names, types, and value ranges, For the PixeICartPole environment, all concepts such as Cart Position, Cart Velocity, Pole Angle, and Pole Angular Velocity continuous. In contrast, the DoorKey environment features discrete concepts like Agent Position (x and y), Key Position (x and y), and Door Open status, each with specifie value ranges. Similarly, the DynamicObstacles environment lists discrete concepts, including Agent and Obstacle positions, with corresponding value ranges.
Precise concept definitions are crucial for enabling correct agent behavior. We visualize the start configurations for DoorKey in Figure illustrate the importance of well-defined concepts in reinforcement learning. DoorKey includes a variety of initial states, each with different positions of the agent, key, and door. This diversity highlights a challenge in concept-basee RL: how to define concepts that are both specific enough to be meaningful and general enough to apply across all possible scenarios. An agent trained with poorly defined concepts might perform well some configurations but fail to generalize others, leading to suboptimal performance in new or unseen environments. As a result, concept engineering is its own separate but important problem.
A.2 LICORICE DETAILS.
In this section, we provide additional implementation details for LICORICE. The model architecture has been mostly described in the main text and we state additional details here. The number of neurons in the concept layer is exactly the number of concepts. For continuous concept values, directly use a linear layer to map from features to concept values. For discrete concept values, since different concepts have different numbers of categories, create linear classification head for each single concept, and to predict the final action, we calculate the class with the largest predicted probability for each concept.
14

---

Published as a conference paper at ICLR 2025
Environment
Concept Name
Type
Value Ranges
GPT-4o Error
Cart Position
Continuous
(-2.4,
0.01 = 0.00
Cart Velocity
Continuous
R
0.31 0.12
PixcICartPolc
Pole Angle
Continuous
(-20095, .2095)
0.01 1 0.00
Polc Angular Velocity
Continuous
R
0.63 = 0.14
Agent Position X
Discrete
5
0.36 = 0.08
Agent Position
Discrete
5
0.41 = 0.12
Agent Direction
Discrete
0.28 0.06
Key Position x
Discrete
0.20 1 0.03
Key Position y
Discrete
6
0.32 + 0.11
Door Position
Discrete
0.37 0.07
DoorKey
Door Position
Discrete
5
0.32 = 0.04
Door Open
Discrete
2
0.38 = 0.09
Direction Movable Right
Discrete
0.30 + 0.03
Direction Movable Down
Discrete
0.19 0.06
Direction Movable Left
Discrete
2
0.33 = 0.06
Direction Movable Up
Discrete
2
0.20 = 0.08
Agent Position X
Discrete
3
0.03 = 0.06
Agent Position y
Discrete
0.04 + 0.06
Agent Direction
Discrete
0.20 0.05
Obstacle Position x
Discrete
3
0.08 + 0.02
Obstacle Position y
Discrete
3
0.19 0.04
DynamicObstacles
Obstacle Position x
Discrete
0.22 0.09
Obstacle 2 Position y
Discrete
3
0.12 0.02
Direction Movable Right
Discrete
2
0.19 0.06
Direction Movable Down
Discrete
0.14 0.05
Direction Movable Left
Discrete
2
0.13 1 0.02
Direction Movable Up
Discrete
2
0.07 1 0.08
Agent Position at Frame
Discrete
160
0.81 1 0.14
Agent Position at Frame 1
Discrete
210
0.93 = 0.05
Enemy Position X Frame
Discrete
160
0.78 + 0.13
Enemy Position y at Frame
Discrete
210
0.90 1 0.07
Boxing
Agent Position at Frame 2
Discrete
160
0.67 1 0.06
Agent Position
Frame 2
Discrete
210
0.88 1 0.04
Enemy Position x at Frame
Discrete
160
0.71 0.05
Enemy Position y at Frame 2
Discrete
210
0.83 + 0.08
Ball Position x
Discrete
176
0.99 - 0.00
Ball Position y
Discrete
88
0.98 + 0.00
Ball Velocity
Discrete
176
0.95 + 0.00
Pong
Ball Velocity y
Discrete
176
0.80 = 0.80 0.00
Paddle Velocity
Discrete
176
0.78 + 0.05
Paddle Acceleration
Discrete
352
0.79 - 0.04
Paddle Jerk
Discrete
704
0.83 0.04
Table 4: Concepts and their possible values for all environments. For discrete concepts, we report the number of categories. We also provide the mean GPT-4o labeling error for each single concept, averaged across 5 seeds and corresponding to the same evaluation protocols as the ones with the largest budgets in Table 7 We use MSELoss for continuous values and accuracy for discrete ones, and in Boxing, we regard one prediction correct if it is within distance to the ground truth. The = [value] part shows the standard deviation. The average errors over concepts are 0.24 + 0.06, 0.30 + 0.02, 0.13 + 0.03, 0.82 + 0.06, and 0.88 + 0.02 respectively for the environments.
The network begins with a CNN-based feature extractor fo X - Rd, comprising three convolutional layers that map input state E X' to d-dimensional feature vector h = fo(x). We then introduce a linear layer 9s d + & for concept prediction (g). mapping extracted features to & concept values. For regression tasks, each of the k predicted concept represented as a real value; for classification tasks, each concept maps to one categorical value derived from a classification head. The action prediction component (f) is implemented
Architecture.
MI ..P with two fully
15

---

Published as a conference paper at ICLR 2025
connected layers, each containing 64 neurons with Tanh activation, taking only as input to produce action logits a = MLP(c). Notably, we share the CNN feature extractor fo between policy and value functions, decision informed by improved performance observed in preliminary experiments.
Value-Based Methods. If we were to use a value- based method the RL backbone, we would need make the following changes. First, we would need to modify [ss, a) or Q(s a) to include concept bottle- neck, such that (s,aa) = (()(s)) Then, we can conduct interleaved training similar way to LICORICE.
Policy Network
Features
Concepts
Feature Extractor. If we use the actor-critic paradigm, we propose to share a feature extractor between the pol- icy and value networks, shown in Figure Intuitively, this choice can offer several advantages compared with using image predicted concepts input for both net- works. Sharing a feature extractor enables both networks benefit from common, rich representation of the input data, reducing the number of parameters to be trained. More importantly it balances the updates of the policy and value networks. In experiments, put for both networks complicated policy learning. Conversely, relying solely on predicted concepts for the value network may limit its accuracy in value estimation, particularly the concepts do capture all the nuances relevant to the value predictions.
Value Network
Figure 7: Architecture of our concept- bottleneck actor-critic method.
observed that directly using the raw image in-
A.3 VLM DETAILS
We detail our prompts for each environment here.
PixelCartPole
Here are the past rendered frames from the CartPole environment. Please use these images to estimate the following values in the latest frame (the last one):
Prompt:
Cart Position, within the range (-2.4, 2.4)
Cart Velocity
Pole Angle, within the range (-0.2095, 0.2095)
Pole Angular Velocity
Additionally, please note that the last action taken was [last action].
Please carefully determine the following values and give concise sure to return an estimated value for each parameter, even the task may look challenging.
by one. Make
Follow the reporting format:
Cart Position: estimated_value
Cart Velocity: estimated_
Pole Angle: estimated value
Pole Angular Velocity: estimated
16

---

Published as a conference paper at ICLR 2025
DoorKey
Here an image of 4x4 grid composed of black cells, with each cell either empty or containing an object. Each cell is defined by an integer-valued coordinate system starting at (1, 1) the top-left cell. The coordinates increase rightward along the x-axis and downward along the y-axis. Within this grid, there is a red isosceles triangle representing the agent, yellow cell representing the door (which may visually disappear the door is open), a yellow key icon representing the key (which may disappear), and one green square representing the goal. Carefully analyze the grid and report on the following attributes, focusing only on the black cells the gray cells excluded from the active black area.
Prompt:
Detailed Instructions:
1.
Identify and report the coordinates (x, y) of the red triangle (agent). En- sure the accuracy by double-checking the agent's exact location within the grid.
Agent Position:
Agent Direction: Specify the direction the red triangle is facing, which the orientation of the vertex (pointy corner) of the isosceles triangle. Choose from 'right', 'down', 'left', or "up'. Clarify that this direction is independent of movement options.
3. Key Position: Provide the coordinates (x, y) where the key is located. If the key is absent, report (0, 0). Verify visually that the key is present not before reporting.
4. Door Position:
Position:
Determine and report the coordinates (x, y) of the door.
Status: Assess whether the door is open or closed (closed means the door is visible as whole yellow cell, while open means the door disappears visually). Report as 'true' for open and 'false' for closed. Double-check the door's appearance to confirm it is open closed.
5. Direction Movable: Evaluate and report whether the agent can move one cell in each specified direction, namely, the neighboring cell in that direction is active and empty (not key, closed door, or grey inactive cell):
Right (x + 1): Check the cell to the right.
Down (y + 1): Check the cell below.
Left 1): Check the cell the left.
Up (y 1): Check the cell above.
Each direction's feasibility should be reported 'true" if clear and within the grid, and 'false" otherwise.
Reporting Format: Carefully report each piece of information sequentially, following the for- mat 'name: answer'. Ensure each response is precise and reflects careful verification of the grid details viewed.
17

---

Published as a conference paper at ICLR 2025
DynamicObstacles
Prompt: Here an image 3x3 grid composed of black cells, with each cell either empty or containing an object. Each cell is defined by an integer- valued coordinate system starting at (1, 1) the top-left cell. The coordinates increase rightward along the x-axis and downward along the y-axis. Within this grid, there is a red isosceles triangle representing the agent, two blue balls representing obstacles, and one green square representing the goal. Please carefully determine the following values and give concise answers one by one:
1. Agent Position: Identify and report the coordinates (x, y) of the red triangle (agent). En- sure the accuracy by double-checking the agent's exact location within the grid.
2. Agent Direction: Specify the direction the red triangle is facing, which is the orientation the vertex (pointy corner) of the isosceles triangle. Choose from 'right', 'down', 'left', 'up'. Clarify that this direction is independent of movement options.
3. Obstacle Position: Identify and report the coordinates of the obstacles in ascending order. Compare the coordinates by their x-values first. If the x-values are equal, compare
by their y-values.
(a) First Obstacle: Provide the coordinates (x, y) of the first blue ball.
(b) Second Obstacle: Provide the coordinates (x, y) of the second blue ball.
4. Direction Movable: Evaluate and report whether the agent can move one cell in each
specified direction, namely, the neighboring cell in that direction is active and empty (not obstacle or out of bounds):
Right (x + 1): Check the cell to the right.
Down (y + 1): Check the cell below.
Left 1): Check the cell the left.
Up (y 1): Check the cell above.
Each direction's feasibility should be reported as 'true' clear and within the grid, and 'false' otherwise.
Reporting Format: Carefully report each piece of information sequentially, following the for- mat 'name: answer'. Ensure each response is precise and reflects careful verification of the grid details viewed.
18

---

Published as a conference paper at ICLR 2025
Boxing
Here are two consecutive rendered frames from the Atari Boxing environment. The game screen is 160x210 pixels. with (0, 0) at the top-left corner. The x-coordinate increases rightward, and the y-coordinate increases downward. For each frame, estimate the following values as integers:
Prompt:
The white player's and coordinates
The black player's and y coordinates
Please carefully determine the following values and give concise answers by one. Make sure to return an estimated value for each one, even the task may look challenging.
Follow the reporting format:
frame white x: estimated_ value
frame white y: estimated value
frame black x:
frame black y: estimated_value
frame 2 white x: estimated_
frame white y: estimated value
frame 2 black x:
frame 2 black y: estimated_vailee
Pong
Here are four consecutive rendered frames from the Atari Pong environment. The game screen 84x84 pixels, with (0, 0) the top-left corner. You need look the ball and the paddle in the right. The x-coordinate increases downward, and the y-coordinate increases rightward.
Prompt:
Please determine the following values and give concise answers one by one. Make sure to return estimated value for each one, even if the task may look challenging.
x coordinate of the ball relative to the paddle in the first frame
coordinate of the ball in the first frame
velocity of the ball calculated from the first and second frames
y velocity of the ball calculated from the first and second frames
velocity of the paddle calculated from the first and second frames
acceleration of the paddle calculated from the first three frames
jerk of the paddle calculated from the four frames
Follow the reporting format in your response.
x coordinate of the ball: estimated value
coordinate of the ball: estimated value
X velocity of the ball: estimated_vall value
velocity of the ball: estimated_vvll value
velocity of the paddle: estimated_ value
acceleration of the paddle: estimated_vallu
jerk of the paddle: estimated value
19

---

Published as a conference paper at ICLR 2025
A.4 EXPERIMENTAL DETAILS
Behavior learning hyperparameters. For the PPO hyperparameters, we set 4 10 total timesteps for PixeICartPole and DoorKey, 10t for DynamicObstacles, 1.5 10" for Boxing, and 10" for Pong- For PixelCartPole, DoorKey, and DynamicObstacles, we use 8 vectorized environments, horizon 4096, 10 epochs for training. batch size of 512, learning rate 3 10 entropy coefficient 0.01, and value function coefficient 0.5. For Boxing and Pong. we use 8 vectorized environments, horizon = 1024, 4 epochs for training. batch size of 256, learning rate 10-1, entropy coefficient 0.01, and value function coefficient 0.5. For all other hyperparameters, we use the default values from Stable Baselines Raflin et al. (0211.
Concept learning hyperparameters, For the concept training, we set 100 epochs with Adam opti- mizer with the learning rate linearly decaying from 3 10 to for each iteration in PixelCartPole, Boxing, and Pong. In DoorKey and DynamicObstacles, ing rate, yct set 50 epochs instead and set early stopping with threshold lincarly increasing from 10 to 20, to incentivize the concept network not to overfit in carlier iterations. The batch size is 32. We model concept learning for PixelCartPole as a regression problem (minimizing mean squared error). We model concept learning for DoorKey, DynamicObstacles, Boxing,and Pong as classification problems.
use the same optimizer and initial learn-
LICoRiCE-specifie hyperparameters. For LICORICE, we the ratio for active learning 10, batch size to query labels in the active learning module h = 20, and the number of ensemble models N for the first three environments. For the complex environments Boxing and Pong, we choose the ratio for active learning = 4. batch size to query labels in the active learning module b /m/5 and number of ensemble models N speed. The default number of iterations chosen in our algorithm is M = 4 for PixelCartPole, M 2 for DoorKey, DynamicObstacles, and M
to make a balance between performance and
for Boxing and Pong. The sample acceptance rate p 0.02 for PixelCartPoleand = 0.1 for the other four environments. The Random-Q baseline uses the same sample acceptance rate p as LICORICE. In the complex environments Boxing and Pong, use both the KL-divergence penalty and PPO loss at the end of the algorithm to improve optimization with B = 0.01, as mentioned in Section 3
Computational resources. We use NVIDIA A6000 and NVIDIA RTX 6000 Ada Generation. Each of our training program uses less than 3GB GPU memory. For Boxingand Pong, each run takes less than 20 hours to finish. For PixelCartPole and DoorKey, each takes less than 9 hours to finish. For DynamicObstacles, each run takes less than hours to finish.
Experimental replicability. For all experiments, train the models. We then evaluate on the environment with seed 42 with 100 episodes and take the average of the reward across the episodes and the concept error across the observations within each episode.
seeds [123, 456, 789, 1011, 1213] to
20

---

Published as a conference paper at ICLR 2025
Algorithm
R1
Error 4
Sequential-Q
0.23 + 0.09
0.10 0.04
Disagreement-O
0.36 + 0.20
0.10 0.04
Random-Q
0.34 + 0.14
0.07 0.02
PixelCartPolc
LICORICE
1.00 0.00
0.02 1 0.00
LICORICE+GPT-40
0.06 0.01
0.25 + 0.08
CPM
1.00 = 0.00
0.01 0.00
Sequential
0.46 1
0.46 0.05
Disagreement-O
0.76 0.12
0.39 + 0.07
Random-Q
0.89 + 0.03
0.27 0.08
DoorKey
LICORICE
0.99
0.05 - 0.01
LICORICE+GPT-40
0.83 + 0.06
0.31 0.01
CPM
1.00 + 0.00
0.00 + 0.00
Sequential-
0.79 0.44
0.01 0.01
Disagreement-O
0.99 0.01
0.00 0.00
Random-Q
0.98 0.01
0.02 0.01
DynamicObstacles
LICORICE
0.99 + 0.00
0.00 + 0.00
LICORICE+GPT-44
0.88 0.09
0.13 0.03
CPM
0.97
0.00 = 0.00
Sequential- Q
0.87 + 0.20
0.32 0.26
Disagreement-i Q
0.83 + 0.14
0.20 + 0.21
Random-Q
0.96
0.14 0.10
Boxing
LICORICE
0.05 + 0.01
LICORICE+GPT-40
0.22 0.01
0.98 0.01
CPM
1.00 0.05
0.00 + 0.00
Sequential-O
0.98 0.02
0.54 0.03
Disagreement Q
0.94 0.09
0.57 - 0.05
Random-Q
0.94 0.05
0.60 + 0.04
Pong
LICORICE
1.00 + 0.02
0.24 0.16
LICORICE+GPT-40
0.13 + 0.17
0.71 + 0.01
CPM
1.00 + 0.00
0.00 + 0.01
Table Evaluation of the reward R and concept error achieved by all methods in all environments. This is a extended table from Figure 2 The reward is reported as the fraction of the reward upper bound. For PixelCanPole, the e error the MSE. For the other environments, the e error accuracy. The first five algorithms are given budget of B [500, 300, 300, 3000, 5000) for each environment, from top to bottom; CPM given an unlimited budget (in practice, it uses 4M, 4M, IM, 15M, 10M concept labels respectively). The + [value] part shows the standard deviation. For each environment, we bold the highest reward and lowest concept error among the budget- constrained methods.
B ADDITIONAL RESULTS
In this section, we present both the numerical results as tables from the figures in the main paper, in addition to experimental results not reported in the main paper.
B. BALANCING CONCEPT PERFORMANCE AND ENVIRONMENT REWARD
In T we present all of the numerical results from Figure 2 including standard deviation. Our method enjoys low variance all environments terms of both concept and reward.
21

---

Published as a conference paper at ICLR 2025
Environment
B
M
R1
e Error +
300
0.28 0.10
0.15 0.07
PixelCartPole
400
0.77+
0.05 0.01
500
1.00
0.02 0.00
100
0.77 0.04
0.26 0.04
DoorKey
200
0.93 = 0.02
0.09 0.01
300
0.99 + 0.01
100
0.96 0.02
0.04
DynamicObstacles
200
0.98 0.01
0.00
300
0.99 + 0.00
0.00+0.00
1000
0.94 0.05
0.11
Boxing
2000
5
0.98 + 0.05
0.10 +.06
3000
1.00 0.02
0.01
1000
0.99 + 0.00
0.65 + 0.04
Pong
3000
5
1.00 = 0.00
0.43 + 0.05
5000
1.00 + 0.02
0.24 + 0.16
Table 6: Performance of LICORICE on all environments for varying budgets. We select M opti- mize R error without additional weighting since they are observed to have the same magnitude. The reward is reported the fraction of the reward upper bound. For PixelCartPole, error is MSE; for the other environments, error is accuracy. The = [value] part shows the standard deviation. This shows more complete version of the results in Figure 3
B.2 BUDGET ALLOCATION EFFECTIVENESS
In Table 6 we present an extension of the results in Figure including the standard deviation. As expected, the budget increases, the standard deviation for both the reward and concept error tends to decrease. The one exception is the reward for PixelCarPPole. Interestingly, the standard deviation highest for B 400. We suspect this is because the concept errors may be more critical here, leading to higher variance in the reward performance.
22

---

Published as a conference paper at ICLR 2025
Environment
B
M
R
L.CORICE+GPP-440 Error
GPT-4o e Error
300
0.06 + 0.02
0.17 0.25 0.15
0.19 = 0.25
PixelCartPole
400
0.07 + 0.02
0.22 + 0.15 0.24 + 0.07
500
0.06 0.01
0.25 2.05 0.08
100
0.71 + 0.04
0.45 + 0.04
0.31 + 0.03
DoorKey
200
0.74 + 0.04
0.33 + 0.01
0.31 + 0.03
300
0.83 0.06
0.31 + 0.01
0.30 + 0.03
100
0.27 0.37
0.20 = 0.03
0.16 + 0.03
OystmicObstacles
200
0.93 + 0.05
0.13 = 0.03
0.12 0.05
300
0.88 + 0.09
0.13 + 0.03
0.13 0.03
1000
0.34 + 0.09
0.98 + 0.01
0.78 0.01
Boxing
2000
0.23 + 0.11
0.98 + 0.01
0.79 + 0.01
3000
0.22 + 0.01
0.98 + 0.01
0.79 0.02
1000
0.00 + 0.00
0.79 + 0.08
0.88 0.01 0.86 + 0.03 0.87 + 0.02
Pong
3000
0.00 + 0.00
0.74 + 0.07
5000
0.13 + 0.17
0.71 + 0.01
Table 7: Performance of LICORICE with GPT-4o integrated into the loop for all environments across different budgets, along with the concept labeling error of GPT-4o as reference. We select M to optimize R- c error without additional weighting since they are observed have the same magnitude. GPT-4o e error is evaluated on a random sample of 50 observations from the same rollout set used for LICORICE+GPT-40. due to API cost constraints. This shows more complete version of the results in Figure 3
Environment
Sequential-Q
Disugreement-QQ
Random-O
R
c Error 4
R
Error
R1
Error
300
0.17 +: 0.03
0.11 + 0.03
0.22 = 0.04
0.20 :: 0.04
0.24 0.07
0.14 +: 0.04
PixelCanPole
400
0.23 1 0.05
0.10
1: 0.03
0.26 1 0.06
0.13
0.03
0.28 0.07
0.09 1: 0.05
500
0.23 1 0.07
0.10
1 0.03
0.39 1 0.28
0.09
0.03
0.35 1 0.16
0.07 0.02
100
0.33 + 0.07
0.51 + 0.08
0.61 + 0.14
0.33 + 0.07
0.70 0.07
0.30 + 0.02
DoorKey
200
0.43 + 0.09
0.49 + 0.03
0.72 = 0.09
0.28
0.07
0.85 = 0.04
0.24 + 0.04
300
0.51 + 0.11
0.45 + 0.08
0.76 0.12
0.41 + 0.06
0.90 =:00=00 0.02
0.27 + 0.08
100
0.43 + 0.47
0.04 + 0.04
0.82 0.18
0.04
0.34 0.46
0.05 + 0.05
DynamicOhstacles
200
0.76 + 0.43
0.03 + 0.03
0.99 + 0.02
0.01 0:1::00 0.00
0.76 0.42
0.03 + 0.02
300
0.78 1: 0.44
0.01 1: 0.01
0.99 1: 0.01
0.00 0.00
0.97 0.01
0.02 1: 0.02
1000
0.84 + 0.20
0.35 + 0.12
0.93 0.05
0.11
0.83 0.24
0.25 + 0.05
Boxing
2000
0.88 + 0.12
0.19 - 0.11
0.93 1 0.06
0.14 0.04
0.83 0.11
0.30 + 0.18
3000
0.87 + 0.20
0.32 + 0.26
0.83 = 0.14
0.21
0.96 196=100 0.04
0.14 0.10
1000
0.91 1: 0.12
0.74 1: 0.07
0.84 0.21
0.68
0.06
0.74 0.21
0.73 1: 0.05
Pong
3000
0.98 + 0.02
0.58 + 0.08
0.98 0.01
0.64 + 0.03
0.96 0.04
0.60 + 0.04
5000
0.98 +: 0.02
0.54 + 0.03
0.94 0.09
0.57 0.05
0.94 0.05
0.60 + 0.04
Table 8: Performance of Sequential-Q. Disagreement-O2 and Random-Q for all environments across different budgets. The reward is reported as the fraction of the reward upper bound. For PixelCart- Pole, error MSE; for the other environments, error accuracy. The + [value] part shows the standard deviation. This shows a more complete version of the results in Figure 3
B.3 INTEGRATION WITH VISION-LANGUACE MODELS
In [ we present an extension of the results in Figure Saureeiin the main paper, including the standard deviation. Interestingly, the standard deviation for the reward obtained by using LICORICE with GPT-4o as the annotator does not always follow the same trend as shown in Tablelo((hhe) access to a more accurate human annotator). Instead, the standard deviation is relatively consistent for PixelCartPole, regardless of the budget. It steadily decreases for DoorKey, as expected. However, in DynamicObstacles we see an increase when B = 300. We are not sure of the cause this. Perhaps at this point the algorithm begins overfitting to the errors in the labels from GPT-4o (the
assume
23

---

Published as a conference paper at ICLR 2025
Figure 8: LICORICE+GPT--0 waits until the coast clear to move to the goal. appears to make small mistake in the bottom left, requiring it to wait slightly longer than necessary to navigate to the goal.
concept eтrоr is the same for 200 and 300 labels). Further investigation is required understand the underlying factors contributing to this anomaly.
Different tasks and concepts have various difficulties for GPT-4o. Table list detailed concept errors for concepts in all environments. In PixelCartPole, cart position and pole angle have smaller errors, while velocities require understanding multiple frames and thus are harder to predict aceu- rately. For DoorKey and DynamicObstacles, different concepts have slightly varying concept errors, indicating visual tasks have different difficulties for GPT-4o. Agent direction has as high as around 0.3 prediction enror for both DoorKey and DynamicObstacles. Direction movable is also hard for DoorKey, with a high concept error even if it is a binary concept. We posit it requires the correct understanding of more than particular object ensure correctness. The concept accuracies in DoorKey are generally higher than DystmicObstucllss, suggesting GPT-4o more struggles with a larger grid.
GPT-4o-trained RL agent recovers from a mistake. In Figure we show an example a GPT-4o-trained RL agent DynamicObstacles, in which the agent appears wait until is safe to move towards the goal: the green square. The image sequence shows the agent (red triangle) starting from its initial position and moving to the right. then cornered by an obstacle (blue circle), then both obstacles. In the bottom left comer frame, it appears to make mistake by turning to the right, meaning missed a window to escape. Finally, it moves the goal when the path clear in the second-to-last and last frames (bottom right). This behavior highlights that the agent may still learn reasonable behavior even if the concept labels may incorrect (causing make mistake, as in the bottom left frame).
B.4 ABLATION
In Table we present an extension of the results Table including standard deviation. Figure shows all learning curves for ablations in all enviromments. In PixelCartPole, we clearly see the ben- efit of iterative strategies on reward: LICORICE, LICORICE-DE and LICORICE-AC consistently increase in reward and converge at high levels, but LICORICE-IT converges at less than 50% of the optimal reward. We also see that LICORICE-IT also achieves higher concept error, indicating that it struggles learn the larger distribution of concepts induced by non-optimal policy. In DoorKey, all algorithms steadily increase in reward. However, we can see a dip at around 10 environment steps where begin the second iteration for LICORICE, LICORICE-AC. and LICORICE-DE. Al- though LICORICE-IT achieves similar reward to LICORICE, LICORICE achieves lower concept error, which is beneficial for the goal of interpretability. Finally, in DynamioObstacles, LICORICE-
24

---

Published as a conference paper at ICLR 2025
Algorithm
R1
Error 4
LICORICE-IT
0.53 + 0.26 0.26
0.08 0.03
LICORICE-DE
0.97 + 0.05
0.03
PixelCartPole
LICORICE-AC
0.02
LICORICE
1.00
0.02
LICORICE-IT
0.99 0.00
0.09
LICORICE-DE
0.78
0.20
DoorKey
LICORICE-AC
0.82
0.10
0.06
LICORICE
0.99 0.01
0.05
LICORICE-IT
+
0.00
LICORICE-DE
0.98 0.01
0.00
DynamicObstacles
LICORICE-AC
0.58
0.00 0.53
LICORICE
0.99 0.00
0.00
LICORICE-IT
0.98 + 0.04
0.04
LICORICE-DE
1.00 + 0.04
0.04
Boxing
LICORICE-AC
0.99 + 0.01
0.09 0.06
LICORICE
1.00 0.02
0.05
Table 9: Ablation study results for LICORICE in the first four environments, where study the effect of each component to compare with LICORICE. This shows a more complete version of the results in Table
PixelCartPole-v1
DoorKey-7x7
DynamicObstacles-55
1.00
% Optimal
0.75
Reward
0.50
0.00
0.4
Error
0.2
2
2
0.2 Environment Steps
0.4
0.6
0.8
1.0
Environment Steps
Environment Steps
LICORICE-IT
LICORICE-AC
LICORIC-DDE
LICORICE
Figure 9: Learning curves for ablations in the first three environments. Shaded region shows 95%
CI, calculated using 1000 bootstrap samples.
AC lags behind the most in terms of both reward and concept error. This result indicates that the active learning component most important this environment.
Impact on data diversity. We hypothesize that data decorrelation enables better performance by increasing the diversity of the training data. As a result, we investigate the role of data decorrelation on state diversity by calculating the average distance to the & nearest samples in the concept label space for all labeled samples. The intuition is that a larger average distance to nearest neighbors indicates samples are more spread out in the space, suggesting higher diversity in the training data. Let D = {(s), ....... (sb, c be the labeled dataset and Neighbor(C,; k) be the index set of nearest neighbors with k 10 for all environments, then calculate diversity
25

---

Published as a conference paper at ICLR 2025
LICORICE
LICORICE-DE
Environment
R
PixelCartPole
0.12 0.00
1.00 + 0.00
0.00
0.10 + 0.01
0.97 0.05
DoorKey
1.12 + 0.18
0.99
0.25 0.00
0.78 0.08
DynamicObstacles
0.93 + 0.04
0.90
0.99
0.98
Boxing
7.20 0.20
+.97
6.53 0.08
0.97
Table 10: Training data diversity (d) and reward (R) with and without data decorrelation. Higher d is better (more diverse). For each environment, the highest d bolded. When the standard deviations overlap, we bold both.
= bk:
Σ
- 1
i= JENeighbor(cr..)
We present d along with the reward in Table 10 for LICORICE and LICORICE- DE. As hypoth- esized, decorrelation increases diversity in 3 of the 4 environments. LICORICE achieves higher reward than LICORICE-DE. In DynamicObstacles, the state diversity and reward is similar for both LICORICE and LICORICE-DE. We believe that, in this environment, decorrelation is less critical because neighboring states are inherently more dissimilar due to the dynamic nature of the obstacles.
26

---

Published as a conference paper at ICLR 2025
0.9
Noraard (R)
0.56
8.00
pAs9, 19.83
past R5
pstat, c78 R3
pdus 178 R3
pa. e78 83
P8L. R3
PC2 8.83
PAO e28
PBI. c20
PBt c2e
no cr8
no oza
282
p82
Ryperparamete Combration
Figure 10: Hyperparameter sensitivity results for Dynamicossttales. Bars represent standard devi- ation. Here, bars grouped by the same p values (acceptance probability). Bars of the same color share the same r value (budget multiplier), and bars with the same pattern share the same value for N (committee size). Generally, p 0.1 tend to perform better overall with greater stability. Having a larger committee (N 5 vs. N = 3) also tends to help with performance.
Budget
100
200
300
Error
R
Error
R
Error
Disagreement 0.96
0.05
0.97
0.01
1.00
0.00
Entropy 0.95
0.07
0.98
0.02
0.99
0.00
Table 11: Comparison of disagreement- and entropy-based active learning strategies for Dynam- icObstacles under different labeling budgets. There is essentially difference between the two strategies.
B.5 HYPERPARAMETER SENSITIVITY
We now seek to understand how sensitive LICORICE is to the choice of hyperparameters. Figure I0 shows the results this experiment for DynamicObstacles. Interestingly, we find that the main difference between the settings is the variation. Overall, LICORICE is relatively robust the choice of hyperparameter settings in this environment, with the final average reward ranging between 96% and 99.5% of the optimal. The concept error for all settings was 0.00 +'t 0.00, do not plot here.
B.6 CHOICE OF ACQUISITION STRATEGY
We now investigate the impact of the choice of acquisition strategy on both reward and concept accuracy. We implemented entropy-based disagreement measurement for the committee and studying its effects on a MiniGrid environment. Specifically, we implement the following vote entropy acquisition function: ((s)
Vigk Visk N log "N' where s; is the input state, K is the number of concepts we learn, Ps is the number of classes for the k-th concept, Vija is the number of votes for the j-th class of the k-th concept for state 8;, and N is the number of committee members. Table IT shows these results on DynamicObstacles. There is essentially no difference between the two strategies terms of both reward and concept error.
Σk> ES
27

---

Published as a conference paper at ICLR 2025
C ADDITIONAL DISCUSSION
C.I LIMITATIONS AND FUTURE WORK
While our approach has demonstrated promising results, there are several limitations to be addressed in future work. One significant challenge is the difficulty VLMs face with certain types of concepts, especially continuous variables. This limitation can impact the overall performance of concept- based models, especially in domains where continuous data is prevalent. Furthermore, although VLMs can be successfully used for automatic labeling of some concepts, there are still hallucination issues (Achiam et al., 2023) and other failure cases, such as providing inaccurate counts. We believe that future work that seeks to improve general VLM capabilities and mitigate hallucinations would also help overcome this limitation. Addressing this issue could involve developing specialized tech- niques or using existing tools and libraries to better complement VLM capabilities.
Another area for future improvement the refinement of our active learning and sampling schemes. Our current method employs an disagreement based acquisition function to select the most informa- tive data points for labeling. While this approach is effective, there is potential for exploring more so- phisticated active learning strategies, such as incorporating advanced apploration-exploiaaaii offs or leveraging recent advancements in active learning algorithms (Tharwat & Schenck (023).
trade-
Furthermore, scaling challenges may arise from environmental complexity, such when only subset of given concepts are relevant or when learning certain concepts is prerequisite to gathering training data for others. Future research could explore using attention mechanisms (Vaswani et al., 2017) or sparse coding (Olshausen & Field 1996) to identify relevant concepts. Another exciting direction is the work in human-Al complementarity and learning-to-defer algorithms (Mozannar eet al. 2023 Verma et al.) (023) to train an additional classifier for deferring labeling to a person when the chance of environment; future work could investigate the use of human VLM teams to determine a reasonable concept set for the environment.
is high. In our work, we assume known set of concepts for the
Finally, designing a concept-based representation for RL remains an open challenge. Our work provides a few illustrative examples, but the exact design of these representations can significantly impact performance, often for reasons that are not entirely clear especially when using VLMs as annotators. Prior work (Das et al.) 2023) proposed some desiderata for concepts RI., but future work could refine these principles, especially in the face of VI.M annotators. Future work could also include systematically investigating the factors that influence the effectiveness of different concept- based representations in RL. This could involve extensive empirical studies, theoretical analyses, and the development of new design principles that guide the creation of effective concept represen- tations. Understanding these factors better will help in creating more reliable and interpretable RL models, ultimately advancing the field and broadening the applicability of concept- based approaches in various RL tasks.
C.2 BROADER IMPACTS
Interpretability in RL. Incorporating concept learning with RL presents both positive and nega- tive societal impacts. On the positive side, promoting interpretability and transparency in decision- making fosters trust and accountability. However, in cases where yields incorrect results, stake- holders might be misled into trusting flawed decisions due to the perceived transparency of the model (Kaur et al. 2020). Unintended misuse could also occur if stakeholders lack the technical expertise to accurately interpret the models, leading to erroneous conclusions and potentially harm- ful outcomes. To mitigate these risks, an avenue for future work is developing clear guidelines for interpreting these models and tools to scaffold non-experts' understanding of the model outputs.
Using VLMs for concept labeling. On one hand, VLMs have the potential to significantly im- prove the efficiency and scalability of labeling processes, which can accelerate advancements in various fields. By automating the labeling of large datasets, VLMs can help reduce the time and cost associated with manual labeling. However, there are important ethical and social considerations to address. One major concern is the potential for bias in the concept labels generated by VLMs. If these models are trained on biased or unrepresentative data, they may perpetuate or even amplify existing biases, leading unfair or discriminatory outcomes. This is particularly problematic in
28

---

Published as a conference paper at ICLR 2025
sensitive applications like hiring, lending, or law enforcement, where biased decisions can have sig- nificant negative impacts on individuals and communities. Furthermore, there are privacy concerns related to the data used to train VLMs. Large-scale data collection often involves personal informa- tion, and improper handling of this data can lead to privacy violations. To mitigate these risks, future work could include developing robust data governance frameworks protect individuals' privacy and comply with relevant regulations.
29


Reviews:
[{"review_id": "pl15CYyXDq", "reviewer": "Reviewer_DRKb", "summary": "The paper introduces LICORICE, a training paradigm for efficient concept-based reinforcement learning. It reduces the need for extensive annotations. LICORICE uses active learning to label only informative data and applies data decorrelation for diverse samples. Experiment results show that LICORICE maintains performance while reducing annotation requirements.", "questions": "1. In Algorithm 1, please clarify the definitions of $s$ and $c$.\n2. In line 187, there is an extra “the.”", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and structured, followed by the motivation and explanation.\n2. By combining iterative training, data decorrelation, and disagreement-based active learning, LICORICE significantly reduces annotation requirements without sacrificing performance.\n3. The model achieves better results compared to baseline methods.", "weaknesses": "1. During concept learning, $g$ is trained using $D_{train}$. However, as the agent's behavior changes, it may encounter new concepts during behavior learning. This can lead to a distribution mismatch between the data used for concept learning and the actual states visited during behavior learning.\n2. The authors mention that LICORICE reduces the need for extensive annotations. I agree with this point. However, I think right now people care more about the computation efficiency instead of the cost of labeling data. It would be better to show the training efficiency with less budget.\n3. In the budget reduction session, the authors initially use a large budget and then reduce it to a smaller one. I suggest starting with a small budget and gradually increasing it to identify the minimum human labeling effort required to maintain an acceptable reward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rKrb994xo", "reviewer": "Reviewer_AVBF", "summary": "This papers proposes an active learning scheme to minimize the required concept labels for training concept policy models, which are RL policy models with concept bottlenecks. The method is simple and requires training an ensemble of concept models to estimate the disagreement for prioritizing samples for labeling. Extensive experiments show the label-efficiency of their proposed scheme over the non-active learning golden CPM, and other simple label-restricted baselines. The authors also explored replacing the concept labeling process with a VLM (gpt-4o) and find mixed results with its success.", "questions": "Here are a couple of clarification questions\n* In P4L120, Line18 of the algorithm states \"Continue training the concept network g on Dtrain, using Dval for early stopping\". Does Dtrain refer to only the new data added in this iteration, or the entire dataset collected thus far?\n* In P6L285, the authors state \"In Disagreement-Q, the agent similarly spends its budget at the beginning of its learning process\". What does the beginning of its learning process refer exactly?\n* In P9L449, Figure 3 only shows the performance of LICORICE and its variants. Including the 3 naive baselines (particularly random) would help readers understand how LICORICE relatively scale with respect to budget.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "## Originality\n\nAll the subcomponents of the proposed method are well-established techniques from their respective fields but chaining them together is an original contribution. The problem of label-efficiency has not been studied for CPM as well (even under-explored in general CBMs).\n\n## Quality\n\nThe paper is extremely well written. Common questions one might have are all answered and extensively studied. This includes basic ablation studies of each component of the technique, to all the possible things one might do to a model with concept bottlenecks (e.g. intervention, automate concept labeling). The detailed documentation of experiment design (particularly the specification of how the validation set is curated in the algorithm) reflects the rigor of the work. Well done.\n\n## Clarity\n\nThe problem statement is well formulated. The motivation is somewhat justified (though I do highly doubt whether label efficiency is the main problem for adopting CPM in place of regular policy networks in RL settings. I digress). The algorithm is well documented. Assumptions and settings are clearly stated.\n\n## Significance\n\nThe weakest part of this paper might be its significance. It is true that interpretability via a concept bottleneck would benefit human-understanding of the decision making of RL agents. However, it does not seem like data scarcity is the biggest issue for implementing CPMs, as current settings with concepts defined all come with sufficient concept labels. If there actually exists a real life scenario where the concept labels are hard to obtain, perhaps the active learning setting makes more sense. That being said, the significant performance improvement of LICORICE over the random baseline is non-negligible.", "weaknesses": "Please refer to the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZ1regC4Qi", "reviewer": "Reviewer_1gdK", "summary": "This paper focuses on concept-based interpretable reinforcement learning. The authors note that existing concept policy model-based methods require a large amount of concept annotations. This is typically too costly in reinforcement learning problems that require extensive interaction for policy learning. To address this issue, the authors introduce active learning, selectively annotating concepts to achieve label-efficient learning.", "questions": "Regarding the present study, how is the aspect of interpretability substantiated? It would be beneficial to provide the specific methods or analyses employed to demonstrate the interpretable nature of the proposed approach.\n\n---- \n\nAfter the rebuttal, most of my concerns have been addressed. I have raised my score to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work studies an interesting problem, making RL policies interpretable is important in practical applications. \n2. The proposed method indeed addresses the data challenges in the RL policy learning process. This includes separating concept learning from policy learning and using data decorrelation to focus on more diverse states. These aspects make this method more than simply applying traditional (disagreement-based) active learning. \n3. The authors validated the effectiveness of the proposed method in several environments.", "weaknesses": "1. This work emphasizes interpretable reinforcement learning, but the authors seem to have directly substituted concept learning for interpretability. I understand that concept learning can, to some extent, help improve interpretability, but the authors should demonstrate more \"interpretable\" results beyond just performance. I noticed that Table 4 provides definitions of concepts for each task, but how do these concepts influence the actions of different policy models? For me, the relationship between concepts and actions is what truly reflects the interpretability of decisions. For example, when the door is locked and the key is to the agent's right, the agent will move to the right; when the door is open, the agent will move towards the door. The authors should provide similar analyses to demonstrate the interpretability of decisions, rather than stopping at the accuracy of concept learning. \n2. The current experiments are mainly conducted on limited concepts. Can LICORICE be validated on a larger concept space? I suggest the authors validate their approach on more complex tasks. The chosen active learning baseline is relatively simple, but considering this is a first exploration for label-efficient concept-based reinforcement learning, it is acceptable.  \n3. Finally, while it may deviate from the topic of interpretable RL, one of the greatest contributions of concept learning to RL is its ability to help RL achieve compositional generalization. I suggest including a related discussion in the section where the concept model is applied to RL.   \nPDSketch: Integrated domain programming, learning, and planning. NeurIPS'22  \nProgrammatically Grounded, Compositionally Generalizable Robotic Manipulation. ICLR'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ruaPjSg2RL", "reviewer": "Reviewer_UqSw", "summary": "Current RL methods lack interpretability, which is necessary for stakeholders to comprehend and trust them. Previous work on concept bottlenecks uses a predefined vocabulary, requiring real-time concept annotation when applied to RL. The authors propose a training scheme (LICORICE) that allows RL algorithms to efficiently learn concept-based policies by querying annotators to label only a small set of data.", "questions": "1. The original concept bottleneck models use images as input, which is already easy for human to understand. Also concept bottleneck models are intended to make the final prediction interpretable by decomposing a complex concept (e.g., an object) into its atomic-level properties, then doing classifications on top of them. But the authors mention “”opaque raw inputs”” in line 46-47. Could the authors explain more about this type of input?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n2. The framework enables interpretability on the state estimation module of the RL framework. Human could interpret the states by reading the intermediate variables.", "weaknesses": "1. This method is not general enough. Because environments like cart pole are very simple, it’s possible to use some textual “words” to describe the state. But what about in the real environment with much more states? Also, some of the states are not describable by language.\n2. In line 73, the paper mentions relying on human labelers is potentially biasing the model training process. But it seems the paper doesn't mention what is the potential ground truth data (without human labeling) that is less biased. The paper proposes using VLMs (obviously, VLMs are also trained on vision-text data with human knowledge) to automate the concept extraction, but it's unclear that it will not bias the RL model training process as human labelers do?\n3. line 76-90, the authors mention two concerns: 1) concept distribution learned from a policy, 2) data diversity.  But in line 91-96, the paper only mentions that the experiments are focused on human annotation and VLM-based annotation. It seems there is not a clear section to address these concerns. Ablation study shows the components are all positive. But it seems no experiment is explicitly talking about training data diversity , e.g., after applying the data decorrelation strategy how does the diversity change?\n4. In line 98-99, authors claim they are the first to investigate “limited concept annotation budget for interpretable machine learning”. But there are some papers like \"LABEL-FREE CONCEPT BOTTLENECK MODELS\" from ICLR2023 propose CBM even without labeled concept data.\n5. In those controlled environment (e.g., cartpole), I believe there might be better ways to collect state data (e.g., velocity) rather than query VLM with prompts (Appendix A.3). First, VLMs are not designed to do accurate estimation like pixel loations. Second, using simple heuristics like pattern matching the pixel of the agent may be more efficient and accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DXdqcE38wU", "reviewer": "Reviewer_mn27", "summary": "This paper proposes a method of learning a concept-based RL policy that utilizes only a small number of annotations. Prior work introduced the concept-based RL learning for the purpose of obtaining interpretable policies, however, the assumption is that all states are annotated with the concepts. This paper proposes to sample the states and select the most informative of them with a disagreement based active learning strategy. As a result, the proposed method achieves very similar performance with much less annotated data both in terms of policy performance and concept prediction as the original method. Moreover, authors study if VLMs can be used as concept annotators in some environments.", "questions": "I would like to understand a bit more about the interpretability of the policies produced by the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper addresses an important problem of making RL policies interpretable for the final users\n- The proposed method removes the unrealistic assumption that all the states are annotated with the concepts. The method combines two lines of work: concept-based reinforcement learning with active learning in a novel way\n- The paper studies the performance both in the case of using good quality labels and noisy labels (in the experiments it corresponds to using the environment state or VLM).", "weaknesses": "I think what I am missing the most in this paper is more explanations on the interpretability of the policy. Given that the novelty of the method is limited in a way that it combines two existing methods, it would be good to demonstrate the importance of the proposed solution more. As the goal is to produce interpretable policies, it would be good to see some examples of interpretable policies, and maybe even verify that users of the policies find them convincing and useful. Also, the paper talks about the \"concepts\" in a very abstract way and I would like to see some more examples of the concepts to make it more concrete. For example, table 1 shows the number of concepts and their type, but does not communicate that each concept can take more than 100 values in some cases.\n\nAdditionally, while the experimental studies are good, I think it could be further improved with some real world experiments (as the original concept-based RL paper presented the experiments in the real world) to increase the impact of the paper.\n\nMinor questions and comments:\n- How do you manage to obtain the diverse predictions in the ensemble given that they are trained in the same way?\n- More details on the architecture of the policy and concept predictor would be useful\n- Data decorrelation seems to be just random subsampling of the data. Is this so or is there any other important aspect? If it is the case, I don't think the authors need to discuss it in so many details.\n- Does each state action pair correspond to only one concept? If there are several, how do you decide based on the disagreement in which concept to select the sample to label?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)