Neural+Symbolic Approaches for Interpretable Actor-Critic Reinforcement Learning
Authors: Yue Yang
Fan Yang
Yu Bai
Hao Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretable
RL
Symbolic
Primary area: interpretability and explainable AI

Abstract: The integration of neural networks into actor-critic frameworks has been pivotal in advancing the field of reinforcement learning, enabling agents to perform complex tasks with greater efficiency and adaptability. However, neural network-based actor-critic models remain opaque ``black boxes,'' concealing their decision-making processes and hindering their use in critical applications where transparent and explainable reasoning is essential. This work introduces an innovative adaptation of the actor-critic framework that unites neural networks with rule ensembles to tackle key challenges in reinforcement learning. We harness the computational power, scalability, and adaptability of neural networks to model the critic, while integrating a rule ensemble system for the actor, ensuring transparency and interpretability for decision-making. Our study establishes a theoretical foundation for integrating rule ensembles into the Advantage Actor-Critic (A2C) framework. Experimental results from seven classic and complex environments demonstrate that our proposed method matches or exceeds the performance of representative RL models, including symbolic methods, while offering self-interpretability and transparency.

Published as a conference paper at ICLR 2026
NEURAL ACTOR-CRITIC FRAMEWORK FOR INTERPRETABLE REINFOR LEARNING
Yue Yang', Fan Yang'; Yu Bai', Hao Wang"t 't
1 Faculty of IT, Monash University, Australia (yue. . yangl, fan. yangl, yu yu.bai3,
Maincode, Australia hao. wang2 @monnsh. edu
ABSTRACT
The integration of neural networks into actor-critic frameworks has been pivotal in advancing the field of reinforcement learning, enabling agents to perform complex tasks with greater efficiency and adaptability. However, neural network-based actor-critic models remain opaque "black boxes," concealing their decision-makinn processes and hindering their use in critical applications where transparent and explainable reasoning is essential. This work introduces an innovative adaptation of the actor-critic framework that unites neural networks with rule ensembles tackle key challenges in reinforcement learning. We harness the computational power, scalability, and adaptability of neural networks model the critic, while integrating rule ensemble system for the actor, ensuring transparency and interpretability for decision-making. Our study establishes a theoretical foundation for integrating rule ensembles into the Advantage Actor-Critie Critic (A2C) framework. Experimental results from seven classic and complex environments demonstrate that our proposed method matches exceeds the performance of representative RL models, including symbolic methods, while offering self-interpretabiiliy and transparency.
INTRODUCTION
Actor-critic reinforcement learning (RL) methods, such as Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO), have delivered exceptional performance across diverse RL tasks (Mnih et al. 2016 Schulman et al.) 2017). largely due to the use of neural networks to effectively manage complex, high digeenimeniional satteaaction spaces. Yet, actor-critic RL models expand to broader real-world domains, such as healthcare, finance, and law, interpretability becomes essential for ensuring trust, transparency, and regulatory compliance in Decision-making By revealing how actions chosen why specific decisions are made, interpretability supports debugging, model improvement, and the ability to explain automated decisions in line with legal and ethical standards.
Despite the effectiveness of neural -ewoor-bbaeed actor-critic models, their opaque nature and lack of transparency can limit their use in sensitive or critical applications where comprehending the rationale behind each decision is crucial (Benítez et al., 1997: Castelvecchi, 2016). Symbolic models, e.g.. expert systems, decision trees, and rule-based systems, rely on clear, well-defined rules and logic to make decisions, allowing for easier tracing of how conclusions are reached (Frmst et al.) 2005 Gupta let al. 2017: Tao et al.] 2018 [Bastani et al.) 2018 IIlanes et al. 2020). These methods use symbols (which can represent objects, properties, or relationships) and logic to process and reason about data, making the decision-makee
process transparent and understandable.
However, substituting neural network-based models with symbolic models in reinforcement learning tasks can be challenging. (i) While symbolic methods are highly valued for their interpretability and transparency, current implementations in reinforcement learning typically rely on predefined knowledge pre-trained models (Garcez et al. [0188] Lyu et al. 2019). (ii) These symbolic methods can face challenges managing the complexity and ambiguity that non- non-symbolic approaches, such as neural networks, are often better equipped to handle (Illanes et al. 2020 Landajuela et al. (021).
"Equal contribution.
1 Corresponning authors: Yue Yang and Hao Wang.

---

Published as a conference paper at ICLR 2026
To tackle the above challenges, we propose new "Neural plus Symbolic" framework for Actor- Critic models, namely NSAC, as shown in Figure [ NSAC integrates the strengths of both neural networks and symbolic rule-based systems. Specifically, the neural critic leverages powerful function approximation capabilities to estimate the value of states and state-action pairs effectively. Meanwhile, the actor component is designed using a rule-based approach, providing intrin- sic interpretability by following explicit, predefined rules that govern its decision- making process. This framework allows for an analysis of how particular fea- tures or rules influence decisions, offer- ing a level of explainability absent in black-box models. Our theoretical anal- ysis proves that NSAC converges to a local optimum. Through empirical ex- periments, our proposed NSAC demon- strates performance levels that match those of widely-used reinforcement learning al- gorithms, such as DQN (Mnih et al.] 2013). PPO (Schulman bow (Hssell et al., 2018). SDSAC (Kong)
Blackbox
Interpretable Symbal-Symbalic Actor Critic
Critic
Action
Action
This oction is chosen because x 100: which is quITs cessanabic decision
decide
Rules
Blackbox
Whitebox Actor
Adion 1 +10 or 100
fb
Action 2-17: 3verd)
Blackbox
Blackbox Critic
Figure I: Comparative illustration of SSurol-Symbolic
(017). 7). Rain-
Actor-Critic (NSAC) and classic Actor-Critic methods.
al. 2021). SACBEF (Zhang al., (2024). and A2C (Mnih et al. 2016). By integrating the adaptability and generalization capabilities of neural networks with the transparency of rule-based decision-making. our model provides a balanced solution that retains high performance while en- abling interpretable and trustworthy decision-making processes. It is worth noting that, work belongs to the broader ceuraa-symbolle family, but differs from compile-logie approaches. NSAC uniquely couples data-driven neural critic with a rule-based symbolic actor, enabling both statistical learning and explicit reasoning within an actor- critic loop. This work makes three main contributions:
We propose novel actor-critic architecture that combines a neural critic for accurate value estimation with symbolic, rule-based actor that delivers intrinsic interpretability.
We provide formal proof of convergence and demonstrate that our rule-based policy satisfies the established interpretability criteria of simulatability, modularity, and low com- plexity (Murdoch al. 2019).
We empirically demonstrate that NSAC matches state-of-the-art RL methods classic con- trol and energy-system tasks, while uniquely preserving transparency and trustworthiness.
2 RELATED WORK
Interpretability has become increasingly important in RL and motivated symbolic approaches that provide transparent reasoning while retaining RL's adaptive capabilities. Existing approaches can be categorized based their interpretability characteristics, knowledge requirements, and learning paradigms, each with distinct limitations. Tree-based policies represent a widely studied approach to interpretable RL. Native tree methods (Ding al. 2020: Ernst et al. 2005 Gupta et al. [201]] Roth et al. 2019 Tao et al.] 2018) learn decision trees directly from environmental interactions. often achieving strong performance
small to medium-scale problems but facing scalability challenges where large trees become difficult to interpret. Recent gradient boosting approaches (Fuhrer et al. 2024) and programmatic tree policies (Kohler et al.) 2024) extend this line by offering editable representations and improved handling of structured features, though interpretabilliyypeefrrmnnne performanee trade-offs remain. Distillation-bass
tree methods (Bastani et af.) 2018 Coppens al., 2019) Gupta al., 2015 Liu et al., 2018) extract tree policies from pre-trained neural networks, but face a size versus fidelity trade-off. Their differentiable variants (Silva et al. 2020) and specialized architectures (Ding al. 2020 Liu al. 2018) attempt address these limitations but may produce mathematical notations and complex tree structures that remain difficult for humans to interpret.
Symbolic RL with predefined knowledge leverages domain expertise to guide policy learning. These methods (Garcez et al., 20188) Ialanes et al.) 2020, Lyu et al., 2019) can maintain interpretability through symbolic rules, modules, or plans, and may achieve strong performance when domain priors
2

---

Published as a conference paper at ICLR 2026
well-specified. However, they require substantial predefined knowledge, making them sensitive to domain assumptions and limiting their applicability to new domains where such expertise may not be available. To remove this dependency, symbolic policies learn from neural teachers or direct search to discover symbolic representations, such as neural-guided symbolic policy discovery (Landajuela et al.) 2021), genetic programming for symbolic expressions (Hein et al. 2018). and program synthesis approaches (Wu et al. [2020] Qiu & Zhu (022). While these methods avoid predefined knowledge, they often yield complex mathematical expressions involving trigonometric functions, logarithms, or program syntax that are challenging for humans to simulate and interpret (Murdoch al. 2019). Recent approaches (Delfosse al. 2023b; Luo al. 2024; (Shnno al.) 2024) improve the interpretability through nerve guide, end-to-end or mixed framework, although they still produce complex models, or external tools are needed to generate rules. Post-hoc approaches additionally may suffer from approximation errors inherited from neural teachers. The importance of interpretable RL is underscored by recent findings on fundamental challenges in agent behavior, including goal misgeneralization (Di Langosco et al. 2022 and systematic failures on task variations (Delfosse et al. 2025), which have motivated approaches such as concept bottlenecks (Delfosse et al. (024) for detecting misalignments and object-centric environments (Delfosse et al. (023a) for visual domains, highlighting the need for transparent and verifiable RL systems.
As discussed, most previous methods either require certain amount of predefined knowledge, which influences the search space or establishes the programmatic rules, or they are post-hoc solutions that learn from a pre-trained model rather than directly from the environment, potentially distorting the learning objectives, or they generate complex representations or rely on external tools for rule genera- tion rather than learning directly from environmental interactions (See Appendix HD. In contrast, our method effectively integrates a neural network- based critic with a symbolic, baseb interpretable actor by learning directly through environmental interactions without requiring predefined knowledge or external rule generation tools. Specifically, we address the tree scalability problem by using additive rule ensembles rather than hierarchical structures, where each rule contributes equally to the decision-mant making process. We employ Orthogonal Gradient Boosting (OGB) (Yang et al. 2024] to discover rule conditions automatically, requiring only basic feature comparisons rather than domain- specific knowledge. Our rule-based actor is trained directly via policy gradients on environmental rewards, ensuring the symbolic representation accurately reflects true pacisiin-makingg process rather than approximating black-box teacher. Our approach achieves interpretability through symbolic rules while learning directly from the environment, addressing both the knowledge requirements and post-hoe limitations of existing methods. Post-hoc explainers are only approximations: they often flag correlates, not causes. SHAP (Lundberg & Lee, 2017), for example, marks a feature as "important" while the model flips under tiny perturbations. Worse, black-box models can even be trained to emit persuasive but misleading rationales (Rudin, 2019). Because these explanations are extrinsic, they guarantee neither consistency, fidelity, nor disti--ditt robustness.
3 PRELIMINARIES
3.1 ADVANTAGE ACTOR-CRITIC REINFORCEMENT LEARNING
The A2C algorithm represents a synergy of two foundational approaches in reinforcement learning: the actor-critic method and the advantage function estimation. In the actor-critic framework, the critic in the A2C framework estimates the value function V(s), which represents the expected return from state s. The critic is updated by minimizing its loss function, typically the mean squared error between the estimated value and the computed target value. The target value is derived from the rewards
obtained and the discounted values of subsequent states: (d) = E (R, + +V((+++)) - (s.)))
where R2 is the reward received after taking action a, in state s, at t, 7 is the discount factor. (s++)) the value estimate for the next state at f++ 1, and @ are the parameters of the critic network. The actor updates the policy by adjusting parameters @ in the direction suggested by the critic's evaluations. The update is guided by the gradient of the policy's performance, estimated using the advantage function: (0) E (V, log (a |s: As], where A, = R. + -V at time t, indicating how much better (or worse) the action a is than the policy's 's average at state st- Through these updates, the actor learns to choose actions yielding higher returns than predicted by the critic, while the critic learns to make more accurate predictions about expected returns, forming a feedback loop that enhances both policy and value estimation in the A2C framework.
V(s)) is the advantage
3

---

Published as a conference paper at ICLR 2026
3.2 ADDITIVE RULE ENSEMBLES
Additive rule ensembles are a class of probabilistic models (Friedman & Popescu 2008 that estimate the expected value of target variable Y E R given input vector X E k which is expressed as E[ X = Ø] = (((())) where, x: is an instance of X, ju: R - IR is an inverse RR is a linear combination of
link function mapping f(x) to the target variable Y, and Rd Boolcan query functions:
k
C1
f(x) Σ (.)((a)
Σ i=1
00 II Pi.(
4=1
In Equation 1 Wi (0 i S k) is the weight of the i-th Boolean query function 9i Rd + {0,1} and di is a conjunction or product of e; Boolean propositions, where Pi.j E {I( ((x21) S sz) : [d], [n]. E {+1}] is a threshold function (αx = (l....,æ} } is the index set). Each term (.)(()) in Equation I can be represented as an 'IF THEN .... rule, where the query defines the condition of the rule, and the weight wy is the rule's consequent.
Boosting an iterative approach for learning additive models (Schapire
1990
Friedman 2001
Dembczyński et al. 2010 Given training set of size n, have [ 9/1 ..... (n.3/a) Yn)), where 2; is the input and y is the corresponding target, through approximately minimizing the empirical risk
Lx( 1 Σ (ff(a;).vv + \//w|/
(2)
n
boosting iteratively adds terms into the model and yields sequence of models f()) f(),..., f(3, The last term of Equation is the regularization term, w = (Wi...., )x) E R is the weight vector, and À 2 0 is the regularization parameter. Note that is loss function derived shifted negative log likelihood, measuring the cost of outputting f(x::) while the true value is 3/i.
4 METHODOLOGY
4.1 FRAMEWORK DESIGN
Actor implemented with Rule-Based Ensembles: As shown in Figure 2 replace the neural sework-based action function approximation with an actor composed of additive rule ensembles. Let F(s) (s) Ya A} represent the collection of ensemble models for action in the action space A at state & E S. The size for A is m. The ensemble fa() for an action a predicts the expected advantages directly, simplifying its expression A(s, a) (s). This prediction is based on a linear combination of Boolean query functions, weighted by coefficients specific to each action E A: Sa(s) Σラ= Wa.jfla.jj (x). where Wa.j are the coefficients for da.j- For each action there are h Boolean query functions qa.j: (j =1.... k)) specific to the action The actor's policy, represented as π, is defined by the probability distribution over action values for a given state, which exp(Ja(e) is derived from FF using the softmax function: (Tass) Er (x)] The softmax function
ensures that (als) is proper probability distribution over the action space.
Critic implemented with Neural Network: In our framework, the critic functions similarly to traditional actor-critic systems, leveraging the computational capabilities of neural networks. It serves as function approximator, implemented using neural network that estimates the value of being in a specific state This neural network, paramount in computing state-value functions, is parameterized by weights d, and the value function it approximates denoted by (s)). This design choice ensures that our critic effectively harnesses the processing power of neural networks to deliver precise value estimations for each state. The pseudocode for the algorithm shown Appendix B and the thcoretical analysis in Appendix C
4.2 POLICY UPDATE
Critie Policy Update:
The critic in our framework estimates the value function V(s) using a neural network, representing the expected return from state s. The critic is updated by minimizing the
4

---

Published as a conference paper at ICLR 2026
Error
pieiruue Posls pas
Recalcuton cocleee--ie the weight raet nutie
Action
It wot if andinne 0.14)
Fratls 12 pes S-n duen (.12)
Fieect pos 0320
Fng O
Fvall -2 (111
Fvllidd
ihat's (4
Stede
Reward
Environment
yow 0.31)
fheils undjpn
(Fvdd and
344
Update
Pree T- (e
State
Actor
Figure 2: NSAC learns an interpretable actor by combining neural critic with a rule-based policy updated via orthogonal gradient boosting. The diagram illustrates the full NSAC training loop, including critic updates, rule mining, rule replacement, and fully-corrective weight optimization steps for the symbolic actor. Blocks and arrows show how trajectories, advantages, and gradients propagated through the system to iteratively refine the rule ensemble; additional algorithmic details provided in Section 4 and Appendix B
loss function, typically the mean squared error between estimated value and the target value. The target value is derived from the rewards obtained and the discounted values of subsequent states: (v) EE (Re + +VV (8++1) V=($)))2 where RG is the reward received after taking action or in state St, y is the discount factor, ($++1) is the value estimate for the next state, and á are the parameters of the critic network.
Actor Update: The learning process involves iterative updates to the coefficient vector wa and the query functions Ga based observed rewards and state transitions, aiming refine the ensembles for optimal decision-making. This approach leverages the interpretability of rule-based systems in the actor's policy formulation, enhancing transparency in complex high-dimensional state spaces. The advantage function, used by the actor for updates, is computed using the critic's value function and the immediate reward: Ac(st, a.) Rg 7V (s++1) (s²). This expression calculates how much better the current action is compared to the average action at state at time considering the immediate reward and discounted future values. The actor updates the policy by adjusting the parameters (w, q) in the direction suggested by the critic's evaluations. The update is guided by the gradient of the policy's performance, which estimated using the advantage function. The loss function derived from the policy gradient is
VL (w, q) = -EE Vw. log (ag St, w, g)Ai) + AV //w.q//Wl22
(3)
To make the loss function locally convex, we add a regularization term Al \/w to the original loss, where À is the regularization parameter (see Appendix D). The actor is updated by modifying each rule ensemble model fa in accordance with the calculated policy gradient objective for each action. To obtain the gradient for each action, we use softmax for action selection, taking the natural logarithm of the policy yields:
m
Va, loga(a se w. wa log
exp( Wa, 9e,).
(4)
Substituting Equation intto Equation 3 we have:
]
m
VL -E V..q.A Wa, qa.
log Σ
exp( Wa, Any))
AV www.qllw/2. (5)
/=1
5

---

Published as a conference paper at ICLR 2026
Treating w and combined vector (w: q), the gradient of VoL(w, q), ie., ViL(w, q) can be derived as
VoLA(W,q) = 0vl = E
V Wa
A,d
Wa, 9o.
log
exp exp(wa, {os))
qo
exp(w W qo)
-B
V Ar Ilama
VV Mas Ab (Naama - T(a 88; w, q))
2j=1 exp(
177
(Wyy(n;))
Taking into account the two distinct situations a = ar and a + ar, the policy gradient for each rule ensemble is follows:
1. When the ensemble model fa corresponds to the action executed in the current step. i.c.,
a at.
V Wa Lx {a) E V wi qu A(st, a1) - π (aa]s)))
(6)
2. When the ensemble model far does not correspond the action selected by the actor that time step, i.e., +
Maraa (W (a) = E V W..gg A(se. at) (ar]s)]
This distinction ensures ensemble updates properly align with the actions taken, enabling targeted adjustments that enhance the actor's policy based on observed outcomes and predicted advantages.
This algorithm has a time complexity of (dPrk) per rule, where d is the observation dimension. It performs at most d iterations to construct a single rule (adding one dimension per iteration), and each iteration, it evaluates all d dimensions as candidates. The bottleneck lies in computing the orthogonal gradient boosting objective objogb (9) = 19'9/ (tlg9/(([9111+ for each candidate, which requires O(nk) operations due to the orthogonal projection computation: 91 = BB g. This represents a k-fold increase in computational cost over standard gradient boosting (which has complexity ((dn)).. The orthogonalization overhead is essential for the algorithm's 's ability to select more general rules and achieve better tisk-complexity trade-offs in practice. Fully-corrective weight calculation has complexity of Ok? from solving convex optimization problem that re-optimizes all weights after adding a new rule. The k2 scaling comes from computing the Hessian on the ls Gram matrix Q" Q. While more expensive than stage-wise approaches, this cost is manageable for small ensemble sizes typically in interpretable rule learning.
Computational Complexity:
5 EXPERIMENTS AND RESULTS
In this section, we aim to answer the following questions: (RQI) How does algorithm perform relative to both black-box and interpretable baselines? (RQ2) Does our framework support inter- pretable decision-makinn in a quantifiable way? (RQ3) What kinds of qualitative insights about the policy can be obtained from inspecting the learned rules?
We evaluated the performance of NSAC in five diverse RL environments: MountainCar Acrobot- v1, CartPole-vI. 1, Blackjack-v1, and Postman (Andrew & Richard S. 2018 Dietterich] 2000). We also evaluated our NSAC algorithm challenging, real-world HVAC control benchmark using Sinergym (see Appendix[]] for details). This environment models a multi-zone office building with coupled thermal dynamics, where each zone's temperature, humidity, and carbon dioxide concentration evolve according to nonlinear heat-transfer and air-exchange equations. The high dimensionality of the state- action space, the strong inter-zone couplings, and the requirement satisfy strict comfort bands make it great testbed for our algorithm.
5.1 PERFORMANCE EVALUATION (RQ1)
We conducted a comprehensive performance comparison against established benchmark algorithms, including Q-Tabular (Andrew & Richard S 2018), DQN (Mnih al., (013). PPO (Schulman et al. 2017). and A2C (Mihh et 2016), all implemented using Stable-Baseling Baselines3 (Raffin et al.
2021 We additionally included Rainbow (Hessel et al. 2018), SDSAC (Kong et al. 2021). and

---

Published as a conference paper at ICLR 2026
Algorithm 1 Neural-Symbolic Actor-Critic Algorithm
1:
Initialize actor with each action rule ensemble function fa = 0; Initialize critic network V(s, ({, with parameters di Initialize environment and observe initial state
2:
repeat
3:
each step of episode do
4:
Choose action a (alss s, 8)
5:
Observe reward and new state 8'
6:
Calculate d +V((s Φ)
((8, Update critic by minimizing (()) 6² for o; in action space
7:
8:
9:
if a, = then
10:
Calculate policy gradient gt: -V low.g(--ogg((4)) w" q)) for fa
11:
else
Calculate policy gradient gr: Vw.q log (als, w q)ó for fo
end if
14:
use Rule Replacement Steps and policy gradient to find a new query for fa (Appendix B
15:
OGB to update the weights for fa (See Appendix end for
16:
18:
end for
19:
if end of episode then
20:
Reset environment and observe initial state
21:
end if
until convergence or maximum episodes reached.
SACBBF (Zhang et al. 2024) as deep RL baselines. To cover symbolic approaches, we also evaluated three interpretable methods: SYMPOL (Marton et al.] 2024). Taffine- (Qiu & Zhu 2022). and D-SDT (Silva et al. 2020). For fair comparison, we use the Gym environments in their original form without any modifications. For each algorithm, we apply the recommended optimal parameters provided by the RLzoo (Raffin & contributors 2020) with Stable Baselines 3 and report performance averaged over 10 random seeds. As shown in Table [ NSAC demonstrates strong and reliable performance across a wide spectrum of tasks, outperforming or matching both classic deep RL methods and symbolic approaches. Compared with classical methods such Q-learning, DQN, A2C, PPO, SAC variants, and Rainbow, NSAC achieves consistently higher returns with lower variance, particularly in settings that demand stability and scalability. Unlike value-based methods that can be brittle in continuous or high digh--iimessonal domains, NSAC maintains robustness while still being competitive
simpler benchmarks. Against symbolic tree-based methods such as SYMPOL, Taf and D-SDT, NSAC shows even clearer advantages: while symbolic policies can excel on select tasks due to strong inductive biases, they often suffer from collapse or sharp degradation when scaled, whereas NSAC delivers stable performance across all environments. This makes it practical and effective alternative that closes the gap between the strengths of classic RL and symbolic reasoning while avoiding their respective weaknesses.
Ablation Study In our ablation study, explored how variations in the rule count and the imple- mentation of a warm start influenced our method. We conducted tests in the CartPole-v environment under various rule configurations, specifically using 5, 10, 12, 20, 30, 40, and 50 rules per action. Each configuration was tested both with and without a warm start. 1. Number of Rules: As depicted in Figure the model configured with 12 rules per action and warm start yields the highest reward. This outcome suggests that having 12 rules per action provides the model with sufficient flexibility to effec- tively capture the essential dynamics of the environment without being overly simplistic. Employing fewer than 10 rules, such as 5, may not offer adequate coverage to manage the environment's complexity, resulting in infe- rior performance. Conversely, a larger number of rules may lead the model to overfit the training data, picking up noise irrelevant patterns and diminishing its generalizability to new contexts. The findings from this ablation study highlight critical balance between the
500
Warm start
No warm start
Rowands
300
200
100
30
Number of Rules
Figure 3: Ablation study on different num- bers of rules and warm start.

---

Published as a conference paper at ICLR 2026
Table 1: Our NSAC policy attains rewards that match or exceed deep RL baselines while clearly outperforming symbolic baselines across most environments. The table reports mean episodic return (#t standard deviation) for NSAC, classic deep RL methods (e.g.. A2C, PPO. Rainbow), and symbolic approaches (SYMPOL, "aftine. D-SDT) on the Gym and HVAC tasks, showing that NSAC achieves comparable performance to strong neural baselines and markedly higher returns than tree-based policies, especially in more challenging environments, such as HVAC. Best results are highlighted in bold, with all numbers averaged over multiple random seeds; further training and evaluation details are given in Appendix
Environment Q-table
DQN
A2C
PPO
SDSAC
SACBBF
Rainbow
SYMPOL
D-SIIT
NSAC
MCar-vo
1 9.52
147.68
+ 16.42
135.07
+ 37.12
-15,55
-150.40
4.05
-14.776 13:91
-139.23 + 17.62
-137.76 39.47
-200
+
-200
+0
-200
4:0
+ 16.08
-132.25
Acrobot-v1
1 13:32
-201.01
-112.68
8.93
1. 29.52
-98.311
-82.629
1. 2.79
-85,124
1. 7.82
-88,708
1. 7.12
-89.753
1. 8.98
I 4.28
-80.02
1. 56.81
-425.77
-212.31
1. 4.19
-87,714
1. 6.95
CartPole-11
1 69.2
219.51
+ 55.2
161.00
453.51
92.5
498.73
2.4
+ 2.71
487.61
+ 10:76
481.07
+ 2.12
498.53
1:0
500
+ 76.82
109
+ 2.12
498.53
= 1.91
499.14
Bluckjack-11
- 0.02
-0.06
-0.06
0.02
+ 0.01
.0007
-0.66
0.01
+ 0.02
-0.07
-0.66
0.01
+ 0.02
.0.06
_ 0.01
.0.66
= 0.01
-0.08
.0.08
0.02
I 0.01
Postman
35.24
2,06
31.91
5.71
24.23
34.35
3.82
34,91
3.77
34.67
25.34
15.23
4.91
18.91
6.28
21,14
3.38
HVAC-1Zone
NA
+ 20183
-1455367
+ 148658
-1334562
18652276
135683
# 145619
-1387692
1423573
+ 34825
1465332
785732
+ 65293
1478783
1 42873
18952886
-1662337
29763
-1251321
+ 85241
HVAC-5Zone
NA
1876253
198263
+ 287237
1984843
1676288
462837
-1582742
227221
+ 241625
11472279
+ 342612
1602142
= 65792
1586352
- 45241
1969663
11766374
= 33468
I 121678
-1463601
Note. Red indicater the best performance acrost all benclaurks. Underscore hightiihts the best performance among symbolic based approaches.
number of rules and overall model performance. They show that merely increasing the number of rules does not invariably improve performance, as an excess can lead to complexity and overfitting. whereas too few can restrict the model's performance. 2. Warm Start: When the number of rules is small, the model's ' representational capability is limited, so the initial conditions (c.g.. the warm start) become more critical. A warm start can guide the model toward better solutions early on, helping it learn an effective policy faster and avoid getting stuck in poor local optima. In contrast, with a larger rule set, the model has greater capacity to explore various configurations. In such cases, relying on an external initialization can sometimes constrain exploration, preventing the model from fully leveraging its increased flexibility.
5.2 INTERPRETABILITY EVALUATION
Quantitative Evaluation (RQ2): Common metrics for interpretability include fidelity, comprehen- sibility, and cognitive load. In our approach, measuring fidelity is unnecessary because it is primarily relevant in post-hoc explanation settings- where one model attempts to approximate and explain the behavior of another. In contrast, both the tree-based approaches (other baselines) and our model are inherently eelf-explai,able, meaning it achieves 100% fidelity by design, as the decisions made by the model directly reflect its underlying logic. For comprehensibility and cognitive load, our rule-based system offers a structured and measurable format. The rule ensembles consist only of boolean propositions formed through simple comparisons (c.g.. feature > threshold) and logical conjunctions. In general, comprehensibility is negatively correlated with model complexity, while cognitive load increases with complexity. We define model complexity in terms of the total number of rules, conditions, and logical operations. Models with more parameters and operations tend to be harder to understand and place a greater cognitive burden. Since the performance of TaBioeDD and D-SDT is not directly comparable, we instead evaluate them in terms of interpretability. Using model size as a proxy for complexity. we observe that across all environments our models require, on average, 61.23 symbolic units (number of parameters) to achieve the reported performance. In contrast, SYMPOL uses 95.6 symbolic units (approximately one per node). According to Murdoch et al. (2019), interpretability involves simulatability, modularity, and low complexity. A model is simulatable if its predictions can be easily followed and computed by humans; modular if its compo- nents are individually understandable; and low-complexity if it uses few parameters or rules. Our rule-based method satisfies all three. Each decision is made via explicit if-then rules (simulatability), each rule is independently interpretable (modularity), and the total number of rules is kept under 30, ensuring the model remains simple and auditable (low complexity). Although symbolic trees like SYMPOL'S are locally simulatable (a single path explains decision), shared internal nodes mean
8

---

Published as a conference paper at ICLR 2026
edits are not truly modular, and trees tend to grow deep and wide harder tasks, so they quickly lose low complexity and thus do not fully satisfy all three criteria simultaneously.
Qualitative Interpretability Case Studies- MountainCar (RQ3): In this section, we use the MountainCar-V environments from OpenAl Gym as examples to illustrate the interpretability of our proposed reinforcement learning algorithm. The Mountain Car environment involves a situated between two hills, tasked with reaching the flag at the top of the right hill. Due to insufficient engine power, the must build momentum by oscillating back and forth. The system state is characterized by two continuous variables: the car's position (p) and its velocity (v). The agent performs one of three discrete actions: push left, do nothing, or push right. For each action, a set of rules evaluate the current state by applying specific conditions based on p and v, assigning corresponding scores that influence the decision-making process. Consider a typical state where 0.02 (moving rightward). For the action
p = -055 (slightly to the left of the center) and Do Nothing, one rule imposes a penalty of - 13.485 since the velocity o exceeds a small positive threshold 0.001, discouraging inaction that could result in insufficient momentum to overcome the hill.
For the action Push Left, only one rule applies, which assigns a positive score as the velocity u is above min- imal threshold, encouraging slight push left to maintain momentum with-
de nothing
nothing
0.07
-19-725 ff 51.
ifv
24.220
0:485 v
0.0
23.003
24,743 F p- 2.0000
out overcompensating. Conversely, the action Push Right activates mul- tiple rules: three contribute +57.59 because the velocity v surpasses dif- ferent threshold values for the cor- responding rules, strongly encourag- ing a push right to build momentum; another rule provides +11.775 since both the position p 2 -0.577 and ve- satisfied, indicat-
0.07
light
niles for right
265215
27:328 g- 5
21,33/ 0
18.688 =-0.000
15766 Fvic0.000
21.341 5-06663
Niftt
rules for lef
18.531 0000
25.331 /fx:s00000
$1.055 -00004
0.0
14.514ff 0.581 5
11.6367f
Siv
13.028 -0-000000
locity is positive ing that a rightward push is beneficial without risking overshooting the val- ley; There is also one rule contributing an additional +1.361 because the po- sition < - -0.449, p 2 -0.874, and S 0.023 hold true, subtly encour- aging a push right to maintain the car within a stable range. The cumulative score for Push Right thus amounts to + +70.725, clearly favoring this action. This comprehensive rule- alle-baed scor- ing system demonstrates that, for the 0.02),
0.07
15.429 0.628 up 0.22
2.43 2212
0.53
-1.00 84 4.40 t.5
K- 02.02222222 0.015
P
-17.992 0192211-0618.-220000 22::0180-82220-00000
oupput
actor
10.00000000
Figure 4: The learned MountainCar rules recover the classic nomentum-building strategy in a compact and interpretable form. The figure visualizes the actor outputs over the (p. e) state space for each action, where higher heatmap values indicate that an action is more strongly pre- ferred in that region. Overlaid squares mark the domains where individual symbolic rules apply, revealing how sim- ple local conditions on position and velocity piece together into the global rocking behavior required reach the goal; additional visualizations and discussion are provided in Ap- pendix M
given state (p -0.5, pushing right is the optimal choice as it significantly builds the necessary momentum to ascend the hill, outweighing the moderate incentives to push left and the discourage- ment from doing nothing. We have also visualized the rules of MountainCar in Figure 4 with all rules presented in Appendix M One can easily identify the corresponding areas associated with a state and locate the applicable rules. To better understand the learned policy, we analyze its behavior in the MountainCar environment in the neighborhood of the valley bottom. The state is two-dimensional, s = (p, v), where p denotes the horizontal position of the car and v its velocity. The car starts in the valley and must build up sufficient kinetic energy by rocking back and forth in order to reach the goal on the right hill. Because the engine is too weak to drive straight up, the policy must carefully coordinate position and velocity to accumulate energy.
In our based policy, one of the most salient decision boundaries appears for positions to the right of the valley minimum, i.e. for p > -0.48, which is close to the bottom of the track. In this region the height of the track changes only slightly, so the gravitational potential is nearly constant and the immediate direction of motion is determined almost entirely by the sign of the velocity e.

---

that the next Published as a conference paper at ICLR 2026
Consequently, the line 0 for -0.48 becomes natural decision boundary in the (p. -llamee States with p > -0.48 and v > correspond to the car already moving to the right; the optimal action is to keep accelerating right to maximize its speed for the upcoming climb. In contrast, states with > -0.48 and u < O indicate that the car has started to roll back to the left after an unsuccessful attempt; the best response is accelerate left and commit rightward run can reach a higher point on the hill. Exactly at = 0 the car is at a turning point, momentarily stationary before choosing whether to move left or right, which is why this line marks the decision boundary used by our policy.
full swing back,
Qualitative Interpretability Analysis Real Time HVAC control (RQ3): Alongside stan- dard Gym benchmarks. we evaluated our algorithm on a challenging, real-world HVAC con- trol problem in a biological laboratory, where maintaining temperature within a narrow band by setting effective heating (htg) and cooling (clg) set points is vital to preserving the viabil- ity of cultured viruses. In such safety-critical settings, interpretability is crucial: when RL governs biocontainment HVAC systems, even minor errors can destroy cultures or compro- mise containment. As shown in Table D our rule-based actor approach delivers performance nearly equal to-or even surpassing that of traditional baseline methods. More than this, our proposed algorithm also delivers interpretability by explaining why and how each decision is made in human-understandable language, enabling users to assess the correctness of its rules.
Consider the very first rule as an example: if outdoor temperature S -4.5; hig setpoint S 20.0; air tem-
oundoor temperauree
Action: htg-17. dg-26
drdo to comperrgaee suur 0U0D0 Aampuaaa
Action: hig 11, cig-28
out Sompor sture oundoor temperature
1eg-22, 20
perature S 21.1503. and
203
HVAC electricity demand rate
Indorr tumperature
indoor temperature
indoer temperature
379
Actions hig-88. 24
Action: Hig -16, clg-26
Action: 1kg-15, 26
> 1452.0250(kWW), +37.6076
dxy ofmont
25-
233
What means, in plain English: When all four of these condi- tions hold: 1. Outdoor tem- perature is extremely low: 2. The previous heating setpoint remains at moderate level; 3.
2
20
tomperature
outdoor temperature
Indoo temperature
Figure 5: The HVAC rule ensemble encodes intuitive, human- readable control patterns that balance comfort and energy use. This figure shows heatmaps of the actor output over key HVAC state variables, where higher values correspond a stronger preference for each heating-coollng action. The resulting patterns align with domain intuition (e.g.. increasing heating when indoor temperatures are low and outdoor conditions are cold), illustrating how NSAC produces a policy that is both effective and easy to interpret; further qualitative examples and time-series analyses are included in Appendix K
The indoor air temperature is still below the desired comfort threshold; 4. The HVAC system is already drawing high power, then this rule "fires," adding +37.6 to the score for choosing (htg 15ºC, clg = C6* i.c., heat turns on below 15 "C and cooling above 26 CC A large positive weight like +37.6 signi- fies that, in this scenario, raising the heating setpoint to 15cC is especially beneficial. Based on the visualized rules of HVAC control in Figure when it is freezing outside, your target heating level is still under typical comfort, the room hasn't yet warmed up. and the system is working hard- so raising the heat setpoint to 15ºC markedly improves comfort, which this rule captures (see details in Appendix K)
CONCLUSION
In conclusion, this paper introduces a neural+symbolic approach within the actor-critic framework. This method leverages both the robust function approximation capabilities of the neural-networks critic and the interpretability provided by rule-based actor models. Users can inspect how the model makes each decision and form a view about its underlying reasoning. The limitations of this work are discussed in Appendix A In the future, we plan to assess the scalability of our method more complex environments across different domains, such robotics, finance, healthcare, or autonomous driving, and adapt with other actor-critic frameworks, such as Soft Actor- Critic.
10

---

Published as a conference paper at ICLR 2026
ETHICAL STATEMENT
This work does not involve human subjects, personal data, or identifiable information. All experiments conducted entirely in simulated environments, including stylized energy-system tasks designed to probe safety and performance trade-offs. These simulations do not interface with or control any real-world infrastructure. Our proposed NSAC framework aims improve the interpretability and trustworthiness of reinforcement learning systems by combining symbolic rules with statistical learn- ing. This direction supports the broader goal of promoting transparency in Al preiiion-makkng. While the symbolic actor promotes explainability, we acknowledge the importance of careful rule design to avoid unintended consequences in safety-critical domains. However, our current implementation is strictly research-oriented and intended for controlled, offline evaluation.
REPRODUCIBILITY STATEMENT
To support reproducibility, we include comprehensive details of our training procedures, hyperparam- eter settings, and evaluation metrics within the main text and appendix. Upon publication, will release the full codebase, including: Random seeds and environment specifications, Training scripts replicate all experiments, Plotting utilities to reproduce all figures. All results can be reproduced using a single command-ling interface without manual intervention. Pretrained models and logs will also be provided facilitate benchmarking and verification.
REFERENCES
Barto Andrew and Sutton Richard S. Reinforcement learning: introduction. 2018.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama, Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
José Manuel Benítez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes? IEEE Transactions neural networks, 8(5): (5):1156-1164) 1997.
Vivek S Borkar and Sean P Meyn. The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38(2):447 2000.
Davide Castelvecchi. Can we open the black box of ai? Nature News, 538(7623):20,, 2016.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies soft decision trees. In Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence, pp. 1-6, 2019.
Quentin Delfosse, Jannis Blüml, Bjame Gregori, Sebastian Sztwiertnia, and Kristian Kersting, Ocatari: Object-centric atari 2600 reinforcement learning environments. arXiv preprint arXiv:2306.086-9) 2023a.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and explain- able logical policies via neurally guided symbolic abstraction. Advances in Neural Information Processing Systems, (66:50838-50835 2023b.
Quentin Delfosse, Sebastian Sztwiertnia, Mark Rothermel, Wolfgang Stammer, and Kristian Kersting. Interpretable concept bottlenecks to align reinforcement learning agents. Advances in Neural Information Processing Systems, 37:66866666855, 2024.
Quentin Delfosse, Jannis Blüml, Fabian Tatai, Théo Vincent, Bjarne Gregori, Elisabeth Dillies, Jan Peters, Constantin Rothkopf, and Kristian Kersting. Deep reinforcement learning agents are not even close to human intelligence. arXiv preprint arXiv:2505.21731, 2025.
Krzysztof Dembezyáski, Wojciech Kotlowski, and Roman Stowiñski. Ender: a statistical framework for boosting decision rules. Data Mining and Knowledge Discovery, 21(1):52-90, 2010.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization deep reinforcement learning. In International Conference on Machine Learning, pp. 12004-12019. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of artificial intelligence research, 13:227-303, 2000.
Zihan Ding, Pablo Hernandez- Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. Cdt:
Cascading decision trees for explainable reinforcement learning, arXiv preprint arXiv:2011.07553, 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-5566 2005.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statisticss, pp. 1189-1232, 2001.
Jerome H Friedman and Bogdan Popescu. Predictive learning via rule ensembles. The annals of applied statistics, 2(3):966-54,, 2008.
Benjamin Fuhrer, Chen Tessler, and Gal Dalal. Gradient boosting reinforcement learning. arXiv preprint arXiv:2407.082550 2024.
Artur d' Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. Towards symbolic reinforcement learning with common sense. arXiv preprint 1VXv:1880.00857, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International conference autonomous agents and multiagent systems, pp. 66-83. Springer, 2017.
Ujjwal Das Gupta, Erik Talvitie, and Michael Bowling. Policy tree: Adaptive representation for policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76: 76:158-169, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila Mcllraith. Symbolic plans as high-level in- structions for reinforcement learning. In Proceedings of the international conference on automated planning and scheduling, volume 30, pp. 540-550, 2020.
Hector Kohler, Quentin Delfosse. Riad Akrour, Kristian Kersting, and Philippe Preux. Inter- pretable and editable programmatic tree policies for reinforcement learning. arXiv preprint arXiv:2405. /4956, 2024.
Yiting Kong, Yang Guan, Jingliang Duan, Shengbo Eben Li, Qi Sun, and Bingbing Nie. Decision- making under on-ramp merge scenarios by distributional soft actor-critic algorithm. arXiv preprint 00t.:2103304355. 2021.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pp. 5979-5989, PMLR, 2021.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 414 429. Springer, 2018.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro- symbolic reinforcement learning with textual explanations. arXiv preprint 2403.140331145 2024.
12

---

Published as a conference paper at ICLR 2026
Daoming Lyu, Fangkai Yang. Bo Liu, and Steven Gustafson. Sdrl: interpretable and data-efficien deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 207022970997 2019.
Sascha Marton, Tim Grams, Florian Vogt, Stefan Lüdtke, Christian Bartelt, and Heiner Stucken- schmidt. Mitigating information loss in tree-based reinforcement learning via direct optimization. arXiv preprint arXiv:2408.068 00000, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXhv:13/2.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillierap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PmLR, 2016.
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning, Proceedings the National Academy of Sciences, 116(44)22071122208) 2019.
Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In The Tenth International Conference on Learning Representations, 2022.
Antonin Raffin and contributors. RL Baselines3 Zoo. https://github.comt RM/
tl-baselines3-2o0, 2020. Accessed: 2025-07-24.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. :abbleb Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://ww...org//pppes///////
10-1364.hee
Aaron Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative q-improvement: Reinforcement learning for interpretable decision tree policy. arXiv preprint arXiv:1907.01180) 2019.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shai Shalev-Shwartz, Nathan Srebro, and Tong Zhang. Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM Journal on Optimization, 20(6))2807-25)) -832, 2010.
Hikaru Shindo, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Blendri: A framework for merging symbolic and neural policy learning. arXiv preprint 10.1
2024.
Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung- Hyun Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference artificial intelligence and statistics, pp. 1865-185. PMLR, 2020.
Yebin Tao, Lu Wang, and Daniel Almirall. Tree-based reinforcement learning for estimating optimal dynamic treatment regimes. The annals of applied statistics, 12(3):1914, 2018.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning, In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33: 13:17617-1766 2020.
13

---

Published as a conference paper at ICLR 2026
Fan Yang, Pierre Le Bodic, Michael Kamp, and Mario Boley. Orthogonal gradient boosting for simpler additive rule ensembles. In International Conference on Artificial Intelligence and Statistics, pp. 1117-1125. PMLR, 2024.
Le Zhang, Yong Gu, Xin Zhao, Yanshuo Zhang, Shu Zhao, Yifei Jin, and Xinxin Wu. Generalizing soft actor-critic algorithms to discrete action spaces. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 34-49. Springer, 2024.
14

---

Published as a conference paper at ICLR 2026
A LIMITATION AND FUTURE WORK
fundamental limitation of our approach lies the natural trade-off between interpretability and performance. While our symbolic, rule-based actor offers clear advantages terms of transparency and simulatability. this structure inherently constrains policy expressiveness compared to fully neural counterparts-partiie
in complex, high -imenssinnii environments. Although NSAC achieves competitive performance across benchmark tasks, there may be scenarios where its symbolic policy cannot match the nuanced optimization achievable by deep learning- baand policies.
NSAC uses rule ensembles over input features, so the candidate rule space grows with the number of attributes and thresholds. In practice, we keep policies compact and simulatable by capping the rule budget and rule depth. This makes NSAC best suited to or to settings with a good compact feature representation; truly high- inputs would require prior feature extraction or a hybrid neural-symbolic architecture. Also, boosting- based rule updates are currently performed per action, so runtime scales roughly linearly with the number of discrete actions. Our experiments use small to moderate action spaces, and control cost by limiting rules per action and exploiting trivial parallelism across actions. Very large action spaces would likely require extensions such shared rule ensembles multi-class boosting. NSAC not well suited to highly partially observable or sparse reward environments, such as Delfosse et al. (2024), that demand rich temporal memory or very high-capacity function approximators.
dimensional feature spaces
Moreover, interpretability is not a binary property but exists on a spectrum. Different symbolic approaches offer varying degrees of clarity, modularity, and simulatability, often depending on the complexity of the rule structure, the nature of the environment, and the domain- speciffc knowledge embedded within the rules. As such, the perceived interpretability of symbolic method may vary across users and use cases. Practitioners must therefore carefully consider the type and level of interpretability required for their braecattonbbllaciinnn traceability with scalability- when choosing or designing symbolic component. Future work may explore adaptive symbolic structures or hybrid neuro-symbolic policy representations to better navigate this trade-off across diverse tasks.
15

---

Published as a conference paper at ICLR 2026
B DETAILED RULE UPDATE
Step J: In time-step i, for fa, we first select a new query d: from a candidate set Q by solving
= arg max qEQ
(b; f((-));
(8)
where adopt OGB
Yang et al. 2024) for the objective function. Let q {0, 1 is the query vector of the query q. and g (fLx(fa-")))) 1/0f (000))))(()))))) (x1), (zi)...., (Lx(fa'-1) 1/af( (xn))" is the gradient vector of the empir- ical risk with respect to the predicted values of f((-1) 91 (t-1) and Qı are the projection of g and q onto the orthogonal complement of the range of Qi-1
(g(zx)....,g( ((x)))
(q....... ..-1)² where qi = (g1()) ((())) E {0,1}* is the query vector of the query for = 1...., We express the objective function follows: obj(g: f objogh (g) 1qTg "0Tlg"">1/10 1 candidate query set Q comprises all possible conjunctions of propositions, where each proposition corresponds condition that splits the attribute space based on attribute values of given data points (e.g., threshold-basee splits). The objective function is maximized using greedy or beam search.
|l. The
Step 2: After adding the updated new query, we re-calculate the optimal weight vector by solving
W; arg min LA (Qt..)) WER!
where Q - Q.-1 U qr is the n x t matrix with the outputs of the selected queries.
Step 3: After there are & rules in the system, instead of continuing to add a new rule for each time, we adopt the post-processing replacement step of boosting algorithm "Shaley-Shwarte et al. 2010). In each iteration, we remove rule with minimal absolute weight, leaving model: 4- = arg minigrs*lu,l and remove the query function qu- together with its coefficient wk.. thereby reducing the ensemble to (( with & - 1 terms. Then, need to recompute the weight vector Wi- (wM ([k][[I.... .k) of f(k-) along with the updated query
rules in the
functions {4r (4-)re(M), w arg min Lx (Qx-ww). where Q = {gr (r is the n x (k (--1) 1) matrix with the outputs of the selected queries.
Step 4: After this, repeat Step to add rules are recalculated (same as and ( in Step 2. the expected loss does not decrease after the post-processing procedure, the model before post-processing restored. The performance of the post-processing with fully-corrective boosting is guaranteed by the following theorem.
rule the system. In the end, the weights of all the
Algorithm Orthogonal Gradient Boosting (OGB)
1: Input: data (m,, y:), number of rules and g:
2: f(0) =0
3: for 1,.... k do
4: q: = arg max, llt.g/////// 1l
5: (t) = arg min (i,. WE)ER LA (2 wggs)
6: ((((() w (-) + w!" 1 gc())
7: end for
8: Output: (k)
16

---

Published as a conference paper at ICLR 2026
Algorithm 3 Rule Replacement Steps
Input: data (æ,, Bi). i I,..., n, original rule ensemble with &o rules (0)) number and g for 1,..., T...,Too do Find the index of the smallest weight absolute value arg minjee,,...... m (111)
maximum iteration
(t-1) arg weRt- (i(EiE :i ()((11))) 9.)
f ()) = Eie( (r) wi (1-1) qc(-)
Find the query Us = arg maxy g 11a /
(()) = arg min L (iiE wiq: + w, (t)
Let 0 = (x(((t) Lx ((())r))))()))) wi + ur (
if > 0 then
fk Eie(k]] w//² q: + u 4F
else
break end if end for
Output: f(()
Actor
Rules For Action!
Score
y +10
5. y27
Action
Rules For Action
Update the Actor Rule Ensembles
Score
Update the Critic
Remove the rule with minimum weight.
Step
TD-error
Network
Critic
Recalculate the weight for all the rules to minimize the policy loss.
Step 2:
States
Value Function
Based on the gradient calculated with TD error, find a new query that could maximize the OGB objective function.
Step 3:
Reward
Step 4:
Recalculate the weight for all the rules to minimize the policy loss.
Environment
Figure 6: Our proposed NSAC algorithm.
C THEORETICAL ANALYSIS
We present comprehensive theoretical convergence analysis of NSAC algorithm. Under standard assumptions, our algorithm converges to a local optimum. We assume the following conditions: Assumption C.I (Step Sizes). Assume ((w.q)) the learning rate for actor and let as be the learning rate for the critic. The sequences (t)( and {a(w.q) (t)} are positive and satisfy
(w (t) (t)
(t) 00,
((w.)) (t) oc, Σ (t) (t)p) < co,
lim
0,
1000
T=
t=
17

---

Published as a conference paper at ICLR 2026
so that the critic update occurs faster timescale.
Assumption C.2 (Markovian Sampling & Ergodicity). We assume mild mixing or ergodicity con- ditions ensuring that each state-action pair is visited infinitely often in the limit, and that standard stochastic approximation methods apply.
Assumption C.3. We assume Lipschitz-smooth approximator for V (s). We assume the "compat- ibility" holds or that any residual bias the critic's estimate does not prevent recovering the true gradient direction in expectation.
Theorem C.A. (Theorem 2.9 in Shalev-Shwark et al. 2010 Let LJJ) be the risk function defined by a B-smooth loss function, where the expectation is with respect an arbitrary distribution over X xY Let X > O be scalar, f is an additive rule ensemble with ls rules. f is reference rule ensemble with kn rules, if k + ko 16/ X²), and assume that is (I: + 1 1+ ko, e-spaseel-stronneee convex. Additionally, let be an integer such that
A(k: + 1-1 ko) log
L(0) - L()
T
(10)
23
Then if the fully corrective boosting is run for l iterations and its last predictor is provided as input for the post-proe pros-procsssing replacement procedure, which is then run for iiterations, then when the procedure terminates at time i, we have I((P()) T.(f)= Se.
Theorem C.S. Let & be the fixed parameters of critic, inducing a fixed value function Vo(s) for each state S. Let f be a parameterized function (e.g., the actor's 's approximation to the advantage) that is updated iteratively according to the update rule, the sequence f&k converges to f.
Proof. We first calculate the risk for rule ensembles for f and fo based on the loss functions: L(f) = log ग y f and L(fa) log / fxk respectively. When R(-) is error bounded, there exists lower bound with a constant ạ > satisfying [(fi) L(f2)] 2 7 Ilfi - -f2ll. for any fi and fz in the parameter space. Based on Theorem C.4 when Equation TO is satisfied, we have
1// fll s + [(fa)) L()l S 5. Hence, bounding the difference in risk [L(h)) L(J) by forces the parameter distance ll fu fll to lie within &.
Theorem C.6 (Convergence of NSAC). Let (Ww,, Φc) be the sequence of actor and critic param- eters updated via Equations 6 and 7 Aunder the Assumptions C.I-C.3 Then [(we, qr), de converges almost surely to a set of stationary points of the associated ordinary differential equation (ODE) sys- tem: e- = FØ; (w, q)), (w q) = Gi(w, q): Φ) where F represents the temporal-differnnc (critic) dynamics, and G represents the policy-gradien (actor) dynamics. In particular, (w,,) converges to a local optimum of J(w, q). where J is the expected cumulative reward.
Proof. We employ the concepts of Two-limescale theorems (Borkar & Meyn 2000 Wu al. 2020 and Theorem C.5 for the proof. Please refer to Appendix for the detailed proof of convergence.
18

---

Published as a conference paper at ICLR 2026
D ACTOR'S LOss WITH L2 REGULARIZATION
Theorem D.1 (Local Strong Convexity of Actor's s Loss with L2 Regularization), Consider the actor's loss function in A2C framework defined
L(0) - aming (A a) log To (a(a)s)] + 1101,
where À > O the regularization coefficient for regularization.
We assume:
1. The policy Ta(a))) is twice continuously differentiable with respect to 0:
2. At local minimum 6" the Hessian of the unregularized loss Lunveg (0)
-s,A"(s,a) log To (alss) satisfies V Lemreg (0)) - 0 (positive semi-definite);
3. The regularization coefficient À is chosen such that X > O.
Then, there exists a neighborhood U around Hr where the regularized loss L(0) is strongly convex. Specifically, within U,
V2I(0) - XI,
where I the identity matrix in Raxd
Proof. We aim to show that the regularized loss function L(0) is strongly convex in neighborhood around 0"
The regularized loss can be expressed the sum of the unregularized loss and the L2 regularization term:
L(0) Lunnc (0) Locg(0),
where
>
Treg (0) 2 11112
Compute the gradient and Hessian of L(0):
VL(6) V Luunre ( + A0,
V2 L() 2 Lunreg (0) + Al.
At the local minimum 0", by assumption:
Lunog (8") 2 0.
Therefore,
( (0") Al Xl.
This shows that the Hessian of the regularized loss at 0" is positive definite, as > 0. Since (a(a is twice continuously differentiable, 4 Tunrey (0) is continuous in 0. Hence, there exists neighborhood U around 0* where:
2 Lunrog (0) z O, VO U.
Within 14
V2I(0) 2 T yunrey (0) X XT.
This inequality holds because V Lunneg () > and Xl is positive definite. Since L² > XZ for all E Ut the loss function L(0) is **strongly c in the neighborhood U around 0* with strong convexity parameter
Theorem D.2 (Local Error Bound under Local Strong Convexity). Let L R.d - B be continuously differentiable and p-strongly convex in neighborhood N of some point f E Rd. In other words, there exists p such that for all N.
L()) L(F)
>
f
2
Then, for every f in that neighborhood N.
2
|| - fll
S V (L() I(f) ((A))
19

---

Published as a conference paper at ICLR 2026
Proof. Since R is p--trongly convex in the neighborhood A around f. we have the inequality
L(f) L(f) 2 11 - f11. -1p. AfEN.
Rearrange this to obtain
2 (T(() (T())
I /1
Taking the square root of both sides yields
2 (UF) V
Il fll
Thus, in the local region N, a small gap in the objective value L(f) - L(f) forces f to be close to This is precisely the local error-bound property.
20

---

Published as a conference paper at ICLR 2026
E PROOF FOR CONVERGENCE OF OUR PROPOSED ALGORITHM
Theorem E.1 (Convergence of our Proposed Algorithm), Let {We, qe), ve) be the sequence of actor and critic parameters updated via Equations 16 ( dr converges almost surely to set of stationary points of the associated ODE system:
(andd(7um under the Assumptions C.7 C.I-C.3 Then
f F (w,g). (vi, - G((ww, Q): Φ).
where represents the temporal-differnce (critic) dynamics and represents the policy gradient (actor) dynamics. In particular, (w, q) converges to a local optimum of [ww, q).
Proof. By Assumption C.1
(w q) (t) os (t) - 0. Hence, on the "fast" timescale, we may treat 8 if were quasi-static. Then the critic update standard temporal-difference learning procedure (or mean-squared error minimization) for fixed policy 7(w.g)-
From classical TD-learnngg results (or generalized linear function approximation theory), know that & converges to the set
T (w, q) Fo" (w,q)): (w, = 0},
provided that (t) diminishes appropriately and under the usual conditions.
Formally, one shows that for each fixed (w, q), the ODE
F(;; (w, q))
has globally asymptotically stable equilibrium at d² (w, q). By the Two-timescale lemma (Borkar Meyn 2000 Wu et al. 2020), the actual sequence Φe tracks this stable equilibrium as t -> DO.
On the slower timescale, consider the actor update:
(w.g)++ (w, q) + ((w.a) (1) ((w.q) log (w..)) (at 8t) A..
As di converges quickly to (w, q), the advantage estimate A, converges to A (8t, at). Hence, up to diminishing approximation errors, the gradient update for our rule hased model is driven from the true policy gradient from the policy gradient theorem. Based on Thcorem C.5 the errors, we have
ignoring the small
G((w, q): d"(w, q)) = V w /(w, q).
Putting the fast and slow processes together, we return the coupled ODE; By the previous steps:
F(o; (w,q) q)) implies == G((w, q); d*) = J q)
(11)
(12)
Hence, equilibrium (0", (w) of the ODE occurs precisely, when
F(ø"; (w, q)) and G((w, q): o*) 0,
which in turn implies
J((w, q) 0, if the advantage estimation unbiased.
Therefore, the equilibrium corresponds a stationary point of J(w, q). Under mild conditions (e.g..
local convexity/concavity arguments or strict monotonicity), one concludes that the limit points local maxima of J(w, q).
We now invoke the standard Two-timesede theorems (Borkar & Meyn 2000 Wu et al. 2020). Then the combined updates converge to an internally consistent equilibrium of the ODE. By the structure of the ODE, the equilibrium stationary point and local optimum of J(w, g).
Li
21

---

Published as a conference paper at ICLR 2026
F HYBRID WARM START FOR ACCELERATING CONVERGENCE
In NSAC, we also adopt a novel hybrid training strategy that incorporates both neural network-basse actor and rule-based actor. This hybrid strategy seeks to merge the quick learning attributes of neural networks with the clear and straightforward nature of rule-based systems.
Step I:
We employ a traditional neural network-based actor-critic model to train the system. In this phase, both the actor and the critic are implemented as neural networks, leveraging their ability to approximate complex functions and capture high -igh-dimessiona patterns in the data. The model is trained for certain number of episodes but not until complete convergence. This pre-training phase designed establish preliminary value function estimation that captures essential features the optimal strategies in computationally efficient manner.
The neural network-baa based actor is replaced with the rule ensemble actor. This rule-based system is designed to provide greater transparency and interpretability in decision-making With the rule ensemble actor in place, training continues, utilizing the previously trained neural network critic. The critic assists the new actor in refining its policy through ongoing feedback and value estimation.
Step 2:
F.1 ABLATION STUDY
In our ablation study, we explored how variations in the rule count and the implementation of a warm start influenced our method. We conducted tests in the CartPole-vl environment under various rule configurations, specifically using 5, 10, 12, 20, 30, 40, and 50 rules per action. Each configuration was tested both with and without warm start.
500
warm start
400
No warm start
Rewards
300
200
100
5
10
12
20
30
40
50
Number of Rules
Figure 7: Ablation study on different numbers of rules and warm start.
Number of Rules: As depicted in Figure 7 the model configured with 12 rules per action and warm start yields the highest reward. This outcome suggests that having 12 rules per action provides the model with sufficient flexibility to effectively capture the essential dynamics of the environment without being overly simplistic. Employing fewer than 10 rules, such as may offer adequate coverage to manage the environment's complexity, resulting in inferior performance. Conversely, larger number of rules may lead the model to overfit the training data, picking up on noise or irrelevant patterns and diminishing its generalizability to new contexts. The findings from this ablation study highlight critical balance between the number of rules and overall model performance. They demonstrate that merely increasing the number of rules does not invariably improve performance, as an excess can lead complexity and overfitting, whereas too few can restrict the model's performance.
Warm Start: When the number of rules small, the model's representational capability limited, so the initial conditions (e.g., the warm start) become more critical. A warm start can guide the model toward better solutions early on, helping learn an effective policy faster and avoid getting stuck in poor local optima. In contrast, with larger rule set, the model has greater capacity to explore various configurations. exploration, preventing the model from fully leveraging its increased flexibility. As result, the model without a warm start may discover a better configuration on its own when there are many rules to learn from.
such cases, relying on an external initialization can sometimes constrain
22

---

Published as a conference paper at ICLR 2026
G CONTINUOUS ENVIRONMENTS
Continuous environments differ fundamentally from the perspective of the theoretical framework we propose. this work, theoretical analysis restricted to discrete action spaces, as our method enables us to establish formal convergence guarantees.
In order to guarantee the convergence of our proposed method, restrict our analysis to discrete action spaces. The convergence proof relies on several technical assumptions suchhssschhhe as bounded gradients, Lipschitz continuity, and stable critic updates-that are typically satisfied in discrete action spaces, but are difficult to guarantee in continuous settings, particularly for actor-critic methods (Borkar & Meyn 2000) Wu al. 2020). Discrete policies (e-g., softmax) yield smooth and bounded log-policy gradients, which facilitate stable actor updates and tractable Bellman backups using finite sums. These same conditions are also required for the convergence the rule-based function class used in OGB. As a
wow.ttttooooooobbsseeeeeeeeeeeee based algorithm, OGB requires the loss function to be Lipschitz continuous, which means that the gradients are bounded. According to Theorem 2.4, 2.8 and 2.9 in Shalev-Shwart learned models to converge to a theoretically optimal model within a certain number of iterations for (fully corrective) boosting methods and the rule replacement steps.
al. (2010). Lipschitz continuity is necessary condition for the
In contrast, continuous action spaces present challenges, such as unbounded gradients (e.g.. the policy gradient can become unbounded as the standard deviation in Gaussian policies), integration- brased Bellman targets, and higher variance in gradient estimates. These factors violate the core assumptions underpinning both the two-timescale stochastic approximation framework and the rule ensemble model used in our convergence analysis.
23

---

Published as a conference paper at ICLR 2026
H
RELATED WORK
Ref
Work (short) Category
Interp. at scale
P/PUE Performance
Ersst
ct
al.
Tree-based
Tree policy Not (laree trees) (native)
N/N/Y Strong
S/M; degrades on L/complex
22055
Gupta batch RL
(2017)
Iao
et
al. Tree RL for Tree policy Not (muetip.muuiisttteee N/N/Y Hard to scale beyond structured (native)
(0188)
DTRs
domains
Bastani al. Policy extrac-
Tree policy Often not interp. (size N/Y/N
Tracks teacher, small typical drop
2018)
tion
(extracted) balloons)
Roth et al Conservative
Tree policy Not (large trees) N/NYY (native)
Strong S/M; degrades on L/complex
2019)
Q- Improvement
Silva et al. Differentiable Tree policy Not (matr nota- N/N/Y
Solid on modest tasks; mixed scaling
2020)
decision trees (diff.)
tion)
Ding
al.
Cascading De- Tree policy Not interpretable N/N/Y (native) scale
Good early; plateaus with plexity
(2200)
cision Trees
Gupta
al.
Policy
Tree policy Often not interpretable N/Y/N (distilled) scale
Tracks teacher; bigger drop for small trees Approximates
(2015)
Tree(distill.)
Coppens et al. (2019)
Soft DT distil-
Tree policy Often not interpretable N/Y/N (distilled) at scale
teacher;
lation
mide-moderate drop
et al.
Linear Model
Tree policy Not (maerp.(math nota- N/Y/N (distilled) tion)
Good local fidelity; may trail overall
2018)
U-Trees
(onl)
Garcez et SRL with Symbolic Interpretable (symbolic Y/N/Y RL (prede- rules)
Strong when priors fit; brittle misspecified
2018)
commonsense
fined)
Lyu et al. SDRL (sym- Symbolic
Interpretable (modules) Y/NYY
Often strong sample-efficient; planner-dependent
2019)
bolic ning)
plan- RL (prede-
fined)
Ialanes et
Symbolic
Symbolic (prede-
Interpretable (plans)
Y/N/Y Good when plans align; limited otherwise
2020)
plans as instr.
fined)
Landajuela
Discovering symb. policies
Symbolic policy (from NN)
Not (mterp.(math nota- N/Y/N Near-teacher seen tasks; tion)
ct (2021)
drops under shift
Hein
et al. GP-based symbolic
Symbolic
Not imter nota- N/N/Y
Competitive on simpler control; search limits scale
2018
policy (di- tion)
rules
rect)
Verma
et al. Programmatic
Program
Not imat nota- N/N/Y Lags in high-dimensional tion)
2018
RI. (PIRL)
synthesis
policy
Ours
NSAC
Symbolic
Interpretablesyymooiic N//YY Often strong & sample-efficient
RI.
rules)
Table 2: Compact comparison of policy representations and learning setups.
We summarise representative interpretable RL approaches across tree-based, symbolic, and pro- grammatic policy classes. Interp. at scale highlights whether interpretability persists as problem size/complexity grows. P/P1/E abbreviates Predefined knowledge? / Post-hoc from pretrained? Learn directly from environment? (Y/N). Performance sketches typical empirical behavior. Overall, native tree policies often face -itaabbiiiy--iitteretabiitt trade-offs, post-hoc distilled trees track teachers but lose fidelity when heavily pruned, symbolic methods offer strong transparency and sam- ple efficiency when priors/plans match the domain, and programmatic policies remain challenging in high- dimenssiniii settings.
24

---

Published as a conference paper at ICLR 2026
I BENCHMARK SETTING
We recognize that certain algorithms might perform better with changes to the environment (e.g.. adjusting the number of parallel environments or modifying rewards). Still, finding a single modifica- tion that consistently benefits all tested algorithms remains challenging. Therefore, we rely on the most basic version of each environment to maintain consistency across all methods. Additionally, it is important to note that the complexity of our model is designed to balance comprehensibility with the cognitive effort needed to understand of the rules involved. Therefore, in our paper, we deliberately selected 12 rules per action model, which helps ensure the interpretability of our proposed method.
All experiments in this paper are conducted on computer with processor "3.1GH. 6-Core Intel Core i5" and a memory of "72GB 2133 MHz DDR4."
25

---

Published as a conference paper at ICLR 2026
J ENVIRONMENT DESCRIPTIONS
CartPole-vl (Gym). The state 8 E IR continuous and consists of cart position, cart velocity, pole angle, and pole angular velocity. The action space discrete with two actions: push left push right. The reward is -1 at each timestep until termination (when the pole angle exceeds the threshold the cart goes out of bounds).
(position, velocity) R is continuous. The action space is discrete with three actions: push left / do nothing push right. The reward is -1 at each timestep until the car reaches the goal position the top of the hill.
MountainCar-v0 (Gym). The state
Acrobot-v1 (Gym). The state R$ is continuous and encodes the joint angles and their angular velocities. The action space is discrete with three torque actions applied at the actuated joint. The reward is -1 at each timestep until the end-effector reaches the target height, at which point the episode terminates.
Blackjack-vl (Gym). The state is a discrete tuple (player sum, dealer showing, usable ace flag). The action space is discrete with two actions: hit / stick. The reward is +1 for a win, - for loss, and for draw.
Postman (grid- The state is discrete and encodes the agent's grid location together with the locations and delivery status of parcels (picked up / delivered). The action space is discrete: move north south east west and pick-up/ drop-off when applicable. The reward is - per timestep and a positive bonus when parcel is successfully delivered, encouraging short, efficient delivery routes.
The state high-dimensional and continuous, including (for each thermal zone) indoor air temperature, humidity, and CO2 concentration, together with outdoor temperature, time-of-day, and occupancy-related features. Actions are discrete combinations of heating and cooling setpoints (e.g.. heating and cooling setpoints chosen from a finite grid of admissible values). The reward at each control step penalizes both energy consumption and comfort violations: large negative penalties are applied when temperatures leave the comfort band, and a smaller penalty is proportional to electricity consumption, the agent is encouraged to maintain comfort with minimal energy use.
HVAC (building control).
26

---

Published as a conference paper at ICLR 2026
K RULES VISUALIZATION FOR HVAC CONTROL WITH SINERGYM
-.2
0u0000 tomporonuu
Action: hig-11, clg-26
Actions Hig-11. dig-28
0400000 S CONTI Cemnsnonu
Action: h2g 2alloc.hi--2..cgoooe clgm25
-1.3
-144
1.5
24
29
24
Action: hig-188 skpm24
Actions higm dig
comperttare
Action: hige1,, cigm2s
-17
moentt
-18
-1.
A2C
Rendew
sbsac
SAcess
0ufldour lemparatar
(a) Performance
(b) Visualization the Rules
Figure& 8: Comparing the performance of NSAC with baseline methods and visualizing the rules.
outdbor temperature
1.5
electricity demand
20000
htg-18,
air humidity
15,00
htgg188 clg-29
50
htg=118
40
10000
000-18.cl-22
-5
5000
clg-12
-10
20
htg-18.
17.5
20.0
22.5
25.0
17.5
20.0
22.5
25.0
17.5
20.0
22.5
25.0
htg==88,
temperature
temperature
temperature
htg==22
htg-221
20
60
htg setpoint
20
humidity
htg==0,,
hour
15
150
htg-18. cig-26
10
40
htg==1,, clg=26
16g=
htg-15. cig-26
indoor temperature
indoor temperature
outdoor temperature
Figure Visualization of action selection for NSAC.
Power Demand (W)
4000
3000
Temperature
2000
00:00
02:00
04:00
06:00
08:00
10:00
12:00
14:00
16:00
18:00
20:00
22:00
24:00
Time of Day
Heating Power
Occupancy
Cooling
Cooling Power
Indoor Temperature
Comfort Range (18-27'C)
Totaal Power
Outdoor Temperature
Figure 10: Average performance of our proposed NSAC algorithm.
Typical rules for HVAC control with sinergym:
27

---

Published as a conference paper at ICLR 2026
Action htg=16, clg=26:
+50.5500 if 11.7075 < outdoor temporratree 2 15.0 & htg ses_settiint 3 20.0 & air temperature 20.8431
+42.3222 hour 5 6.0 -9.5 2 ouldoor_temperature I 4.3200 & tir_reeprr I 21.1095
+33.7433 if hour - 6.0 & outdoor tepoorrttmeeetteeeeee 3.0 & eleeeeecrrriittyddddaddd.
_emandd rate 3524.0383
+30.0530 air temperature v 19.4323 & HVAC_
demand pate 2 3492.0372
+23.8401 if hour - 6.0 & outdoor temoortttumeeeureeeee -6.0 & HVAC_ LLAACCelectricity... .eriidd__eddddddeeee rate 2 1836.0156
+19.5165
- -5.0 & air air_temperature < 20.8367 & air hir_huuiiity 2 34.4557
+19.3194 if outdoor Mundoor_temerraurre < 4.3200 & clg_setpoint- - 26.0 & air temperature & 20.7468
air air__aumin hur_huiiyity/ 2 44.9061 & LVAAC_elearrcitt__eeeaeee rate Σ $328.7136
+17.1997 if outdoor ceadoor_temmerature < -6.0 & air tur_teeerratree 20.6569 & air sur_huii 2 34.4701
+12.2539
< 1.1000 & air "et_mmerrature < 19.8760 & air humidity 2 48..4554
HVAC_ CVACCelectriccty.... [ demand rate 2 2194.6457
+10.1033 dla__o_monite < 16.0 & outdoor_tempae jemperrturr & -9.5 & air jir_temperature - 21.48222
+9.9095 if outdoor saddoor_temperaturee - -1.2000 & clg_setpoint < 28.0 & air sir_temperature 18.6297 &
2259 < air humidity < 57.8947
+6.5348 if hour - 5.0 & outdoor suudoor_temperatur -6.2000 & air humidity 2 46.6687 &
ler__eeccrritty_eeneeeee rate < 6047.4648
Action htg=18, clg=26:
+52.4710 outdoor_ teaiborrtempeaatuuee - 2.0 htg_setpoint 2 18.0 air jir_temperature 2 24.6918 & 36.4788 S aai__hh jon_huuddity s 59.9498 & 655.1863 < HVAC @VVA_electricityy electricity_demmandddmo caatand_rate rate < 3286.0521
+31.8345 - 7.3000 & outdoor_temperatee temperature S 2.4880 & blg_seeppinttt 2 28.0 & air tem_eerrrr 2 22.7824 &
air humidity S 53.0045 & HVAC cocctricitydemmanee rate 2804.0908
+25.6196 if day_of_ _f__ththhhh < 23.0 & hour 2 21.0 & htg_setpoint 2 18.0 &
air_ jir_temperature < 21.2843 & 22.2264 air humidity 57.4836 &
HVAC -lce_cccctr _demanddddddddddeeee rate < 4727.3044
+ +14.6693 if hour S 16.0 & clg setpoint 2 28.0 & 24.3635 air tii_ttemeerture 25.3421 &
air_ air_humidity Σ 43.6038 & 1974.6005 I UWAC_eeectricity_deeeeee _ate Σ 2847.7105
+12.8951 if ddy__oo month > 17.0 & 1.0 < hour 5.0 & hit_setpoint < 16.0 &
air_tenn tirrtemperature < 19.6280 & HVAC gleetricity_demaeee rate 2 728.1273
+6.6281 if 14.0 < hour - 15.0 & outdoor caalloor_temprraturr -2.4800 & 24.1909 air ter_temeerrture 25.1225 rate < 2856.4850
air air_humidity - 44.0097 & HVAC_electricity_ddee
+3.6962 if outdoor tecoooottttmeratuureee < -2.4800 & air temperature 25.9993
+2.2511 if hour 2 18.0 & - 5.0 I outdoor teniooortemperatureeee < -2.4800 & air air_temmpaaa tet_pereertureee 2 22.3801 & air air_ humidity 52.5242 & 1164.7935 <
wtlpeeeerritt__eemeeeeeee rate 1443.5973 +1.7191 - 7.3000 < sonddorr_emmeraturre < -3.0 & clg_setpoint 2 30.0 &
air humidiuttttti - 52.1981 & 1411.8365 < HVAC Snbettrciit_demmaeee rate 2783.3056
+1.6791 5.0 < ofy_of_monte < 28.0 & 3. 44000 <
< 6.6000 &
22.4682 < air hir_temperature < 23.2356 & HVAC_ WWWACCeeectrccty_ddmaddd.ao < 1559.4128
28

---

Published as a conference paper at ICLR 2026
Action h1g=18, clg=25:
+76.0682 month $ 2.0 & day_ moy_offmothhh - 29.0 & - 2.4800 Σ autdoor_temperee Σ 15.0 &
htg_setpoint 21.0 & air_temperature v 20.4542
7-35.7797 if month < 2.0 & day_of_month < 28.0 & outdoor_temperature < 15.0 &
air_lemperature - 20.4547 & HVAC_electricity_deeee httplettrreitty_ddmm___eeeeee 888.4346
+ +19.5429 if hour - 17.0 & air hir_humidity 22.1544 & SWVACCellctriccty__emaaaa
& 4.3200 & air_temperature 2 21 21.9922
3353.0663
+11.0059 if outdoor_temperatue < -7.3000 & air_temperature 2 23.0022
HVACC_llettrictty_demme letrriicttyddmmad_____m pate 2 3343.0970
+ +10.4447 month & 1.0 & guudoor_temperatte N 0.0 & air_temperature 2 22.3607
air hur_humyy 35.8469 & INVAC_electrccity_demaaee _ate 3237.7531
+9.2884 if may_of_monti < 28.0 & - 2.4800 - oudooo__temperature - 15.0 & clg_setpoint S 25.0
air air_temperature - 20.4644
+8.4585 if hour < 16.0 & air_temperature 2 23.5872 & IVAACellcticctt_ddmmmn_aaaaa 2 3378.0424
+7.6878 if 2.0 may_of_month 8.0 & hour V 15.0 & outdoor__emperraure S -6.7000
air temperaturee 2 22.9565 & HVAC_electricit Selectricityddemandddoo __aterttt_____ < 5120.2045
+6.0944 if hour < 16.0 & outdoor_temperrauree -9.5 & air air_temperature 22.3765
+6.0912 if - 2.4800 < outdoor_temperatue < 1.0 & clg_setpoint < 29.0 & air air_temperature < 18.5048
28.1472 S. air hir_iiiyy S 57.5569
+4.1747 if air temperature 25.1225 & HVAC -eccCticcctiiittytcooo __elldddddd____e__eeee caamand_ratt> rate > 4135.6304
+2.7032 2.0 day_of_mont 29.0 & hour 1.0 & 11.7075 outdooor_temperauure < -5.6000
& 18.0977 S. air_ air_temperature < 24.9366 & 4026.8635 < LaAAA_eelctricit__eemaadd
7054.0210
Action htg=18, clg=28:
-447.2007 huudoor__emperature < 1.1000 air_temperature < 21.3847
+42.5021 if month 2.0 & outdoor_tent _eepperauuee S 13.0150 & clg_setpoint 29.0 &
air sir_temperature < 20.8932 & CWAC_ellctricciyy. Beciiitdddmmdd_____e aaanand_ratt_ S 9236.5036
+37.5818 if month 2.0 & htg_setpoint < 21.0 & 19.0381 air_temperature 22.2181
air_ air_humidity S 61.5915
+37.5560 if month 2.0 & htg_setppint - 21.0 & 19.0971 air teumteerrrttee < 24.1215 air humidity < 62.4223
126.1161 if air terpeeerrreree & 21.6418 & HVAC_elcctricity_ Berpllddddmmdddd__eee _rtee_rte___ 2 1616.5198
< -3.4000 & 17.0 < het_oetpoint < 18.0
+25.3651 if month 2.0 & hour < 6.0 & butdoor_temperree
8.8000 & air temperaturee & 21.0920 &
air_humidity < 51.7392
+21.4283 if hour < 6.0 & outdoor_ terboorrtemeraturee -200 & htg_setpoint 18.0 & air_temperature s 20.0191 & WVAC_clectricity demand rate 2 1666.1656
+19.3831 if air temperature 19.0718
+18.9771 cel_setppoin 2 28.0 & air air_temperature 22.6279 & 3313.9887
rate S. 9994.2258
+15.9370 outdoor_ temporruue Σ 1.0 htg_setpooin < 20.0 air temperature Σ 21.1680
seeeerricitt_ddmmaddd... & air humidity 46.0692 & EVVC_electricitty demand rate 3766.4288
+11.4827 outdoor temperature < 1.1000 & air jir_temperature < 18.9835 outdoooottemerreeee & 2059. 0059.7858 HVAC_ HVAC_elcctricitty _emmadd demand rate < 8526.6964
+11.1224 3.0 < day_of month < 28.0 & hour 2 22.0 outdoor temperature & -1.0 & htg_setpoint $ 18.0 & 20.5696 air son_huiddity < 37.0262 & wwwceeeiectyddeeeeeeee _ate rate 2007.6986
29

---

Published as a conference paper at ICLR 2026
L
RuLES VISUALIZATION FOR BLACKJACK-V V1
stick
rules for stck
12-
14-
With Usable ace
With Usable ace
20
LOME 6337k3
12-
12
18
20-
14
14
niles for hit
AMM
20
16
24-
18
18
20
20
20-
dealer
dealer
10
20
Figure 11: Visualization for action, value, and rules for Blackjack with Ace.
Usable
Abcck
rules for slick
Without Usable
Without Usable
20.0
stick
17.5
14
6585ifa221
15
8
15.0
18-
4557 re Exehipst
10
niles for
player
12
12
10.0
Eshhllighhaa
14-
7.5
16
16
15
18
18
2.5
20
0.0
Seweer
dealer
10
6
1.0
dealer
dealler
Figure 12: Visualization for action, value, and rules for Blackjack with no Ace.
Action 0 (Stick):
+1.0012
if
+1.0376
if < and 12
+4.4833
acc 2 and p 2 14
+7.9438
if and 17
+3.9105
if 17 and p 2 13
+0.9851
2 21
+2.6981
2
-0.1967
if and d and < 8 and p 2 8
-8.313
acce $ 0 and p < 13 and 2
+0.0882
if ace 0 and d > 10 and p 16 and p 2 14
30

---

Published as a conference paper at ICLR 2026
Action (Hit):
+0.6347
if < 9 and - 8
00.481
if and p _ 6
00.231
if < 8 and - 7 and p 2 6
+1.6016
if and 2 and p S 8 and p 2
-0.5885
if < 9 and _ 7
+0.9266
if and p -
+0.3394
if 2 9 and - 10 and 2 9
+0.9844
if and p <
-0.3849
if 2 3 and S 6
-5.0124
if d and
p 2 17 and p 2 18
31

---

Published as a conference paper at ICLR 2026
M MOUNTAINCAR RULES
do nothing
rules for do nothing
0.07
119.725 pf 0.510 v=
-13.485 =
24.220 p
-23.371 v
24.743 if p= v 0.000
-26.515 pf 0.633 v
-0.07
right'
rules for right'
26.215 if pf v 0.077
0.07
27.518 pf 00402 0.000
21.307 v
18.688 vf 00000
19.766 if 0.000
21.341 if s -0.633
11.775 if = -0.577 0.001
-0.07
16.511 v
left
rules for left
0.07
25.311 if 000001
-10.568 if 0.502
51.035 if 0.0004
0.0
-14.514 if 2 v
-11.616 s v
17.688 -0.00
-0.07
16.429 if.4..2....88
-1.20
-0.8
-0.48
-0.12
0.24
0.60
11.20
-084
-0.48
0.24 0.60
&-0.012
p
-17.952 if 7f
0.0001 v.0000
actor
21.474 5
output
50
50
100
150
Figure 13: Visual representation of the rules for MountainCar, Higher actor output values in the heatmap indicate that the corresponding action is more strongly preferred that state. The squares indicate the regions of the state space where the corresponding rules apply.
M.1 CASE STUDY: DECISION BOUNDARY AROUND P > -0.48 IN MOUNTAINCAR
To better understand the learned policy, we analyze its behavior in the MountainCar environment in the neighborhood of the valley bottom. The state is two-dimensional, the horizontal position of the car and v its velocity. The car starts in the valley and must build up sufficient kinetic energy by rocking back and forth in order to reach the goal the right hill. Because the engine is too weak to drive straight up, the policy must carefully coordinate position and velucity accumulate energy.
(p, v). where denotes
32

---

Published as a conference paper at ICLR 2026
goal (p=0.5)
I 0: Right
<
turning point
v=0
P
Figure 14: MountainCar Environment Near the Valley Decision Boundary. This figure shows the MountainCar track with the goal flag on the right hill and the car rendered near the bottom of the valley at P -.48. The vertical dashed line marks the region P -0448 where the policy's decision depends on the sign of the velocity: for v > 0 the agent acceleraes to the right to build speed toward the goal, whereas for v < 0 accelerates left commit full swing back and gain momentum for the next climb.
In our rule-based policy, of the most salient decision boundaries appears for positions the right of the valley minimum, i.c. for -0.48, which is close to the bottom of the track (see Figure [4p. In this region the height the track changes only slightly, so the gravitational potential is nearly constant and the immediate direction of motion is determined almost entirely by the sign of the velocity v.
Consequently, the line v 0 for p > -0.48 becomes a natural decision boundary in the (p. ul-pnane. States with p > -0.48 and action is to keep accelerating right to maximize its speed for the upcoming climb. In contrast, states with p > -0.48 and < indicate that the car started to roll back the left after an unsuccessful that the next
correspond to the car already moving to the right; the optimal
attempt; the best response is to accelerate left and commit rightward run can reach a higher point on the hill. Exactly at v = 0 the car is at a turning point, momentarily stationary before choosing whether to move left right, which is why this line marks the decision boundary used by our policy.
full swing back,
Figure 14 visualizes this situation directly in the environment: the car is rendered near p -0.48 the bottom of the valley. and the vertical dashed line indicates the region > -0.48 where the policy switches between "go left" and "go right" depending the sign of the velocity.
33

---

Published as a conference paper at ICLR 2026
M.2
THE RULES
Action Left:
+25.3111
if 0.00664715572250255288
10.5677
p -..502344751335503222
+53.8697
if True
+51.0351
if - 0.0042659555922713333
-44.5140
p 0.581898997584415111.aa and 0.00680037022499915122
and 2 0.0008947385551188666999
-11.6162
if I 44117757730222750055 and v 0.000666100077709983399
+17.6882
if 2 L.55032877753289977 10
+16.4293
if' 0.252408774422791444 and p 0.62755403357184143
and v S 0.0151030002204772997 and v -0.01191
17.99516
if 0.44011819665567779 and -0.615831172466278
and v - 0.00759 and v -0.004150 and 0.00126822
+21.4738
if 0.00040988854762209496666
-236676
p 2 :
and -000341004066300022555
+1.2316 p I -0.1402401 and p -0.4994079709053003
and & 0.02341402264452294402
and -0.016320005203559933
No Action:
199.7248 if 2 -05100113883301143377.maa and v 0.006803370246991555522
and Z 0.0055730004455550773
-13.4848
if r 2 0.00071
+24.2199
if 2 0.48485273112287477 and v 0.00018948809495699776
and v 2 0.009227755142339277022
23.3706
i S 0.003299118250699934414
-6,6878
i < 9. 9.655811237431741 x 10-5
+24.7429
pf < 0.40226433873176637
and e Σ .000209522749110077777
26.55154
ifp Z 0.631155845229633999 and < 0.008872396966151048
and 0 2 0.006244536223303331099
-8.7515
if Z 0.392600633713886777
and
0.001906221889116333699
+0.7214
if < 0.00014433928881444977
-0.9839
if True
+6.8736
p 2 0.544775545978546155mm and e & 0.009966-69033820033555555
+1.0000
if Z 0.00100968902887776618
34

---

Published as a conference paper at ICLR 2026
Action Right:
+26.2150
p - 0.31011388301849997 and e - 0.00680337022669915552
+6.3636
if 2 0-44448573122277777 and 2 -0.0092771512233997702
+6.1251
if 2 Q.481566555449224038 and Z 0.00329218225069983444
+24.0837
Tr'u
+27.5176
if < - 0.4022643333733166377 and S 0.0012711382005366774
+21.3072
i 2 0.0007191195115773555
+18.6883
if - 0.000552022999966338333
+19.7662
i 2 0.00044220104228125554
+21.3412
ifp - 0. 9.63292798995597666
+11.7750
if 2 0.37676002970228866 and 2 0.00122622389265222241
+16.5113
if 2 0.00040000004776200999966
+1.3610
i S 0.44944077000553038 and p 2 0.87444302277996631 and v S 0022341402294552244002
35

---

Published as a conference paper at ICLR 2026
N
EXPLANATION OF CART-POLE RULES
left
rules for left
-0000000
-6822 if a..00.5
15.726 if
3.470 0.366.
4.015
-7.361 2 & pa 2000884
-7.779 if 00.000
00.422
7.091 0.217 500.00 & -0.020
right
rules for right
0.41
-3554 co 0.044
-3.776 <p1-0.121
0.0
2.038
.9.522 cD o.0047
202-0006 pa.-00.036
10.287 of.000.00000
0.0-0.034
2.050 0.000.004
1f
-0.43
5.118 qp 0.402 & 000000
20.400022
-4,80
-2.40
0.00
2.40
4.80
4.80
-2.40
0.00
2.40
4.80
p
P
-10
Figure 15: Visual representation of the rules for CartPole involving position and velocity.
The CartPole environment consists of a pole mounted on a cart that moves along a track. The objective is balance the pole upright by controlling the cart's movement left or right. The state space includes the cart's position(p), velocity(#)), pole angle(f), and angular velocity(0), while the action space includes applying force to move the cart in either direction. Figure [5]is the visualization of our actor's rule regarding the position and velocity. Please refer Appendix O for all the resulted rules.
Here will typical state in the Cart-Pole game, where the optimal action is to the cart the right, characterized by the parameters p -0.05, 0.1, / 0.05 and Ø 0.3, to illustrate the decision-making process for our proposed method. In this scenario, pushing right reduces the pole's angular velocity, helping stabilize the pole by balancing the torque induced by the tilt. In our model, three rules become active when considering a leftward action; each rule incurs negative score because the pole's angle is positive, decisively discouraging left move. contrast, the overall score for moving right is 2.3, notably higher than that for moving left. Although the car's slight leftward position imposes small penalty, the pole's 's rightward angular velocity plays a critical role: rule alone contributes +19.8 to the total, effectively countering any negative effects from the other rules. Consequently, given the pole's 's rightward tilt and velocity, the agent's 's optimal choice is to push right, thereby maintaining stability and preventing undesirable outcomes.
example
36

---

Published as a conference paper at ICLR 2026
O
CARTPOLE RULES
Left:
-6.8310
pa 2 -0.06048
-6.82240
if pa 2 0.05751
+14.59180
if pa Σ 0.04701 and pav & 0.26781
+15.72630
if pav -0.52576
+3.47030
cf - 0.36470 and cp 2 0.03103 and cv - 0.20766 and ev Z 0.00066
and pa - and pav - 0.28779 and pav -0.05232
+0.88280
cp 2 0.24231 and cv 2 -0.40319 and pa I -0.03636 and pa 2 -0,05565
-7.36080
if cp 2 0.36775 and 2 -0.00425 and pa 2 -0.03416 and pav S 0.12249
+11.59770
cp 2 0.25174 and pa S 0.00381 and pa 2 -0.02022 and pav & 0.11855
and pav 2 -0.37647
-7.77910
if pa 2 -0.02969
-0.66900
if cp - 0.16311 and ep - 0.30448 and cp 2 -0.11618 and cv I -0.01313
and cv < 0.17905 and pa & 0.03930 and pa 2 -0.02479 and pav - 0.14835
-3.02560
if cp 2 -0.42154 and ev 0.32329 and ev 2 -0.01879 and pa -0.02018
and pa 2 0.01293 and pav & 0.13896 and pav 2 00.0136
+7.09090
if cp < 0.21659 and < 0.21317 and 2 -0.00898 and pa S -0.00355
and pa 2 -0.01990 and pav < -0.29080
Right:
-3.54700
if ep < 0.04146
+19.18060
cp 2 -0.07721 and pav 2 0.20059
3.77380
if cр 2 -0.12061
+2.03760
cp 2 0.05832 and cv S 0.06116 and pa 2 -0.06911 and pav I -0.05532
-9.54210
if v 0.04726
-3.07880
cp 2 -0.01648 and cv Σ 0.16053 and cv 2 -0.19434 and pa S -0.03636 and pa 2 -0.05565
+ +10.28650
if cp 0.36775 and cv -0.00425 and 2 0.20400 and pa S 0.00317
and pa 2 -0.03416 and pav < 0.12249 and pav 2 -0.37842
+2.04960
if cv 2 -0.34930 and pa 2 0.00381 and pav 0.11855
5.77560
cv 2 -0.36800 and pa < -0.00339
+5.11750
if cp 0.40156 and 0.39100 and pa -0.00914 and pav & -0.33129
+0.00510
if cp 2 0.02529 and cv 2 -0.01879 and pa I 0.03998 and pa 2 0.01293
and pav 0.13896
-4.80880
if cp & 0.21659 and cp 2 0.00944 and cv S -0.28634 and pa S 0.00355
and pav 0.30114 and pav 2 -0.01324
37

---

Published as a conference paper at ICLR 2026
P ACROBOT RULES
+8.7846
if w1 2 0.09082 and u2 S 0.22991
+9.3353
if cos(02) 2 0.89409
10.4999
(i2) - 0.44980
+8.2458
if sin(O2) < 0.37771 and wi 2 -0.12610 and w2 s 0.15699
11.6541
True
+6.8474
if sin(O)) < 0.24220 and w2 2 0.04877
+1.2850
cos(02) z 0.99115 and sin(01) v -0.01332
+1.7239
if sin < -0.01472 and W2 0.02869
+3.5813
cos(01) 2 0.88816 and sin(01) 2 -0.28723 and w1 S 0.77240
+1.3335
if cos(01) 0.48252 and wi S 1.78991 and w1 2 -1.7088
+1.0000
cos(02)
- 0.95137 and sin(0;) 2 0.01719
-3.0199
if (02) & 0.99319
-3.3464
if c0)) S 0.99048 and W2 S 0.12677
-8.6917
if cos(01) 2 0.81787 and wh - 0.92265
55.697
if True
99.3263
if wh & 0.21755 and w2 -0.10354
-5.5746
if s0n) < 0.37771 and w1 S 1.28679 and wy 2 0.15699
+5.4709
if s01) 2 -0.26647 and sin(O2) s 0.44500 and w1 2 -1.07104 and uy S 0.04877
8.2010
if s0n) S 0.04400
+1.5784
if cos((1) 2 0.99341
-9.7897
if s0.) I 0.15513 and sin(0:) -0.70008 and sin(02) -0.17949
and < 1. 78999 and w1 2 11700888
-1.2848
if cos(f) (0)) 2 0.99674
+1.0000
if cos(02) < 0.99350 and sin(0;) < -0.21224 and 2 0.11238
and W2 S -0.22888
cos(0)) 2 0.99689 +3.9980
+2.5385
cos(O))
0.99048
+ 16.7525
True
-377130
if cos (0) 0.99999 and sin(O2) <
-0.01789
-1.9920
sin(02) < -0.00190
12.0937
if w < -0.02180 and w2 0.04994
+ -9.6334
if W2 2 0.04994
+1.1957
if cos( 0.24062 and wi -0.32756 and w2 < 1.64803
-3.6800
if w/ 2 -0.12540
15.8530
cos(O1) 0.88816 and w2 -0.41702
+ +1000707
cos(0)) < 0.99674 and (0s) 2 0.92457
+1.6685
cos(02) S 0.99350 and sin(0;) < -0.21224 and wi > 0.11238
and up < -0.22888
38

---

Published as a conference paper at ICLR 2026
Q INTERPRETABILITY OF TREE-BASED AND RULE-BASED POLICIES
Following the widely cited definition of interpretability by Murdoch et al. (2019), an interpretable model should satisfy simulatability, modularity, and low complexity. Simulatability means that a human easily compute the model's output from its explicit expression, given input and the model parameters. Modularity means that individual components of the model can be understood in isolation. Complexity can be quantified by, for example, the total number of terms parameters in the model.
SYMPOL learns axis-aligned decision tree policy directly with policy gradients in standard on-policy RL setting, yielding an interpretable tree-based policy class. Conceptually, our method plays similar role but uses rule-based policy class instead of trees. While trees and rules closely related-each root-to-leaf path can be seen as a rule-they differ in how they are used and in the kind of interpretability they afford.
From the perspective of interpreting individual decisions, both trees and rule sets provide simulatabil- ity. For given state, a tree explains its choice via a single path of the form:
(condition node I)A(condition node 2) -ccondiicnn M at node M) take action LEFT.
rule-based model explains the same decision by indicating which rules fired and how they were combined, for example: "Rule I and Rule 7 are activated and jointly give the highest weight; therefore the policy selects LEFT."
In terms of complexity, rule-based models tend to be slightly easier to keep compact: we can directly control the number and length of rules. Tree-based policies, by contrast, often grow deep or wide as task complexity increases, which can quickly erode practical interpretability even if the representation remains transparent in principle.
For modularity, the difference is more pronounced. In tree, can treat each root-to-leaf path as a "module" (a rule), but internal nodes are shared across many paths, so modifying a single split typically affects large portion of the state space. In rule set, each rule is self-contained piece of logic. This makes the rule-based policy highly modular: individual rules can be inspected, added, or removed without having to mentally re-parse globally restructure the entire model.
This modularity also makes certain safety-style queries much easier to answer with rules than with trees. For example, consider the property: "Is there any situation where the policy chooses RIGHT while |0] > 0.25 and x > 1.2? so, list those situations." In rule-based policy, we can simply scan for rules whose antecedent includes |0] > 0.25 and 1.2 and whose consequent is action = RIGHT. If such rule exists (say, Rule V1), we can immediately say: "Yes, a violation occurs whenever Rule VI fires." " If no such rule exists, we can cleanly state: "The rule set guarantees the safety property: in rule does [0] > 0.25 A > 1.2 appear with action RIGIIT." For an equivalent tree policy, answering the same question requires an exhaustive traversal and aggregation of all relevant paths, which is considerably less direct in practice.
R
PONG ENV
R.1 RULES
Do nothing
39

---

Published as a conference paper at ICLR 2026
Aspect
SYMPOL (tree-based policy)
Rule-based policy (ours)
Policy class
Axis-aligned decision tree policy; each internal node threshold on one ture, and leaves output action logits. High: for one state, follow single root- to-leaf path clear "if AA BA ..... then action."
Explicit rule set rule list; cach rule conjunction of conditions = action (op-
tionally with weights or priorities..
Simulatability (single decision)
High: for one state, see which rulc(s) fired and how they combine (e.g., "Rule for LEFT").
+ Rule
Global structure
Clear hierarchy of decisions: top splits show the most salient features.
rigid; rules form flat or lightly structured set, regimes are inferred by grouping similar rules.
Global picture
Modularity
Each path can be read as rule, but in- teraal nodes shared: changing split affects many paths at once.
Highly modular: each rule is self- contained unit. One inspect, add, remove rules without restructuring the whole model.
Complexity control
Control via depth limits, pruning, and tree size; trees can still grow large and become hard to read.
Control the number of rules and rule length; it casier to set an explicit rule budget and keep individual rules short.
Table 3: Comparison between the SYMPOL tree-based policy and our rule-based policy from an interpretability perspective (simulatability, modularity, and complexity).
+ 0.9642
if Un,b 2 0.1
0.0053
vf,6 -0.1 A Uy 2 -0.1
+ 0.9613
of & 0.61874997615881221
4.2170
if yo 0.65238094332963338 y. 0.54761194448877312
- -3.880
if Uy 2 0.1 0.652238009339933333
4.1575
if uf 2 -0.2
+ 0.9699
if 2 445714295999734199
331777
if Ug A. S 0.54761904448873112
0.0000
if 2 0.800000011920929
3.1775
if 0.1 Am 044875000011992022366 A 0.5476190447807317
4.2170
if < 0.618749996615814211
Aye <
Aye 2 0.5476190444507312
0.9693
i 0.78755000388418579
+ 0.0005
if xs 2 0.78755000388418559
3.3783
if ra 0.487550001192092836
+ 0.9695
ify. & 2.6428571343421110
0.9695
ify. S O. 6523889432983399
0.1303
if < 0. 899999997611884221 A 22 2 0.1
0.0000
ify, Z 77809553340088365
-.0000
if < 0.300000011992092296
A 2 0.909523885114288! ^ 3y - 0.300000011920998896
-443359
Up 0.300000011922092996
40

---

Published as a conference paper at ICLR 2026
Up
1.0691
if true if d I -0.1
+ 9.5734
9.7684
if d _ 00.1 2y -0.1 A 9o 2 0.7380955233003356
3.6835
U2. 2 -0.1 A dg -0.1
1.8689
if - 0. 1.80000001 1920929
+ 6.6212
if 2 0.800000011920920 AY. 2 0.54776190447897312
9.8937
if 2 - -0.1 y 2 0.78809523940088333
+ 3.2888
if - 0.0 A 2 0.80000000119229999 A.
0.54701994478877112
2.1446
if x - 0.793779988079071
-2.7148
U2S 2 0.1 A 44 0.1 5 00011777777711111111100000000000
10.3563
if 12 2 0.1 A yb 2 71176600077773322222222200011010
+ 2.2118
rf 2 1. 01t.:9001111111110..0.0000071413 A ye 0.776190459728241
1.1660
if eg - -0.2 4h S 0.788095233400066333
+ 9.0361
if & 0.800000011920992) ^ zb 0:4577500011992992866 A y6 2 0.5476190447807312
10.8639
if d4 2 0.1 A 4g.b 2 0.1 A y6 S 0.78800533900066366
+ 6.7201
if - 0.800000011920992) A yb 0.78095533400863655 A ye 0.776190459928241
1.9128
bf 2 0.45714285969933419
2₁ 2 0.4875000111920022866
+ 12.6463
ify: 2 0.9095237851142888
+ 0.8748
bf Z 0.128571435809113 43580913544 A "p S 0.60000000238411557) y6 2 0.7809523940086363
1.1194
if Up S 0.3000000119200000000000 A Ua,b < 0.1 S 0.6523809432083338
Down
0.1153
true if Un,b 0.1
+ 0.2371
1.1869
if uy & 0.1 y.
y. 2 0.64285713343421988
+ 6.6905
if 0.80000000119209999
1.6155
if & 0.20000000029880222244 A 1466 S 0.1 14,5 2 0.1
7.3399
if 0.618749976115812211AA 2 0.65238094329683398 A A.e > 0.6428571343421938
1.0679
if < 5..17777758. Ay,
0.54761904448877312
0.1710
if fy -0.2
1.6300
if y 2 0.547619904488073124y 1t0 2111110011111
0.0564
if 0.1 < 00818774997611181441111
0.64285711344421938 62
0.2991
ify. 2 0.9095237851142883
6.6643
if 0.300000011122099999000
0.64285711334211936
+ 6.4798
v& 2 0.1 A F S 0.88000000119909999) A m. 2 0,487500011992092886
0.6501
if 0. 1,0000000111001112222,,,,, 0.61874997615811221
yo 0...200.00000000 0.7809523940086363
0.7654
if 2 0.1187749971158812211 0.776190459722241 |
0.0399
if vy - 0.776190459728241
0.0802
if uy S -0.2 2 0.642285714434219986
1.2114
if 0.1 0.48750001192002288 Ay, 0.64285713434219996
-.0488
if 2 0.97500000388418577 A A3p & 0.1185574455099135444
1.000
if if 0.61874997615814221 Ab 0.7809523940088336
41

---

Published as a conference paper at ICLR 2026
Feature importance analysis. To diagnose why this particular rule ensemble performs poorly, compute a simple global importance score for each feature by summing the absolute values all rule coefficients in which it appears. This tells us which variables the actor relies on most when selecting actions. For the rules in previous section, we obtain:
ball xxposition (zr): 89.18, opponent y-position (y+): 88.85,
ball y-position (y6): 77.05, ball x-velocity (Va,6): 65.8, ball y-velocity (Vg.6): 47.9,
paddle velocity (xp): 19.7, block fraction (bf): 3.9, paddle y-position (1p): 0.05.
These numbers show that the policy is dominated by the geometry of the ball and opponent, with the ball's horizontal position Ia and the opponent's vertical position Ye emerging the most influential features. In contrast, important control variables such the paddle velocity Up and especially Yp contribute much less overall. Although focusing on ball-opponent geometry not unreasonable, closer inspection shows that many of the largest coefficients involving Ye appear in nules that trigger clearly suboptimal actions. In other words, the ensemble has learned strong but misaligned dependencies ye, effectively overfitting to spurious thresholds on the opponent's position. This explains why the resulting policy is both interpretable and systematically wrong: it selects actions for transparent, but ultimately bad, reasons.
42

---

Published as a conference paper at ICLR 2026
S ADDITIONAL EXPERIMENTS ON OCATARI
S.1 PERFORMANCE
We also evaluate NSAC on the OCAtari-Breakon environment, a low-dimensional variant of the classic Atari Breakout game exposed through the OCAtari Delfosse et al. (2023a). The agent observes summary features such as the ball position (ballyy, rel_x), ball velocities (vx, vy), paddle position (padole_ede) _eegee), and brick-related indicators (brick_pree brick_cur) and must choose between three discrete actions: Fire/Do nothing, Right, and Left. Performance this environment is reported the total game score, averaged over 5 runs with different random seeds for each method. Among all baselines, PPO achieves the highest overall score on OCAtari- Kari-Breakoutt Nonetheless, NSAC also attains strong performance, with average score above 300 over 5 random seeds, making it competitive with neural baselines while clearly outperforming all other symbolic methods. This demonstrates that NSAC can retain much of the raw performance of powerful deep RL algorithms, while still yielding an interpretable rule ensemble.
Environment Breakout
Q-male
DON
A2C PPO
SDSAC
SACBBF 267.00+ 27.12
Rainhow
SYMPOL
wathne-D
D-SDT 23.86 + 34,26 14.76
KSAC 312.76
41.23
227.23+
344.234
389.21 + 342.00
761
227-00=
14:20
25.78
31.78
14.33
39.62
31.37
35.76
7.86
31.60
Table 4: Performance comparison on OCAtari-Breakonn
S.2 REPRESENTATIVE RULES AND THEIR INTERPRETATION.
To illustrate the kind of behaviors captured by the learned rule ensemble, we highlight two represen- tative rules, the detailed rules are listed in the following section.
Rule (Action 1: Fire/ Do nothing). One of the highest-weight rules for the "do nothing" action
+6.1152 0.5000 < ball_y_our < 0.7714, rel_x_cur S -0.5375, vy_prev I -0.0190, vy_cur 2 -0.0381.
Here, wwa.bbllllll___ee in 0.5, 0.77 indicates that the ball is still relatively high on the screen, while rel_x_ _0l_x_cur_ cur < -0.5375 means it is clearly to the left of the paddle. The vertical velocities vy_ vyee S -0.0190 and cur 2 -0.0381 show that the ball is moving downwards, but not extremely fast. In this regime, the rule strongly favors not moving the paddle: it is too early and too uncertain to chase the ball aggressively, and keeping the paddle near a default position preserves flexibility for a later, more accurate intercept. This encodes an intuitive "do not overreact too early" timing heuristic.
Rule (Action Left). strongly negative rule for the "move left" action is
-5.7469 0.4143 < ball_y 0.8381, rel_x_prev < -0.0063,
0.0500 < vx_prev < -0.0375, vy-prev S -0.0190, vy_cur S 0.0286.
In this situation, the ball is in the middle-to- topulerr region of the screen, already to the left of the paddle trel_x_prev < -0.0063), moving leftwards (vx_prev < 0) and downwards (vy_prev: < 0). The large negative coefficient indicates that moving the paddle further left in this configuration is strongly discouraged. Intuitively, shifting left would risk moving away from the eventual intercept point overshooting it; instead, the policy prefers to stay more central or delay horizontal motion until the ball is lower and the crossing point is clearer. This rule thus captures a natural "do not move away from incoming b behavior.
43

---

Published as a conference paper at ICLR 2026
S.3 RULES
Action 1: Fire/ / Do nothing
-2.0037
if pfpaddle_edr__prev 2 0.1375
4.0445 vf 0.0190 and vy_eur 000019
+2.6453 ifball_ Baclllpppp______ 0.7714 and ball_y________ __eeuu - 0.7714 and rel _______crr_______ cur S 0020000
and - 0.0500 - vx_prev < 0.0500 and - 0.0286 2 vy_cur 2 0.0381
+5.9613
@aeell_above_pre_ 2 1.0000 and
/ 0.8381 and rel_x_prev < -0.0063 -000063
and 0.0500 - vx_cur S 0.0500 and vy_prev 2 -0.0190 and vy_cur & 0.0048
-1.1160
ifball_y_prev_ 0.7714 and rel_X_cur 2 00.415 and vy_prev & 0.0381 and vy_cur & -0.0190
-233335
rel____cuur 2 00.000 and Sy_prev
+3.6887
ifball_y_prev - 0.7714 and ball_y_cur & 0.7048 and rel_x_prev -0.1000 -0.1000
and vxl_x_rrev2 2 -0.0355 and vy_prev 2 -0.0381 and vy_cur S 0.0190
+2.4687
ifbally_prev 2 0.6381 and vy_prev - -0.0286
+3.8568
ifbll_y_reev& - 0.7714 and rel_x_cur v -0.0063 and vx_prev 2 -0.0500
and vx_cur -0.0500 and vy_prev < 0.0048 and vy_cur 2 -0.0190
-27113
ifball_y_cur= < 0.7048 and brick_cur 2 0.0900 and vy S 0.0381 and vy_cur S 0.0048
+1.9062
ifball_ - 0.5000 and ball < 0.5000 and srick_preyy 2 0.0900
undbrick_euz 0.0900 and
v -0.0063 and prev 0.0500 and vx_cur 2 0.0500
-085400
if ifball_yprrev 0.8381 and rel rel_x_prev S -0.1000 and
0.0375 < vx_prev < 0.0500 and vy-prev < 0.0190
-28244
ifball_y_ccur 0.4143 and vy_prev 0.0048
+2.4663
lfball_y_pree < 0.8381 and vx_prev 00.550 and vy_prev S -0.0190
and - 0.0381 < vu_cur S 0.0381
-1.5553
ifball_y_prev - 0.6381 and vx_ & -0.0375
-1.8001
if ifball_y_ 2 0.6381 and vx_cur 2 -0.0375 and vy_cur 2 -0.0286
+6.1152
if 0.5000 < ball__cur 0.7714 and rel_x_cur -0.3375
and vy_prev < 0.0199 and vy_cur 2 00.081
33.4766
if ifbally_prev > 0.4143 and 0.5000 <
0.7714
0.000000 vx_prev 0.0500 and vx_cur S 0.0500 and vy_prev S. 0.0381 and vy_cur > -0.0381
11.944
rf 0.6668 and rel_x_cur < -0.6688
0.7953
if 0.4143 ball_y_cur 5 0.8381 and brickbrcuurrrrr S 0.1100 and rel_x__________ _eli___peee___ S 0.1000
and rel rel______u_______ 100000 and vx_prev 2 -0.0355 and vy_prev & 0.0381 and vur 2 - 0.0381
1.2368
if 0.4143 < baal_a_cuuu 0.7714 and paddle__dgg__reee_____e 2 0.1375
and rel_____p____ 2 -0.6688 and vx vx_ jre_prev_5 < 0.0500 and _ur & 0.0500
+2.5430
if ba __a_e_y_rev 2 0.5000 and 0.5000 < ball _aallyyccur_ - 0.8381 and rel rel_x_prev 5 0.2000
and vx_prev 0.0000 and vx_cur -0.5500 and 0.0381 vvy_prev < 0.0048
+0.7993 if ifball_above_prev 2 1.0000 and ball ball_aaaal ball_above_cur 1.0000 and rel_x_prev 2 -0.3775 and 0.0000 < vx_prev < 0.0500 and vx_cur 2 -0.0775 and vy_prev & 0.0286
+1.7013
if bbaa_aabove_pree > 1.0000 and hall_y_curt 2 0.6381 and rel_x_cur -0.2000 and
vy_prev < 0.0048
-1.1456 if rre__x_prev' 2 -0.3063 and - 0.3063 < rel_x_cur I -0.0063
and vy_prev 0.0048 and vy_cur -0.0381
44

---

Published as a conference paper at ICLR 2026
Action 2: Right
-3.7358
if ifball_y_cu= 0.8381
+3.3079
if rel_x_cu 2 -0.3775 and vy_prev 2 0.0190 and vy_cur 2 0.0048
-9.162
ifball_above_cux 2 1.0000 and vxev & 0.0500 and vy_cur 2 0.0381
+4.6580
if ifball_above_prer 2 1.0000 and balr_y_cur 2 0.7048
and 0.0190 and vy_cur -0.0190
+2.7250
if dfpaddle_edge_pzer 2 0.1375 and vx_pzey 2 0.0000 and vx_cur 2 -0.0875
and vy__rev 0.0090 and vy_cur 2 00.0881
-1.7765
ifball_y_cur 0.7714 and vy-prev 0.0190
+4.5538
if ifbal__ypprev 2 0.5714 and rel_x_press 2 00.6688 and vrep S 0.0500
and vy_prev 0.0286 and vy_cur -0.0881
+3.0658
if rel_x_cur -0.3063 and 0.0375 < vx_prev S 0.0500
and vy_prev 2 -0.0190 and vy_cur 2 0.0048
+2.8135
if bfbal_above_prev 2 1.0000 and rel_x_e______ v 0.0063
and andvx_prrv -0.0500 and vy_prev 0.0381 and vy_cur 2 0.0190
+2.8650
ifball_above_preee 2 1.0000 and ba ball_y_prey 2 0.7714
and ree____eur_0 2 -0. -0.6685 and vy_prev 0.0190 and vy_cur 2 -0.0381
-4.3001
ifball_y_prev 0.8381 and rel_x_cur < -0.4225
6.5868
if ifball_above_cur- 2 1.0000 and vy_prev S 0.0381 and vy_cur S -0.0190
2.9138
ffbally__pres 2 0.7048 and rel rel_ prev v -0.0063 and rel __cur S -0.0063
and cu_cur < 0.0375 and vy_prev 2 0.0048
2.0466
if bfaall_bbve_cour > 1.0000 and laalllllpprrr < 0.8381 and rel_x_cur S -0.2000
and cur 0.0000 and vy_prev 2 0.0048
+3.3836 if ifball_y_cur 2 0.7714 and rel_x_prev < 0.1000
and vx_cu -0.0375 and vy_cur 0.0048
-18377 ifball_above_cuz 2 1.0000
and - 0.0375 - vx_pprev & 0.0500 and - 0.0375 S vx_cur> S 0.0000 and vy > 0.0048
+3.7873
ifball_y_cur= 0.8381 and 0.0190 S vy_cur S 0.0381
-2.4499
ifball_above_cut 2 1.0000 and rel_x_ < -0.5375 and cur 0.0375
-2.7407
if ifbal_y_prev < 0.7714 and ball_y_eur 2 0.4143 and rel_x_prey 2 -0.6688
and mudvx_prev < 0.0500 and vy_prev 2 0.0190
-25117
ifballly_cur 0.5714 and brbbbbricerrrr 2 0.1000 and rel_x_prev 2 -0.1000
-1.7701
ifball_above_cut 2 1.0000 and brick_rucu 0.0900
and vi_vyeree < 0.0286 and vy_cur 2 0.0190
+1.5424 boallly_prev2 > 0.5000 and ball_y_cur= 2 0.6381 and bri 2 0.0900
and rel_x_preev > -0.2000 and vx_cur < 0.0375 and vy_prev 0.0190 and vy_cur > -0.0381
+1.4835 if bfball_y_cur 2 0.4143 and brick_prev 2 0.0900 and rel_x_prev 2 -0.6688
and rad_re___cur < -0.1000 and vx_cur 2 0.0000 and vy_prev 2 0.0048 and vy_cur 2 -0.0286
+1.8123 ifbally_prev 2 0.7048 and brick_cur < 0.1300 and vx_cur < 0.0000
and vdd_yreev 2 0.0381 and vy_cur 2 -0.0381
+1.4038 if rel_x_prev's 2 -0.3063 and vu_cur 2 0.0000
and vy_prev 0.0048 and vy_cur? -0.0381
45

---

Published as a conference paper at ICLR 2026
Action 3: Left
+3.2914 ifball_y_cur / 0.8381 and paddle_edse__pev_ 2 0.1375
-5.9441 ifball_y__co - 0.7714 and rel_ _______pee______________________ _ 00.0063 and _el_x_cu -0.1000
and - 0.0500 v vx_prev S 0.0500 and vy_prev & 0.0048 and - 0.0286 S vy_cur 0.0381
-3.0919
yur 0.7714 and rel cur 2 0.6668 and vy_cur 2 0.0048
+3.1300
if ba fbbal_above__cur22 1.0000 and
0.1375 and vx_prev 0.0500
and v 0.0381 and vy_cur < 0.0381
0.038)
2.78833
ifball_above_pa_
2 1.0000 and
> 0.7048 and rel__________ x_prev 2 -0.3663
2.4206
if ba bbaallabboe__prev 2 1.0000 and rel_x_prev I -0.3063 and rel_x_cur S 00.0663
and cur < 0.0500 and vy_prev -0.0381 and vy_cur S 0.0048
1.4360
if ree___preet 2 -0.6688 and vx_prev 2 00.0375
and 0.0375 v vx_cur 0.0500 and vy_prev 0.0286
-5.7469
if 0.4143 < _all_y_cur 0.8381 and
S -0.0063
and - 0.0500 S vx prev -0.0375 and prev - -0.0190 and vy_cur < 0.0286
33.718
vx 0.0000 and vx_cur 2 0.0000 and cur 2 0.0048
3..132
ifball_y_cur= 0.5714 and cur 000375 and vy_prev S 0.03881
1.9788
vfvx_prev 0.0000 and 0.0500 < cur I 0.0000 and vy-prev 2 0.0228
+3.7481
ifball_above_ctu 2 1.0000 and ball_y _rree 2 0.4143 and ball_y__hh 0.5000
and vx__rev < 0.0500 and vy_prev 0.0190 and -0.088 - cur S 0.0286
+1.8542
ifballyy_cur - 0.7714 and paddle_edge _eeeppeeeee_ee prev 2 0.1375 and rel_ __r_x_curr -0.5375
and vx wwwppppeae__...._ - 0.0500 and vu_cur - 0.0500 and vy_ & 0.0381
+0.6150 if bbaall_abov__cur 2 1.0000 and re lee____pee_ 00.4125 and re rel_x_curs S 00.4125
and vy_prvy -0.0381 and vy_cur 2 0.0286
+2.3428
if rerl x x_peee 2 -0.5375 and rel_x_cut cur 2 0053375
and 0.0375 - vx_cur < 0.0375 and vy_cur v -0.0190
-1.2518 if fabll abbve_preveevveeeeeeeeeeeeeeeeee 2 1.0000 and rel_x_ellee__eeeee < -0.1000
and - 0.5375 - rel_x_ - -0.1000 and 0.0500 S vx_cur S 0.0500
and pree -..028 and vy_cur 0.0286
-2.2239
if ba bball_y_pres 0.7714 and ball_y_cur < 0.7048
and vx__revr 2 -0.0375 and vydvyyrrev 0.0190 and vy_cur S 0.0190
-0.8174 bbbal_abbove_pree 2 1.0000 and ball_y_cur 2 0.6381
and vndvx_prev55 0.0375 and vy_prev 2 000381
-..6264 if ifball_above_prev 2 1.0000 and rre_x_preve 2 -0.3375 and rel_x_cur -0.5375
and vod_vxprevve 0.0000 and vx_cur -0.5500
+2.5593 if ifballyyprrev < 0.7714 and rel_x_cur < -0.3063
and vy__rev 2 0.0381 and vy_cur 2 -00199
+0.9760 ifball_above_ctt 2 1.0000 and bail Y and 0.3063 < rel x prev < 0..003 and rel x __cur < -0.1000
2 0.4143
and vy < 0.0381 and - 0.0881 < vy_cur < 0.0381
+1.6089 ifball < 0.6381 and rel_x_prev > -0.2000 and re rel_x_out -.2000
-0.7929 xuu -0.5375 and vx_prev 2 -0.0500 and vy_cur 0000190
-0.5734 if y prev 2 0.5000 and brick cur < 0.0900 and vx prev 2 -0.0500
and - 0.0500 < vx_cur < 0.0375 and vy_cur 2 -0.0381
-1.0272 bf y y 0.8381 and lallbaalll___ > 0.7714 and vy_prev S 0.0381
46

---

Published as a conference paper at ICLR 2026
T STATEMENT ON LLM USAGE
Large language models (LLMs), such as ChatGPT, were used solely for editorial assistance this work. Their role was limited improving grammar, rephrasing sentences, and enhancing clarity and readability of the authors' original text. No LLM was used to generate original scientific content, analysis, or results. The authors take full responsibility for the integrity and validity of the work presented.
47


Reviews:
[{"review_id": "JdItmbRzlz", "reviewer": "Reviewer_VfDp", "summary": "The paper introduces a novel interpretable RL method based on actor-critics. The main idea is: Keep neural networks for the critic for accurate value estimation, and replace the neural actor with an interpretable one (but has been implemented e.g. by NUDGE, c.f. Weakesses). \nFor interpretability the authors chose rule ensembles, which learn to select and weight multiple symbolic boolean rules (e.g. x > y) for each action at the current state. The final advantage of each action is then computed by summing over all rule-weights. The action with the highest advantage is executed.\nNext to the definition of the actual methodology, the authors provide a local convergence proof and an empirical evaluation of the performance and interpretability.\nOverall, this work develops and evaluates a simple, but intriguing reinforcement learning algorithm that shows both good performance and interpretability.", "questions": "* How well does the method scale when we have a larger action space (e.g. 16 actions)?\n* After equation (5), the regularization term does not exist anymore. What happened?\n* Can you explain for equation (6) and (7) how the different cases (a=a_t or not) influence the gradient and how the updates encourage/discourage taking specific actions? (e.g. will case 1 increase the likelihood of choosing the action in future if A is large?)\n* Are you willing to add a clarification/visualization of the detailed rule update?\n* How did you decide on the symbolic baselines to compare to?\n* Can you share your code up-front (e.g using an. anonymous git repo)?\n* Why/How is the local convergence proof useful?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Combining a neural critic with rule-ensembles as actor allows for a good trade off between an accurate and flexible critic, and an interpretable actor. \nThe overall quality of the work is good and the methodology itself is well thought through. The visual representation of the learned rules help interpreting the policy. The work and paper have many avenues for improvements, detailed after, but constitute a sound research idea.", "weaknesses": "The paper is overall comprehensible, but could be improved greatly by restructuring the paper a bit and providing further visualizations. For example, the detailed rule update in section 4.2, which is difficult to follow, would really benefit from a visualization or intuition. \n\nAdditionally, the authors may want to think about restructuring the paper by moving the theoretical convergence analysis to the appendix, and instead add the warm start idea + an analysis of the effect of the total number of rules to the main paper.\n\nIn detail, I advise the authors to follow this general structure for their research:\nThe way many readers (including myself) dive into a paper is by reading the title, abstract, then figures and table (with their captions). \nHence, I advise the authors to have a list of scientific research questions at the beginning of the experimental evaluation section (often denoted RQ1, RQ2, ... etc), that are each answered in different paragraphs. For example:\n* (RQ1) Is our framework outperforming existing baselines?\n* (RQ2) Does our framework allow for interpretable decision-making?\n* (RQ3) How important is module A ? (conducting an ablation study)\n\nFrom what I extracted, RQs would here be e.g.:\n\n* How does the algorithm perform compared to non-interpretable and interpretable methods? \n* Is the method interpretable?\n\nThe paper is overall comprehensible, but could be improved greatly by restructuring the paper a bit and providing further visualizations. For example, the detailed rule update in section 4.2, which is difficult to follow, would really benefit from a visualization or intuition. Additionally, the authors may want to think about restructuring the paper by moving the theoretical convergence analysis to the appendix, and instead add the warm start idea + an analysis of the effect of the total number of rules to the main paper.\nIn detail, I advise the authors to follow this general structure for their research:\nThe way many readers (including myself) dive into a paper is by reading the title, abstract, then figures and table (with their captions). \nHence, I advise the authors to have a list of scientific research questions at the beginning of the experimental evaluation section (often denoted RQ1, RQ2, ... etc), that are each answered in different paragraphs. For example:\n* (RQ1) Is our framework outperforming existing baselines?\n* (RQ2) Does our framework allow for interpretable decision-making?\n* (RQ3) How important is module A ? (conducting an ablation study)\nFrom what I extracted, RQs would here be e.g.:\nHow does the algorithm perform compared to non-interpretable and interpretable methods? Is the method interpretable?\n\nWhile the second order results are appreciated, they demonstrate that many results are completely insignificant as the benchmark used is saturated. **A baseline whose std range overlap cannot be considered as outperforming another one.** Hence, only experiments on MCart-v0 and HVAC-1Zone show that your method outperform the evaluated interpretable baselines (and not by a big margin, thus not very confidently). \nThe significance could be further improved by tackling more complex environments, and above all, **relational environments** e.g. non-symbolic/pixel-based ones, or at least discussing how this may be solved in future (e.g. use object-centric extraction methods -> e.g. OCAtari [4] for Atari games). Demonstrating the interpretability of a method on such environment, that necessitate relational reasoning, is way easier than on continuous control ones (such as cartpole).\nI strongly advise adding experiments on e.g. the object-centric Pong to verify if their method also allows detecting or prevent the misalignment problem that learning agents encounter on this environment [2,3]. This object centric environment is also already available in JAX: https://github.com/k4ntz/JAXAtari.\n\n\nA limitation section should be included (also including scalability with growing number of attributes and overall run time) of this method.\n\n\nThe authors chose not to share code upfront, which makes checking the methodology more complicated and reduces reproducibility.\n\n\nExtreme lack of related work:\nThere are 2 papers from 2024, and the rest is from 2021 or older. Much research on interpretable RL has been conducted and publish. I strongly advise the authors to look up this research and compare to this work in a dedicated Related Work section. I provide pointers to the existing relevant literature, that was mentioned above as well. Logic actors encoded in First order logic have also already been introduced in [5] and further improved in [6]. Attempts to use interpretable decision trees as policies have also been explored in other works than SYMPOL [7, 8, 9], and other form of interpretable baselines (programs or using LLM to help decode them) have also been explored. While comparing to all of them is tedious, at least mentioning these lines of work seem necessary to help the readers (and apparently the authors) to situate this work :\n\n[1] Di Langosco et al. \"Goal misgeneralization in deep reinforcement learning.\" International Conference on Machine Learning. PMLR (2022).\n\n\n[2] Delfosse et al. \"Interpretable concept bottlenecks to align reinforcement learning agents.\" NeurIPS (2024).\n\n\n[3] Delfosse et al. \"Deep reinforcement learning agents are not even close to human intelligence.\" arxiv (2025).\n\n\n[4] Delfosse et al. \"Ocatari: Object-centric atari 2600 reinforcement learning environments.\" RLJ (2024).\n\n\n[5] Delfosse et al. \"Interpretable and explainable logical policies via neurally guided symbolic abstraction.\" NeurIPS (2023).\n\n\n[6] Shindo, et al. \"BlendRL: A Framework for Merging Symbolic and Neural Policy Learning.\" ICLR (2025).\n\n\n[7] Bastani et al. \"Verifiable reinforcement learning via policy extraction.\" NeurIPS (2018).\n\n\n[8] Fuhrer et al. \"Gradient boosting reinforcement learning.\" arXiv (2024).\n\n\n[9] Kohler et al. \"Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning.\" arXiv (2024).\n\n\n[10]  Luo et al. \"End-to-End Neuro-Symbolic Visual Reinforcement Learning with Language Explanations.\" ICML (2024).\n\n\nWhile the weakness section is extensive, I value the presented work and really believe that incorporating the provided feedback will make this work very valuable to ICLR readers. I would be eagger to revise my score if changes are operated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "evN6oFttvF", "reviewer": "Reviewer_oca3", "summary": "This study proposes an interpretable RL algorithm (NSAC) built upon A2C framework. As a variant of A2C, NSAC consists of an actor and a critic, but it is distinct from baseline A2C in that an actor is built with an “additive rule ensemble”. As individual rules of the ensemble are symbolic and interpretable, NSAC can natively explain chosen actions.", "questions": "As NSAC is tested against simple RL tasks, the actors’ decisions in this study can be easily modeled by additive rule ensembles. However, it is unclear to me if additive rule ensembles can also be applied to model more complex actors. Can the authors provide some insights into how generalizable NSAC can be?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "One notable point of this study is that the authors effectively make additive rules of the ensemble differentiable and propose updated rules. Thus, NSAC can be trained using a gradient descent or its equivalents. The manuscript also provides theoretical analysis of the training of the actor.", "weaknesses": "At first, this straightforward idea sounded intriguing, but I am not convinced that NSAC is truly interpretable because the actor’s decisions are dependent on the critic, which is still a black box. Without knowing how the critic operates, can we explain how NSAC make its decisions? For instance, if adversaries inject some adversarial attack vectors to the critic in NSAC, they can manipulate the critic’s assessment of current state, and thus the actor would make its decision based on incorrect information. In this case, can we trust the explanation provided by the actor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMXA7XW63T", "reviewer": "Reviewer_2xmA", "summary": "This paper presents a hybrid neuro-symbolic extension of Actor-Critic, resulting in the \"NSAC\" framework (for \"Neural plus Symbolic framework for ActorCritic\"). Concretely, the proposed method retains neural network for the critic, but replaces the actor with a symbolic method in order to enhance interpretability while maintaining high-performance. The stated goal, alongside performance, is to achieve the proposed interpretability critieria from Murdoch et al. (2019) of \"simulatability, modularity, and low complexity\". In particular, the actor is replaced with what are called \"additive rule ensembles\", that estimate a variable by incrementally adding rules in the vein of boosting. The actor is implemented as a softmax over these additive functions, which are tasked with estimating the action-values for each action. The critic is a standard estimate of the value function implemented using a neural network, and is implemented using standard techniques. By contrast, the actor is updated by individually updating the query functions of the rule ensemble based on the gradient of the advantage of the policy (with a regularization term added to ensure local convexity). The paper then carries out a convergence analysis of NSAC subject to relatively standard conditions (step size annealing, ergodicity, smoothness). Lastly, an experimental evaluation is conducted, studying two distinct aspects of NSAC: First, the performance of NSAC is contrasted to a variety of baselines, and Second, an interpretability study is conducted, showcasing the rules underlying the decisions of the actor in MountainCar and an HVAC domain.", "questions": "Q1: Why is the actor effectively a Q-function estimator? Does this not partially defeat the point of making use of a baseline in Actor Critic (see cited work by Garcin et al.)?\n\nQ2: How are simulatability, modularity, and complexity evaluated in your experiments (or analysis)?\n\nQ3: What, precisely, are the numbers (and variance-like terms) measuring in Table 1? \n\nQ4: Is there a reason to present performance results as a table rather than learning curves? I can appreciate you can fit more domains into one table than say one row of learning curves in a \\begin[figure}... block, but I believe learning curves will be far more useful to the reader.\n\nQ5: How might you make contact with the stated contribution about modularity, simulatibility, and complexity regarding the interpretability of your method?\n\n\nAnd, one minor typo:\n- Missing space around line 453: \"...viruses.In such\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "There is a lot to appreciate about this paper. It sets out with a very clear goal: extend Actor-Critic to retain its performance, while also ensuring interpretability. The premise to make use of the boosting-like additive rule ensembles is quite clever, and not something I have seen much of in deep reinforcement learning. I have a few questions on that aspect of the method that I return to below under the \"Questions\" box.\n\nOverall:\n1. Clear scope, and delivery on that scope. The claims mostly match the evidence in the paper (some caveats, see weaknesses).\n2. Relatively light-weight and simple algorithmic idea at its core: I can see this proposal having high potential for adoption given the popularity of actor critic methods at the moment, plus the demand for interpretable methods.\n3. Convergence analysis is interesting, and to my reading appears correct.\n4. The experimental design is thoughtful (some caveats, see weaknesses).", "weaknesses": "A few weaknesses stand out, though I suspect they can be fixed with some discussion and some iteration on the manuscript.\n\nW1: First, I find it puzzle that the actor is effectively implemented as a Q function (and thus, more of a critic) in its own right, with a softmax at the very end turning this into a policy, rather than a value estimator. This seems to deviate from the basic premise of Actor Critic, in that the Actor is allowed to specialize to behavior, whereas the Critic needs to learn about values---this specialization is an important piece of Actor Critic. For example, see recent work by Garcin et al (2025), who investigate the ways in which the representation of the Actor and Critic each specialize. I will end up posing this as a question below, but I wanted to call it out as a perceived mystery of the design of the method.\n\nW2: In some cases, claims do not exactly match the evidence. This is largely down to claims around interpretability. The start of the paper states \"We ... demonstrate that our rule-based policy satisfies the established interpretability criteria of simulatability, modularity, and low complexity\". While aspects of these criteria are touched on, for instance computational complexity is discussed around line 334, this thread is left largely untouched throughout the paper. In particular, the results discussion does not revisit these critieria specifically. A simple change to the paper will fix this: revisit simulatability, modularity, and complexity in the discussion, and tie together the evidence available that supports these claims (or, if there is none, soften the claim). If \"complexity\" in the intro refers to computational complexity, I would be explicit about that.\n\nW3: The presentation of the paper could be strengthened at points. I find the most confusing part to be the \"Detailed Rule Update:\" in section 4, but I'm actually not convinced it needs to be in the main paper. I believe the NSAC pseudo-code from Appendix B would be better suited to the main text.\n\nW4: The experimental results are slightly difficult to interpret. I find the table to be a difficult way to read the results (even with the color / bold). Alongside the table, I would encourage the use of standard learning curves with confidence intervals that can showcase the evolution of performance over time. Similarly, there is currently no precise description of what numbers are actually being shown---I only see in line 371 \"higher returns\". Are these the returns the last episode? Over the entire training run? The last k-episodes? Similarly, what are the $\\pm$ values? This is critical to include to interpret and reproduce the plot.\n\nW5: Lastly, I found the interpretability results to be mischaracterized by the section title, \"Human Interpretability Analysis\". This leaves the impression that you have conducted a user study to evaluate the level of interpretability actually afforded to people. Since this is central focus of the paper, I actually believe this is relatively important. While I can appreciate the visual representations provided by Figure 2 and 3, they are a much softer version of an interpretability analysis than one involving actual people. Additionally, I find Figure 2 extremely difficult to read and parse.\n\nOverall, I still believe this is a solid paper, but the above limitations should be addressed before publication.\n\nReferences:\n- Garcin, S., McInroe, T., Castro, P. S., Panangaden, P., Lucas, C. G., Abel, D., & Albrecht, S. V. _Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning_. ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRDNJYOsfN", "reviewer": "Reviewer_gu5z", "summary": "This paper presents a neuro-symbolic framework for interpretable reinforcement learning. The method relies on neural network function approximation to learn an expressive critic, while the actor follows a symbolic, rule-based design. The proposed symbolic actor can be trained end-to-end via Policy Gradient combined with Orthogonal Gradient Boosting, allowing it to discover symbolic rules automatically without relying on prior domain knowledge. The authors prove convergence under two-timescale assumptions, give algorithmic details (OGB + rule-replacement, pseudocode), and empirically evaluate NSAC across seven benchmarks, showing competitive performance in terms of return. The interpretability of the method is analyzed on two environments by examining the learned rules in the actor.", "questions": "- What is meant by the \"parameters\" of NSAC and SYMPOL (line 409)?\n- What exactly are the scores in Figures 2 and 3? Is the color coding the advantage function?\n- How is the advantage obtained from the scores that each rule provides, e.g. in the right column in Figure 2? For a given state, a number of rules hold, and the advantage for each action is obtained as the sum of scores given by all active rules?\n- Are the explanations that your method provides conceptually different from those in SYMPOL or other tree-based policies?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Since the interpretability of the actor stems from its intrinsic architecture, the proposed method avoids the challenges and limitations associated with fitting a post-hoc explainable actor to a RL-learned neural network actor.\n- Since the symbolic actor discovers rules automatically through end-to-end training, it doesn't require domain knowledge. The proposed approach is, to the best of my knowledge, novel and provides a principled approach for rule-based actor learning.\n- The paper provides a strong theoretical analysis and derivation of the rule-based actor learning procedure, proving convergence to the global optimum under mild assumptions.\n- The empirical evaluation is thorough.\n- The proposed approach performs on par even with non-symbolic SOTA methods.\n- The paper provides algorithmic pseudocode and promises the release of the full codebase, which strengthens reproducibility.", "weaknesses": "- The actor update procedure is hard to follow. The connection between the PG update and OGB is difficult to grasp. The connection only became apparent to me after studying the algorithmic pseudocode in the appendix to understand the procedure. I strongly suggest improving the clarity of this procedure by outlining the key algorithmic steps. For example, how and when are equations (6) and (7) used? When are Algorithms (2) and (3) invoked?\n- The symbolic baselines are not discussed sufficiently. The paper should describe the key distinctions between them and the proposed method.\n- The environments used are under-specified. What are the state and action spaces and reward functions? Are any of the action or state spaces continuous?\n- The scalability of the method seems limited. The update procedure must be executed for each action separately, since the advantage of each action, $f_a(s)$ is modeled with a separate ensemble. \n- The human interpretability analysis is hard to follow. Figures 2 and 3 are not sufficiently explained. See questions section...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)