Piecewise Linear Parametrization of Policies: Towards Interpretable Deep Reinforcement Learning
Authors: Maxime Wabartha
Joelle Pineau
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning
interpretability
control
navigation
transparency
discrete
Primary area: reinforcement learning
TL;DR: We propose a neural policy constrained to express a small number of linear behaviors, and show that it leads to improved interpretability while performing comparably to baselines in several control and navigation tasks.

Abstract: Learning inherently interpretable policies is a central challenge in the path to developing autonomous agents that humans can trust. Linear policies can justify their decisions while interacting in a dynamic environment, but their reduced expressivity prevents them from solving hard tasks. Instead, we argue for the use of piecewise-linear policies. We carefully study to what extent they can retain the interpretable properties of linear policies while reaching competitive performance with neural baselines. In particular, we propose the HyperCombinator (HC), a piecewise-linear neural architecture expressing a policy with a controllably small number of sub-policies. Each sub-policy is linear with respect to interpretable features, shedding light on the decision process of the agent without requiring an additional explanation model. We evaluate HC policies in control and navigation experiments, visualize the improved interpretability of the agent and highlight its trade-off with performance. Moreover, we validate that the restricted model class that the HyperCombinator belongs to is compatible with the algorithmic constraints of various reinforcement learning algorithms.

Published as a conference paper at ICLR 2024
PIECEWISE LINEAR PARAMETRIZATION OF POLICIES: TOWARDS INTERPRETABLE DEEP REINTTTCEMENT LEARNING
Maxime Wabartha" McGill University, Mila
Joelle Pincau
McGill University, Mila, FAIR at Meta
ABSTRACT
Learning inherently interpretable policies central challenge in the path de- veloping autonomous agents that humans can trust. Linear policies can justify their decisions while interacting in dynamic environment, but their reduced ex- pressivity prevents them from solving hard tasks. Instead, we argue for the use of piecewise-lines linear policies. We carefully study what extent they can retain the interpretable properties of linear policies while reaching competitive performance with neural baselines. In particular, we propose the HyperCombinator (HC), a piecewise-linee neural architecture expressing a policy with controllably small number sub- cobb-poliiis. Each sub-policy is linear with respect to interpretable features, shedding light the decision process of the agent without requiring an additional explanation model. We evaluate HC policies in control and navigation experiments, visualize the improved interpretability of the agent and highlight its trade-off with performance. Moreover, we validate that the restricted model class that the HyperCombinator belongs to is compatible with the algorithmic constraints of various reinforcement learning algorithms.
INTRODUCTION
Artificial intelligence (AI) enables practitioners to automate processes by replacing human decisions and predictions with computational ones. The value potential is high, since the rules are learnt from data: predictions can be made more accurate (Silver et al.) 2017: Nam et al. 2019) or quicker (Le- Cun et al. 1989 Lindsey et al,, (018)) than their human counterparts. To achieve these feats, models have rapidly grown complexity. This trend has come the cost of interpretability: recent models are opaque decision makers. As a consequence, they are untrustworthy in many situations, either because their predictions ize (Wabartha et al. 2021), the actual sub-policies differ from the expected one (Amodei et al., 2016 Ibarz et al.] 2018), or because one cannot easily extract the logic behind their decisions.
unstable (Goodfellow et al. 2014; Lu et al. 2020), fail general-
Often overlooked, in our quest for ever improving performance, the transparency and explainabil- ity of AI models might soon become indispensable component of their democratization, amid growing oversight by regulatory bodies (European Union 2016: United States of America [022] United Kingdom| 2023a b). But interpretability is also fascinating scientific topic in itself, giving us the opportunity to strengthen our understanding of our models and their effects. Therefore, it important to systematically explore the methods that let us interpret machine learning models. Deep reinforcement learning (DRL) is particularly interesting domain in which to study inter- pretability. In fact, a precise description of how policies interact with an environment can inform other algorithms designed to improve the fairness, accountability, or transparency of the deployed model (Doshi Velez & Kim 2017). Transparent policies could also help better diagnose the agent's 's interactions. which can be helpful if it appears to malfunction. Such functionally interpretable policies are especially of interest as they do not need a separate algorithm providing potentially approximate explanations for policy's decisions (post-hoc interpretability) (Rudin) 2019)).
In this work, we examine the viability of piecewise-liness policies as a beneficial compromise be- tween interpretability and performance in DRL. Piecewise- linear policies such multi layer per- ceptrons (MLPs) with ReLU activations directly extend linear policies, which hold favorable inter-
Work partially done visiting researcher for FAIR at Meta. maxi waba tha@mai 1 m.mogi1

---

Published as a conference paper at ICLR 2024
pretability properties (Molnar et al. 2021). However, the extremely large number of linear sub- policies usually learnt by MLPs (Hanin & Rolnick 2019) hinders their interpretability, even many of these linear sub-policies might in fact be superiluous for end performance Akrour et al. (2018). We focus on extending the analysis of piecewise-linear policies in the context of interpretabil- ity. To that end, propose the HyperCombinator (HC), neural architecture that parametrizes a piecewise-linear policy with a controllably small number of linear sub-policies. The HC agent selects a linear sub-policy at every interaction with the environment, an important property miss- ing from previous works. The advantages of our piecewise- are many: we can first guarantee that the agent, which can be thought of as a mixture of linear experts (Jacobs et al. 1991). interacts with the environment exclusively through one of the learnt sub-policies. Moreover, each sub-policy can inspected individually. Finally, we can probe the policy at any given interaction to transparently analyze the computation leading to the decision.
implementation of the policy
Our contributions are the following: (1) we characterize properties that an interpretable piecewise- linear policy should ideally satisfy, (2) we present the HyperCombinator, actor architecture that implements a piecewise-linear policy with a specifiable number of linear sub-policies and compat- ible with most RL algorithms, (3) we characterize the interpretability properties inherited by HC, (4) we extensively evaluate the model on control and navigation tasks and observe a sustained per- formance of the model despite its greatly reduced expressivity. In addition, (5) we leverage the two levels of interpretability of HC to propose two visualizations. The first one lists the different ways the policy can react an input by cataloging exhaustively each linear sub-policy and its coefficients into a table. The second condenses the sequence of decisions into the sequence of sub- sub-policies used, surfacing the temporal abstractions emerging from the task.
BACKGROUND
2.1 DEEP REINFORCEMENT LEARNING AND INTERPRETABLE POLICIES
In reinforcement learning, an agent learns to interact with its environment to maximize the accu- mulation of a learning signal, the reward (Sutton & Barto 2018). The environment is modeled as a Markov Decision Process (MDP) defined as a tuple (K, R, P. Pu). At each timestep i, the agent in state Iz E X' takes the action a; to dx+1 P(-|xe, a.) and receives reward r++1 = R(a,, de, TT+1). We denote with Po the starting state distribution and represent a trajectory r as (Fo. ag; .1,..., .TT), where is the last timestep of the trajectory if the environment is episodic, and T = else. The discounted sum of rewards is G(r) = Ei' 7'TE41, and we denote its expectation J E, [G(T)]- Fun- damental in reinforcement learning is the function Q", for which for a given state and action, (T a) Exx DTT R(r,, at, 1) = x, ag = a Actor-critic algorithms solve an MDP by modelling explicitly both and Q". They aim at finding policy x² such that n²" arg
((x)) E A. The agent then transitions
Both Q" and π can be parametrized by deep neural networks (NNs) to handle high-dimensional in- put spaces and increase the performance of the agent (Mnih et al. 2015). However, this modelisation limits our understanding of the action choice of an agent, due to the opacity of NNs.
In this work, tackle the task of making the policy interpretable each timestep. Since we focus on functional interpretability, we constrain the class of functions that the policy belongs to. That is, design policy architecture that gives exact insights about the decision applied to state d4- The last layer of deep policies is often non-linearity μ. such as the softmax function, to cast the prediction of the NN to the right action space. In similar fashion to the interpretation of logistic regression, we focus on interpreting the pre non-linearity part of the policy, which we denote by म. Finally, we remark that an interpretable model is of limited utility if the imput features themselves are not interpretable. Therefore, we consider robotics environments with varying levels of difficulty and where the input to the policy is the proprioceptive state representation, each feature representing physical variable describing the agent (Tassa et al. 2018).
2.2 FROM LINEAR TO PIECEWTSE-LIIEAN POLICIES
To achieve functional interpretability, one can reduce the size of the class that the policy belongs to (Rudin) 2019). Classes of functions that can be expressed through small number of parame-
2

---

Published as a conference paper at ICLR 2024
ters, such as linear functions, are often considered to be interpretable due to the reduced amount of information required to understand the function they realize (Molnar et al. (021). We write linear policies as +((x)) = (0, 2z), where 9 designates the linear coefficients, the input is assumed inter- pretable, and (A, b) is the dot product between A and b both are vectors, and the matrix-vector product if A is matrix instead. Linear policies have several desirable properties related to inter- pretability, such as being summarizable by their coefficients, or having each coefficient explicitly quantify the influence of given feature and expressing the same valid explanation for all inputs. However, despite showing strong performance in certain benchmarks (Rajeswaran et al. 2017). their flexibility is generally not sufficient to handle more complex tasks (Mnih et al. 2013).
We therefore consider the larger class of piecewise-linees policies. They are direct extension of A with:
linear policies, as they can be written as
(wwp x) (Owi,
if E we if x E w1
a(x)
(0wK-1;2)) x)
if E wK-1
where K is the number of linear sub-functions, and where the partition of the input space Q (wo,...,e -K... the set of linear coefficients 0 {ow. w E 52} and the function mapping an input to the corresponding subset w. a X + Ω describe स. Therefore, स is locally linear, and some interpretability properties of linear policies carry over. However, both the value of K and the complexity of Ω and a influence the interpretability of æ.
MLPs using only linear layers (including linear, convolutional or layer normalization layers) and ReLU activations (Fukushima) 1975) Nair & Hinton 2010) already offer a piecewise-lines parametrization of policies (Pascanu al. 2014)) learnable with gradient descent. The rightmost plot of Fig. [ illustrates the induced partition, and Hanin & Rolnick 2019) analyse more pre- cisely the nature of the partition. Growing the width and the depth of the network quickly increases the approximation power of MLPs by increasing the number of linear sub- functions that compose them (Pascanu et al., 2014) and the complexity of the partition, in turn penalizing the interpretabil- ity of т. Conversely, it is only possible to loosely control an upper bound on the number of linear sub-policies through the architecture of the MI.P (Hanin Rolnick 2019). Moreover, policy with a large number of linear sub-policies implies that each sub-function only applies to a few inputs, limiting the generalization of the explanations provided by the linear coefficients. For instance, we show in Appendix that even baseline RL agent provided with small neural policy uses unique linear sub-function around 950 times 1000 step-long trajectory, meaning that a different set of coefficients is applied at almost every step. Restricting the number of linear sub-functions while maintaining high level of performance and interpretability is therefore a major challenge, that we explore next.
METHODS
As noted in Sec. 2.2 piccewise-lincariy is not sufficient requirement to guarantec that we obtain an interpretable policy. Therefore, we begin by formulating properties that deployed, interpretable policy should ideally satisfy.
the input to output sub-policy computation is sasy-to-understen (P2) Assignation transparency: the input to sub-policy choice computation is easy to understand.
(P1) Sub-policy transparency:
(P3) Separation: the policy uses single sub-policy at each interaction (training and evaluation).
(P4) Counterfactual reasoning:
compare the sub-policy taken to the other possible ones. (P5) Generalization: the sub- policies generalize to similar inputs.
(P6) Causality: want to justify the causal influence of each feature w.E.t. the task success.
Given the subjective nature of interpretability, a single definition might not be sufficient (Doshi- Velez & Kim 2017). and several desiderata relate to different notions of interpretability. To satisfy as many of these properties as possible, we design the Hypercombinator, a piccewise- linear neural architecture with small, controllable number of linear sub-functions.
3

---

Published as a conference paper at ICLR 2024
Tree
MLP
High-lovol interpretabiiity
Low-level intrpretabiltt:
Temporal abstraction
transparent
2 E d' (inteppeeabbl inpul)
Gumbel network
confiicinnss
Gumbel Gumbel-Softmaa Softmax
(((())) RAM
a E 12
Batch
X,
X;
MLP
Linear
HyperCombinator
Xi
Figure/ 1: (Left) Proposed actor architecture. Layers are colored in yellow, vectors in blue. (Right) Intuition of the partition of 2D space into linear regions by different models. Each polygon corre- sponds distinct linear region, and each color set of unique linear coefficients.
3.1 PROPOSED POLICY ARCHITECTURE
Our starting policy class is expressed by an MI.,, realizing a piecewise- function with respect to an input x E Rh assumed interpretable. Following Sec. S.c. the prediction can be decomposed as (=) = (fa())) 2)), where a IR" 52 assigns an input x to subset of the partition induced by ft. For general MLP, both and 0 are implicitly defined by स and have complex dependency on its parameters. This intricate relationship illustrates two major hurdles in making neural networks in- terpretable, namely the complexity of the induced partition and the plurality of linear fubnfunciions. Both relate to the complexity of the policy class.
Parametrization We focus on reducing the number of sub-po sub-policies while retaining the complexity of Ω. To do so, we make the functions and 0 explicit in the output computation through new parametrization of the NN. We first view f linear function coefficients. Secondly, we model a explicitly within म by an MLP followed by Gumbel-Softmax layer (Jang et al. 2016 Maddison al., 2016), and refer the composition of both Gumbel network à. The Gumbel-Sortmax is a technique typically used to continuously learn a categorical variable. In this context, predicts which sub-policy should an input x be mapped. Furthermore, we use the straight-through estimator (Bengio et al. 2013) to force à to produce strict assignations, i.e. one-hot encodings in the forward pass while being trainable with gradient descent. We note that because the output of the Gumbel network is always one-hot encoding, we can use any activation function within the Gumbel network without breaking the piecewise- property of the policy. The resulting architecture is
Ω E rather than set of
7(x) = (0((())) 2), n ( (()())),
(3)
which we refer to as the HyperCombinator,, illustrated in Fig, aalong with the partition it induces.
This formulation is advantageous for several reasons. First, the MLP preceding the Gumbel-Softmax induces the partition of the input space which π inherits, thus retaining some of the predictive flex- ibility of NNs. Second, we explicitly control the number of unique sub- sub-policies of through the dimension de of the Gumbel-Softmax layer. Moreover, the Gumbel network à is piecewise-constant thanks to the straight-througg estimator. Therefore, л interacts with the environment only through one of the de sub sub-policies, the contrary previous work AArour et al. 2018).
Policies modeled with the HyperCombinator differ from MLPs in that they are usually not contin- uous at the border between linear regions, which we do not find to be problem for stability in practice (Appendix D.6). Moreover, reducing the width or the height of the NN is an alternative and more direct way to reduce the number of unique coefficients. Yet, the number of linear sub-functions grows very quickly with the size of the architecture, both theoretically and empirically. We refer to Appendix B for more in-depth study of this approach.
4

---

Published as a conference paper at ICLR 2024
3.2 INTERPRETABILITY CHARACTERIZATION OF THE HYPERCOMBINATOR ARCHITECTURE
The HyperCombinator policy class has the following structural properties:
स is a linear function of the interpretable input (PI). A stakeholder can therefore access trans- parently the computation that led to the decision, given the sub-policy. The coefficients ((((z))
are available as an output of our approach.
The policy is piecewise- -eneea during evaluation and training thanks to the use of the straight- through estimator. This ensures that HC stays interpretable for all environment interactions (P3).
The number of unique sub-policies of the policy is set by the hyperparameter de (P4), which lets a stakeholder control the performane-imerpretaabllllyy interpretability trade-off.
Subsequently, our architecture needs to re-use the few sub-policies of very low complexity in order to solve complex tasks. The sub- col-pooicies are thus in practice constrained to generalize across a wide range of states, which implies P5. This is not true for MLPs, due to their huge number of linear regions and unique coefficients. MLPs additionally do not satisfy P4.
Our approach is locally interpretable, it explains the sub-policy applied to any given example. Conversely, while the explanations generalize to the inputs that are mapped to the same sub-policy, the fact that the partition is induced by a neural network prevents us from having a globally inter- pretable model; our approach does not satisfy P2. As an important consequence, the HyperCombi- nator does not provide feature importance or any notion of causal explanations and does not satisfy P6. Indeed, just like MLPs, a feature 2; of x can be involved in the computation of all the coef- ficients ((ā(x)) and simultaneously have the value of its linear coefficient ((ā(z)); be O. Instead, an HC policy provides a transparent justification of how it interacted with the environment for a given input, which is governed by the transparent sub-policy. In other words, our approach can be understood as interpretable conditioned on the choice of a sub-policy.
Two levels of interpretability In summary, an HC policy can perhaps best be understood as two consecutive modules, each corresponding to a different level of interpretability. The first module selects linear sub-policy given an input, while the second retrieves the linear coefficients of the selected sub-policy and applies them to the input. The second module provides the exact linear coefficients that used by the policy interact with the environment, quantifying the influence of each interpretable feature on the chosen action. We characterize this as the low-level interpretability of the HC policy. Simultaneously, one can observe the sequence of choices of sub-policies made by an agent in order to analyze its behavior, thanks to the first module of न. The forced re-use of sub-pole collppoicies leads to the emergence of cycles and temporal abstractions which let us analyze how the agent solves the task, that we call high-level interpretability. We illustrate the complementary existence of both level of interpretabilities Fig. (left), using green and purple to respectively locate the high and low-level interpretability of æ. We will see in Sec. X that these two levels of interpretability enable insightful visualizations of the agent's policy, such as the identification of the behavior modelled by each linear subpolicy (Appendix. D.7 and the observation of the emergence of temporal abstractions (Sec and 4.2
3.3 APPLICATION TO DEEP ACTOR- ARCHITECTURES
We can use the HyperCombinator the actor of any actor-critic algorithms, provided that com- peting assumption is made on the architecture of the policy. This means that the critic or the potential other parts of the RL algorithm can be modeled by complex neural networks without affecting the interpretability of the policy. We note that in practice, all the linear sub-policies share the same bias vector. This limits the fiexibility of the sub-policies, but also increases their interpretability since they all reduce to the same default sub-policy at x = O. We adapt μ to the action space of inter- est. In addition, we note that the Humbel-Softtaa layer can destabilize training. For instance, its predictions may collapse to single mode, making â equivalent to a linear policy. We alleviate this issue through different forms of regularization. First, we increase the temperature parameter and add weight decay to the last layer weights of the Gumbel network MLP. Both limit the magnitude of the input fed to the Gumbel-Softman layer. Secondly, we maximize the entropy Hl of the aver- age sub-police assignation over a batch of size B, that is H ΣE= a(z;)). This encourages the Gumbel network to the diversity of sub-policies at its disposal.
5

---

Published as a conference paper at ICLR 2024
Reacher easy
Ball in cup catch
Walker walk
Finger spin
Cartpole swingup
Cheetah run
Episode reward
1000
750
500
HC8
HC64
250
SAC
0
0.5M
1M0
0.5M
1M0
0.5M
1M0
0.5M
1M0
0.5M
1M0
0.5M
Training steps
Figure 2: HC actors match or approach the performance of SAC for most environments.
4
EXPERIMENTS
In this section, we evaluate if the structural constraints of the HyperCombinator offer a reasonable compromise performance, and visualize the improved interpretability properties of our approach.
4.1 CONTROL
We evaluate how well HC policies can control proprioceptive variables such as the joints of a robot through the DeepMind Control Suite benchmark (Tassa al., 2018). We aim to (1) ensure that the HC architecture can be trained to solve control tasks, (2) investigate the improved interpretability of the agent and (3) compare the performance of HC actors to their baseline algorithm using their usual neural policy. We replace the actor in the SAC algorithm by HC actor with final non- linearity μ which predicts, for state æ, the mean of the action distribution π T(x) = (x)())). In other terms, is linear w.r.f the interpretable proprioceptive features (Sec. 2.1 The Gumbel network follows a [1024, 1024, 1024] architecture and uses a sub-policy assignation entropy coefficient of 0.001. The remaining hyperparameters are the defaults from Yarats & Kostrikov (2020), that detail in Appendix D We consider 2 variants of the HC actor, HC8 and HC64, with a maximum of respectively& 8 and 64 sub-policies. The latter model is more expressive, at the cost of interpretability, since practitioner now needs to factor in 64 different possible sub-policies instead of 8.
Performance We perform an in-depth experimental analysis of the performance of piecewise- linear policies in DM Control, extending previous work (Akrour et al. 2018). Figure 2 presents
the evolution of the evaluation return over training for a subset of 6 commonly used environ-
ments (Yarats al. 2021), while we defer the results for all environments to Appendix D.2 HC
policies approach or match the performance of SAC in most environments, with Inter-Quartile Mean (Agarwal al. 2021) scores (95% CI) of .81 (.78-883,, .85 (.83-87) and .87 (.86-888)
spectively for HC8, HC64 and SAC. We highlight in particular the limited drop in performance of HC8, which performs at most 3% worse than SAC in 57% of the tested environments (HC64: 78%). These results show that drastically reducing the expressivity of the actor is only penalized by mod- erate drop in performance: HCS achieves a comparable return to SAC in most environments despite having restricted the interaction capability of the actor to only 8 different linear functions. We can improve further these results by trading some of the interpretability the results of HC64. We further analyze the performance-ineepretabbiitt gap in Appendix D In addition, the HC architecture requires the sub-policies to be reused for a large number of inputs, which can help sample efficiency. We observe this effect in the Cheetah after training for 5M time-steps. We find that HC8 and HC64 improve over SAC during early parts of the training in 3 of these tasks. In the Humanoid Stand environment in particular, HC8 and HC64 converge several million steps before SAC. We further illustrate these results in Appendix [ Conversely, the performance curves of HC policies suffer higher variance than SAC's, which presume is due to the presence of the Gumbel-Softmax layer in the network.
the agent, as illustrated by
and Humanoid tasks
Interpretability The low-level interpretability of HC actors, thanks to the linearity of the sub- policies, gives insights about the influence of the features in each decision. In the Cartpole swingup environment, we can for instance precisely analyze how some sub- sub-policies swing the cartpole up while others stabilize the cartpole once in an approximately straight position. We defer the precise analysis of this result to Appendix [D7]] The discrete bottleneck introduced by the Gumbel network also lets us visualize the high-level interpretability of HC through the sequence of chosen sub-

---

Published as a conference paper at ICLR 2024
Actual sub-policy taken
Coeff. for action 0 logits
Sub bab-pooicy
10
18
11
12
7.5
11B
011
11
13
5.0
eefieeet value
75
100
125
150
175
200
16
18
0.0
Zoom, 25 t 50
2.5
Sub cob-ppoliyy
112
-5.0
715
-7.5
30
35
Sub-policy #
Trajectory step
Figure 3: (Left) Sub-policy sequence of the first 200 timesteps of HC8 on Cheetah run. A temporal abstraction emerges as HCS reuses the same sub-policy for several timesteps before switching to a new one in a cyclical fashion. (Right) All sub-policies coefficients, for one dimension of a's output space. This view exposes the similarities of sub-policies (2 and 3 have same-sign coefficients for features 1 to 5) or their disparities (2 and 3 have opposite sign (hence effect) w.r.t feature 0.
policies. For instance, HC8 learns control the joints of robot maximize its speed in the Cheetah run environment. The learnt movement is cyclical by nature, which is adapted to the strong sub-policy re-use of HC. We find again this periodicity in the sequence of sub-policies taken by HC8, in the left part of Fig. 3 The successive diagonals in the sub- sequence illustrate the HC8 has learnt chain its restricted sub-policies to learn well-performing policy. We then combine this visualization with an exact description of which feature the chosen sub-policy used to interact with the environment, and with which magnitude (right part of Fig. 3 corresponding to the low-level interpretability of HC). In contrast, we emphasize that these visualizations do not bear any meaning for a regular actor. Indeed, SAC reuses the same linear coefficients in average 50 times over 1000 timestep trajectory Cheetah run, in spite of the cyclicity of the task. The information learnt by SAC is hidden by the interdependence of its sub-policies, preventing us from observing the emergent temporal abstraction phenomenon from Fig. (see further details in Appendix C)
4.2 NAVIGATION
We now focus on maze navigation tasks (Nasiriany et al., 2019). We want to discover if (1) HC actors can solve hard exploration mazes by being combined with dedicated algorithms in spite of the restricted architecture and if (2) using HC actors in mazes can elicit temporal abstractions in the sub-policy sequence, similar to the control setting. The agent, quadruped which state is described by proprioceptive features, is provided with a goal state to reach. The sparse reward signal makes the mazes hard to explore and require dedicated algorithms. In this context, we extend RIS (Chane- Sane et al. [02I], goal-conditioned DRL algorithm. Given a state and goal, RIS encourages during training the policy to match the action predicted by a "prior" policy (an empirical moving average of the actor) to reach intermediate goal. We replace the neural policy of RIS with HC actors with 8, 16 and 64 sub-policies, and otherwise follow the same training and evaluation protocol, which detail in Appendix E All HC actors use (64, 64, 64) architecture and use several regularization techniques. Compared to RIS, HC actors are trained with a reduced batch size (to accommodate computing resources) and use stronger prior regularization hyperparameter (denoted o)).
Performance We observe in Fig. 4 that RIS as well as HC64 perform well in all the mazes: the constrained expressivity of HC actors does not prevent them from being trained on mazes using a dedicated navigation algorithm. While both RIS and HC64 get stable performance for the U and S-shaped mazes, the wider confidence intervals for the other two mazes illustrate the difficulty of the tasks. Furthermore, HC8 and HC16 solve most of the time the U-shaped maze, and HC16 the S-shaped maze. All reach the goal at least one of the seeds, showing that such architectures are not too constrained to express goal-reaching policy. However, all HC actors require more interactions than RIS to solve the respective mazes, and the failure rate is overall higher. Compared to DM Control, the higher difficulty of the task (and of the assignation function needed to solve it) is one possible explanation to the reduced sample efficiency. Solving the maze requires high-level navigation in addition to low-level control, whereas only the latter was previously required.
7

---

Published as a conference paper at ICLR 2024
U-shaped
S-shaped
n-shaped
w-shaped
Episode reward
1.0
0.5
HC16
HC64
0.5M 1M
1.5M
2M
0.5M 1M
1.5M
2M
0.5M 1M
1.5M
2M
0.5M 1M 1.5M
Training steps
Training steps
Training steps
Training steps
Figure 4: HC actors learn solve the navigation tasks with reduced sample efficiency.
111
01:0
Tejeectory timestep
Figure 5: (Top left) The quadruped must go down the left corridor, and up the right corridor. (Bot- tom) HC8 sub-policy sequence after 2M steps. We note the appearance of motifs (repetitions of subsequences of sub-policies) that match the behavior observed the agent: rotation, then straight movement, before rotation to change corridor and a straight movement. (Top right) Trajectory of sub-policy choices superposed with the maze: each colored point refers to the sub-policy chosen that location. The grouping of colors illustrate the empirical emergence of temporal abstractions.
Interpretability We finally provide an example of a sequence of sub-policies followed by HC8 during one successful trajectory, in Fig. This visualization emphasizes how temporal abstraction emerges from the hierarchical nature of HC. HC learns to alternate between sub-policies O and to move straight, and between 2 and 3 to rotate, illustrating how the high-level interpretability of HC lets us identify behaviors followed by the agent to solve the maze. For instance, the usage of sub- policies 2 and for the first steps indicates that the agent first needed turn. comparison with a video of the agent highlights that was initialized facing the wrong direction, justifying this first rotation. We analyze wider set of sub-policy sequences other mazes in Appendix
5 RELATED WORKS
Interpretability in (deep) reinforcement learning Closest to our work, (Akrour et al. 2018 ex- plore very similar piecewise-linen policy architecture in the control setting, though not in the scope of interpretability. In particular, the choice of sub sub-policy is determined by softmax function, such that the partition the state space induced by the sub-policy assignation function is soft. On the con- trary, the HyperCombinator induces hard partition which guarantees that only single sub-policy
8

---

Published as a conference paper at ICLR 2024
is used at any interaction with the environment. Previous work also showed that linear (Rajeswaran et al. 2017) and discrete (Tang & Agrawal (020) policies performed surprisingly well in control environments, at the cost of reduced flexibility and therefore likely reduced ability to solve com- plex tasks. LMUT (Liu et al. 2019) generates functions that live in the same space as HC. However, LMUT is applied to the Q-function rather than the policy, is introduced in a distillation setting, and the partition it induces, while interpretable, is not learnt by gradient descent. Programmatically interpretable reinforcement learning algorithms (PiRL) train highly structured policies such as deci- sion trees or finite state machines (Koul et al. 2018). Verma et al. (2018) and Bastani et al. (2018)
distill an expert policy into an inherently interpretable one. Trivedi et al. (2021) Qiu & Zhu (2021);
Liu et al. (2023) use policies defined over discrete actions (either from the environment or learnt)
to solve high-level planning. On the contrary, HC aims to learn functionally interpretable policies that handle both to low-level continuous control and high-level planning. Contemporary to HC, Hr. fanos & Lelis (003) learn an interpretable tree policy derived from a small actor. Other methods use non gradient-descen optimization methods to learn inherently interpretable policies (Hein et al. 2017 2018). In high dimensional domains, the network can be intertwined with prototypes to jus- tify inherently its decisions (Kenny al., 2023). An attention module was added to the IMPALA algorithm, highlighting the parts of the image used for decision making (Mott et al. 2019). In our approach, we foeus on proprioceptive states, and rather propose a policy with transparent interac- tions with the environment. Further from our work, there has been efforts to make the Q-function interpretable (Annasamy & Sycara, 2019) Juozapaitis et al.] 2019). Tangentially, a recent work studied the evolution of the linear regions of a ReLU MLP in DRL (Cohan et al. 2022).
Hierarchical methods Several navigation methods incorporate a hierarchical component in the
policy training, which can inform the policy architecture (Nasiriany et al. 2019 Nachum et al.) 2018
Faust et al. 2018) or guide it during training Cchae--aane et al. (2021). We focus on making the pol-
icy more interpretable by building off the later method, restricting its expressivity. Options (Sutton 1999. Bacon et al. 2017) are another type of temporal abstraction. In comparison, our method
et al. does not define initiation termination set, instead belonging to the usual RL formalism, with simple policy architecture change. Close to our work, SENN (Alvarez Melis & Jaakkola (0188) uses a similar architecture to HC, but focuses on supervised learning and adds penalty to smooth the linear coefficients, while we structurally enforce the latter to be piecewise-constant and analyze the effects of the gained piecewise- linearity on the policy interpretability in RL. Finally, hierarchy can
manifest directly in the network weights. Our architecture can be thought of as hypernetwork (Ha) et al. 2017) predicting the coefficients of a policy linear w.r.f. interpretable features.
DISCUSSION
In this article, we studied whether piecewise-linear policies can be a suitable compromise between performance and interpretability in DRI. To this end, we presented the HyperCombinator archi- tecture, an implementation of a piecewise- diecewsse-linea actor that combines a controllably small set of inherently interpretable sub-policies to interact with the environment in transparent manner. We showed that in several control and navigation environments, the performance loss due to the reduced expressivity of our policy class was limited. Moreover, illustrated how new visualizations, pos- sible thanks to the two levels of interpretability provided by HC policies, give us insights on the interactions of an agent with its environment. We also noticed the empirical emergence of temporal abstractions in the policy in both the control and select navigation settings.
This work could be improved in several ways. Currently, the assignation function learnt by the policy is not interpretable, which prevents explaining why a sub-policy was chosen. Moreover, the Hyper- Combinator interacts transparently with the environment, but is not causal architecture. And so our policy ultimately does not fully answer "why" an action was taken, instead giving an interpretable decision conditioned on the sub-policy choice. Nonetheless, this is a significant step compared to current DRI. algorithms, which are far from meeting the most basic standards of explainability.
The growing presence of AI models our daily lives has raised the stakes about the abilities they should exhibit. In the face of an increasing need for interpretability, both by citizens (Vasconcelos (et al., 2023) and by lawmakers, is imperative to develop models that can accurately justify their decisions. Through the HC architecture, we showed that appropriately parametrized piecewise- linear policies can be an important first step towards meeting this expectation for DRL agents.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the Fonds de Recherche du Québec Nature et Technologies (FRQNT), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR Al chairs program. The authors thank Pascal Vincent, Alessandro Sordoni, Harsh Satija, Clara Lacroce, Jonathan Lebensold, Samuel Lavoie and Emmanuel Bengio for fruitful discussions, as well as colleagues from FAIR Montréal and Mila and anonymous reviewers for insightful comments and suggestions on earlier drafts of this document. Finally, we thank the Python (Van Rossum & Drake Jr 1995). numpy (Van Der Walt et al. (011), matplotlib (Hunter 2007) and PyTorch (Paszke al. 2017) communities that developed the tools this work is built on.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 14:29304-29322 2021.
Riad Akrour, Filipe Veiga, Jan Peters, and Gerhard Neumann. Regularizing reinforcement learning with state abstraction. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 539. IEEE, 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances neural information processing systems, 31, 2018.
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv: (80.: 2018
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: 1606.v66000060 06505, 2016.
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Raghuram Mandyam Annasamy and Katia Sycara. Towards better interpretability in deep q- networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33. pp. 4561- 4569, 2019.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
Osbert Bastani, Yewen Pu, and Armando Colar-Lezamm. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint (eXXi::300..3 3432, 2013.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference Machine Learning, pp. 1430-1440) PMLR, 2021.
Setarch Cohan, Nam Hee Kim, David Rolnick, and Michiel van de Panne. Understanding the evolu- tion of linear regions in deep reinforcement learning. Advances in Neural Information Processing Systems, 35:10891 08911-00903 2022.
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint 1702.080000000 2017.
European Union General data protection regulation, 2016.
Aleksandra Faust, Kenneth Oslund, Oscar Ramirez, Anthony Francis, Lydia Tapia, Marek Fiser, and James Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling based planning. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 5113-5120. IEEE, 2018.
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 8052-2066 PMLR, 2019.
Kunihiko Fukushima. Cognitron: self-organizing multilayered neural network. Biological cyber-
netics, 20(3-)):21-136, 1975.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
David Ha, Andrew M Dai, and Quoe V Le. Hypernetworks. In International Conference on Learn- ing Representations, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019.
Daniel Hein, Alexander Hentschel, Thomas Runkler, and Steffen Udluft. Particle swarm optimiza- tion for generating interpretable fuzzy reinforcement learning policies. Engineering Applications of Artificial Intelligence, 65:87-98, 2017.
Daniel Hein, Steffen Udluft, and Thomas Runkler. Interpretable policies for reinforcement learn- ing by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 133302150.. 2015.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, (03):90-95, 2007.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-88 -7, 1991.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv:1611 144, 2016.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi Velez. Explainable rein- forcement learning via reward decomposition. In IJCAI/ECAI Workshop explainable artificial intelligence, 2019.
Eoin M Kenny, Mycal Tucker, and Julie Shah. Towards interpretable deep reinforcement learning with numan-friendle prototypes. In The Eleventh International Conference Learning Repre- sentations, 2023.
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not enough, learn to criticize!
criticism for interpretabilltyy Advances in neural information processing systems, 29, 2016.
Anurag Koul, Sam Greydanus, and Alan Femn Learning finite state representations of recurrent policy networks. arXiv preprint 1811 2018
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 144)5541-55,, 1989.
Robert Lindsey, Aaron Daluiski, Sumit Chopra, Alexander Lachapelle, Michael Mozer, Serge Sic- ular, Douglas Hanel, Michael Gardner, Anurag Gupta, Robert Hotchkiss, et al. Deep neural net- work improves fracture detection by clinicians. Proceedings of the National Academy of Sciences, 115(45): 1551-)111111111111 1596, 2018
11

---

Published as a conference paper at ICLR 2024
Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung- Lee, and Shao-Hua Sun. Hierarchi- cal programmatic reinforcement learning via learning to compose programs. arXiv preprint arXiv:2301.12950, 2023.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingean Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part II 18, pp. 414-429. Springer, 2019.
Ming Yu Lu, Zachary Shahn, Daby Sow, Finale Doshi-Velez, and H Lehman Li-wei. Is deep rein- forcement learning ready for practical applications in healthcare? a sensitivity analysis of duel- ddqn for hemodynamic management in sepsis patients. In AMIA Annual Symposium Proceedings, volume 2020, pp. 773. American Medical Informatics Association, 2020.
Chris Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.0077 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski. et al. Human-level control through deep reinforcement learning, nature, 5 518(7540):5995333 2015.
Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable machine learning-a brief history, state-of-the-an and challenges. In ECML PKDD 2020 Workshops: Workshops of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2020): SoGood 2020, EDFI.2020, MLCS 2020, NFMCP 2020, DINA 2020, EDMI. 2020, XKDD 2020 and INRA 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, pp. 417- 431. Springer, 2021.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 607-617, 2020.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems, 32, 2019.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hicrarchical reinforcement learning. Advances in neural information processing systems, 31, 2018
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Ju Gang Nam, Sunggyun Park, Eui Jin Hwang, Jong Hyuk Lee, Kwang-Nam Jin, Kun Young Lim, Thienkai Huy Vu, Jae Ho Sohn, Sangheum Hwang, Jin Mo Goo, al. Development and valida- tion of deep learning-bases automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology, 290(1):218-22) 2019.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
Spyros Orfanos and Levi HS Lelis. Synthesizing programmatic policies with actor-critic algorithms and relu networks. arXiv preprint arXiv:2308.00722 2023.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-253 -22955, 2021.
12

---

Published as a conference paper at ICLR 2024
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimie: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv: 2015.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with miece-wise linear activations. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In International Conference on Learning Representations, 2021.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M Kakade, Towards general- ization and simplicity in continuous control. Advances in Neural Information Processing Systems, 30, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-215, 2019.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstart- ing deep reinforcement learning. arXiv preprint arXiv: arXiv: 00030000333333 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359-3399 2017.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180-186, 2020.
Dylan Slack, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. Counterfactual explanations can be manipulated. Advances in neural information processing systems, 34:62-75. 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. $319-3328 PMLR, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211. 1999.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the aaai conference on artificial intelligence, volume 34, pp. 5981-5988. 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1001::18000006900 2018.
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34: 251 1514-221163 2021.
13

---

Published as a conference paper at ICLR 2024
United
Kingdom.
Establishing
pro-innovation approach to regulating ai,
2023a.
URL
stae Setabbishing--ee ortp::ah...a.....oooooooooooonnr otg.:p/ccc.cccciii....ooooooooer
United
Kingdom.
pro-innovation approach to ai regulation.
2023b.
URL
ai aegulationn---aaeeeeee. 1wo.ooomooiaaapppraaaa
United States of America. Blueprint for an bill of rights: Making automated systems work for the american people, 2022. URL https:/ /www///wwww/..////////////////// http://wwtwwtttttt www.whouse..weeoeoomm gov/ostp/
ai ai-bill-of-riqe
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in science & engineering, 13(2):22-30, 2011.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Guido Van Rossum and Fred Drake Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995.
Helena Vasconcelos, Matthew Jörke, Madeleine Grunde- Munde-MMLauughin, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. Explanations can reduce overreliance on ai systems during decision-making. Proceedings of the ACM Human-Computer Interaction, 7(CSCW1): 1-38, 2023.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Maxime Wabartha, Audrey Durand, Vincent Francois-Lavet, and Joelle Pineau. Handling black swan events in deep learning with diversely extrapolated neural networks. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences Artificial Intelli- gence, pp. 2140-2147, 2021.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation pytorch. https:// 2020.
github. wtt.:/ccccde.ci.. http://w.....eeellyaaaaaaaprpppe
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10674-10681, 2021.
14

---

Published as a conference paper at ICLR 2024
A POST-HOC, DISTILLATION AND FUNCTIONAL INTERPRETABILITY
We begin by setting the context of interpretability in machine learning. We wish to explain de- ployed model and its predictions to users of varying expertise. There exists three main approaches to do so:
A post-hoc method is an auxiliary model that extracts explanations from a complex (potentially black box), well-performing deployed model (Ribeiro et al. 2016 Kim et al. (016] Waccter et al.) 2017 Sundararajan et al. 2017, Mothilal et al. 2020). We can expect good performance from the deployed model. However, the explanations are provided by the auxiliary model, which raises questions about their reliability, their accuracy their robustness (Alvarcz-Melis & Jaakkola 2018 Rudin 2019 Slack et al. 2020 202T).
Distillation methods first train complex "teacher" model, and then fit and deploy a simpler "student" model on the predictions of the teacher (Hinton al. 2015 Liu al. 2019). The explanations come from the interpretable nature of the student (e.g. a shallow decision tree or linear model). Yet, defining the data distribution fit the student on can be challenging. especially in the DRL setting (Fujimoto et al. 2019: Ostrovski et al. (021).
Functional methods (Doshi- & Kim 2017) directly train interpretable model, which is deployed and explain its decisions (Rudin 2019 Kenny et al. 2023). The end-to-end approach to
interpretability often imposes limits expressivity that can imply performance trade-off.
In this work, we focus functional interpretability. To achieve it, can reduce the size of the class that the model belongs to (Rudin) (209).. Classes of functions that can be expressed through a small number of parameters, such as linear functions shallow decision trees, are often considered to be interpretable due to the reduced amount of information required to understand the function they realize (Molnar et al., 202T). Yet, designing such a model is challenging, since inherently interpretable models are at odds with the complexity and expressivity required to solve hard tasks. For instance, a linear policy might not be expressive enough to solve an intricate maze. Moreover, the inclusion of discrete components, while often involved in interpretable models (Doshi- Velez & Kim, 2017), can make training unstable.
B REDUCING THE SIZE OF THE NETWORK TO LIMIT ITS COMPLEXITY
B.1 THEORETICAL ANALYSIS
Reducing the width or the height of an MLP an alternative and more direct way than our proposed approach reduce the number of unique coefficients. In fact, reducing the size of the MLP also reduces the size of the partition, iee the number of linear regions, on the contrary to our proposed solution. The main issue with this approach is that controlling the number of unique coefficients becomes impractical, since the maximum number of linear regions (which is the quantity we have control over) grows at least exponentially w.r.f. the depth and polynomially w.r.f. the width of the network (Montufar et al. 2014 Corollary 5). For instance, relatively small MLP with a one- dimensional input and 2 layers of 64 neurons each can express a maximum of over 4000 linear regions (Montufar al., 2014 Theorem 4). Finally, accessing the linear coefficient for given feature 2; requires a backward pass when using an MLP, while can be directly outputted as part of our solution, since we compute this quantity explicitly in the forward pass.
B.2 EMPIRICAL ANALYSIS
We complement the theoretical analysis of the upper bound by an empirical analysis of the actual number of unique linear regions used by small variants of the Soft Actor Critic algorithm (SAC). To do so, we train variants of SAC with an MLP actor architecture of depth 1, 2 and 3 and width of 2, 4, 8. 16, 32, 64, 128 and 256 units. We then roll out each variant in the Cheetah run environment (prone to sub-policy re-use due to the periodicity of the task) 10 times. We repeat the procedure for 10 seeds, which gives us, for each variant, 100 measures of the episode return (performance metric), and 100 number of unique sub-policies used (USPU) during the trajectory (complexity metric). We then produce Pareto plot illustrating the trade-off between complexity and performance for small
15

---

Published as a conference paper at ICLR 2024
Pareto plot: performance against re-usage of sub-policies
HC8
HC64
HyperCombinator
SAC-[2]
SAC-[4]
800
SAC-(8]
SAC-[16]
700
SAC-[32]
SAC-(64]
SAC-[128]
600
SAC-[256]
Average retum
SAC-12,
SAC-14,
500
SAC-(8, 8)
SAC-(16, 16]
SAC-[32, 32]
400
SAC-164, 64]
SAC-1128, 128]
SAC-1256, 256]
300
SAC-12, 2, 2]
SAC-[4, 4, 4]
200
SAC-(8, 8, 8]
SAC-116, 16, 16]
SAC-[32, 32, 32]
100
SAC-(64, 64, 64]
SAC-[128, 128, 128]
1000
800
600
400
200
SAC-1256, 256, 256]
Average unique sub-policto used
Figure 6: Pareto plot comparing the HyperCombinator actor and SAC with different small architec- tures performance (y-axis, higher is better) and number of sub-policies used during trajectory (X-axis, lower is better). Note the inversion the x-axis, such that the best models are located the top right part of the plot (with high return and low number of unique sub-policies used). We display error bars representing the 95% bias corrected and accelerated bootstrap confidence intervals (9999 resamples).
SAC architectures, in Fig. 6 We also compare the small variants of SAC with variants of the HyperCombinator, respectively with 8 (HC8) and 64 (HC64) possible sub-policies.
We observe that overall, the performance of SAC is correlated with the number of unique sub- policies used during trajectory. As consequence, models with small USPU metric tend to perform significantly worse than the more complex variants. For instance, to reach an average return of 700, variants of SAC need least 200 unique sub-policies. This means that average, each sub-policy is used only 5 times during trajectory. In addition, if we limit the complexity to 100 USPU, the average return of SAC variants barely surpasses 600, well below the average return of 800 reached by the more complex SAC variants. Conversely, our proposed approach offers better trade- off between complexity and performance SAC. Indeed, HCS and HC64 located further than the SAC Pareto front, as they manage to obtain significantly higher return for comparable complexity, or equivalently a comparable return with a much smaller number of unique sub-policies used. While in this particular example, HC8 outperforms even the SAC variants with the biggest architectures (see also Fig. 12 we insist specifically on the excellent performance of HC8 and HC64 w.r.t. SAC variants that comparable number of USPU.
In conclusion, simply limiting the architecture of SAC is not an adapted solution to our problem, since cannot exactly control the number of unique sub-policies that will be used, and since the performance decreases severely for low number of unique sub-policies used (<100), even more so with very low number of unique sub-policies used (<10).
16

---

Published as a conference paper at ICLR 2024
B.3 L1 PENALTY
We consider adding LI penalty the parameters of the actor. The goal is to encourage sparse policies, that would hopefully use less linear sub-policies within a trajectory. We consider various 1.1 penalties, as well as a small (1 layer of units), medium (one layer of 32 units) and big (two layers 1024 units) architectures. We illustrate the results in Table [.3] We observe that for the combinations that were tried, the LI penalty was not sufficient to reach satisfying performance- interpretability trade-off.
architecture & penalty
episode reward (95% CT)
unique count (95% CT)
big. À le-4
0.8 (0.7, 0.9)
1.0 (1,
big, X = le-5
814.8 (804.0, 824.7)
917.4 (902.6, 928.9)
big. À le-6
760.7 (724.5, 789.1)
965.9 (948.5, 978.5)
big, À le-7
822.6 (782.2, 838.8)
962.1 (940.4, 978.8)
med, = le-1
0.1 (0.1, (0.1,
1.0 I)
med, À = 1
(1, I)
med, = 2.5
0.1 0..1,
11,
med, À 10
0.1 (0.1,
(1,
small, lc-4
390.8 (368.1,
(3.9..449)
small, À le-5
428.7 (413.2,
9.7 (8.8. 10.5)
small, = le-6
417.4 (393.7,
5.6 5.99
small, À 1e-7
384.7 (338.3, 425.5)
4.5 (4.2,
B.4 DISTILLATION
An alternative to learning a small policy with reinforcement learning is to first train a complex policy with reinforcement learning (a teacher) and then fit in a supervised learning fashiong small policy (a student) to imitate the actions predicted by the teacher. We now compare the HyperCombinator to such strategy in the Cheetah rum task.
We first train to convergence a SAC agent with two hidden layers 1024 units, following the base- line hyperparameters detailed in D.I We repeat the experiment over 3 seeds, and select the seed with the highest episode return averaged over 10 rollouts. The corresponding agent referred to the (fully trained) teacher. We save the replay buffer, which contains the sequence of observa- tions that the teacher encountered over training. We then label each observation with the action predicted by the fully trained teacher. This methodology yields dataset of 1M observations and the corresponding ideal action.
We now focus on training the student on this dataset. We minimize the mean squared error between the student predicted action and the teacher predicted action for each observation in the dataset. We detail the hyperparameters Table addition, we compare results for different batch sizes (256
and 1024) and II penalty coefficients (ALI {le - 6, le 5, le 4, le 3))-
Value
Hyperparameter name
Student actor architecture Optimizer
[32]
Adam
Learning rate
1e-3
Table I: List of hyperparameters in the distillation experiments.
Performance is evaluated several times during training by rolling out the student in the Cheetah run task and recording the rollout return. We repeat this operation 10 times and compute the average rollout return. We repeat the experiment over different seed and average the results, and compute 95% bootstrap confidence intervals. We illustrate this curve in Fig. 7 where also plot the return curve for HC8. We observe that all the different students failed at getting an acceptable performance in the task. We observe that combination of hyperparameter led well-performing student in
17

---

Published as a conference paper at ICLR 2024
Evaluation performance (with HC8 plotted)
Evaluation performance (without HC8 plotted)
800
120
episade roward
episade reward
80
400
200
20
1.0
of training samples
1e6
* of training samples
1e6
HCB
sac@batch sac@baach__izee11222255 Batpeel___eett_o_e_eeeeeee cooff:1eee5
sac@ batch 0tt:::/5ggr0.0001.k.. ptrppppootrrrto1eeeeeeeeaaatttee cwt.f:leefffeee
sac@batch h00p pttpa/aarr....0000pmmeemoma.m ccoff.0.001
boo@batchsizee255eeee oot.lalleeeattyyeeeeeooooo
sac@batch wwwe:/56@/... 001@11 Bttpeeelaatyycooelllllloooooe coef1ee-5
sac@batch 000 penattt p coefile-06
sac@ batch htt.hhiiie:225g10000001010 |0__eeatty_cceef.0000
Figure 7: Students trained to imitate the teacher fail catastrophically.
Student training loss
100
loss
0.0
0.2
0.4
0.6
0.8
1.0
training samples
1e6
Figure 8:
Training loss curves for each student combination.
this task. Students that were distilled from the teacher did not manage to learn to solve the task. In addition, high L1 penalty coefficient degraded performance.
We also plot in log scale the training curves in Fig. g to illustrate that the students do learn to minimize the error with the teacher predictions. The lower capacity of the students, which forces them to make prediction errors at potential key states in the trajectory, is possible reason for their catastrophic performance.
18

---

Published as a conference paper at ICLR 2024
Evaluation performance
sac@batch_ sac@batch_size:25ee 0iz::::6000000000000000 http:/000000e11111leeeeeeeae
100
90
80
episode reward
70
60
50
40
30
0.0
0.2
0.4
0.6
0.8
1.0
of training samples
1e6
Figure 9: Input augmentation does not improve the performance the agent.
We also consider training periodically the student inputs that it generates itself, which is referred to "input augmentation" in Verma et al. (2018). We evaluate the student every 80 student updates (corresponding to 20480 data points with batch size of 256). The 10 evaluation rollouts generate each 1000 observations, leading to 10000 inputs that were generated by the student interacting with the environment. We train the student these inputs the end of the 10 evaluation rollouts, and then resume training on the teacher data. We show in Fig. that input augmentation does not improve the performance of the student in case.
Finally, we presented one way to distill a teacher into student network. We note that there are other ways we could implement distillation algorithms, for instance by mixing on on-policy distillation and a reinforcement learning loss (Schmitt et al. 2018), or adding a feature regression objective to the reinforcement learning loss (Parisotto al. 2015). All of these methods add additional inductive bias compared to our algorithm. One could also modify the training data, for instance by using more exploratory expert policy to gather a wider range of data points.
19

---

Published as a conference paper at ICLR 2024
Student training loss
sac@batch 2011 Baapooa@l_epeaalllooellae..
100
loss
0.0
0.4
0.6
1.0 1e6
# of training samples
Figure 10: Training loss curve with input augmentation. We hypothesize the spikes due to training the low-capacity student on the data it generated, which differs significantly from the data generated by the teacher.
20

---

Published as a conference paper at ICLR 2024
Re-usage of subpolicies in trajectory
Cumulative unique sub- policcoliiee used
1000
SAC
HC8
HC64
800
600
400
200
0
200
400
600
800
1000
Trajectory step
Figure 11: During trajectory, SAC actor visits new linear region (and uses new locally linear sub-policy) at almost every timestep. On the contrary, our HC agent can only express one of the or 64 sub-policies at his disposal, which forces it to learn sub-policies that are valid for wide range of inputs.
C SUB-POLICIES REUSE
Since SAC uses an MLP as policy, the policy is also a piecewise-lineat function. We study in this section how often does the policy re-use the same linear region in trajectory.
We perform the experiment using the trained models of Sec. 1.1 on the Cheetah run environment, which is particularly favorable to the reuse of sub- sub-policies. We rollout the trained policies of SAC, HC8 and HC64 for an entire trajectory and record, for each timestep, the number of unique sub- policies used to interact with the environment until the current timestep. This yields non-decreasing sequence of integers per algorithm. The lower the curve, and the less sub-policies are being used by the policy. We repeat the same procedure 10 times, and compute the mean curve well as 95% confidence interval using the bias corrected and accelerated bootstrap (9999 resamples).
We display the results in Fig. The HyperCombinator cannot express more than de sub-policies, which caps the corresponding plots for both HC8 and HC64. We observe that the curve correspond- ing to SAC linearly increases almost without failure. This indicates that SAC barely reuses past sub-policies i.e. that new locally linear function is applied at practically each timestep. Moreover, this sub-policy shared by all seeds, given the thinness of the confidence interval.
This does not mean that SAC strictly overfitted the trajectory or completely memorized the sequence of policies to apply during the trajectory: indeed, there is a great amount of parameter sharing between the linear coefficients in 0. Thus, there is generalization happening between sub-policies, even different linear region used at each timestep. Yet, the precise generalization mechanism obscure, and is not clear to what extent updating sub-policy will affect the other sub-policies. Perhaps more importantly to our use case, one still needs to enumerate all of the sub-policic when describing the possible interactions of the policy with the environment, given that even though they might share parameters, each sub-policy uses different set of linear coefficients. As a consequence, this makes the SAC policy highly uninterpretable, the contrary to the HyperCombinatoor where the size of the exhaustive list of sub-policies is capped by de.
21

---

Published as a conference paper at ICLR 2024
D CONTROL EXPERIMENTS
D.I IMPLEMENTATION DETAILS
We base ourselves open-source PyTorch implementation of SAC (Yarats & Kostrikov 2020) with its default hyperparameters, listed in Table 3 We detail below the modifications needed to implement the HyperCombinator and replicate the experiments, and keep the rest of the existing code for training and evaluating SAC and HC.
SAC algorithm We modify the actor of the Soft Actor Critic algorithm (SAC) (Haarnoja et al., 2018), strong continuous control agent learning from proprioceptive observations. The original actor is modeled as an MLP that predicts vector of size Al. The output vector then split into two parts, the mean of the action predictive distribution, ग and its log standard deviation, log We note that this implies that the mean ग and the log std log a share some of their parameters through the shared structure of the actor. The log standard deviation is transformed to belong to the interval [log a min: log o max) using the following formula:
t t(u) = log min + (log max log o min)
(4)
2
Finally, the action predictive distribution for state x is defined as a squashed normal:
/4 (N exp (xx)))
(5)
where μ is the non- -inear transformation ensuring that the action belongs to the environment action space. In practice for the DeepMind Control Suite, we use tanh as μ to force the predicted actions to belong 1-, 1]-
The critic is learnt via double Q learning (ann Hasselt et al. 2016). Each critic network is a 2- layer MLP. The strength of SAC entropy regularization (represented by the hyperparameter a:) is automatically learnt during training. We use Adam to optimize all parameters. All weight matrices initialized using orthogonal initialization. All bias vectors initialized to O.
Hyper Combinatnt modifications With the HyperCombinator, we model the mean ß and the log std log a with independent networks. HC first models log a as an MLP with the same architecture as in SAC, but that only outputs a vector of size IA). Then, HC models स as a Gumbel network, that is, the composition of an MLP, Gumbel-Softman layer (with the straight-through estimator) and a linear layer. The rest of the computation of the action predictive distribution is left unchanged. In particular, for x, the mean of the squashed normal is in both cases (x)) and does not depend on log (z:).
We indicate in blue the modifications due to using a HyperCombinator actor. We remark that the only major modification is in the architecture of the actor. Therefore, the majority the structure the SAC algorithm is left unchanged. We notice the use of ttop-grad to detach tensor from the computation graph.
We detail the modified SAC algorithm using Alg. I
Training details Classically, Gumbel noise is used in the Gumbel-Softmn layer to stochastically select the sub-policy. After sufficient training. this (Gumbel) trick ensures that ultimately sam- ple from the likeliest sub-policy. We only activate Gumbel noise during the actor update, when computing the action predicted by the actor. When the agent interacts with the environment, not use Gumbel noise, both for training and for evaluation. Therefore, the Gumbel network selects the most likely sub-policy for all interactions with the environment. This ensures that the mean of the action predictive distribution is deterministic, like SAC. It also improves the consistency of the agent over an episode, since for state x, fixed policy leads to the same sub-policy. This not to be confused with the action predicted by the HyperCombinane which is sampled from the squashed normal defined in Eq. duriig training.
do
Model choice We experiment with different combination of hyperparameters values (Gumbel net- 1024,1024)) and strength of regularization of the average
work architecture from [64,64] to assignation entropy Aessig from 0 to 1) on Cheetah run, from where selected the best set of hy- perparameters according to the return curve. We then evaluate the HyperCombinator using the same fixed set of hyperparameters for all the environments.
22

---

Published as a conference paper at ICLR 2024
Evaluation details We evaluate the agent every 10000 timesteps by rolling it out for 10 episodes and taking the average return. During evaluation, all actors act deterministically using only the mean of the predicted action distribution ((((x)), without exploration noise (as opposed to sampling from the predictive action distribution during training). Hence, is sufficient for p(f) to be piecewise- linear get the desired form for the actor at evaluation time. We report all results and curves in Sec. [..] using 10 seeds for each agent. We draw the mean performance as colored line, as well as 95% bias-corrected and accelerated bootstrap confidence interval in lighter shade (9999 resamples).
Compute We ran all the experiments on an internal cluster. All the GPUs were NVIDIA Tesla V100, with 16GB memory available. The CPUs were Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Each seed was allocated GPU, 10 CPUs, and 64GB of RAM. We detail the compute budget to reproduce the experiments in Table. 2
Experiment
# models
# envs
# seeds
Avg. duration
Compute
Full results (Fig.
12
3
23
10
hours
173 GPU days
Longer horizon (Fi..
16
3
10
34 hours
170 GPU days
Perf.-interpret gap (Fig. 17
8
1
10
17 hours
57 GPU days
Small SAC (Fig.
24 (*)
10
hours
60 GPU days
Table Compute budget for the control experiments. (*) the HyperCombinator plots were already computed in the full results.
Hyperparameter name
Value
Common (SAC defaults) Yarats & Kostrikov 2020)
Action repeat
Discount factor Learnable &
0.99
True
Initial a
0.1
a learning rate Aa
le-4
Adam momentums
[0.9, 0.999]
Actor learning rate AS
le-4
Actor Adam momentums
[0.9, 0.999]
Actor update frequency
Critic architecture
[1024, 1024]
Critic learning rate XQ
le-4
Critic Adam momentums
[0.9, 0.999]
Critic exponential moving average ratio
0.005
Critic target update frequency
Batch size
1024
log σ min
-
log max
SAC actor-specific
Actor architecturd
[1024,1024]
HyyerrCombinato--peeffff
Gumbel net architecture Sub-policy assignation entropy coefficient Aussig Gumbel temperature
[1024, 1024, 1024]
0.001
Table 3: Full list of hyperparameters in the control experiments.
Sharedd between the and the log std net
23

---

Published as a conference paper at ICLR 2024
Algorithm SAC (with HyperCombinator actor)
Require: Replay Buffer
Require: Actor parameters
Require: Double critic parameters 7/. 7/2
Require: Double critic target parameters 7) 722
Require: Hyperparameters from Table 3
Require: N
Maximum number of timesteps Initial state
Require: 80
while < N do
a₁ ~ (s)()
St+11 P(-(st: at)
Sample the next state from the environment
T++1 = R(s;, Ct, S1+1)
D D (St: des To+1s (441,d+1)
Update replay buffer; de+1 indicates a terminal
transition
s', ~D
t> Sample from replay buffer
Launch routine UpdateCritic
See paper (Haarooj al. 2018 and code
if l. % actor update frequency == 0 then
Launch routine UpdateActoorAdAll pha (see Alg. end if
t critic target update frequency == 0 then
7/1 (1 critic ratio) 1)1 critic ratio 1/1
1/2 = (1 critic ema ratio) * 772 + critic ema ratio 7/2
end if end while
Algorithm 2 UpddateActorAndalll..
Require: s
Batch states sampled from the replay buffer p(N (f(s), (x)(((lg ñ is Gumbel network and log an MLP instead of being jointly parametrized as an MLP the base case.
Lssis = H (mean( (ā(s)))
b Compute entropy of average sub- sub-policy assignation
Q min (s, a), Qup a))
C. mean (stop grad (a) log o (als) Q) -Aassig & Do not backprop. gradients through
6 = 1---p-p-- X₂ V.G. Lr
if learn then
C. = mean(o a stop grad(log (als) + (A))
a= Xa V L...
end
24

---

Published as a conference paper at ICLR 2024
D.2
RESULTS FOR ALL DEEPMIND CONTROL ENVIRONMENTS
Acrobot swingup
Ball in cup catch
Cartpole balance
Cartpole balance sparse Cartpole swingup
1000
HC8
750
HC64
SAC
250
0.5M
1M
0.5M
1M 0
0.5M
1M O
0.5M
1M
0.5M
1M
Cartpole swingup sparse
Cheetah run
Finger spin
Finger turn easy
Finger turn
1000
750
500
250
0.5M
0.5M
1M 0
0.5M
1M 0
0.5M
1M
0.5M
Hopper hop
Hopper stand
Humanoid run
Humanood stand
Humanoid walk
Episode reward
1000
750
0.5M
1M
0.5M
1M
0
0.5M
1M 0
0.5M
1M
0.5M
1M
Pendulum swingup
Quadruped run
Quadruped walk
Reacher easy
Reacher hard
1000
750
500
250
0.5M
0.5M
1M 0
0.5M
1M 0
0.5M
1M
0.5M
Walker run
Walker stand
Walker walk
1000
750
250
0.5M
1M
0.5M
1M
0
0.5M
1M
Training steps
Figure 12: Complete results for the DM control environments.
D.3 DETAILED QUANTITATIVE RESULTS
To compute the results, we followed the methodology proposed in the r iable library (Agarwal al. 2021). We used the final scores after 990000 frames for each run.
We first investigate the scores produced by rliable in Fig. [3 This lets us quantify the trade-off in performance that follows the reduction in expressivity of HC8 and HC64 according to several metrics. In particular, the overlapping confidence intervals between SAC and HC64 indicate that overall, very small amount of performance is foregone when one is willing to guarantee that the agent will interact with the environment in at most 64 different ways.
Fig. 14 illustrates the close performance between SAC and HC64, especially in the environments where the normalized score is at least 80% the maximum (i.e,, in the games where SAC excels to begin with). HC8 follows similar curve, albeit sensibly lower.
Finally, Fig. 15 shows the fraction of environments where the normalized score IQM of HC8 (re- spectively HC64) is least percentage of the baseline SAC score. This visualization is helpful
25

---

Published as a conference paper at ICLR 2024
to distinguish the relative performance of the HC actors, compared SAC, the different envi- ronments. We remark that both for HC8 and HC64, there a sudden drop around the 0% relative performance mark, which indicates that HC8 and HC64 have close IQM scores SAC in several of the environments. Moreover, HC8 and HC64 perform worse than SAC on most environments by the end of training, expected.
Median
IQM
Mean
Optimality Gap
HC64
HC8
0.80
0.85
0.90
0.81
0.84
0.87
0.675 0.700 0.725 0.750 6.250 0.275 0.300 0.325
Normalized
Figure 13: Performance metrics computed using the r rliable library.
1.00
HCS
HC64
0.75
Fraction runs with
0.50
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
Normalized Score (t)
Figure 14: Performance profiles.
Fraction of environments where HC improves over SAC
1.0
HC8
0.9
HC64
0.8
Fraction environment
0.7
0.6
0.5
0.4
0.3
0.2
0.0
-25%
-00%
-15%
-10%
-5%
5%
10%
15%
20%
25%
IQM loss/gain compared SAC
Figure 15: Relative performance loss/gain of the HC actors compared to SAC.
26

---

Published as a conference paper at ICLR 2024
D.4 LONGER HORIZON
In this section, we train the different actors for 5M steps instead of 1M as in the rest of Sec. 4.1 We focus our study environments: the first Cheetah run, observe if the benefits brought to early training by the usage of the HyperCombinator are also transferred to later during the training. The other other environments, Humanoid stand, walk and run, are chosen for their increased difficulty. We illustrate the results in Fig. T6 We remark that HC actors improve over SAC for the first IM timesteps in the Cheetah run environment. After approximately 1M timesteps, the performance of the SAC actor exceeds the HyperCombinatorr s. For Humanoid run and stand, both Hypercombinatorr improve over SAC approximately between IM and 3M timesteps, with a significant improvement noted for Humanoid stand. For Humanoid walk, only HC64 improves over SAC, here again ap- proximately between 1M and 3M timesteps. The competitive performance of the HyperCombinator actors, added their improved performance in these hard environments early during training. hints at the benefits of sub-policies re-use. Still, we note that given sufficiently many samples, the more complex neural policies of SAC outperform HC. This recalls the trade-off between interpretability and performance observed the long run, induced by choosing a lower complexity policy when using HC.
Cheetah run
Humanoid run
800
Episode reward
600
400
HC8
200
HC64
SAC
Humanoid stand
Humanoid walk
800
Episode reward
600
400
200
0
1M 2M
3M
4M
5M
0
1M
2M
3M
4M
5M
Training steps
Training steps
Figure 16: During early training, HC actors can demonstrate better sample efficiency thanks to the re-use the same sub-policy for several inputs. Over longer timeframe, the higher expressivity of SAC leads to matching or better performance.
27

---

Published as a conference paper at ICLR 2024
D.5 PERFORMANCE-INEERRRETINN
GAP
In this section, evaluate how the HyperCombinator agent's performance evolves when increasing
the number of sub-policies. We run a HyperCombinator actor with 4, 8, 16, 32, 64, 128 and 256 sub-policies Walker walk for 2.5M timesteps, and observe the results in Fig.
We remark that HC4 reaches a relatively high return, in average higher than 900. This shows that even with small number of sub-policies, the HyperCombinator can learn well performing policy in certain environments. As we trade-off interpretability for performance by increasing the number of sub sub-policies, we see the performance curve increase as well and approach the performance curve of SAC. This trend culminates with 128 sub-policies. We notice that the return curve of HC256 decreases compared HC128.
A HC agent with higher number of available sub-policies can produce more finely grained policy, since each sub-policy can specialize to chain high number of sub-policies might not be straightforward. This last point is one possible explanation lower performance of HC256.
smaller part of the state space. Conversely, learning how
Walker walk
1000
Episode reward
600
400
HC4
HC8
HC16
HC32
200
HC64
HC128
HC256
SAC
0
0.5M
1M
1.5M
2M
2.5M
Training steps
Increasing the capacity tends to increase the performance, as the policy is now able to exhibit wider variety of sub-policies. A sufficiently high capacity can help reach comparable performance to the base algorithm. here SAC. We notice that too many sub-policies can however prove harmful: the performance of HC256 sub-policies decreases compared to HC128.
Figure 17:
D.6 ROBUSTNESS TO PERTURBATIONS
this section, we evaluate the robustness of the HyperCombinator to perturbations. We start with regularly trained agent on the Cheetah run task. At evaluation time, after 100 timesteps (out of a 1000 timesteps trajectory), we ignore the action predicted by the agent and instead interact with the environment through an action sampled uniformly at random from the action space. We do so for 100 consecutive timesteps. At timestep 200, we resume normal evaluation and follow the action predicted by the agent. This perturbation throws the agent off its trajectory.
28

---

Published as a conference paper at ICLR 2024
SAC
#***
Method
HC8
HC64
800
-600
400
-200
Performance loss
Figure 18: When perturbed with 100 consecutive random actions, HC actors perform overall no worse than SAC. The box plot summarizes the results of the experiment repeated 10 times for each seed.
[83]
11
III
B
1/1
1
111
100
200
300
400
500
Figure 19: After being perturbed for 100 timesteps, the HC actor manages to resume the task. sub- policy choices between t=100 and 200 (in turquoise) are ignored since a random action is used instead.
We repeat this evaluation 10 times per seed, for each of the 10 seeds of algorithm, for total of 100 data points.
We now compare how this perturbation affects HC8 and HC64, compared the baseline, SAC. In Fig. 18 we compute for each model the distribution of performance losses, defined as the perfor- mance of the perturbed evaluation subtracted the performance of the unperturbed evaluation. We then illustrate the distribution of these performance losses. We see that all models react similarly to the perturbations. This illustrates the fact that the HyperCombinator no less robust perturbations than the base neural policy, despite its non-continuity. A similar conclusion was already found when comparing linear models to neural baselines (Rajeswaran et al. 2017). Accordingly, diversifying the initialization distribution could improve the robustness of the model.
We illustrate in Fig. 19 how the agent, after being perturbed for 100 timesteps, manages to resume the cyclical choice of sub-policies that lets it solve the task.
D.7 ANALYSIS OF THE LINEAR COEFFICIENTS INTERPRETABILTT IN CARTPOLE
We now analyze the low-level interpretability benefits of our method when applied to the Cartpole environment.
In this environment, the agent controls the translation of a cart on an axis in order to swing up and then stabilize pole. We write the sub-policy i as: b w / w sin 0 w cos 0 w + w for an input (Ir, (r, sin(f), cos(O), .., 0) and sub-policy coefficients w'. Note that there a single action: for a given positive feature, the contribution tends to move the cart towards the right the corresponding coefficient is positive and left if negative.
We visualize the results in Fig. The left plot an alternative visualization of the sub-policy coefficients w; (to be compared with the heatmap in Fig. 3 right). This is an easy way to note similarities between sub-policies and understand how they are combined. For instance, HC mainly uses sub-policy BO (in orange) to stabilize the pole that has been swung up. This is translated by a negative coefficient in front of the cart velocity, such that the agent moves towards the right (causing the pole tilting the left), the sub-policy will push the agent towards the left balance the pole.
We also see in Fig. 20 (right) that the agent sometimes oscillates between using BO (orange) and sub- policy 1 (grey). We can observe in the left plot that they both give a large coefficient to cos which around ( when the pole is stabilized. The positive sign of the coefficient indicates that if the pole
29

---

Published as a conference paper at ICLR 2024
Alternative sub-policy representation
Sub-policy chosen
6
X
B2
Features
(o5(9)
Subpolicy
B3
(in
84
86
87
1.5
-1.0
-0,5
0.0
0.5
1.0
1.5
2.0
200
400
600
800
1000
Linear coefficient
Trajectory timestep
Sub-policy
BO
Figure 20: Evaluation of HC8 in cartpole swingup. (Left) alternative visualization of the sub-
policies. (Right) sub-policy choice sequence.
is tilting towards the right, the agent will tend to move the cart towards the right to balance the pole. Finally, Fig. (left) shows that B2 (red) and B6 (purple) are extremely similar. Simultaneously. we see in Fig. (right) that the only setting in which B6 is used is during an extended period of usage of B2.
We think that this figure is an example of how using HC actor (rather than a neural one) can improve the interpretability and our understanding the task.
D.8 COMPARISON WITH PROGRAMMATIC POLICIES
We compare in this section the HC architecture with the programmatic policy presented in Qiu & Zhu (202I), called -PPRL, Their algorithm uses TRPO (Schulman et al.) 2015) to learn a tree policy in two phases. In the first phase, the algorithm alternates between learning the probability distribution over the possible trees ("architecture iteration") and learning the parameters of each tree ("program iteration").. A training step is sequence of (1) gathering data from rollouts, (2) an architecture iteration, (3) gathering data from rollouts, (4) a fine tuning iteration. At the end of this first "fusion" phase, the most probable tree is extracted, and its parameters second "fine-tuning" phase. We refer to the original paper for more precise details about the training scheme of their method. The authors propose several controllers to apply in the leaves of the tree. We focus on affine controllers as they are the one that compare to our method, since each of HC's subpolicies are affine functions.
fine-tuned during
We focus on the Cheetah run experiment from DM Control. We modify the code provided by the authors to be able use affine policies in each leaf, similarly to HC. We parameterize the program- matic policy to consider the full state when choosing which branch to follow at an intersection (Eq. I of their paper). We repeat the experiments over 10 seeds.
We use the original paper's 's hyperparameters when available, and we detail the full list in Table 4
We illustrate the results of the algorithm resulting a soft and hard (discrete) tree respectively in Fig. 21 and Fig. 22 In both cases, the algorithm does not manage to learn strong policy in Cheetah run, despite having access to times as many samples as HC and SAC (15M vs M,, respectively). The algorithm converges in fact to lincar architecture, hence the same plots for the soft and hard trees.
We also plot the performance of a neural policy with two hidden layers and 256 hidden units each trained with TRPO on Cheetah run to showcase a likely upper bound on the performance of 7-PRL, in Fig. 23 Note that we use 10 times more training samples for TRPO than for HC or SAC in order
hhtps///hhthaalcmmmmiiimmmmymaah sac
30

---

Published as a conference paper at ICLR 2024
17- H1-RRL (affine) performance over 10M + 5M training steps
7-PRL affine, fusion
350
72-PRL affine, fine-tuned
300
episode reward
250
150
100
50
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
# of training samples
1e7
Figure 21: In Cheetah run, x-PRL (affine) did manage learn strong policy,
71-PL (affine, fine-tuned discrete) performance over 10M + 5M training steps
nt-RL affine, discrete, fusion
350
rt-RL affine, discrete, fine-tuned
300
episode reward
200
150
100
50
0.0
0.4
0.6
0.8
1.0
1.2
1.4
of training samples
1e7
Figure 22: Cheetah run, x-PRL (affine and hard decision tree) did manage learn strong policy.
31

---

Published as a conference paper at ICLR 2024
Hyperparameler name
Value
Action repeat
Discount factor
0.99
Max graph depth
GAE À
0.97
Num of consecutive architecture iterations
Num of consecutive program iterations
Total number of "fusion" training step
1000
Total number of "fine-tuning" steps
1000
Samples per data gathering rollout Baseline architecture
5000
[64,64]
Baseline num epochs
Baseline learning rate
1e
Baseline batch size
64
Table 4: List of hyperparameters in the programmatic policies comparison.
TRPO performance over 10M training steps
TRPO [256,256]
400
300
episode reward
200
100
0.0
0.2
0.4
0.6
0.8
1.0
# of training samples
1e7
Figure 23: In our setting, TRPO plateaus slightly below 400.
to obtain good performance with TRPO (10M vs IM). TRPO and m-PRL plateau below episode return of 400 (lower than SAC or HC).
Since the results of the algorithm are high variance (potentially because of the usage of policies of different capacity depending on the probability distribution over architectures that learnt), we analyze in Fig. 24 the performance of the best performing seed in the affine, discrete case. We notice that the performance rises quickly to around 400, but stays stuck this local optimum.
Finally, note that 7-PRL is restricted small architectures (Orfanos & Lelis (003)), while HC can use deep networks to partition the input space, and instead restricts the number unique affine sub-policies that can be used.
32

---

Published as a conference paper at ICLR 2024
77-PRL (affine, fine-tuned discrete) performance over 10M training steps (seed 0)
12-PRL affine, discrete, fusion
12:PRL affine, discrete, fine-tuned
400
episode reward
300
200
100
0.8
of training samples
1e7
Figure 24: Performance of the best seed, which gets maximum score of 450.
D.8.1 ADAPTING PI-PRL TO SAC
We adapt PRLL to trained with the SAC algorithm. We re-use the hyperparameters from Table and repeat the experiment 8 times. We used half of the training samples budget for the first fusion phase, and half for the second fine-tuning phase. We note that we test --PRL (SAC version) unfairly compared to HC, since -PPL is in this experiment a soft decision tree, as mentioned in Eq. of Zhu (202I). As a consequence, the predicted actions from r-PRL are mixture of all the actions in each leaf, which impedes interpretability. On the other hand, HC uses single linear sub-policy at each timestep.
We show the results in Fig. 25 T-PRL. seems to perform below HC. The parameterization that we chose for HC is able to quickly improve the quality of the policy, comparatively to the other algorithms. We recall that even besides this control experiment, as mentioned before, the same architecture of HC is able to handle high level planning tasks, while 7-PRL does not. In addition, the end-use has the possibility with HC to change its performance interpretability trade-off by increasing the number of sub- sub-policies to use for a marginal cost by simply increasing the size of the Gumbel layer, while the same operation comes at great computational cost for π T-PRI (Qiu and Zhu, 2022, Section 3, Complexity; Orfanos et al, Section 2., paragraph 2).
We note that, besides the first 200k timesteps, the SAC version of
33

---

Published as a conference paper at ICLR 2024
n-PRL, depth=9
800
7t-PL,, d
HC8
SAC
600
episode rewward
400
200
0.0
0.2
0.4
0.6
0.8
1.0
num samples
1e6
Figure 25: Comparison between the SAC version of x-PRL and HC.
34

---

Published as a conference paper at ICLR 2024
E NAVIGATION EXPERIMENTS
E. IMPLEMENTATION DETAILS
We base our experiments on the open-source code provided by RIS (Chane-Sane et al. 2021 and use most of its default hyperparameters. We detail in Table the common hyperparameters and the hyperparameters we change for the HyperCombinator.
RIS algorithm RIS belongs the family of goal-conditioned RL algorithms, that learn policy (s. RIS aims at guiding the policy, during training, to produce the same action to reach a goal as the action reach imaginary subgoal located halfway through the trajectory. In this case, the notion of distance implied in "halfway" means that as many steps are needed to go from the current state to the imaginary subgoal than from the imaginary subgoal to the goal (as opposed an Euclidean notion of distance that would not be suitable in mazes).
To do this guiding, prior policy Tprior is defined as an empirical moving average of the online, interacting policy. A neural network (denoted high-level policy in the original paper) learns online to generate likely subgoals 8, given a state and a goal g. Then, given state s, the policy T(s g) is constrained to stay close to the prior policy to reach subgoal 8g. i.e. (5,
Similarly to SAC, the RIS actor composed of an MLP on top of which two heads predicting the mean of the action predictive distribution, [z), and its log standard deviation log (zz). t' clamps log or (xx) between log min and log maxx The action is then sampled from a squashed normal, similarly to SAC. Two crities, each modeled by a MLP, are learnt with double Q learning. The hyperparameter a ((Chane-Sane et al., 2021 Eq. 9)) is fixed during training. The algorithm uses Hindsight Experience Replay (HER) (Andrychowicz
al. 2017) to facilitate training. and refer to the RIS paper for further details on the specific implementation (Chane-Sane et al. 2021). All weight matrices are initialized using orthogonal initialization. All bias vectors are initialized to
CopperComiinaoo modifications We replace से by a Gumbel network. Similarly to the control experiments, log a is still defined by an MLP, but does not share parameters with 7. As a conse- quence of the modelisation of न., the prior policy priorr also takes the form of HC policy. We do not modify the rest of the algorithm, including the high-level policy that learns to predict the inter- mediate goals used for training. Due memory constraints, HyperCombinator instead of the base 2048. We train RIS with batch sizes of both 1024 and 2048 and ensure that the performance of RIS with 2048 (the base value) improves over the performance with a batch size of 1024. We detail in Alg. 3 the RIS algorithm and add in blue the modifications that we apply.
batch size of 1024 to train the
Training details The agents interact with the environment for maximum of 600 timesteps, after which the episode is interrupted. We do not use Gumbel noise for any interaction with the environment, which guarantees that the most likely sub-policy is consistently selected. Therefore, sub-policies are stochastically chosen only during the update of the actor, in the routine. During training, the actions are sampled from the action pre- dictive distribution paN (T(s. g), exp(!' (log (s, to encourage exploration. We use three dif- ferent types of regularization to prevent the Gumbel network from collapsing into the prediction of a single sub-policy:
UpdateActorAndA.
We increase the temperature of the Gumbel-Softmax, controlled by the "Gumbel temperature" parameter in T
We regularize the entropy of the sub-policy assignations averaged over a batch, controlled by the Aasig hyperparameter.
We penalize the magnitude of the last layer weights of the MLP preceding the Gumbel-Softman through the Aweight decay hyperparameter. This forces the input to the Gumbel-Softmax layer have a smaller magnitude, and therefore encourages the selection of a more diverse set of sub- sub-policies.
35

---

Published as a conference paper at ICLR 2024
Model choice We experiment with different combinations of the architecture of the Gumbel net- work (from [32.32] to 11024,004,10244), the sub-policy assignation entropy coefficient Assig (be- tween 0 and 0.1), the Gumbel temperature (from .5 to 10), the weight decay coefficient (between O and 0.001) and the RIS hyperparameter a (between 0.05 and 0.4) on the U-shaped maze and the w-shaped maze (we found that the relative simplicity of U-shaped maze compared to the other vironments meant that design architectures useful to solve the U-shaped maze would not always generalize to the other tasks) and select the final set of hyperparameters based on the return curve. We then run the final experiment on all environments, keeping the same hyperparameters for all variants of the HyperCombinator (i.e. HC8, HC16 and HC64).
A notable difference with the control experiments is that we found that smaller Gumbel network architectures worked best. Therefore, we selected [64, 64, 64] architecture for the navigation ex- periments, as opposed to [1024, 1024, 1024] for the control experiments. We have overall observed in both control and navigation experiments that wider and deeper Gumbel network architecture usually requires stronger regularization to perform well on the task. It is possible that increasing even more the regularization could help the [1024, 1024, 1024]-architecture solve the maze tasks. The success of the smaller architecture and the failure of the bigger one, which such high regular- ization the Gumbel network, is another clue that the key to solving the maze might be to find a trainable and sufficiently regularized architecture that efficiently combines the different sub-policies.
Evaluation details During evaluation, all actors act deterministically using only the mean of the predicted action distribution (s. g)), without exploration noise (as opposed to sampling from the predictive action distribution during training), which guarantees the sereewsse-linearii of the policy. We remark that this departs from the base code, that evaluated stochastic agents.
Every 10000 steps, we roll out 5 evaluation episodes and report the mean success score, i.e. 1 if the agent reached the goal and else. We report all results and curves using 10 seeds for each agent. We draw the mean performance bootstrap confidence interval in a lighter shade (9999 resamples).
colored line, well 95% bias-corrected and accelerated
Compute We ran all the experiments
internal cluster. All the GPUs were NVIDIA Tesla
V100, with 16GB memory available. The CPUs were Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Each seed was allocated GPU, 10 CPUs, and 64GB of RAM. We detail the compute budget to reproduce the experiments in Table, 5
Experiment
models
# #envs
# seeds
Avg. duration
Compute
Full results (Fig.
23 hours
153 GPU days
Temperature ablation (Fig, 30
10
11 hours
37 GPU days
Table 5: Compute budget for the navigation experiments.
*Shared between the net and the log std net
"http:///ithub..c http:///ghthbcoooomclllcccnnnece 1/RIS
36

---

Published as a conference paper at ICLR 2024
Algorithm RIS (with HyperCombinator actor)
Require: Replay Buffer
Require: Actor parameters yp
Require:
Double critic parameters 7/1, 7/2
Require: Double critic target parameters 7/1 7/2
Require: High-level policy parameters
Require: Hyperparameters from Table
Require: N
D Maximum number of timesteps Initial state
Require: 80. go
while < N do
a, ~ (().
5c+ P(- P(. (t)
Sample the next state from the environment
T1+1 R(s,, at, $1+1)
D = D u (st, ae, re+1 St11, s441,
Update replay buffer; de+1 indicates terminal
transition
(s',,r,s',d,g) D
Sample from replay buffer using HER
Launch routine UpdateCritic
b See paper (Chane-Sane al. 202 and code
Launch routine UpdattetiggheeeePoo...
>> See paper (Chane-Sane et al. 202 and
code
Launch routine UpcåtteAtoorkndApple (see Alg.
7/1 = (1 critic ema ratio) * V/ + critic ema ratio 7J1
7/2 (1 critic ema ratio) 7/2 critic ema ratio
end while
Algorithm UpddateeAttornnnhhllha
Require: s.g
>> Batch of states and goals sampled from the replay buffer is a Gumbel network and log a an independent MLP instead of being jointly parametrized MLP in the base case.
g), (#,ss
Lusig H (menn ((s, g)))
Compute entropy of average sub- sub-policy assignation > Magnitude of the Gumbel network MLP last layer weights W
Lweight docay W 3
Q min Qn, (s. a), Q (s, a))
L. mean (a log ग (a|s, 9)
Q
Mussigassii + Aweight decay weeight decay
Minimize the
magnitude of the Gumbel network last layer weights
x = 8 Xj Vx
37

---

Published as a conference paper at ICLR 2024
Hyperparameter name
Value
Common (RIS defaults)
Discount factor
0.99
Success distance threshold
0.5
Burn in
le4
Replay buffer size
le6
Learning for the high-level policy Aeve
le-4
Learning rate for the critics XQ
le-3
Learning rate for the policy X; Critic architecture
le-3
[256, 256]
Critic empirical moving average ratio
0.005
Critic target update frequency High-level policy architecture sub-policy assignation entropy coefficient
[256, 256]
le-4
le-16
HER replay buffer goals ratio
0.5
0.1
log min
-20
log 0 max
RIS actor-specific
Actor architecture
[256, 256]
Batch size
2048
0.1
HypperComiinato-ppecffie
Gumbel net architecture
(64, 64, 64]
Batch size
1024
sub-policy assignation entropy coefficient A
0.01
Weight decay coefficient Aweight decay
0.0001
Gumbel temperature
0.3
Table 6: Full list of hyperparameters the navigation experiments.
38

---

Published as a conference paper at ICLR 2024
E.2 ADDITIONAL POLICY VISUALIZATIONS
In Fig. f illustrated a typical sequence of sub-policies followed by HC16 on the S-shaped maze. In this section, we propose wider variety of sub-policy sequences, showing examples where the emergent temporal abstraction appears clearly, and some where this temporal abstraction is harder to spot, or, interestingly, is lost over training. For each situation, we show sub-policy sequences taken during evaluation after 5M, 1M, 1.5M and 2M training steps.
HC8, U-shaped maze (Fig. 26 This figure shows a well-performing HC8 policy U-shaped
maze. We note the clear separation of three phases in the bottom plot, as the quadruped learns to go down the corridor in the first phase, using mostly sub-policy 0 and 1, then turns right and navigates the bottom corridor with sub-policies and 3, before re-using sub-policies 0 and to go up the right corridor and solving the maze.
Actual sub-policy taken, 0.5M steps (goal not reached 600 steps)
Subpollcy
100
200
300
400
500
600
Trajectory step
Actual sub- policy taken, 1M steps (goal reached in 408 steps)
Subpolicy
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, 1.5M steps (goal reached steps)
Subpollcy
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, 2M steps (goal reached in 255 steps)
Subpolicy
100
200
300
400
500
600
Figure 26: Example of well-performing HC8 agent in the U-shaped maze.
39

---

Published as a conference paper at ICLR 2024
HC64, w-shaped maze (Fig. 27 This figure illustrates the policies learnt by HC64 the chal- lenging w-shaped maze. We particularly note the appearance "diagonals", where the same se- quence of sub-policy is repeated several times. These diagonals recall the sub-policy sequence obtained for Cheetah run in the control experiments, as the agent learnt to chain its sub-policies to move itself. After 100 timesteps in the bottom figure (final evaluation after 2M training steps), HC64 switches from one set of repeated diagonals to another, and barely re-uses the sub-policies forming the first of repeated diagonals until the end of the trajectory. We conclude that the first set of repeated diagonals was specialized to the first phase of the trajectory.
Actual sub sub-policy taken, 0.5M steps (goal not reached in 600 steps)
Subpollcy *
20
40
60
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, 1M steps (goal reached In 600 steps)
Subpolicy
20
40
60
100
300
500
Trajectory step Actual sub- sub-pooicy taken, 1.5M steps (goal reached in 481 steps)
Subpolicy #
20
40
100
200
300
400
500
600
Trajectory step
Actual sub bub-policy taken, 2M steps (goal reached in 354 steps)
Subpolicy
20
40-
60
100
200
300
400
500
600
Figure Example run of HC64 solving the shapeed maze. We note the appearance of "diagonals" that are reminiscent of the control experiments.
40

---

Published as a conference paper at ICLR 2024
We have in the previous example that structure emerged in the sub-policies, learning to repeat chains of sub policies ("diagonals") to move itself. Fig. (8 illustrates the possible effects of long training this structure. This structure is particularly present after .5M timesteps, and the sub-policies in the second part of the trajectory are not used during the first part of the trajectory. However, training progresses, the structure is progressively forgotten, which makes the bottom sub-policy sequence plot harder to read. One possible explanation is that the agent learns "overfit" the task, resulting in lower re-use of sub-policies, but better performance (as evidenced by the progressively lower number of timesteps required to solve the task). This might also be due the high regularization that we enforced on the Gumbel network, forcing agents to use as many sub- sub-policies possible.
HC64, U-shaped maze (Fig. 28
Actual sub-policy taken, 0.5M steps (goal reached 410 steps)
Subpolicy
20
40
60
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, steps (goal reached 327 steps)
Subpolicy
40
60
500
Trajectory step
Actual sub-policy taken, 1.5M steps (goal reached In 295 steps)
Subpolicy
20
40
60
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, steps (goal reached steps)
Subpolicy
20
40
60
100
300
500
Figure 28: The learnt structure (top plot) can get lost as the agent learns to solve efficiently the maze (bottom plot).
41

---

Published as a conference paper at ICLR 2024
Fig. 29 finally provides example of a policy where structure is difficult to observe. HC16 learns to solve the I[-shaped maze after 1.5M timesteps. but no particular structure emerges. In addition, the bottom plot. We note that despite this fact, the HyperCombinatoor architecture guarantees that bring transparency to the interaction of the agent with the environment, conditioned on the knowledge of the sub-policy chosen. Notably, we still have access to all the sub-policies that define how HC16 interacts with the environment.
HC16, II-shaped maze (Fig. 29
example of HC16 forgetting how to solve the maze in
Actual sub-policy taken, 0.5M steps (goal not reached in 600 steps)
lil With II
Subpollcy #
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, 1M steps (goal not reached in 600 steps)
Subpolicy
10
15
100
200
300
400
500
600
Trajectory step
Actual sub-policy taken, 1.5M steps (goal reached in 319 steps)
Subpolicy *
10
15
100
200
300
400
500
600
Trajectory step
Actual sub policc taken, steps (goal not reached in 600 steps)
mill
Subpolicy *
11
10-
15
100
200
300
400
500
600
Figure 29: HC16 can learn solve the maze without displaying obvious structure in its sequence of sub-policies (3rd plot).
42

---

Published as a conference paper at ICLR 2024
E.3 EFFECT OF GUMBEL TEMPERATURE ON PERFORMANCE
We run an ablation to see the effect of changing the temperature hyperparameter, all other hyperpa- rameters being held the variants for temperature values of .5, .66, 1, 2, 3, 5, 7 and 10 in the U-shaped maze. In general, higher temperature value tends to increase the stochasticity of the Gumbel network, meaning that the Gumbel network will be more uncertain of which sub-policy to choose (and will also increase gradient sharing during the backward pass).
in the navigation part of Sec. 4 We evaluate the HyperCombinator
We visualize the results of the ablation in Fig. 30 Overall, low temperature values impede learning,
and the resulting agents do not succeed the U-maze. Higher temperature values led better performing agents, though the overall return curve remains noisy in all cases.
U-shaped maze
0.8
HC64, T-0.5
Episade reward
HC64, T=0.66
HC64, T-1
HC64, T-2
HC64, T=3
HC64, T-5
0.4
HC64, T=7
HC64, T=10
Training steps
1e6
Figure 30: Low temperature values lead to insufficient regularization and HC64 fails to solve the maze, Increasing the temperature overall leads better performance, except for the highest temper- ature.
43


Reviews:
[{"review_id": "54eG9FQBt8", "reviewer": "Reviewer_KsbR", "summary": "The authors propose a framework for interpretable reinforcement learning that attempts to balance interpretability, performance, and computational complexity. They propose learning a set of piecewise linear policies, which have the benefit of interpretability of linear models. They ensure that the number of piecewise linear policies is not too large which would inhibit practical interpretability. They present the empirical performance of their framework, HyperCombinator, in control and navigation tasks.", "questions": "How complex of environments can the HC framework handle without fully losing performance?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The communication in the paper is clear. The authors clearly describe the desiderata for the ideal interpretable RL methods, and are clear about how their proposed approach seeks to address these points.\n- The authors' commitment to maintaining linear policies at some level is good, as these will always be very interpretable on their own.\n- The authors provide nice visualizations of their approach and their empirical results.", "weaknesses": "- The contribution level is low in this paper. Pasting together a set of linear policies does not seem that differentiated from prior work.\n- It's not clear practically how interpretable the resulting model is. I suppose the user is supposed to inspect the linear model coefficients to understand what the policy is doing. However, there is not much discussion of this. For example, how should I interpret the coefficient heatmap in Figure 3?\n- As it stands, it seems the approach is less of a compromise between interpretability and performance and more of a deterioration of performance while only gaining a little bit of interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYAb1h98lZ", "reviewer": "Reviewer_8HLj", "summary": "The paper proposes to make Deep RL more interpretable by training a policy which is a piecewise linear parameterization of policies. It is supposed to solve most of the set of constraints an interpretable policy should satisfy (Section 3). The method works by using a Gumbel Softmax to select the different linear polices and then interpreting predictions based on the linear models. Some experiments show the  ability of their model to converge is more or less equal to normal methods, and arguments are made that is interpretable.", "questions": "* Sorry if I missed it, but does this work for pixel-based input data?\n* Do you see this being useful for debug, teach, regulate, calibrate reliance, or offer recourse etc.,? If so, why didn't you demonstrate this.\n\nI think overall, given the positives and negatives, I lean a bit towards acceptance, but will await the rebuttal etc. to mutate my score later. Thanks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strength of this paper is that is is proposing a reasonable solution to a very difficult problem of interpretable deep RL. I like the general idea and it makes sense, and results are reasonable enough. It should also pave a way for counterfactual reasoning as the authors suggest, and several other possible future directions.", "weaknesses": "There are two main weaknesses of the paper in my view.\n\nFirstly, there appears to be clear limitations of the method performance wise, it appears to really struggle to latch onto the similar abilities of the black-box, which can be problematic as reducing performance has catastrophic effects on user trust (and then appropriate reliance etc.) Or at least, HC64 etc. is needed to get reasonable performance as opposed to HC8.\n\nSecond, the authors don't actually show the method being particularly useful for anything. It isn't always necessary to do this, but I feel here it would have helped a lot. Usually, explainability methods are used to debug, teach, regulate, calibrate reliance, or offer recourse etc., but this method isn't shown to be able to do any of those things.\n\n### Other\n* There is a misspelling in the title of parameterization.\n* I wouldn't limit your Section 3 to just \"counterfactual reasoning\", there are many other types of contrastive explanation which your method could also generate.\n* I also don't fully get (P2) here.\n* p6: Please don't say \"remarkable\", let the readers decide themselves.\n* Would be good to briefly elaborate in Fig2 why Finger spin does differently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NGDdMMwXAB", "reviewer": "Reviewer_yR55", "summary": "### Problem Statement\n\nThe paper addresses the challenge of crafting interpretable policies in Deep Reinforcement Learning (DRL) to foster the development of trustworthy autonomous agents. The conventional linear policies, while interpretable, lack the expressivity to tackle complex tasks. In light of this, the authors advocate for piecewise-linear policies that aim to marry the interpretability of linear policies with enhanced performance akin to complex neural models.\n\n### Methodology\n\nTo this effect, they introduce the \"HyperCombinator\" (HC), a neural architecture that embodies a piecewise-linear policy with a controlled number of linear sub-policies. Every interaction with the environment engages a specific linear sub-policy, enhancing the interpretability while maintaining competitive performance. The architecture is a preset number of sub-policies, each in the form of a set of linear coefficients, gated by a MLP with one-hot output with Gumbel-softmax reparameterization allowing end-to-end training.\n\n### Main Contributions\n\nThe key contributions encapsulated in the paper are as follows:\n\n- They delineate the attributes that a desirable interpretable piecewise-linear policy should exhibit.\n- They design the HyperCombinator, a novel architecture that encapsulates a piecewise-linear policy with a defined number of linear sub-policies, and which is amenable to a broad spectrum of RL algorithms.\n- They analyze the interpretability of HyperCombinator.\n- They conduct evaluations of the model on control and navigation tasks, showcasing that the model retains a robust performance despite its curtailed expressivity.\n- They leverage the interpretability of HC to develop two visualizations that elucidate the policy's reactions to inputs and unveil the temporal abstractions in task execution through tracking the sequence of sub-policies employed.", "questions": "- In the Cheetah control setup, is the action space continuous? If so, what does the \"action 0\" in Figure 3 mean? Is it one dimension of the action? Then it is not proper to refer to it as \"for one action of $\\tilde{\\pi}$\".\n- Why are SAC and RIS chosen as the base alogrithm for RL in the experiments?\n- Are subpolicies trainable? Besides the diversifying regularizations for the MLP, would similar regularizations encouraging a diversified sub-policy help with the learning and model capacity?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality and significance\n\nThe problem of interpretable Deep Reinforcement Learning is of great practical value. Despite the simplicity of the proposed approach, it attains competitive performance while providing much more interpretability compared to conventional deep neural networks. The formulation of desired properties of an interpretable policy is also insightful.\n\n### Writing\n\nThe paper is very well written. The organization is logical, presentation accurate and efficient. First laying out the desired properties and then evaluating the method against the properties makes the paper very easy to follow.", "weaknesses": "- The method has limitations that prevent it from being applied to complex problems of larger scale: The policy loses interpretable as soon as the number of linear sub-policies grows large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lYpehdPKG3", "reviewer": "Reviewer_j8Wd", "summary": "The paper presents a novel architecture for actor-critic algorithms with the goal of increasing the interpretability of the learned policies. The proposed architecture learns a model that maps each input to a linear function, which is used to predict the agent's action. The underlying assumption is that, if we can understand the set of linear functions used to produce actions, then the model is \"partially\" interpretable. \n\nThe new architecture, HyperCombinator (HC), is evaluated with SAC, where the actor model is replaced with an HC. The SAC-HC algorithm is evaluated on the DeepMind Control Suite benchmark. The empirical results show that the performance of SAC-HC in terms of sample efficiency is similar to that of SAC. \n\nThe paper also presents plots showing how different linear functions are used across different episodes of problems such as Cheetah.", "questions": "1. How does HC architecture compare with other works from the literature, especially those on the PiRL line of work? \n\n2. How can one properly evaluate the interpretability of these models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Policy interpretability is an important topic, since policies we can understand and verify are important in real-world scenarios. \n\nIt is also interesting to see empirical evidence that it is possible to perform well in commonly used benchmarks with models with less capacity. \n\nAnother strength of the paper is its clarity and ease of understanding. While the topics discussed aren't necessarily simple, the authors have done a good job presenting their results.", "weaknesses": "I have two main concerns with this paper. The first is about a whole body of work on programmatic policies that the paper overlooks. The second is about the weak evaluation related to the interpretability of the proposed model. I'll detail each concern below.\n\n**Missing Related Work**\n\nBelow are some related works that I think the paper missed. The paper should cite some of these to better place its contributions within the existing literature. Others should serve as baselines in the experiments, as I explain later.\n\nThere's a line of research called Programmatically Interpretable RL (PiRL), see Verma et al. The goal in this line of research is to create programs that encode policies for RL problems. One of the key motivations behind PiRL is interpretability. Both Verma et al. and Bastani et al. use imitation learning to train interpretable models. Later, Qiu and Zhu found a way to learn similar policies with a fully differentiable approach, so there's no need for imitation learning. The architecture of Qiu and Zhu is essentially an oblique decision tree, which could also be trained with ReLU neural networks, as discussed in the work of Lee and Jaakkola and of Orfanos and Lelis. \n\nThe properties of ReLU networks share a lot in common with what's mentioned in the paper for the HC architecture:\n\n1. \"We make the functions $a$ and $\\theta$ explicit through a new parametrization of the NN.\"\n\nThis is also possible with ReLU networks. In these networks, the function $a$ is the set of weights in the hidden layers, while $\\theta$ is the function the model learns when the activation pattern is fixed (see Lee and Jaakkola for more). So, what makes the HC different from ReLU networks?\n\n2. \"We explicitly control the number of unique sub-policies of $\\pi$ through the dimension $dG$ of the Gumbel-Softmax layer.\"\n\nThis is also the case with the number of neurons in ReLU networks. Another way to reduce the number of unique sub-policies in ReLU networks is to use a single hidden layer with strong L1 regularization (refer to Orfanos and Lelis).\n\n3. \"Policies modeled with the HyperCombinator differ from MLPs in that they usually aren't continuous at the border between linear regions.\"\n\nI'm not sure why this is crucial, but ReLU networks also switch the linear function that gives the prediction when you move from one region to another.\n\n4. \"Our approach is locally interpretable, explaining the sub-policy applied to any given example.\"\n\nThis is true for ReLU networks mapped to Oblique Decision Trees too. The function that gives the prediction is at the leaf node. The path in the tree might not be easy to understand, but the linear function at the leaves is just as clear as those in the HC architecture.\n\nGiven all these similarities, small ReLU networks should be used as baselines in the experiments. Specifically, the architecture by Qiu and Zhu showed strong results on the same benchmark problems used in this paper.\n\nOther papers I mention below might be less crucial but are related. So, it would be good to see where this paper stands within the broader literature. For example, Inala et al. discuss how to learn interpretable finite state machines for RL problems. Koul et al. learn FSM policies for RL problems from recurrent networks. Aleixo and Lelis look at programmatic policies in multi-agent RL. Trivedi et al. and Liu et al. learn a latent space of a domain-specific language which can be used to search for programmatic policies. All these papers explore potentially interpretable policies for RL problems, making them relevant to this submission.\n\n**Lack of Evaluation on Interpretability**\n\nThe paper mostly gives anecdotal evidence when it comes to the interpretability of the policies. I appreciate the plots showing how the sub-policies work, but they're just examples. For some reason, the literature on interpretable policies doesn't focus much on evaluating interpretability. The papers I've listed below are weak in this area too. But none of them emphasize interpretability as much as this one does. The big unanswered question is: are these policies really interpretable, and if so, to whom?\n\n**References**\n\n1. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Proceedings of the International Conference on Neural Information Processing Systems, pages 2499–2509. Curran Associates Inc., 2018.\n\n2. Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing programmatic policies that inductively generalize. In International Conference on Learning Representations, 2020.\n\n3. Anurag Koul, Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. In International Conference on Learning Representations, 2019.\n\n4. Guang-He Lee and Tommi S. Jaakkola. Oblique decision trees from derivatives of relu networks. In International Conference on Learning Representations, 2020.\n\n5. David S. Aleixo and Levi H. S. Lelis. Show me the way! Bilevel search for synthesizing programmatic strategies. In Proceedings of the AAAI Conference on Artificial Intelligence. 2023.\n\n6. S. Orfanos and Levi H. S. Lelis. Synthesizing programmatic policies with actor-critic algorithms and relu networks, 2023.\n\n7. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. 2018.\n\n8. Abhinav Verma, Hoang M. Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. In Proceedings of the International Conference on Neural Information Processing Systems. Curran Associates Inc., 2019.\n\n9. Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In International Conference on Learning Representations, 2022.\n\n10. Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning to synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34:25146–25163, 2021.\n\n11. Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, and Shao-Hua Sun. Hierarchical programmatic reinforcement learning via learning to compose programs. arXiv preprint arXiv:2301.12950, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)