Strategically-Linked Decisions in Long-Term Planning and Reinforcement Learning

## Abstract

Abstract Long-term planning, as in reinforcement learning (RL), involves finding strategies: actions that collectively work toward a goal rather than individually optimizing their immediate outcomes. As part of a strategy, some actions are taken at the expense of short-term benefit to enable future actions with even greater returns. These actions are only advantageous if followed up by the actions they facilitate, consequently, they would not have been taken if those follow-ups were not available. In this paper, we quantify such dependencies between planned actions with strategic link scores : the drop in the likelihood of one decision under the constraint that a follow-up decision is no longer available. We demonstrate the utility of strategic link scores through three practical applications: (i) explaining black-box RL agents by identifying strategically linked pairs among decisions they make, (ii) improving the worst-case performance of decision support systems by distinguishing whether recommended actions can be adopted as standalone improvements or whether they are strategically linked hence requiring a commitment to a broader strategy to be effective, and (iii) characterizing the planning processes of non-RL agents purely through interventions aimed at measuring strategic link scores—as an example, we consider a realistic traffic simulator and analyze through road closures the effective planning horizon of the emergent routing behavior of many drivers.

## 1 Introduction

Figure: (a) Optimal Strategy
Refer to caption: /html/2505.16833/assets/figures/raw/gridworld-wsimple.png

A fundamental goal in reinforcement learning (RL) is to find long-term strategies—sequences of interdependent actions that work together to accomplish an objective as opposed to disjointed actions driven by their own immediate outcomes. This interdependency is a key aspect of strategic decision-making, where early actions are often taken at the expense of short-term benefit to enable future actions with even greater overall returns. When that is the case, those early actions would only be advantageous if followed up by the actions they facilitate, consequently, if an intended future action were to become unavailable with some advanced notice, a strategic decision-maker would adjust their earlier actions accordingly. In this work, we aim to formalize such strategic dependencies between decisions:
to what extent a current action is taken to set up a future one.

As a concrete example, consider a simple navigation task (Figure [1](#S1.F1)), where the straightforward path to the objective is long and winding. Slightly off that path, however, there is a key that unlocks a significant shortcut. In this scenario, the plan to retrieve the key—incurring a small delay—and then take the shortcut—saving more time than lost in the delay—is a strategic one. This is made evident by the fact that, if the shortcut were to somehow be blocked, an optimal agent would no longer choose to retrieve the key. With such strategic plans, we refer to “retrieving the key” as the set-up decision and “taking the shortcut” as the pay-off decision, which brings us to our core contribution:

- We quantify the strength of the strategic link between the set-up decision and the pay-off decision as the drop in the likelihood of the set-up decision under the constraint that the pay-off decision is no longer available, and we call this quantity the strategic link score between the two decisions.

This drop would be large when the set-up decision is not advantageous on its own but rather made primarily to facilitate the pay-off decision. In contrast, if the optimality of the two decisions is not contingent on each other— that is, the “set-up” decision is advantageous regardless of whether it is followed up by the “pay-off” decision—then the drop would be zero.
By providing a quantitative definition of strategic links, we enable a range of practical use cases. In this paper, we highlight three applications and demonstrate each through experiments:

#### Providing Planning-Level Explanations

Strategic links can help explain black-box policies learned by RL algorithms. Existing explainability methods typically provide either state-level explanations—for instance, important features of a state that most influence the corresponding action (Greydanus et al., 2018)—or policy-level explanations—for instance, key trajectories that summarize a policy’s behavior (Amir and Amir, 2018). Meanwhile, strategic links offer planning-level explanations, as they are not the result of a given policy alone but instead reflect the broader planning process and how that policy would have differed if certain decisions had been unavailable—in other words, strategic links depend on counterfactual policies as well.

Given an RL agent, we identify strategic links between its decisions by searching for set-up and pay-off pairs with high strategic link scores along a trajectory of likely decisions. When a goal-driven agent’s behavior is only observable through demonstrations—like human experts—we show that one can still identify strategically linked decisions by first inferring a reward function—via inverse reinforcement learning (IRL)—that captures the agent’s underlying objective and serves as a model of their planning process. While reward functions learned through IRL are typically used for imitation learning—that is, to recover the demonstrated policy (e.g. Abbeel and Ng, 2004; Ziebart et al., 2008)—our application highlights they are also effective at generating counterfactual policies to compute strategic link scores.

#### Improving Policies Safely via Strategy-Aware Recommendations

Decision support systems that aim to improve policies can be made safer by determining whether recommended actions are strategically linked. Suppose a system recommends two changes to a decision-maker’s policy. If the changes are strategically linked—meaning one is only effective when the other is also adopted—there is a risk that the decision-maker may unknowingly implement just one and end up worse off as a result. In such cases, the system should ideally communicate that the changes form a single strategy and must be implemented together or not at all. Conversely, if the changes are strategically independent—meaning each one is effective on its own merit regardless of whether the other is adopted or not—the decision-maker can safely choose which recommendations they want to implement. Through experiments, we show that providing strategy-aware recommendations leads to greater improvements, both on average and in worst-case scenarios.

#### Characterizing Planning Behavior Through Interventions

Strategic link scores are not limited to RL agents or reward-based planners, and they offer a means of characterizing how any agent forms long-term plans as long as it is possible to perform interventions that prevent them from taking certain actions. This is particularly useful when the agent’s inner workings are inaccessible otherwise. For instance, suppose we want to understand how far ahead an agent plans. If it were an RL-based planner, we could have directly examined its discount factor. But regardless of what type of planner they are, we can still analyze strategic links between their decisions—specifically, the time distance between strongly linked decisions—to gain insight into their effective planning horizon.
We demonstrate this process in a realistic traffic simulator, where many drivers make individual routing decisions that collectively produce an emergent routing policy—one that determines how traffic flow is distributed throughout the road network. By performing interventions in the form of road closures, we characterize the implicit planning horizon of this policy and highlight how it differs from an RL-based policy.

## 2 Related Work

Applications of strategic links are related to explainability in RL, which can be address at state, policy, or planning levels, and safe policy improvement.

#### State-Level Explanations

At this level, explanations focus on individual decisions: why a particular state is mapped to its corresponding action. A common approach is to adapt techniques from supervised settings, such as extracting saliency maps (e.g. Greydanus et al., 2018; Iyer et al., 2018), collecting human annotations (e.g. Ehsan et al., 2018), fitting white-box meta-models (e.g. Liu et al., 2018), or generating counterfactual examples (e.g. Olson et al., 2021; Chen et al., 2022; Huber et al., 2023). While this last group of works consider counterfactual states, strategic links involve counterfactual policies that could have been optimal had the environment conditions been different.

#### Policy-Level Explanations

These explanations consider all the decisions of a given policy collectively. Some methods summarize those decisions via key trajectories (Amir and Amir, 2018) or policy graphs (Topin and Veloso, 2019); others contrast the policy with a baseline (van der Waa et al., 2018; Yao et al., 2022); and many aim to learn white-box policies directly (e.g. Khan et al., 2009; Shu et al., 2018; Verma et al., 2018; Hein et al., 2018; Silva et al., 2020; Sun et al., 2023). All of these methods are limited in scope to a single policy and ignore the planning process behind it. In contrast, planning-level explanations such as strategic links consider policies in context of others related through a shared planning process.

#### Planning-Level Explanations

In RL settings, the planning processes are largely characterized by the reward function being optimized and the objective it encodes. Accordingly, a few methods explain policies by relating actions to particular dimensions of the reward function (Juozapaitis et al., 2019; Erwig et al., 2018) or to the impact they have on the probability of success (Madumal et al., 2020; Yau et al., 2020; Cruz et al., 2023). In doing so, these methods provide insight not only into the policy itself, but also into why maximizing a given reward or achieving a given objective requires that particular policy. Strategic links also provide planning-level explanations, however, they relate actions to other actions rather than rewards or objective. Hence, their application is not limited to RL settings like the other methods. Strategic link scores can be computed for any type of planner, as long as their planning process can be intervened on.

#### Safe Policy Improvement

Improving upon policies already in deployment is a common use case of RL. A key challenge in this setting is safety: ensuring the new policy does not perform worse than the original when deployed. This is usually a risk caused by the poor coverage of the available data collected by the original policy. So, prior work tends to focus on solutions at the training stage—for instance, limiting deviations from the original policy to keep new policies within coverage region (Laroche et al., 2019; Wu et al., 2022; Sharma et al., 2024). Strategy-aware recommendations, on the other hand, address the post-training risk of improper or partial adoption of new policies by flagging changes that are strategically linked and should be implemented together.

## 3 Preliminaries

#### Environments

We consider decision-making environments $\mathcal{E}=(\mathcal{S},\mathcal{A},\sigma,\tau)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\sigma\in\Delta(\mathcal{S})$ is the initial state distribution, and $\tau\in\Delta(\mathcal{S})^{\mathcal{S}\times\mathcal{A}}$ is the transition dynamics. At the start of each interaction, the environment is in some initial state $s_{1}\sim\sigma$, and at each time step $t\in\{1,2,\ldots\}$, it transitions into a new state $s_{t+1}\sim\tau(s_{t},a_{t})$ depending on the current state $s_{t}\in\mathcal{S}$ and the action $a_{t}\in\mathcal{A}$ taken by a decision-maker.

#### Policies & Planners

Decision-makers take their actions by planning policies and then executing those policies. Policies $\pi\in\Delta(\mathcal{A})^{\mathcal{S}}$ map each state into an action distribution such that $a_{t}\sim\pi(s_{t})$ when the policy is executed in an environment, and $\pi(a|s)$ is the probability of taking action $a$ at state $s$. Planners $\smash{\mathcal{P}^{\dagger}}$ output policies given an environment $\mathcal{E}$, which we denote with $\smash{\pi^{\dagger}}=\smash{\mathcal{P}^{\dagger}}(\mathcal{E})$. For instance, given a reward function $r\in\mathbb{R}^{\mathcal{S}\times\mathcal{A}}$ and a discount factor $\gamma\in(0,1)$, the optimal planner $\smash{\mathcal{P}^{*(r,\gamma)}}$ aims to find the policy that maximizes the expected cumulative discounted rewards $R$:

$$ $\displaystyle\pi^{*(r,\gamma)}=\mathcal{P}^{*(r,\gamma)}(\mathcal{E})=\operatorname{argmax}{\!}_{\pi}~\mathbb{E}{}_{\pi,\mathcal{E}}[~R\doteq{\textstyle\sum_{t=1}^{\infty}}\gamma^{t-1}r(s_{t},a_{t})~]$ (1) $$

Since our definition of strategic links involves policies planned under constraints, we denote with $\smash{\pi^{\dagger:\mathcal{C}}}=\smash{\mathcal{P}^{\dagger:\mathcal{C}}}(\mathcal{E})$ the policy planned by $\mathcal{P}^{\dagger}$ under a given constraint $\mathcal{C}$. For instance, when states and actions are discrete, a key constraint we consider is preventing a particular action $\tilde{a}\in\mathcal{A}$ from being taken at a particular state $\tilde{s}\in\mathcal{S}$ such that $\pi(\tilde{a}|\tilde{s})=0$. Under that constraint, we can express the optimal planner’s new policy as $\smash{\pi^{*:\{\pi(\tilde{a}|\tilde{s})=0\}}}=\smash{\mathcal{P}^{*:\{\pi(\tilde{a}|\tilde{s})=0\}}}(\mathcal{E})=\operatorname{argmax}{}_{\pi:\pi(\tilde{a}|\tilde{s})=0}\mathbb{E}{}_{\pi,\mathcal{E}}[R]$.

## 4 Strategic Link Scores

When an agent takes an action $a\in\mathcal{A}$ at some state $s\in\mathcal{S}$, we consider this to be a decision—that is, decisions are state-action pairs $(s,a)\in\mathcal{S}\times\mathcal{A}$. Then, for a given planner $\mathcal{P}^{\dagger}$, the strategic link score between set-up decision $(s,a)$ and pay-off decision $(\tilde{s},\tilde{a})$ is defined as

$$ $\displaystyle\mathfrak{S}^{\dagger}_{(s,a)\to(\tilde{s},\tilde{a})}=\pi^{\dagger}(a|s)-\pi^{\dagger:\{\pi(\tilde{a}|\tilde{s})=0\}}(a|s)$ (2) $$

which is the drop in the likelihood of the set-up, $\pi(a|s)$, when the pay-off is constrained, $\pi(\tilde{a}|\tilde{s})=0$.

#### Continuous States & Actions

If either the state space or the action space is continuous, this definition becomes problematic as a point constraint such as $\pi(\tilde{a}|\tilde{s})=0$ would practically have no impact on the planned policy $\smash{\pi^{\dagger}}$. In such cases, we allow a region of pay-off decisions, $\smash{\tilde{\mathcal{S}}}\times\smash{\tilde{\mathcal{A}}}\subseteq\mathcal{S}\times\mathcal{A}$, and work with the constraint $\pi(\tilde{a}|\tilde{s})=0,\forall\tilde{s}\in\smash{\tilde{\mathcal{S}}},\forall\tilde{a}\in\smash{\tilde{\mathcal{A}}}$.

The choice of a meaningful region will depend on the application. For instance, the traffic scenario we consider in Section [5.3](#S5.SS3) involves determining the (continuous) traffic flow $a\in\mathcal{A}=[0,1]$ through junctions $s\in\mathcal{S}=\{1,\ldots,10\}$. If we are interested in whether the flow through the first junction $s=1$ is strategically dependent on sufficient flow at the last junction $s=10$, we can consider the pay-off region: $\smash{\tilde{\mathcal{S}}}=\{10\}$ and $\smash{\tilde{\mathcal{A}}}=(a^{*},1]$ for some threshold $a^{*}$. This encodes the constraint that the flow $a\sim\pi(s=10)$ cannot be larger than the threshold—that is $a\leq a^{*}$ almost surely.

Figure: Figure 2: Dynamics of the toy example. For initial state $S_{1}$, the action sequence $(A_{2},A_{2})$ is optimal under both reward functions $\smash{r_{\alpha}}$ and $\smash{r_{\beta}}$. For $\smash{r_{\alpha}}$, the two actions are strategically linked as $A_{2}|S_{1}$ is not optimal unless $A_{2}|S_{2}$. For $\smash{r_{\beta}}$, there is no strategic link as $A_{2}|S_{1}$ is optimal regardless of the action taken at state $S_{2}$.

#### A Toy Example

Consider a simple environment with two states $\mathcal{S}=\{S_{1},S_{2}\}$ and two actions $\mathcal{A}=\{A_{1},A_{2}\}$. The transitions are deterministic as shown in Figure [2](#S4.F2) and the initial state is always $S_{1}$. We are given two different reward functions, $r_{\alpha}$ and $r_{\beta}$, also shown in Figure [2](#S4.F2). Suppose two optimal planners, $\mathcal{P}^{*(r_{\alpha})}$ and $\mathcal{P}^{*(r_{\beta})}$, aim to each maximize one of these reward functions, for a fixed time horizon of two steps and no discounting.

In either case, the optimal policy is exactly the same: take action $A_{2}$ at $S_{1}$ and then $A_{2}$ at $S_{2}$, which can be expressed as

$$ $\displaystyle\pi^{*(r_{\alpha})}(S_{1})=\pi^{*(r_{\beta})}(S_{1})=A_{2}$ (3) $\displaystyle\pi^{*(r_{\alpha})}(S_{2})=\pi^{*(r_{\beta})}(S_{2})=A_{2}$ (4) $$

Despite having the same policies, for planner $\mathcal{P}^{*(r_{\alpha})}$, these two decisions are strategically linked. Under $r_{\alpha}$, the immediate reward of $A_{1}$ is larger than $A_{2}$ at $S_{1}$: $r_{\alpha}(S_{1},A_{1})=1>r_{\alpha}(S_{1},A_{2})=0.5$. However, $\mathcal{P}^{*(r_{\alpha})}$ still takes action $A_{2}$ to be able to transition into $S_{2}$, where an even larger pay-off is available, namely $r_{\alpha}(S_{2},A_{2})=2.5$ over $r_{\alpha}(S_{1},A_{1})=1$. If taking action $A_{2}$ at $S_{2}$ had not been an option, taking action $A_{2}$ at $S_{1}$ would no longer have been optimal, therefore

$$ $\displaystyle\mathfrak{S}^{*(r_{\alpha})}_{(S_{1},A_{2})\to(S_{2},A_{2})}\>=\>\underbrace{\pi^{*(r_{\alpha})}(A_{2}|S_{1})}_{~~=\,1}\>-\>\underbrace{\pi^{*(r_{\alpha}):\{\pi(A_{2}|S_{2})=0\}}(A_{2}|S_{1})}_{~~=\,0}\>=\>1$ (5) $$

Meanwhile, for planner $\mathcal{P}^{*(r_{\beta})}$, there is no strategic link. Under $r_{\beta}$, transitioning into $S_{2}$ is not important as both $A_{1}$ and $A_{2}$ have the same reward in either state: $r_{\beta}(S_{1},A_{1})=r_{\beta}(S_{2},A_{1})=1$ and $r_{\beta}(S_{1},A_{2})=r_{\beta}(S_{2},A_{2})=1.5$. Taking action $A_{2}$ at $S_{1}$ is optimal regardless of what action is taken at the next state $S_{2}$, therefore

$$ $\displaystyle\mathfrak{S}^{*(r_{\beta})}_{(S_{1},A_{2})\to(S_{2},A_{2})}\>=\>\underbrace{\pi^{*(r_{\beta})}(A_{2}|S_{1})}_{~~=\,1}\>-\>\underbrace{\pi^{*(r_{\beta}):\{\pi(A_{2}|S_{2})=0\}}(A_{2}|S_{1})}_{~~=\,1}\>=\>0$ (6) $$

This toy example underlines why strategic link scores lead to planning-level explanations: While policies $\pi^{*(r_{\alpha})}$ and $\pi^{*(r_{\beta})}$ are the same, they exhibit different strategic links depending on the planner, $\mathcal{P}^{*(r_{\alpha})}$ or $\mathcal{P}^{*(r_{\beta})}$. It also underlines why such strategic analysis is important for safe policy improvement. Suppose the current policy is suboptimal and always takes action $A_{1}$ with total value $r(S_{1},A_{1})+r(S_{1},A_{1})=2$ under either reward function. Then, suppose we recommend the optimal policy instead: taking action $A_{2}$ first at state $S_{1}$ and then at state $S_{2}$. Under $r_{\alpha}$, when the recommendations are strategically linked, implementing the first one without implementing the second would result in worse performance: $r_{\alpha}(S_{1},A_{2})+r_{\alpha}(S_{2},A_{1})=1.5<2$. Under $r_{\beta}$, when there is no strategic link, the same implementation would still improve performance: $r_{\beta}(S_{1},A_{2})+r_{\beta}(S_{2},A_{1})=2.5>2$.

## 5 Practical Applications

Having defined the strategic link score $\mathfrak{S}$, we now illustrate three of its applications introduced earlier: providing planning-level explanations (Section [5.1](#S5.SS1)), improving policies safely via strategy-aware recommendations (Section [5.2](#S5.SS2)), and characterizing planning behavior through interventions (Section [5.3](#S5.SS3)). We present empirical results for each application, with detailed setup given in the appendix.

### 5.1 Providing Planning-Level Explanations

#### Setup

We consider grid-based maze environments, called GridWorld, as in our introductory example in Figure [1](#S1.F1). In these environments, the agent is tasked with navigating to a target cell, and they incur a constant negative reward each time step they have not reached the target. As actions, the agent can move either up, down, left, or right, and by first walking over keys, they can later pass through previously locked doors. In this application, we aim to explain policies learned by soft value iteration (Haarnoja et al., 2017) for solving various maze layouts. We make sure none of the layouts become unsolvable if a particular decision is constrained.

#### Approach 1: Links from Known Planner

Given a planner $\mathcal{P}^{\dagger}$, we explain the policy $\pi^{\dagger}$ by first selecting a trajectory representative of its behavior, and then computing the strategic link score between each pair of decisions along that trajectory. We consider trajectories, rather than random decisions, as they are more likely to exhibit strategic links. While any trajectory of interest can be analyzed in the same way, for this demonstration, we use the most likely trajectory, obtained by taking the highest-probability action at each step. When computing scores, we treat the earlier decision in each pair as the set-up and the latter as the pay-off. The complete procedure is given in Algorithm [1](#alg1).

Figure: (a) Optimal Trajectory
Refer to caption: /html/2505.16833/assets/figures/raw/gridworld-wsimple.png

#### Results for A Simple Layout

Figure [3](#S5.F3) shows our approach in action for the maze layout from our introduction. In Figure [3(a)](#S5.F3.sf1), we plot the most likely trajectory under the soft optimal policy: The target can be reached via a long and winding path, but a quicker strategy is to pick up a key that is slightly out of the way to unlock a door, creating a significant shortcut. We know the key and the door are strategically linked because if we block the door, picking up the key is no longer optimal, see Figure [3(b)](#S5.F3.sf2). Preventing another decision that is not related to the key does not lead to the same outcome, see Figure [3(c)](#S5.F3.sf3). In Figure [3(d)](#S5.F3.sf4), we plot the strategic link scores between all potential set-up and pay-off decisions along the optimal trajectory, which makes it clear that moving towards the key (1^st and 2^nd decisions, labeled “K”) are indeed a set-up for going through the door (9^th decision, labeled “D”).

Figure: (a) Layout with Independent Keys
Refer to caption: /html/2505.16833/assets/figures/raw/gridworld-unlinked.png

#### Results for Linked & Unlinked Keys

In Figure [4](#S5.F4), we consider two other layouts, each with an additional key and its corresponding door. In the first layout (Figure [4(a)](#S5.F4.sf1)), the two keys unlock separate shortcuts, so whether one of them is retrieved or not has no effect on the optimality of retrieving the other. In Figure [4(b)](#S5.F4.sf2), we see that each key (“K1” and “K2”) is strategically linked to its corresponding door (“D1” and “D2”), but not to the other key-door pair. In the second layout (Figure [4(a)](#S5.F4.sf1)), there is just a single shortcut and unlocking it requires both keys to be collected. Now, the keys are strategically linked: If one of them is not retrieved, there is no point in retrieving the other (seen in Figure [4(d)](#S5.F4.sf4)).

These two layouts also highlight the importance of strategy-aware recommendations. If an agent is not taking advantage of any shortcuts by collecting keys, we might recommend them to pick up both keys. In the independent case, it is safe for the agent to only follow through on just collecting one of the recommended keys. In the strategically linked case, however, if the agent were to collect only one key without the other, they would unnecessarily waste time as the shortcut would remain inaccessible.

Figure: (a) GridWorld

#### Approach 2: Links from Demonstrations

When a goal-driven planner is not available as an explicit algorithm but only observable through demonstrations—that is, trajectories generated by its unconstrained policy—we can still identify strategic links by first inferring a reward function that captures the planner’s objective. We achieve this via maximum entorpy IRL (Ziebart et al., 2008). Because the coverage of demonstrations is critical for IRL, we experiment with different levels of stochasticity in actions by varying the temperature parameter in soft value iteration. We evaluate inferred rewards using the EPIC distance, (Gleave et al., 2021), which accounts for equivalent reward expressions up to shaping, and measure the accuracy of strategic link scores computed based on those rewards using mean square error (MSE). In addition to Gridworld, we also consider two other environments from later applications: Shortcuts, abstract environments with similar key-shortcut dynamics (Section [5.2](#S5.SS2)), and ArterialHighway, a realistic traffic simulation (Secion [5.3](#S5.SS3)).
We run each experiment five times with different seeds to obtain 1-sigma error bars.

In Figure [5](#S5.F5), we see that strategic link scores inferred from demonstrations improve in accuracy as the variation in those demonstrations increases. This is similar to the trend with inferred reward functions with one notable difference: When policies are so stochastic that they essentially pick actions uniformly at random, reward functions become unidentifiable, hence the sudden spikes in the corresponding error. However, strategic link scores are unaffected by this issue as recognizing policies to be uniformly random—without finding the exact reward structure—happens to be sufficient to correctly conclude that no decision is strategic. These results show that demonstrations alone can carry enough information to model not just the policy of a decision-maker but also their planning process in the form of a reward function, and thereby identify their strategies.

### 5.2 Improving Policies Safely via Strategy-Aware Recommendations

#### Environment

For this application, we introduce Shortcuts, a procedurally-generated collection of abstract environments that are designed to have dynamics similar to key-shortcut relationships in from our previous application with more complex strategic links between different shortcuts.

Figure: Figure 6: An example environment that has 5 nodes, 3 shortcuts, and 4 preparation actions. After taking the required preparation actions, the agent can jump via shortcuts to move more efficiently towards the target.

Figure [7](#S5.F7) shows an example Shortcuts environment. The agent starts at some initial node and needs to move one node at a time until they reach the $N$-th node. Each move costs one unit of time but reaching the target returns a reward of $N$. What makes the environment interesting is shortcuts that jump over multiple nodes at once. Each shortcut becomes available only after performing certain preparation actions beforehand (similar to keys in GridWorld). These preparation actions do not cause the agent to move but they still cost a fixed amount $C<1/2$. They become advantageous through shortcuts they enable: Jumping forward via a shortcut that spans $n$ nodes with $k$ required preparations costs $n-k\cdot 2C$ rather than $n$ individual moves normally needed. We generate 100 environments with 10 nodes, 5 shortcuts, and 5 preparation actions, setting $C=0.1$. Which nodes the shortcuts jump over and which preparations they require are randomized, while preparation actions are always placed at the initial node.

#### Objective

Our goal is to recommend actions to a suboptimal agent to improve their performance. We consider an agent that knows of shortcuts and makes use of them when they are available, however, their behavior suboptimal because they are not aware of the required preparations—that is, they never take a preparation action unless one is recommended by us. Hence, a recommendation is essentially a set of specific preparation actions. For instance, in Figure [7](#S5.F7), the two shorter jumps happen to be more efficient, so the optimal recommendations are $\{\texttt{prep}_{1},\texttt{prep}_{2},\texttt{prep}_{4}\}$.

Figure: Algorithm 2 Strategy-Aware Recommendations

#### Methods

Given a set of recommendations, we are interested how to best present them to the agent, and we consider three approaches.
(i) Pick-and-Choose: One can present them directly without any explanation, allowing the agent to pick and choose which recommendations they want to implement. During evaluation, we consider all possible combinations and report average as well as worst-case performances. This approach has a risk: If two recommendations are strategically linked, the agent might unknowingly implement only one and end up worse off as a consequence.
(ii) All-or-Nothing: One can inform the agent that the recommendations might be related, and hence should either be implemented all together or not at all. This can be overly cautious: If the agent is not willing to make large changes in their behavior, they might miss out on individual improvements that are not reliant on other recommendations.
We propose (iii) Strategy-Aware: Each recommendation is considered as a potential set-up decision, and it is grouped together with every other recommendation that is strategically linked to it as a pay-off decision (see Algorithm [2](#alg2)). As an example, consider again the recommendations for the environment in Figure [7](#S5.F7). Strategy-Aware would group them as $\{~\{\texttt{prep}_{1},\texttt{prep}_{2}\},\allowbreak~\{\texttt{prep}_{4}\}~\}$, while Pick-and-Choose is equivalent to the grouping $\{~\{\texttt{prep}_{1}\},~\{\texttt{prep}_{2}\},~\{\texttt{prep}_{4}\}~\}$, and All-or-Nothing to the grouping $\{~\{\texttt{prep}_{1},\allowbreak\texttt{prep}_{2},\allowbreak\texttt{prep}_{4}\}~\}$.

#### Results

For each strategy, we consider all subsets of recommendations the agent can implement without breaking groups, record the average as well as the worst-case performance, and take their mean across the 100 random environments we have procedurally generated. Figure [7](#S5.F7) shows the results broken down with respect to the number of individual recommendations implemented. We see that, indeed, the worst-case performance of Pick and Choose tends to be worse than that of the original policy. All-or-Nothing is safer, never resulting in worse performance, as it does not allow for partial adoption of recommendations. However, it leads to significant improvements only if a large number of recommendations are implemented. Meanwhile, Strategy-Aware, which takes advantage of our concept of strategically-linked actions, is safe and also effective even when the agent is only willing to make small changes to their policy.

### 5.3 Characterizing Planning Behavior Through Interventions

Strategic link scores offer a way to characterize how agents plan their behavior purely through interventional data, without needing to know anything about their decision-making process. We demonstrate this by examining driving behavior in a realistic traffic simulator, UXsim (Seo, 2025). Our goal is to understand how far ahead drivers plan when making routing decisions. Although the simulator has explicit rules for an individual driver’s behavior, how these rules—executed by many drivers—collectively shape the traffic flow through a road network is difficult to track formally. Instead, we rely only on interventions and use them to measure strategic link scores.

Figure: Figure 8: The traffic scenario. ‘Entry’ to ‘exit’, drivers need to decide whether to stay on an arterial road or divert to the highway at junctions ‘J1’ to ‘J10’. Despite having a lower speed limit, the arterial route is shorter hence overall quicker. But, if one of its segments were to be closed off, it is better to divert to the highway the soonest to make use of its higher speed limit for longer.

#### Road Network

We consider the road network in Figure [8](#S5.F8), which we call ArterialHighway. Each driver enters the network through the same road and aims to reach the same exit, either using a highway or an arterial road. Along the arterial road, there are junctions, “J1” to “J10”, where drivers need to decide whether to stay on the arterial road or switch to the highway via on-ramps. The highway has a higher speed limit but it is overall a longer route as the on-ramps add to the distance. Under no congestion, the shorter arterial route still takes less time despite the lower speed limit, hence staying on the arterial road at every junction is the optimal policy. However, if any segment of the arterial road were to be closed off, then drivers would have no choice but to eventually divert to the highway, and to take advantage of the higher speed limit for longer, diverting as early as possible would become optimal. This pressure to act early makes this road network particularly suitable for measuring planning horizon.

#### Intervention

Taking advantage of the road structure, we can measure the planning horizon of drivers through a simple intervention: closing off the arterial road at the last junction and then observing how early drivers tend to change their route. What we measure is the strategic link score between routing decisions at earlier junctions, as set-ups, and the decision at the last junction as the pay-off. In formal terms, let $\pi^{\dagger}:\mathcal{S}=\{\text{J1},\ldots,\text{J10}\}\to\mathcal{A}=[0,1]$ be the collective policy that emerges out of individual drivers’ decisions such that $\pi^{\dagger}(\text{JX})$ is the frequency with which drivers take the arterial road at junction JX. Then, we are interested in the following strategic link score, modified slightly to fit the continuous action setting: $\smash{\mathfrak{S}_{\text{JX}\to\text{J10}}^{\dagger}}=\pi^{\dagger}(\text{JX})-\smash{\pi^{\dagger:\{\pi(\text{J10})=0\}}}(\text{JX})$.

Figure: Figure 9: Vehicle counts. The rate at which vehicles pass through each junction and still stay on the arterial road changes significantly after the intervention at J10.

#### Simulation

We run a simulation for 50k time steps, where the intervention to close off the arterial road at J10 is performed at time step 10k. Figure [9](#S5.F9) shows our raw data: the number of vehicles over time that pass through each junction and stay on the arterial road; we see a clear change in behavior after the intervention. By taking the average slope of these count-over-time plots, we obtain traffic flow rates—number of vehicles per time step—at each junction, before and after the intervention, either staying on the arterial road or switching to the highway (Figure [10(a)](#S5.F10.sf1)). By normalizing the flow rates at each junction so that they add up to one, we obtain the pre- and post-intervention routing policies (Figure [10(b)](#S5.F10.sf2)). Taking the difference between the two policies gives us the strategic link scores we are after (Figure [10(c)](#S5.F10.sf3)).

Figure: (a) Traffic Flow Rates

Figure: (a) Routing Policy

#### Results

First, we see that the link score for J10 is one. This is by definition of link scores; every decision is strategically linked to itself with a score one. Since the intervention is performed at J10, every driver reaching J10 has no choice but to continue to the highway. Besides J10, we see that the most significant link score is at J9, corresponding to a 25% percentage-point drop in the rate of drivers that still decide to stay on the arterial road and continue to J10. This suggest that the collective behavior of drivers respond rather myopically to changes in traffic conditions. Compare their behavior to the optimal routing policy shown in Figure [11](#S5.F11), which we compute via RL and would only be possible if the drivers were to all cooperate. Besides J10, the only strategic link is to J1—the earliest diversion point—which requires a planning horizon of at least ten junctions.

While the emergent behavior of drivers is limited in horizon when diverting from the arterial road, one long-horizon effect still stands out: the significant negative link score at J1. This reflects drivers who were avoiding the arterial road even before the intervention, and whose avoidance was strategically linked to the road connection following J10. That connection is what made majority of the drivers prefer the arterial road, leading to congestion that slowed down the traffic enough for the longer highway route to become competitive. That is why some drivers decided to avoid the arterial road altogether. Once the intervention caused drivers to leave the arterial road, reducing congestion, this avoidance strategy became less frequent—hence the negative strategic link score.

## 6 Conclusion

We introduced strategic link scores, which formalize a key aspect of long-term planning: actions being taken to enable future actions. By identifying strategic links, we were able to provide planning-level explanations of RL agents, and even directly improve decision-making performance by guiding safe implementation of action recommendations. Beyond reward-based settings, we also used strategic link scores as a tool for characterizing planning processes that are given as explicit algorithms.

#### Limitations

While the definition in ([2](#S4.E2)) works well in discrete settings, it may need to be interpreted differently in continuous settings depending on environment structure and objective—for instance, our final application provided one interpretation tailored to the traffic scenario we analyzed. Moreover, not all strategic links are captured by ([2](#S4.E2)); some strategies may involve higher-order dependencies: If A is taken to enable either B or C, constraints put on B or C only would not reveal the full relationship.

#### Societal Impact

Strategic links are a general concept in planning, not tied to a specific application. Our strategy-aware approach to decision support is one way they can positively impact society.

## References

- Abbeel and Ng (2004)
Abbeel, P. and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in *International Conference on Machine Learning*, 2004.
- Amir and Amir (2018)
Amir, D. and O. Amir, “HIGHLIGHTS: Summarizing agent behavior to people,” in *International Conference on Autonomous Agents and Multiagent Systems*, 2018.
- Chen et al. (2022)
Chen, Z., F. Silvestri, G. Tolomei, J. Wang, H. Zhu, and H. Ahn, “Explain the explainer: Interpreting model-agnostic counterfactual explanations of a deep reinforcement learning agent,” *IEEE Transactions on Artificial Intelligence*, vol. 5, no. 4, pp. 1443–1457, 2022.
- Cruz et al. (2023)
Cruz, F., R. Dazeley, P. Vamplew, and I. Moreira, “Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario,” *Neural Computing and Applications*, vol. 35, no. 25, pp. 18 113–18 130, 2023.
- Ehsan et al. (2018)
Ehsan, U., B. Harrison, L. Chan, and M. O. Riedl, “Rationalization: A neural machine translation approach to generating natural language explanations,” in *AAAI/ACM Conference on AI, Ethics, and Society*, 2018.
- Erwig et al. (2018)
Erwig, M., A. Fern, M. Murali, and A. Koul, “Explaining deep adaptive programs via reward decomposition,” in *IJCAI/ECAI Workshop on Explainable Artificial Intelligence*, 2018.
- Gleave et al. (2021)
Gleave, A., M. Dennis, S. Legg, S. Russell, and J. Leike, “Quantifying differences in reward functions,” in *International Conference on Learning Representations*, 2021.
- Greydanus et al. (2018)
Greydanus, S., A. Koul, J. Dodge, and A. Fern, “Visualizing and understanding Atari agents,” in *International Conference on Machine Learning*, 2018.
- Haarnoja et al. (2017)
Haarnoja, T., H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in *International Conference on Machine Learning*, 2017.
- Hein et al. (2018)
Hein, D., S. Udluft, and T. A. Runkler, “Interpretable policies for reinforcement learning by genetic programming,” *Engineering Applications of Artificial Intelligence*, vol. 76, pp. 158–169, 2018.
- Huber et al. (2023)
Huber, T., M. Demmler, S. Mertes, M. L. Olson, and E. André, “Ganterfactual-rl: Understanding reinforcement learning agents’ strategies through visual counterfactual explanations,” *arXiv preprint arXiv:2302.12689*, 2023.
- Iyer et al. (2018)
Iyer, R., Y. Li, H. Li, M. Lewis, R. Sundar, and K. Sycara, “Transparency and explanation in deep reinforcement learning neural networks,” in *AAAI/ACM Conference on AI, Ethics, and Society*, 2018.
- Juozapaitis et al. (2019)
Juozapaitis, Z., A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez, “Explainable reinforcement learning via reward decomposition,” in *IJCAI/ECAI Workshop on Explainable Artificial Intelligence*, 2019.
- Khan et al. (2009)
Khan, O., P. Poupart, and J. Black, “Minimal sufficient explanations for factored markov decision processes,” in *International Conference on Automated Planning and Scheduling*, 2009.
- Laroche et al. (2019)
Laroche, R., P. Trichelair, and R. T. Des Combes, “Safe policy improvement with baseline bootstrapping,” in *International Conference on Machine Learning*, 2019.
- Liu et al. (2018)
Liu, G., O. Schulte, W. Zhu, and Q. Li, “Toward interpretable deep reinforcement learning with linear model U-trees,” in *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, 2018.
- Madumal et al. (2020)
Madumal, P., T. Miller, L. Sonenberg, and F. Vetere, “Explainable reinforcement learning through a causal lens,” in *Proceedings of the AAAI conference on artificial intelligence*, vol. 34, no. 03, 2020, pp. 2493–2500.
- Olson et al. (2021)
Olson, M. L., R. Khanna, L. Neal, F. Li, and W.-K. Wong, “Counterfactual state explanations for reinforcement learning agents via generative deep learning,” *Artificial Intelligence*, vol. 295, p. 103455, 2021.
- Seo (2025)
Seo, T., “UXsim: lightweight mesoscopic traffic flow simulator in pure Python,” *Journal of Open Source Software*, 2025.
- Sharma et al. (2024)
Sharma, A., L. Benac, S. Parbhoo, and F. Doshi-Velez, “Decision-point guided safe policy improvement,” *arXiv preprint arXiv:2410.09361*, 2024.
- Shu et al. (2018)
Shu, T., C. Xiong, and R. Socher, “Hierarchical and interpretable skill acquisition in multi-task reinforcement learning,” in *International Conference on Learning Representations*, 2018.
- Silva et al. (2020)
Silva, A., M. Gombolay, T. Killian, I. Jimenez, and S.-H. Son, “Optimization methods for interpretable differentiable decision trees applied to reinforcement learning,” in *International Conference on Artificial Intelligence and Statistics*, 2020.
- Sun et al. (2023)
Sun, H., A. Hüyük, D. Jarrett, and M. van der Schaar, “Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples,” *Conference on Neural Information Processing Systems*, 2023.
- Topin and Veloso (2019)
Topin, N. and M. Veloso, “Generation of policy-level explanations for reinforcement learning,” in *AAAI Conference on Artificial Intelligence*, 2019.
- van der Waa et al. (2018)
van der Waa, J., J. van Diggelen, K. van den Bosch, and M. Neerincx, “Contrastive explanations for reinforcement learning in terms of expected consequences,” *arXiv preprint arXiv:1807.08706*, 2018.
- Verma et al. (2018)
Verma, A., V. Murali, R. Singh, P. Kohli, and S. Chaudhuri, “Programmatically interpretable reinforcement learning,” in *International Conference on Machine Learning*, 2018.
- Wu et al. (2022)
Wu, J., H. Wu, Z. Qiu, J. Wang, and M. Long, “Supported policy optimization for offline reinforcement learning,” *Conference on Neural Information Processing Systems*, 2022.
- Yao et al. (2022)
Yao, J., S. Parbhoo, W. Pan, and F. Doshi-Velez, “Policy optimization with sparse global contrastive explanations,” *arXiv preprint arXiv:2207.06269*, 2022.
- Yau et al. (2020)
Yau, H., C. Russell, and S. Hadfield, “What did you think would happen? explaining agent behaviour through intended outcomes,” in *Conference on Neural Information Processing Systems*, 2020.
- Ziebart et al. (2008)
Ziebart, B. D., A. L. Maas, J. A. Bagnell, A. K. Dey *et al.*, “Maximum entropy inverse reinforcement learning.” in *AAAI Conference on Artificial Intelligence*, 2008.

## Appendix A Experimental Details

This appendix outlines the implementation details of our experiments. All experiments were run on a personal computer with 64GB of RAM and no GPUs. Each individual experiment took less than a couple hours to complete.

### A.1 Details for Providing Planning-Level Explanations

#### Approach 1: Links from Known Planner

In GridWorld, the state space consists of the agent’s position, represented as row and column indices $(i,j)$, and a binary flag ${f_{k}}$ for each key $k$, indicating whether the corresponding key has been retrieved. The action space consists of the four cardinal directions: up, down, left, and right. Transitions are deterministic. At each time step when the agent is not in the target cell, they receives a reward of $-1$. When collecting the most likely trajectory in Algorithm [1](#alg1), we let the agent to interact with the environment until they reach the target, which determines the value of $T$.

As the planner, we consider soft value iteration (Haarnoja et al., 2017), setting the discount factor as $\gamma=0.99$, the number of iterations as $250$, and the inverse temperature parameter as $\beta=100$. When computing strategic link scores, we constrain the decision $(s,a)$ by setting the reward value at that decision to negative infinity: $r(s,a)\leftarrow-\infty$. In GridWorld, we ignore flags $\{f_{k}\}$ and only consider the agent’s position $(i,j)$ to be part of the state. That is, we consider moving a specific direction in a specific cell to be the same decision, regardless of which keys have been picked up so far. Accordingly, when constraining the decision $(i,j,\{f_{k}\},a)$, we set $r(i,j,\{f^{\prime}_{k}\},a)\leftarrow\infty$ for all $f^{\prime}_{k}\in\{0,1\}$, independent of the values $\{f_{k}\}$.

#### Approach 2: Links from Demonstrations

For these experiments, we compute strategic link scores using reward functions inferred from demonstrations via IRL, rather than using the true reward functions. The specific scores we evaluate depends on the environment: For GridWorld, we consider the same score matrices as in “Links from Known Planner”; for Shorcuts, we consider the same scores that needed to be computed for the application in Section [5.2](#S5.SS2); and similarly for ArterialHighway, we consider the same scores computed in Section [5.3](#S5.SS3).

Although compute the same scores as in our applications, we use slightly smaller versions of each environments to reduce the computational demand of performing IRL: For GridWorld, we consider only the simple layout shown in Figure [3](#S5.F3); for Shortcuts, we generate $10$ environments with $5$ nodes, $3$ shortcuts, and $3$ preparation actions instead of $100$ environments with $10$ nodes, $5$ shortcuts, and $5$ preparation actions; for ArterialHighway, we consider a road network with $5$ junctions instead of $10$. In ArterialHighway, using soft value iteration requires us to quantize the continuous action space. We normally use $100$ quantization points, but for these experiments, we reduce it to $10$.

As demonstrations, we sample $10,\!000$ trajectories following optimal policies computed via soft value iteration. In GridWorld, we fix the time horizon of each trajectory to $T=W\times H$, where $W$ is the grid width and $H$ is the grid height; in Shortcuts, we fix $T=N\times L$, where $N$ is the number of nodes and $L$ is the number of preparation actions; and in ArterialHighway, we fix $T=J+1$, where $J$ is the number of junctions. For each environment, the number of soft value iteration steps is set accordingly. We vary the inverse temperature parameter to obtain demonstrations with different levels of stochasticity—see Figure [5](#S5.F5) for the specific temperature values considered.

As the IRL algorithm, we use maximum entropy IRL (Ziebart et al., 2008). The RL algorithm used in the inner loop of maximum entropy IRLis again soft value iteration with the same hyperparameters (number of iterations and inverse temperature) as those used for generating demonstrations. Since all our environments have deterministic transitions, we consider the setting where the true transition dynamics are known. Meanwhile, reward functions are represented as $|\mathcal{S}|$-by-$|\mathcal{A}|$ matrices; initialized to all zeros; and updated via gradient steps over $10,\!000$ iterations using the Adam optimizer with hyperparameters $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-8}$. The learning rate is set to $10^{-4}$ for GridWorld and Shortcuts, and to $5\cdot 10^{-3}$ for ArterialHighway.

### A.2 Details for Improving Policies Safely via Strategy-Aware Recommendations

In Shortcuts, the state space consists of the node that the agent is currently at, along with binary flags corresponding to each preparation action, indicating whether that action has been taken or not. The action space consists of moving forward (move), jumping via the $i$-th shortcut ($\{\texttt{jump}_{i}\}$), and or taking the $j$-th preparation action ($\texttt{prep}_{j}$). Letting $n$ be the current node and $f_{j}$ be the flag corresponding to action $\texttt{prep}_{j}$, the transition dynamics are given by the following rules:

$$ $\displaystyle n$ $\displaystyle\leftarrow n+1\quad$ $\displaystyle\text{if}\quad a=\texttt{move}$ $\displaystyle\wedge\mathcal{V}_{\texttt{move}}$ $\displaystyle\doteq\{n\neq N\}$ (7) $\displaystyle n$ $\displaystyle\leftarrow n_{\texttt{jump}_{i}}^{\text{(to)}}\quad$ $\displaystyle\text{if}\quad a=\texttt{jump}_{j}$ $\displaystyle\wedge\mathcal{V}_{\texttt{jump}_{i}}$ $\displaystyle\doteq\{n=n_{\texttt{jump}_{i}}^{\text{(from)}}\}\wedge\{f_{j}=1,\forall j\in\mathcal{J}_{\texttt{jump}_{i}}\}$ (8) $\displaystyle f_{j}$ $\displaystyle\leftarrow 1\quad$ $\displaystyle\text{if}\quad a=\texttt{prep}_{j}$ $\displaystyle\wedge\mathcal{V}_{\texttt{prep}_{j}}$ $\displaystyle\doteq\{n=n_{\texttt{prep}_{j}}\}$ (9) $$

where $\mathcal{V}_{a}$ denotes the validity condition for action $a$, $n_{\texttt{jump}_{i}}^{\text{(to)}}$ is the destination of shortcut $i$, $n_{\texttt{jump}_{i}}^{\text{(from)}}$ is the origin of shortcut $i$, $\mathcal{J}_{\texttt{jump}_{i}}$ is the set of preparation actions required to use shortcut $i$, and $n_{\texttt{prep}_{j}}$ is the node where action $\texttt{prep}_{j}$ is located at. Meanwhile, the reward function is given by

$$ $\displaystyle r(n,\{f_{j}\},\texttt{move})$ $\displaystyle=\begin{cases}-1&\text{if}\quad n+1\neq N\\ -1+N&\text{if}\quad n+1=N\end{cases}$ (10) $\displaystyle r(n,\{f_{j}\},\texttt{jump}_{i})$ $\displaystyle=\begin{cases}-1&\text{if}\quad\neg\mathcal{V}_{\texttt{jump}_{i}}\\ -(n_{\texttt{jump}_{i}}^{\text{(to)}}-n_{\texttt{jump}_{i}}^{\text{(from)}})+|\mathcal{J}_{\texttt{jump}_{i}}|\cdot 2C&\text{if}\quad\mathcal{V}_{\texttt{jump}_{i}}\wedge n_{\texttt{jump}_{i}}^{\text{(to)}}\neq N\\ -(n_{\texttt{jump}_{i}}^{\text{(to)}}-n_{\texttt{jump}_{i}}^{\text{(from)}})+|\mathcal{J}_{\texttt{jump}_{i}}|\cdot 2C+N&\text{if}\quad\mathcal{V}_{\texttt{jump}_{i}}\wedge n_{\texttt{jump}_{i}}^{\text{(to)}}=N\end{cases}$ (11) $\displaystyle r(n,\{f_{j}\},\texttt{prep}_{j})$ $\displaystyle=\begin{cases}-1&\text{if}\quad\neg\mathcal{V}_{\texttt{prep}_{j}}\\ -C&\text{if}\quad\mathcal{V}_{\texttt{prep}_{j}}\end{cases}$ (12) $$

We generate these environments randomly according to following procedure, letting $I$ be the number of shortcuts and $J$ be the number of preparation actions:

Figure: Algorithm 3 Shortcuts Environment Generation

When using soft value iteration to plan policies, we set the discount factor as $\gamma=0.99$, the number of iterations as $30$ (twice the total number of nodes and preparation actions), and the inverse temperature as $\beta=100$. When checking whether a strategic link score is significantly greater than zero, as in Line 5 of Algorithm [2](#alg2), we set the significance threshold to halfway between 0 and $1/5$ (one over the number of preparation action). This is because: When all preparation actions are recommended together, a soft-optimal policy assigns them equal probabilities at the initial state, since the order in which they are taken does not matter. Therefore, the strategic link score between any two preparation actions becomes at most $1/5$, as the probability of a recommendation cannot drop more than its original value after a constraint.

### A.3 Details for Characterizing Planning Behavior Through Interventions

#### Simulation

We refer to Seo (2025) for the details regarding the mechanisms underlying the traffic simulator. In our experiments, the simulation parameters are set as follows: $\texttt{deltan}=5$, $\texttt{reaction_time}=1$, $\texttt{duo_update_time}=500$, $\texttt{duo_update_weight}=0.5$, and $\texttt{duo_noise}=0.01$. Each road segment along the highway, as well as the on-ramps leading to it, has two lanes, a length of $1000$, and a free flow speed of $20$. Road segments along the arterial have the same number of lanes and the same length, but a lower free flow speed of $20\sqrt{J/(J+1)}$. We choose this value so that the arterial road is reasonably quicker (takes less overall time): If the speed on the arterial road were to be $20\cdot J/(J+1)$, both the arterial and highway routes would have taken the same amount of time as their length ratio is $J/(J+1)$. If it were to be $20$ instead, so that the two routes have the same speed, shorter length would have always meant a quicker route, requiring no strategic routing. The actual free flow speed is the geometric mean of these extreme scenarios. Finally, the incoming traffic flow into the road network through the “entry” is set to be $2.0$.

#### Reinforcement Learning

In addition to simulated agents, we also use RL to find the optimal routing policy that minimizes the average travel time throughout the whole road network. Each road segment has a travel time that depends on the traffic flow it carries, which we denote as $T_{A}(f)$ and $T_{H}(f)$ given flow $f$ for the arterial and the highway (including on-ramps) respectively. Since the incoming flow must be equal to the outgoing flow at each junction, the initial entry flow $f_{0}$ determines the flow throughout the network, conditioned on a routing policy $\pi$.
Letting $f_{A,J_{i}}$, $f_{R,J_{i}}$, and $f_{H,J_{i}}$ denote the traffic flow on the arterial segment following junction $J_{i}$, the on-ramp leaving that junction, and the highway segment that follows, we can write

$$ $\displaystyle f_{A,J_{1}}$ $\displaystyle=f_{0}\cdot\pi(J_{1})$ $\displaystyle\hskip 12.0ptf_{R,J_{1}}$ $\displaystyle=f_{0}-f_{A,J_{1}}$ $\displaystyle\hskip 12.0ptf_{H,J_{1}}$ $\displaystyle=f_{R,J_{1}}$ (13) $\displaystyle f_{A,J_{i}}$ $\displaystyle=f_{A,J_{i-1}}\cdot\pi(J_{i})$ $\displaystyle\hskip 12.0ptf_{R,J_{i}}$ $\displaystyle=f_{A,J_{i-1}}-f_{A,J_{i}}$ $\displaystyle\hskip 12.0ptf_{H,J_{i}}$ $\displaystyle=f_{H,J_{i-1}}+f_{R,J_{i}}$ (14) $\displaystyle={\textstyle\sum_{i^{\prime}=1}^{i}}f_{R,J_{i^{\prime}}}=f_{0}-f_{A,J_{i}}$ (15) $$

Notice that the flow rates past each junction, $f_{A,J_{i}}$, $f_{R,J_{i}}$, $f_{H,J_{i}}$, can be determined solely from the flow rate coming into that junction, $f_{A,J_{i-1}}$. Leveraging this structure, we find the policy $\pi(J_{i})$ that would result in the shortest average travel time efficiently via the following three steps:
First, we include in our state space the incoming flow $f$ to junction $J_{i}$, for which we can write the following deterministic transition function:

$$ $\displaystyle\tau(J_{i},f,a)=(J_{i+1},f^{\prime}=fa)$ (16) $$

Second, we use value iteration to compute the optimal flow-dependent policy $\pi^{*}(J_{i},f)$, which captures optimal routing under any flow condition, not just for our specific entry flow rate $f_{0}$.
Third, we roll this policy out, starting from the entry flow rate $f_{0}$, to obtain a flow-free policy $\pi^{*}(J_{i})$, which would remain optimal as long as the entry flow rate does not change:

$$ $\displaystyle\pi^{*}(J_{1})$ $\displaystyle\leftarrow\pi^{*}(J_{1},f_{0})$ $\displaystyle\hskip 24.0ptf_{A,J_{1}}$ $\displaystyle\leftarrow f_{0}\cdot\pi^{*}(J_{1})$ (17) $\displaystyle\pi^{*}(J_{i})$ $\displaystyle\leftarrow\pi^{*}(J_{i},f_{A,J_{i-1}})$ $\displaystyle\hskip 24.0ptf_{A,J_{i}}$ $\displaystyle\leftarrow f_{A,J_{i-1}}\cdot\pi^{*}(J_{i})$ (18) $$

We use the following reward function

$$ $\displaystyle r(J_{i},f,a)$ $\displaystyle=\underbrace{fa\cdot T_{A}(fa)}_{\text{arterial}}+\underbrace{f(1-a)\cdot T_{H}(f(1-a))}_{\text{on-ramp}}+\underbrace{(f_{0}-fa)\cdot T_{H}(f_{0}-fa)}_{\text{highway}}$ (19) $$

which ensures that cumulative rewards correspond to the average travel time across all road segments, weighted by the actual traffic flow carried by each segment. When using value iteration, we quantize the continuous actions space $\mathcal{A}=[0,1]$ using $100$ equally-spaced quantization points.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Authors should include at least two baselines (state-of-the-art) to verify their methods against them; if such methods do not exist, it should be clearly stated.

The manuscript mentions that inverse reinforcement learning (IRL) is used for strategic link scores using reward-based modeling, but it is not clear whether the strategic link score is used as an analysis tool applied to the results of IRL or whether it is integrated in the maximum entropy IRL objective function.

The paper's core objective is to analyze the cause-and-effect relationship between a setup action ($a$) and a future payoff action ($\tilde{a}$). This raises the causal relation among the actions and is fundamentally a question of causal influence. However, the manuscript fails to connect this work to the well-established and highly relevant field of Causal Reinforcement Learning [2], [3]. The authors should position their work within this literature, compare their proposed score to existing methods in Causal RL for identifying action influence, and justify why a new metric is needed.

[1] Bertoin, David, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. "Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning." Advances in neural information processing systems 35 (2022): 30693-30706.
[2] Seitzer, Maximilian, Bernhard Schölkopf, and Georg Martius. "Causal influence detection for improving efficiency in reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 22905-22918.
[3] Madumal, Prashan, Tim Miller, Liz Sonenberg, and Frank Vetere. "Explainable reinforcement learning through a causal lens." In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 03, pp. 2493-2500. 2020.
soundness: 2
strengths: The paper addresses the critical and challenging problem of policy interpretability in long-horizon sequential decision-making. The idea of quantifying the link between a "setup" action ($a$) and a "payoff" action ($\tilde{a}$) is an intuitive and valuable contribution to the field of Explainable RL (XRL).

The core contribution, the "strategic link score," is clearly and formally defined. This provides a concrete, computable metric for what is often not a very clear, qualitative discussion of "strategy."

The authors connect their proposed metric to an established field by showing how strategic link scores can be used to analyze the results of Maximum Entropy (MaxEnt) IRL, providing a practical use case.
summary: This paper introduces a novel interpretability metric, "strategic link scores," designed to quantify dependencies between actions in policies, $\pi$, derived from long-horizon reinforcement learning. The score is formally defined as the decrease in the likelihood of selecting an "enabling" action, $\pi(a|s)$, when a related, future "follow-up" action, $\tilde{a}$, is unavailable. The authors motivate this by noting that optimal policies, $\pi^*$, often involve short-term sacrifices (i.e., selecting $a$ where $r(s, a)$ is low) to enable long-term gains. The utility of this score is demonstrated through policy analysis in several tabular planning environments and within the Maximum Entropy (MaxEnt) Inverse Reinforcement Learning (IRL) framework.
weaknesses: The paper is not very clear on the computational expense of the calculation of the strategic link score. Can the authors elaborate on the computation cost and the increase in the complexity analysis? Possibly in $\mathcal{O}(|\mathcal{S}|, |\mathcal{A}|)$
 
I believe that the definition of set-up action and pay-off is not very rigorous and requires to be more mathematically grounded in order to increase the applicability of the method to more applications. Additionally, this definition should connect the actions clearly to the defined MDP.
 
Moreover, the limitations of the paper are not clearly stated. I believe the inclusion of the limitations would further improve the readability of the manuscript.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Positioning / novelty. How is the strategic link score not just causal influence / Q-value sensitivity / option precondition discovery? Can you provide (a) a small formal example where traditional credit assignment says two actions are related but your link is zero, and vice versa, and (b) a concise statement of what is genuinely new?

2. Computational tractability. What is the computational complexity of computing all pairwise link scores along a trajectory of length $T$? How many re-plans do you need, and how does this behave for large neural policies where replanning is nontrivial? Do you have an approximation approach (e.g. masking logits in a learned policy network instead of full replan)?

3. Quantitative evaluation. Can you report precision/recall or AUROC for recovering “true” enabling relations (e.g. key→door, shortcut-setup→shortcut-payoff) vs baselines such as (i) temporal proximity, (ii) pure co-occurrence frequency, or (iii) reward-drop ablations? This would turn your qualitative figures into measurable impact.

4. Robustness / false positives. Does a high link score always imply that the setup action is useless without the payoff, or can it also just mean that disabling the payoff collapsed a whole profitable branch of future behaviors that share structure with that payoff? In other words: how specific is S? Are there systematic false positives?

5. Clinical external validity. In Hypotension, the “reward” is a shaped physiological stability score rather than actual patient-centered outcomes, and the human decision-maker is simulated by randomly dropping some actions from the agent’s bundle. How confident are you that this result will generalize to (a) real ICU objectives like mortality and (b) real clinician partial adoption patterns (e.g. “I’ll titrate fluids but I won’t start pressors yet”)? Can you discuss limitations here explicitly, so readers don’t overinterpret?

6. Black-box setting. Section 6 is very interesting. Under what assumptions does the IRL-based reconstruction of strategic links match the ground-truth planner’s links (e.g. identifiability or coverage assumptions)? Right now Fig. 12 suggests performance can improve with more stochastic behavior, but this is only briefly described. Can you elevate and formalize this claim?
soundness: 2
strengths: ### 1. Clear, appealing problem statement.
The paper frames a real gap: existing explanations in RL mostly tell me why an action is “good long-term,” but not “this action is only worthwhile if we’ll later do that other action.” The authors focus on action-action contingency, which is intuitive to humans (“I grabbed the key only because I planned to open the door”). This is a crisp interpretability goal.

### 2. Actionable downstream use.
The decision-support angle is compelling. It’s not just interpretability for storytelling; the paper operationalizes it by grouping linked actions into bundles and shows that this “strategy-aware” interface can improve worst-case outcomes when a human only partially follows the agent’s advice, especially in environments with fragile multi-step maneuvers (Shortcuts), and at least does not hurt in a clinically flavored hypotension scenario.

### 3. Breadth of domains.
The method is demonstrated in (i) toy GridWorlds with keys and doors, (ii) a structured traffic network, (iii) a chess sequence with castling and follow-up play, (iv) a simplified ICU hypotension management setup. That breadth helps argue generality and shows the method isn't tied to a single planner or reward function.
summary: The paper proposes a method to expose multi-step strategies in sequential decision-making systems by defining a strategic link score between pairs of actions in different states. A link score $𝑆_{𝑠,𝑎→\tilde{s},\tilde{𝑎}}$ measures how much the policy’s probability of taking an earlier “setup” action $(𝑠, 𝑎)$ drops if a later “payoff” action $(\tilde{s},\tilde{𝑎})$ is made unavailable. Intuitively: “Would the agent still bother doing step 1 if it couldn’t later do step 2?” 

The paper argues this reveals strategic dependencies between actions, in a way that reward attribution / value explanations don’t. The paper (1) visualizes these links to interpret behavior in GridWorlds, chess, and traffic and (2) uses them to build “strategy-aware decision support,” where instead of recommending individual actions to a human operator, the system recommends bundles of linked actions so that partial compliance is less dangerous. This is tested in “Shortcuts” and a discretized hypotension treatment environment that mimics ICU decision support. 

Finally, the paper claims these links can be estimated even without direct access to the original planner, either by intervening in a simulator (traffic) or via IRL on demonstrations.
weaknesses: ### 1. Novelty vs existing ideas is under-defended.
I am not fully convinced (yet) that “strategic link score” is fundamentally distinct from prior notions like (i) causal influence of future constraints on earlier policy choices, (ii) credit assignment / Q-value sensitivity, (iii) options / skills / subgoal discovery. One might say: “You’re just measuring how much I stop wanting setup action A if payoff action B is impossible,” which sounds like standard counterfactual policy sensitivity. The paper needs a sharper, explicit contrast with causal RL explanations and hierarchical RL. Right now, that contrast is mostly implicit and could be challenged. 

### 2. Scalability / practicality.
Computing the link score requires re-planning under counterfactual interventions that forbid specific future actions at specific future states. That’s cheap in tabular / toy planners and in the softmax-over-Stockfish construction, but it’s unclear how this scales to high-dimensional continuous-control agents or giant neural policies without a fast differentiable planner. There is no complexity analysis, runtime table, or approximation method.

### 3. Validation is mostly qualitative.
Evidence that high link scores correspond to “true strategy” is presented as visual heatmaps (e.g. keys↔doors, shortcut pre-commitment, chess piece mobilization) and narrative interpretation. There’s no quantitative benchmark like precision/recall of discovered links against ground-truth enabling relations in GridWorld / Shortcuts, no human annotation study (“does this look like a setup-payoff pair?”), and no false-positive / false-negative analysis. That leaves room for the criticism that this is anecdotal. 

### 4. Chess experiment credibility.
The chess case study may look less impressive to experts, because the “policy” is generated by softmaxing Stockfish evals over legal moves rather than reading Stockfish’s true multi-ply principal variation. So it is not obvious that it is revealing latent plans; it might just be reconstructing obvious tactical follow-ups visible from engine eval deltas.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 4
questions: 1. In Figure 7, how is the baseline performance (black horizontal solid line) determined when you randomly generate the MDP? Is it the average of the performance of the 100 suboptimal agents? 
2. For pick-and-choose and strategy-aware recommendation, how do you pick which of the recommended (group of) actions to incorporate when the number of recommendations implemented is smaller than the number of recommendations.
3. I do not follow how, in Figure 12, the error in strategic link score can further go down when the error in reward function increases as stochasticity becomes the highest.
4. The paper writes: "This is by definition of link scores; every decision is strategically linked to itself with a score one." However, by Eq. 4, isn't the link score of a state-action pair $(s, a)$ to itself $\pi(a | s) - 0 = \pi(a | s)$? When $\pi(a | s)$ was not 1 to begin with, how is the link score to itself 1?
5. What is the shaded region in Figure 12?

### Concluding Comments
Despite the paper being very well-written, due to the critical issue (see Weaknesses 1), I have provided an initial score of 4.
soundness: 3
strengths: - The paper is visually appealing and very well-written, and key concepts are explained very clearly.
- Claims are well supported by evidence through well-explained demonstrations in a number of environments.
- The strategic link score could be of interest to researchers in explainable RL.
summary: The paper addresses the problem of explaining RL policies at the planning level, and proposes the strategic link score, which captures the temporal dependencies of the state-action pairs of a policy in terms of set-up and pay-off. The paper demonstrates how the strategic link score captures such dependencies in GridWorld and Chess environments. The paper also applies the proposed score for action recommendation, where the proposed score enables an appropriate level of granularity, forming a nice middle ground between recommending atomic actions and recommending full policies. The paper further demonstrates the utility of the proposed score for general non-reward-based planners, e.g., traffic simulators.
weaknesses: 1. There seems to be a critical issue with the proposed method, which is that the strategic link score seems to fail whenever there exists more than one possible future pay-offs.
  - To give a concrete example, consider a GridWorld environment similar to that in Figure 3a. But now, instead of having a single door, there are two neighboring doors unlocked by the same key leading to the shortcut. Now, if we block one of the doors, the agent can still access the other door, so picking up the key is still part of the optimal strategy. Then, the probability of picking up the key is not decreased, and the strategic link score is 0, failing to connect the action of picking up keys to either of the future doors. 
  - Then if we were to perform strategy-aware recommendation in this setting, it is possible for the agent to adopt the recommendation of picking up the key without adopting the actions of opening doors, causing a failure similar to pick-and-choose.
  - Whenever the future pay-off of a set-up can be realized in more than one way, the strategic link score fails to link the set-up to future pay-offs. This is because the score can only handle one-to-one correspondence of set-up and pay-off, and is incapable of linking a set-up to a combination of parallel/independent pay-offs.
2. The strategic link score seems to be limited to small tabular environments.
  - Assuming access to a planner (e.g., an RL algorithm) that returns a policy given an environment, to compute the strategic link score, you first use the the planner to return a policy, which is then used to generate a trajectory that is most likely under the returned policy. Then, for every state-action pair $(s, a)$ in the trajectory, you re-solve the environment while forbidding $(s, a)$ from being chosen by the policy. Then, the complexity of the method is $O(TP)$, where $T$ is the length of the trajectory, and $P$ is the complexity for the planner to solve the environment. In RL, for example, you would need to run an RL algorithm to solve MDPs $T$ times.
  - Such computation seems intractable for large MDPs that's costly to solve, and MDPs that have a long/infinite horizon. 
  - Also, can the authors comment on the method's application on continuous state or action spaces? Is there a way to forbid an action in a continuous action space?
3. In relation to the above weakness, limitations is under-discussed in the paper. 
4. Metrics of statistical significane (e.g., confidence intervals) are not reported in Figure 7.


Some minor issues

5. In the final paragraph on page 5, "In the second layout (Figure 4a)". Should be 4c instead of 4a.
6. On line 288, there's an unfinished sentence: "We can make this interpretation because"
7. Typo in the caption of Fig. 10: "When it comes to the decision of saying"    saying $\longrightarrow$ staying
8. First letter of "markov" should be capitalized in the references.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Could you comment on the connection between your concept and the concepts from the planning literature above?
2. Could you comment on the soundness concerns?1. The related work ignores the literature on the same and related concepts that come from the field of planning
2. The authors present a very simplistic and unrealistic view of planning, as actions that are there to enable other actions in pairs
3. The main concept, "strategic link score" is not well-justified, as I mention above, as it is not a function of two actions
4. As a result of the former, the empirical evaluation does not provide meaningful insights
soundness: 1
strengths: 1. The authors present the ideas clearly
2. The paper include formal definitions
3. The paper presents a wide empirical evaluation
summary: # Summary

The paper focuses on quantifying the causal dependencies between actions on a plan (policy).It formalizes the concept of strategically linked actions and defines strategic link score of two actions (producer and consumer).

It is worth noting that causal links were introduced in planning more than 30 years ago [1].
It seems like the authors have a very simplistic view of planning. While I understand that your setting does not assume state features, consider for a moment state features and actions having preconditions (in the simplest form, which of the state features are true) and effects (change in state features), as in e.g., classical planning. Then, an action would "produce" a feature that some other actions along the way would "consume" (require in the precondition). But it does not have to be pairs of actions, in the sense that a single action can achieve precondtions used by multiple actions along a plan (e.g., two doors opened by the same key), there can be alternative plans of the same cost that would use different actions that consume the same precondition (like two alternative doors). My point is, this property is not a property of action pairs. Looking at action preconditions and effects makes you see it more clearly. There was quite a body of research in planning on causal links, might worth a look.
Another closely related topic is action justification on a plan (e.g., [2,3,4,5,6]), talking about the reason for an action to be on a plan (providing some precondition for at least one following action or achieving a goal fact). The literature describes perfectly justified plans as a plan that cannot be reduced by removing actions and keeping it a plan. It seems like the authors are looking for similar concepts.


# Soundness

The main idea, strategic link score, as defined in Equation (4) does not make sense for planning in general, for the reasons described above. For instance, it is not hard to craft examples where the score will not drop when only one "pay-off" action is blocked. Of course, there are families of problems for which that is not possible, but I am not sure how to characterize such problems. 

# Novelty

Novelty is limited, for the reason expressed above.

# Scholarship

[1] Weld, D. S. (1994). An Introduction to Least Commitment Planning. AI Magazine, 15(4), 27.
[2] Fink, E.; and Yang, Q. 1992. Formalizing Plan Justifications. In Proc. CSCSI 1992.
[3] Fink, E.; and Yang, Q. 1993.  A spectrum of plan justifications.  In Proceedings of the AAAI 1993 Spring Symposium,23–33.
[4] Lindner, F.; and Olz, C. 2022.  Step-by-Step Task Plan Explanations Beyond Causal Links. In31st IEEE InternationalConference on Robot and Human Interactive Communication, RO-MAN 2022
[5] Sreedharan, S.; Muise, C.; and Kambhampati, S. 2023. Generalizing Action Justification and Causal Links to Policies. In Proceedings of the Thirty-Third International Conference on Automated Planning and Scheduling (ICAPS).
[6] Salerno, M.; Fuentetaja, R.; and Seipp, J. 2023. Eliminating Redundant Actions from Plans using Classical Planning. In Proc. KR 2023, 774–778.

# Clarity

The presentation of ideas is sufficiently clear.

# Evaluation and Reproducibility

The evaluation is quite extensive, but meaningfulness of the results is questionable.
weaknesses: 1. The related work ignores the literature on the same and related concepts that come from the field of planning
2. The authors present a very simplistic and unrealistic view of planning, as actions that are there to enable other actions in pairs
3. The main concept, "strategic link score" is not well-justified, as I mention above, as it is not a function of two actions
4. As a result of the former, the empirical evaluation does not provide meaningful insights


Decision: Reject