Predicting Future Actions of Reinforcement Learning Agents
Authors: Stephen Chung
Scott Niekum
David Krueger
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe reinforcement learning
deep reinforcement learning
agent predictability
Primary area: reinforcement_learning
TL;DR: We evaluate two approaches to predict the future actions of agents trained by different types of reinforcement learning algorithms.

Abstract: As reinforcement learning agents become increasingly deployed in real-world scenarios, predicting future agent actions and events during deployment is important for facilitating better human-agent interaction and preventing catastrophic outcomes. This paper experimentally evaluates and compares the effectiveness of future action and event prediction for three types of RL agents: explicitly planning, implicitly planning, and non-planning. We employ two approaches: the inner state approach, which involves predicting based on the inner computations of the agents (e.g., plans or neuron activations), and a simulation-based approach, which involves unrolling the agent in a learned world model. Our results show that the plans of explicitly planning agents are significantly more informative for prediction than the neuron activations of the other types. Furthermore, using internal plans proves more robust to model quality compared to simulation-based approaches when predicting actions, while the results for event prediction are more mixed. These findings highlight the benefits of leveraging inner states and simulations to predict future agent actions and events, thereby improving interaction and safety in real-world deployments.

Predicting Future Actions of Reinforcement Learning Agents
Stephen Chung"
Scott Niekum
David Krueger Mila
University of Cambridge
University of Massachusetts Amherst
Abstract
As reinforcement learning agents become increasingly deployed in real-world scenarios, predicting future agent actions and events during deployment is impor- tant for facilitating better human-agent interaction and preventing catastrophic outcomes. This paper experimentally evaluates and compares the effectiveness future action and event prediction for three types of RL agents: explicitly planning, implicitly planning, and non-planning. We employ two approaches: the inner state approach, which involves predicting based the inner computations of the agents (e.g., plans or neuron activations), and simulation-basel approach, which involves unrolling the agent learned world model. Our results show that the plans of explicitly planning agents are significantly more informative for prediction than the neuron activations of the other types. Furthermore, using internal plans proves more robust to model quality compared to simulation-bart when predicting actions, while the results for event prediction are more mixed. These findings highlight the benefits of leveraging inner states and simulations to predict future agent actions and events, thereby improving interaction and safety real-world deployments.
approaches
Introduction
As reinforcement learning (RL) becomes increasingly applied in the real world, ensuring the safety and reliability RL agents is paramount. Recent advancements have shown that agents can exhibit complex behaviors, making crucial to understand and anticipate their actions. This is especially important in scenarios where misaligned objectives [ or unintended consequences could result in suboptimal or even harmful outcomes. For instance, consider an autonomous vehicle controlled by an RL agent that might unpredictably decide to run a red light to optimize travel time. Predicting this behavior in advance would enable timely intervention to prevent potentially dangerous situation. This capability is also beneficial in scenarios that require effective collaboration and information exchange among multiple agents [244]. For example, if passengers and other drivers know whether self-driving car will turn left or right, it becomes much casier and safer to navigate the roads. Thus, the ability accurately predict an agent's future behavior can help reduce risks and ensure smooth interaction between agents and humans in real-world situations.
In this paper, explore the task of predicting future actions and events when deploying trained agent, such as whether an agent will turn left in five seconds. The distribution of future actions and events cannot be computed directly, even with access the policy, because the future states unknown. We consider two methods for predicting future actions and events: the inner state approach and the simulation-basee approach. We apply these approaches to agents trained with various RI. algorithms to assess their predictability!!
"Correspondence mhc488cam. ac.uk
Full code is available at /httts://ggithii http:ttw..........eaaaaaaaaaaaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In the inner state approach, we assume that we have full access to the inner state of the agent during deployment. Here, the inner state refers to all the intermediate computations required determine the final action executed by the agent, such as the simulation of the world model for explicit planning agents or the hidden layers for agents parametrized by deep neural networks. We seek to answer the following questions: (i) How informative are these inner states for predicting future actions and events? (ii) How does the predictability of future actions and events vary across different types of RL agents with different inner states?
As an alternative to the inner state approach, we explore a simulation- approach by unrolling the agent in a learned world model and observing its behavior. Assuming we have a sufficiently accurate world model that resembles the real environment, this simulation should provide valuable information for predicting future actions and events in the real environment. We seek to answer the following question: (iii) How do the performance and robustness of the simulation-base approach compare to the inner state approach in predicting future actions and events across different agent types?
We conduct extensive experiments to address the above research questions. To summarize, the main contributions of this paper include:
1. To the best of our knowledge, this is the first work to formally compare and evaluate the predictability of different types of RL agents in terms of action and event prediction.
2. We propose two approaches to address this problem: the inner state approach and the simulation-basel approach.
3. We conduct extensive experiments to evaluate the effectiveness and robustness of these approaches across different types of RL agents, demonstrating that the plans of explicitly planning agents are more informative for prediction than other types of inner states.
2 Background and Notation
We consider Markov Decision Process (MDP) defined by a mple (SP.P 7, do), where S is a set of states, A is a finite set of actions, P S x A x S - [0, 1] is a transition function representing the dynamics of the environment, R S x A - R is a reward function, 7 [0, 1] is a discount factor, and do S + [0, 1] is an initial state distribution. Denoting the state, action, and reward at time by S,, Ar, and Rg respectively. P(s, a, s') Pr(( s s. A, a). R(s,a) E[R[ = s, A₁ = a]. and (s) = Pr(So = s), where / and do are valid probability mass functions. An episode a sequence of (S,, A.. R2), starting from reaching the terminal state, special state where the environment ends. Letting G. >x= > denote the infinite-horizon discounted return accrued after acting at time 4, an RL algorithm attempts to find, or approximate, policy : S A [0, 1). such that for any time / > 0. selecting actions according to ग (s, a) = Pr(A, S,S, s) maximizes the expected return [T.
and continuing until
In this paper, planning refers the process of interacting with an environment simulator world model to inform the selection of subsequent actions. Here, a world model is a learned and approximated version of the environment. We classify an agent, which is defined by its policy, into one of the following three categories based on the RL algorithm by which it is trained:
Explicit Planning Agents. In explicit planning agents, an environment simulator or world model used explicitly for planning. We consider two explicit planning agents this paper, MuZero 5 and Thinker (6) given their superior ability in planning domains. MuZero is state-of-the-arr model-based RL algorithm that combines a Icarned model with Monte Carlo Tree Search (MCTS) g for planning. During planning, MuZero uses the learned model to simulate future trajectories and performs MCTS to select the best action based on the predicted rewards and values. Thinker is a recently proposed approach that enables RL agents to autonomously interact with and use learned world model to perform planning. The key idea of Thinker is to augment the environment with a world model and introduce new actions designed for interacting with the world model. MuZero represents a handcrafted planning approach, while Thinker represents a learned planning approach.
Implicit Planning Agents. In implicit planning agents, there is no learned world model nor an explicit planning algorithm, yet these agents still exhibit planning-lik behavior. A notable example the Deep Repeated ConvLSTM (DRC) 99, which excels in planning domains. DRC agents are trained

---

by actor-critic algorithms [T0] and employ a unique architecture based on convolutional-LSTT with internal recurrent steps. The authors observe that the trained agents display planning like properties. such as improved performance with increased computational allowance, and so argue that the agent learns to perform model-free planning-
Non-plane Vonnplanning Agents. In non-planning agents, there is neither a learned world model nor an explicit planning algorithm, and these agents do not exhibit planning-like behavior. Typically, these agents perform poorly in planning domains. Examples include most model-free RL algorithms,, such as the actor-critic and Q-learning. In this paper, we focus exclusively on IMPALA [0], a variant of the actor-critic algorithm, chosen for its computational efficiency and popularity.
We believe that this distinction between RL agents, adopted from previous work [T]. is useful for investigating their predictability. We hypothesize that the plan made by an explicit planning agent should be more informative of future actions events than that of implicit planning agent, the plan in an explicit planning agent typically human-interpretablee whereas the plan for an implicit planning agent stored in hidden activations. Nevertheless, the computation in these two types of agents provides indications of their future actions and thus should carry more information than the hidden activations of non-planning agent, which lacks future plans and may merely serve more compact representation of the state.
Problem Statement
Given a fixed policy we aim to estimate the distribution of a function of the future trajectory. For example, we may want to estimate the probability of agent entering a particular state or performing specific action within a certain horizon. Mathematically, let H; (St. A,, Re) denote the transition at step I, and let H.T- = {He, He+t,..., Hr} denote the future trajectory from step / to the last step T. Let H denote the set of all possible future trajectories. We are interested in estimating the distribution a random variable [HHTT conditioned on the current state and action:
F(f(H(r) At),
(I)
where H + T is function specifying the variables to be predicted.
This paper focuses on predicting tion prediction, where we want to predict the action of the agent in L steps, i.c., f(Hrr) = (A44),442,.... A.4.) and the problem becomes estimating:
particular types of information. The first type is future ac-
P(Ac+1, Ar+2...... 1 S., Ar).
An example of action prediction is whether an autonomous vehicle is going to turn left or right in the next minute. The second type is futture event prediction, where we want estimate the probability of a binary indicator AS) + {0,1) being active within L steps, and the problem becomes estimating:
(Ü (( k=1
O Actx) = 1 Sr, At..Al)
P
which is equivalent to the case [(H.T) = max g(S.+k; probability of the event defined by g occurring within L steps. An example of event prediction is predicting whether an autonomous vehicle will run red light within a minute.
In other words, 3 the
Event prediction shares resemblance to the generalized value function [12]. where f(Her) Σ * (Sr+*,, Ac+1) and we estimate its expectation ([ St, At). When g is binary indicator, this expectation is equivalent to the discounted sum of the probabilities of the event defined by This is arguably harder to interpret than (3); for example, it can be larger than and thus is not a valid probability.
To learn these distributions, we assume access to some transitions generated by the policy л as training data. The transitions may come from multiple episodes. In the case of future event prediction, we assume that g(,,, A²) is also known for each transition. We further assume that the policy π and the inner computation for each action A, within is known.
In this work, we assume that π is already a trained policy and is fixed. This is the case where the agent is already deployed, and transitions during the deployment are collected. In cases where the training and deployment environments are similar, we can also use the transitions when training the agent training data for predicting future actions and events, but this is left for future work.
3

---

4 Methods
Since we already have the state-action (S,, At) and the target output ((H))) ) in the training data, we can treat the problem supervised learning task P In particular, We can train neural network that takes the state-action pair as input to predict ((H).). This network trained using gradient descent on cross-entropy loss.
Besides the state-action, there can be additional information that may help the prediction. For example, the inner computation of the policy may contain plans that are informative of the agent's s future actions, especially in the case of an explicit planning agent. We refer to this information that available before observing the next state Si+1 as auxiliary information and denote it as Iz. We will consider two types of auxiliary information: inner states and simulations.
4.1 Inner State Approach
In the inner state approach, we consider choosing the agent's 's inner state as the auxiliary information. Here, the inner state refers to all the intermediate computations required to compute the action A, As inner states different across different types of agents and may not all be useful, we consider the following inner state to be included in the auxiliary information:
1. MuZero: Since MuZero uses MCTS to search in a world model and selects action with the largest visit count, we select the most visited rollout as the auxiliary information. Rollouts here refer to the simulation of the world model and are composed of sequence of transitions ($e41, ALtz, Re+))<1gE It should be noted that the agent may not necessarily select the action sequence of this rollout, the MCTS is performed every step, and the search result at the next step may yield different actions. We do not use all rollouts, as MCTS usually requires many rollouts.
2. Thinker:
We select all rollouts and tree representations during planning the auxiliary information. We do not choose a particular rollout because, unlike MCTS, in Thinker, is generally unknown which action the agent will select at the end, and Thinker usually requires only a few rollouts.
3. DRC: We select the hidden states of the convolutional-LSTM at every internal tick as the inner state, as it was proposed that the hidden state contains plans guide future actions (
4. IMPALA: We select the final layer of the convolutional network as the inner state, is neither too primitive which may only be processing the state, nor too refined which may only contain information for computing the current action and values.
Experiment results on the alternative choices of inner states can be found in Appendix D
4.2 Simulation- Approach
As an alternative to the inner state approach, we can train a world model concurrently with the agent. Once trained, we can simulate the agent in this world model using the trained policy ı generate rollouts. These rollouts can then be utilized as auxiliary information for the predictor. In this paper, consider using the world model proposed in Thinker (6. which is RNN that takes the current state and action sequence as inputs and predicts future states, rewards, and other relevant information. For both implicit and non-planning agents, the world model is trained in parallel with the agents but is not used during their selection of actions. Instead, the world model is solely employed to generate rollouts auxiliary information for the predictors.
the learned world model closely resembles the real environment, we expect these rollouts to yield valuable information for predicting future actions and events, as the agent's behavior in the world model should be similar to its hehavior in the actual environment. In the ideal case where the world model perfectly matches the environment dynamics, we could compute the exact future action and event distribution without needing any prediction. However, we do not consider this scenario in the paper, as this assumption is impractical for most settings.
'Temporal- "Temporal-difference difference methods are not directly applicable here, as both the action and event prediction tasks involve limited horizon L and do not sum over variables.

---

It should be noted that in the simulation- paselttion--bedd approach. the world model must always predict the same state space as the input to the agent, enabling the simulation of the agent within the world model. Since agents typically receive raw state inputs (with exceptions such as Dreamer [3], where agents receive abstract state inputs), the world model should also make predictions in the raw state space rather than in latent state space.. Consequently, the simulation- based approach may not be suitable for situations where learning a world model in the raw state space is challenging, such as predicting camera input in real-world autonomous driving scenarios.
5 Experiments
We conduct three sets of experiments to evaluate the effectiveness and robustness of the discussed approaches. First, we apply the inner state approach to predict future actions and events. We compare it to the case where only state-action information is provided to the predictor so as to evaluate the benefits of the proposed inner state in the prediction. Second, we apply the simulation- based approach and compare it with the inner state approach to evaluate the benefits of these two different types of auxiliary information. Finally, we consider model ablation setting, where deliberately make the world model inaccurate to see how the different approaches perform under such conditions.
We consider the Sokoban environment, where the goal is to push all four boxes into the four red- bordered target spaces as illustrated in Fig [ We choose this environment because (i) a wide range of levels in Sokoban make action and event prediction challenging. and we evaluate the predictors on unseen levels to evaluate their generalization capability; (ii) there are multiple ways solving a level: (iii) Sokoban is a planning- based domain, so it may be closer to situations where we want to discern plans of agents in more complex settings.
Figure 1: Example levels of Sokoban, where the goal is to push all four boxes into the four red- bordered target spaces. box can only be pushed, not pulled, making the level irrecoverable if the boxes get stuck. We paint a random empty space blue (which still acts as an empty tile) and predict whether the agent will stand on the blue location within steps.
We choose the prediction horizon L to be 5 in all experiments. For action prediction, try to predict the next five actions A1+1, A ..., A445. For event prediction, we randomly select an empty tile in the level and paint it blue. That blue tile acts an empty tile to the agent and serves no special function. We define the event that is to be predicted as the case where the agent stands on the blue location. In other words, try to predict whether the agent will go that blue location within L steps.
We train four different agents using MuZero, Thinker, DRC, and IMPALA. All agents are trained for 25 million transitions. To ensure that the result would not be affected by the particular choice of the world model, we uniformly employ the world model architecture proposed in Thinker, as the world model in Thinker predicts the raw state and is suitable for both MuZero and the simulation based approach. We train separate world model for each agent. For DRC and IMPALA, the world model is not needed for the policy and will only be used in the predictors in the simulation-
approach.
After training the agents, we generate 50k transitions, where part or all of it will be used as the training data for the predictors. We evaluate the performance of predictors with varying training data sizes: 1k, 2k, 5k, 10k, 20k, 50k. We also generate 10k transitions as a testing dataset. For simplicity, we use greedy policies, where we select the action with the largest probability instead of sampling. The predictor uses a convolutional network to process all image information, including the current state and states in rollouts (if they exist). The encoded current state, along with other auxiliary information such as encoded states, rewards, and actions in rollouts (if they exist), will be passed to
5

---

three-layer Transformer encoder [14], and the final layer predicts the next L actions or the probability of the event within steps. More details about the experiments can be found in Appendix A
5.1 Inner State Approach
Tigure2preseer the final accuracy of action prediction and the Fl score of event prediction using the inner state approach. The error bars represent the standard deviation across three independently trained predictors. The accuracy here refers to the percentage correctly predicting all the next five actions, with credits awarded if any action is predicted incorrectly. The graph also shows the performance of the predictors when they only receive the current state St and action A, as inputs, as indicated by 'baseline'. Several observations can be made.
First, when access to the plans is available, the prediction accuracy for both action and event is significantly higher for explicit planning agents. For example, with 50k training data, the action prediction accuracy of the MuZero agent increases from 40% 87% when given access to the plans. Agents using handcrafted planning algorithms (MuZero) or learned planning algorithms (Thinker) show similar performance gains. This is perhaps not surprising, as these explicit planning agents tend to follow the plans either by construction or by learning, and the explicit nature of planning facilitates easy interpretation of the plans.
Second, the case for implicit planning agents (DRC) and non-ppanning agents (IMPALA) is more nuanced. For action prediction accuracy, both receive moderate improvement from accessing the hidden state. There are two possible explanations: (i) plans of the agents are stored in the learned representations that are informative of future actions; (ii) the hidden states or hidden layers contain a latent representation that is easier to learn from, compared to the raw states. To discern between the two cases, interpreting the nature and the underlying circuit of the inner states is required. We leave this to future work interpretability is outside the scope of this paper.
Third, in contrast action prediction, the inner state does not improve event prediction for DRC and IMPALA, likely because the blue location in the environment does not affect the reward, and the agent may ignore it in its representation. This suggests an advantage of explicit planning agents, as in explicit planning agents, we explicitly train the world model and can train it to attend not just to the reward-relevant features but to all features (or features we deem useful) in the environment. This may be important for cases where the reward function is not well designed, leading to the agent ignoring certain features that are, in fact, important to us.
Accuracy of Action Prediction
F1 Score of Event Prodiction
1.0
0.6
FJ Scoee
0.4
0.2
0.0
160
10a
10a
Training Dara Size
Taining Dota Size
MuZem Inner state Trincer- - Inner state
BRC Inner state
IMPLAA Inner state IMKLAA bassline
HuZcro bascine
Trinker baseine
DRC base inc
Figure 2: Final accuracy of action prediction and F1 score of event prediction with inner state approach on the testing dataset. The error bar represents two standard errors across 9 seeds.
5.2 Simulation-based Approach
We now consider applying the simulation-base approach to both implicit planning (DRC) and non-planning agents (IMPALA). We unroll the world model for L= 5 steps using the current policy and input this rollout as auxiliary information to the predictors. We can use a single rollout, as both the policy and the chosen world model are deterministic, all rollouts will be the same.

---

We do not apply the simulation- manued approach to explicit planning agents because (i) rollouts already exist as an inner state within the agent and can be input to the predictor, and (ii) requires training a world model that can be unrolled for 27, steps instead of only I. steps, as the agent needs to perform planning on every step in the simulated rollout. For a fair comparison, we assume we can only train a world model that is unrolled for I. steps in all setups.
Figure shows the final accuracy of action prediction and the Fl score of event prediction for the simulation- prasee approach of DRC and IMPALA. For easy comparison, we also include the result of the inner state approach of explicit planning agents in the figure.
Accuracy Action Prediction
Score Fvent Prediction
1.0
0.8
0.6
15500
0.4
0.2
10a
10a
10
10
Training
Taining
MuZero
Thinker state
DRC simulition
IMPILA smulttioo IMILLA- basaline
MuZero baselne
Thinker basalnn
DNC baseinn
Figure 3: Final accuracy of action prediction and Fl score of event prediction with simulation-bass approach (DRC and IMPALA) on the testing dataset. The absolute performance can be found in Appendix C. The error bar represents two standard errors across seeds.
We observe that the predictors for DRC and IMPALA agents in the simulation basedd approach perform very well, with performance surpassing that of the explicit planning agents with the inner state approach. This because the world model we trained is very close to the true environment, so the behavior in the rollout is almost equivalent to that in the real environment. The high-quality world model also enables accurate prediction of when the agent will stand on the blue location, resulting in excellent event prediction performance.
5.3 World Model Ablation
Learning an accurate world model may not be feasible in some settings, such as auto-driving in the real world. An inaccurate world model will affect the plan quality of explicit planning agents, rendering the plan less informative in the inner state approach. An inaccurate world model will also affect the quality of rollouts in the simulation-based approach, leading to inconsistent behaviour between rollouts and real environments. As such, it is important to understand how the inner state approach and simulation-baase approach differ when the learned world model is not accurate.
To investigate this, we designed three different settings where learning an accurate world model is challenging. In the first setting, we use world model with much smaller size, making it more prone to errors. the second setting, we randomly replace the agent's action with a no-operation 25% of the time, introducing stochastic dynamics into the environment. However, since the world model we use is deterministic, it cannot account for such stochastic transitions and will yicld errors. In the third setting, consider partially-observaal randomly display the character at a position within one step of the true character location. As the world model we only observes the current state, this will lead to uncertainty over both the true character location and the displayed character location. We repeat the above experiments in these three different settings.
Markov decision process (POMDP) case, where
Figure 4 shows the change in the final accuracy of action prediction and the score of event prediction for the model ablation settings compared to the default setting. We observe that in terms of action prediction, the accuracy generally drops less in the inner state approach of explicit planning agents than in the simulation- based approach of the two other agents. This is likely because planning
7

---

does not necessitate accurate world model, as plans can still be made without the ability to perfectly predict the future. For example, MCTS, only values and rewards need to be predicted well, but the state. In contrast, if we simulate an agent in poor world model, the agent may be confused as the states may be out of distribution and never encountered during training. This leads to inconsistent behavior compared to the agent's 's behavior in the real environment.
In contrast, the results for event prediction are more nuanced, with the inner state approach sometimes performing better and the simulation-bain based approach performing better at other times. We conjecture that because the world model is not accurate, the event under consideration often not predicted correctly. As such, more informative plans that can predict future actions do not help in event prediction, leading mixed results.
0.2
0.2
A0om Oharge 9r.8mt
-0.2
-0.5
0.3
Risen Chang te tweee Preditton
-0.2
-04
-10
Figure 4: Change in the final accuracy of action prediction and FI score of event prediction for the world model ablation settings. The error bar represents two standard errors across 3 seeds.
Related Works
Our work is related safe RL, where try to train agent to maximize rewards while satisfying safety constraints during the learning and/or deployment processes [15). A wide variety of methods have been proposed in safe RL, such shielding, where manually prevents actions that violate certain constraints from being executed [16), and Constrained Policy Optimization (CPO), which performs policy updates while enforcing constraints throughout training [17). Many works in
Safe RL:
safe RL. are based on correcting the action when deemed unsafe. Dalal et al. [8] fit linear model to predict the violation of constraint functions and use it to correct the policy; Cheng al.
[99 project
the learned policy to safe policy based on the barrier function; Thananjeyan et al.
20 guide the
agent back to learned recovery zones when it is predicted that the state-action pair will lead to unsafe regions; Thomas et al. 21 use world models to predict unsafe trajectories and change the rewards to penalize safety violations.
In contrast to the works in safe RL, we are solely interested in predicting future actions and events of trained agents. The actions events do not necessarily need to be unsafe. In the case of unsafe action or event prediction, our work allows for preemptive interruption of the deployed agent, which be used last resort in addition the above safety RL works.
Opponent Modelling in Multi-agent Setting: In a multi-agent setting, modeling the opponent's behavior may be beneficial in both competitive and cooperative scenarios. He et al. [22 use the
opponent's inner state to better predict Q-values multi-agent setting. Foerster et al. 2 update
an agent's policy while accounting for its effects other agents, and Raileanu et al. 24 predict the

---

other agent's 's actions based on the same network that outputs the agent's own action. In contrast to these works, our research involves predicting agent actions multiple steps ahead and does not involve a multi-agent setting or learning policy.
Recent research has highlighted the importance of predictability in enhancing human-agen interaction and collaboration. The agents in these studies are not necessarily RL agents but are often hardcoded to follow certain rules. Daronnat al. B demonstrated that higher predictability in agent behavior facilitates better human-agent interaction and collaboration, particularly in real-time scenarios. Dragan al. 2 found that legible motion, which makes an agent's 's intent clear, leads to more fluent human-robot collaboration. Kandul et al. [25] found that humans are better at predicting human performance than agent performance, raising concerns about human control over agents in high-stakes environments. Finally, Ahrndt et al. 4 discussed the significance of mutual predictability in auman-agent teamwork. These works support the motivation that predictability of agents is important concern for human-agent interaction.
Predictability for Human-Agent Interaction:
7 Conclusion
In this paper, we investigated the predictability of future actions and events for different types of RL agents. We proposed and evaluated two approaches for prediction: the inner state approach and the simulation-basee based approach. The simulation-bal model but is less robust when the world model quality is compromised. Conversely, the performance of the inner state approach depends on the type of inner states and the agents. Internal plans of explicit planning agents are particularly useful compared to other types of inner states. These findings highlight the importance of leveraging auxiliary information to predict future actions and events. Enhanced predictability could lead to more reliable and safer deployment of RI agents in critical real-world applications such as autonomous driving, robotics, and healthcare, where understanding and anticipating agent behavior is important for safety and effective human-agent interaction.
approach performs well with an accurate world
Future research directions include extending our analysis to more diverse environments and RL algorithms, exploring safety mechanisms to modify agent behavior based on action prediction, and developing RL algorithms that are both predictable and high- tigh-performing.
Limitation
The paper only evaluates the proposed approaches in a limited set of environments. Including additional enviromments would provide better understanding of agent predictability, but this requires finding or designing new benchmark environments with diverse states. Additionally, the paper focuses on only four different RL algorithms. Evaluating broader range of RL algorithms could allow for better comparisons of their predictability.
Broader Impact Statement
This work involves predicting the actions and events of trained agents during deployment, It is important to consider the risk of false alarms, where the predictor forecasts that an agent is going to perform an unsafe action, but in fact, the agent would not be doing it. This may lead to improper responses (such as shutting down the agent) that are not warranted.
References
[1] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004-12019, PMLR, 2022.
[2] Anca D Dragan, Shira Bauman, Jodi Forlizzi, and Siddhartha Srinivasa. Effects of robot mo- tion human-robor collaboration. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, pages 51-58, 2015.
[3] Sylvain Daronnat, Leif Azzopardi, Martin Halvey, and Mateusz Dubiel. Impact of agent reliability and predictability on trust real time human-agent collaboration. In Proceedings of the 8th International Conference on Hiaman-Agent Interaction, pages 131-139, 2020.

---

[4] Sebastian Ahmdd, Johannes Fähndrich, and Sahin Albayrak. Human Sluman-agent teamwork: what predictabilitty, why is important? In Proceedings of the 31st Annual ACM Symposium on Applied Computing, pages 284 184-286, 2016.
[5] Julian Schritwiesser loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-6/)) 2020.
[6] Stephen Chung, Ivan Anokhin, and David Krueger. Thinker: Learning plan and act. Advances in Neural Information Processing Systems, 36, 2024.
[7] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference machine learning, pages 282 293. Springer, 2006.
[8] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
[9] Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, et al. An investigation of model-free planning. In International conference on machine learning, pages PMLR, 2019.
[10] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
[11] Thomas Moerland, Joost Brockens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[12] Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 2, pages 761-766 2011.
[13] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions.
[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[15] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[16] Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Nickum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[17] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[18] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint 1rXivv/801.088) 08757, 2018.
[19] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3387-3395 2019.
10

---

[20] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 49):4915-4922, 2021.
[21] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34:13859-13880 2021.
[22] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In International conference on machine learning, pages 1804-1813. PMLR, 2016.
[23] Jakob N Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Picter Abbeel, and Igor Mordatch. Learning with opponent-learnigg awareness, arXiv preprint arXiv: / 770.004366, 2017.
[24] Roberta Raileamu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference machine learning, pages 4257-4266. PMLR, 2018
[25] Serhiy Kandul, Vincent Micheli, Juliane Beck, Thomas Burri, François Fleuret, Markus Kneer, and Markus Christen. Human control redressed: Comparing ai and human predictability in real-effort task. Computers Human Behavior Reports, 10:100290 2023.

---

A Agent Details
In this section, we describe the details of how we trained the four types of agents discussed in the paper. Most of them follow the procedure outlined in the original paper:
1. MuZero: We the same agent configuration in the original paper, except for the world model, where we adopt the architecture and training method proposed in Thinker. This is to ensure that the results are not affected by the choice of the world model. We conducted 100 simulations for each search
2. Thinker: We use the same default agent as described in the original paper.
3. DRC: We use the DRC(3,3) described in the original paper.
4. IMPALA: We use the large architecture but omit the LSTM component described in the original paper.
All the hyperparameters are consistent with those in the original paper, and the agents are all trained using 25 million transitions. For each RL algorithm in the default model case, we train three separate agents with different seeds. For the model ablation case, train only one agent due to computational cost.
World Model:
The world model utilizes the dual network architecture proposed in Thinker, as it enables the prediction of raw states, values, and policies, allowing its use in both simulation-bai based approaches and planning in MuZero and Thinker. We also found that using the dual network results in better performance in the environment than the original network proposed in MuZero, likely due to the addition of learning signals from predicting the raw state. In the small model ablation case, we reduced all channel sizes the RNN block from the default 128 to 32.
We follow the training procedure discussed in Thinker to train the world model, except that we added an additional loss based on L2-distance between the predicted raw state and the true raw state. This ensures that the world model focuses on all features of the raw states, not just those relevant to rewards. Consequently,
nent--eard-affecting features, such as the blue location, can still encoded and predicted by the world model. We found that the addition of this loss does not negatively impact the agent's 's performance in the environment. It should be noted that we do not assume we have knowledge about the event when training the world model or the agent. We train a separate world model following the same training procedure for each agent.
Figure [ shows examples of model outputs for both the default setting and the model ablation setting. In the default case, the model predicts the states accurately, albeit with slight blurring. In the small model case, the agent erroncously pushes the box across the wall, which should not be allowed, and the blue location is missing, likely due the model's limited capacity preventing it from fitting into the representation. In the stochastic case, the agent gradually fades due to the uncertainty of its position. Lastly, in the POMDP case, the agent is completely missing, attributable the difficulty of ascertaining the agent's 'ss true position. These three model ablation cases thus showcase the different failure modes of the model.
The learning curves of the agents in both the default setting and the model ablation setting can be found in Figure We observe that in the default setting, the Thinker agent performs the best, with results closely replicating those of the original paper. The MuZero agent here outperforms the MuZero agent in the Thinker's paper due to the use of dual network as the world model. In the small model setting, the performance of the DRC and baseline agents is similar to that in the default case, as the world model is not used in the policy. In the POMDP case, all agents perform poorly, likely because they cannot be certain of their own location, making the problem too challenging.
Learning Curve:
B Predictor Details
Predictor architecture The raw states and predicted states (if they exist) are processed by a separate convolutional encoder with the same architecture. The encoder shares the same architecture as follows:
We count each node traversal simulation, as opposed new node expansion
12

---

Step O
Step 1
Step 2
Step 3
Step
Step
F
-
Defouit
T
5"
Small Modil
T
Stochastic
PONDP
Figure 5: The predicted states output by the trained world model, where the starting state is shown in the leftmost column and the input action is five consecutive UP actions.
Convolution with 64 output channels and stride 2, followed by ReLu activation.
residual blocks, each with 64 output channels.
Convolution with 128 output channels and stride 2. followed by ReLu activation.
residual blocks, each with 128 output channels.
Average pooling operation with stride
residual blocks, each with 128 output channels.
Average pooling operation with stride 2.
All convolutions then flattened and passed to a linear layer with an output size of 128. The encoded state then concatenated with the action selected that state to form embedding. For the inner state approach of Thinker, we also concatenate the tree representation [6] to the embedding.
kernel size of 3. The resulting output shape is (128, 6, 6). The output is
For the simulation-basee approach applied to all agents and the inner state approach used by explicit planning agents, embedding the current state combined with the rollouts forms sequence of embeddings of size 1 total rollout length. For the inner state approach the DRC agent, we encode the hidden state each internal tick using the same convolutional encoder mentioned above, but without average pooling. Since there four internal ticks (t 0, 1, 2, 3) in DRC(3,3), they form sequence of embeddings size four. This sequence is concatenated with the current state embedding to form sequence of five embeddings. Similarly, for the inner state approach we encode the hidden layer using the same convolutional encoder but also without average pooling. This is concatenated with the current state embedding to form sequence of two embeddings. Note that the embedding of the current state is always positioned at the first slot in the sequence.
the IMPALA agent,
In all cases, the sequence of embeddings passes through three-layer Transformer encoder with dimension 512. The output from the Transformer encoder at the first token is then passed to linear layer, which predicts the required probabilities using either softmax sigmoid output units.
13

---

Default
Small Model
80%
80%
Solving Ratu
Solving Rate
60%
60%
20%
20%
0%
2/0
2.5
0%2
Frames
le?
Frames
1el
100%
100%
Solving Rate
60%
6076
Salving
40%
40%
20%
0.5
3.0
1.5
2.0
2.5
0.0
0.5
2.0
2.5
Figure 6: Rumning average solving rate the last 200 episodes in Sokoban in both the default setting and the model ablation settings. For the default case, the shaded area represents two standard errors across seeds.
Training: We generate 50,000 training samples, 10,000 evaluation samples, and 10,000 testing samples using the trained agents. We perform stochastic gradient descent on the cross-entropy loss to train both the action predictors and incident predictors. We utilize a batch size of 128 and an Adam optimizer with learning rate 0.0001. Training is halted when the validation loss fails to improve for 10 consecutive steps. For each agent with a unique seed, we train three independent predictors. As we have three separately trained agents for the default model case, this leads to a total of 9 runs.
14

---

C
Experiment Details
Figure shows the final accuracy of action prediction and the F1 score of incident prediction for the model ablation settings. The change in performance shown in Figure is computed as the difference
between the performance in the default model case and the performance shown here in Figure
Small Model
Stochastic
POMOP
0.6
0.4
0.2
0.0
Thairing Data
Theining Size
Thviring Data
1.0
0.8
0.6
10
102
10
M.Zero imner state
Thinker Inter shate
DRC simufation
IMPALA- simulittion
MuZore bascline
Thinker bosaline
DRC basoline
IMPALA: baselinc
Figure 7: The final accuracy of action prediction and FI score of incident prediction for the world model ablation settings. The error bar represents two standard errors across 3 seeds.
Computational Resources: Each agent is trained using single A100 GPU, with training time varying by algorithm. MuZero, Thinker, DRC, and IMPALA take approximately 7, 3. 2, and days, respectively. to complete training. The world model is trained concurrently with the agent on the same GPU. For training the predictors, we also use single 100 GPU, and takes about days to complete training across all auxiliary information settings for a single agent and single seed.
Code: The code used for these experiments is available https://gi
com/
httpp//ewhhunnnmmmepeeettttttaia and is based on the public code released in Thinker [6].
15

---

D Ablation on Inner State Approach
We consider alternative choices for inner states and repeat the experiment shown in Figure The following inner states are considered:
1. MuZero: We considered using the top 3 rollouts ranked by visit counts against only the top rollouts (the default case).
2. DRC: We considered using the hidden state at all ticks (the default case) against only the hidden state the last tick.
3. IMPALA: We considered using the output of all three residual blocks against only the last residual block (the default case).
The results can be found in Fig(8) 8 We observe that the results are similar with different chosen inner states, except that (i) using top 3 rollouts in MuZero leads slightly lower event prediction accuracy. possibly because the top rollout is sufficient to make the prediction, and (ii) using all residual blocks IMPALA gives slightly better performance event prediction, likely because lower residual blocks still encode the blue location that is helpful for predicting the chosen event.
Accuracy of Action Prediction
Prediction
0.8
Accuracy
0.6
F1 Scare
0.2
0.0
10a
10
10¹
10a
Training
Training
MuZero top rollout
DRC tick
IMPALA last block IMPALA blocks
MuZero rollouts
all
Figure 8: Ablation experiments on the chosen inner state. The error bars represent two standard
errors across seeds.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are supported by experiments in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experiment details are provided in both the main paper and the appendix, and the code is released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
Code is provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com /tte://bb///uudddd///eeeeeiessss ssioiilicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experiment details can be found in the appendix and the code. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: 1-sigma error bars are included in the figure. The train/test split is explained in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details can be found in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing assets properly cited in the appendix.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The codes well documented.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "3G3uhroZm6", "reviewer": "Reviewer_7yyB", "summary": "The authors present two approaches for predicting future states and actions in an RL setting. This is done in a supervised setting, where the inputs are the state and action and the target is either the future action sequence or the time of occurrence of some event. The first approach leverages using the inner-state as additional information to the state action input. The second approach uses a model trained to simulate the environment to generate a rollout to serve as the additional information.", "questions": "The reviewer invites the authors to kindly clarify if they have misunderstood any parts of the paper leading to the weaknesses outlined above. Beyond those related to the weaknesses, the reviewer does not have any other questions.", "limitations": "The authors acknowledge the limited range of RL agents and environments tested.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "While look-up tables of reward-action have been utilized before, the reviewer is not aware of any prior work that includes entire rollouts, or leverages external hidden states as predictors of future events or actions. The authors’ method is clearly explained. The experiments clearly demonstrate the authors’ claims and both metrics illustrate the capability of their approach in predicting future actions and events. The problem statement, prediction of future actions and the events that may be a result, is undoubtably a significant issue in the larger ML community as a whole.", "weaknesses": "The reviewer’s primary concern is with data leakage that could occur with the rollouts used as additional information when training the model, especially with the ‘Thinker’ agent. If the action space is rolled out and appended to the input, would this not result in the potential future actions already being included with the input?\n\nThe reviewer is also unsure as to the benefits of the method over, for example, training a world model, rolling out several series of actions and choosing the branch that accumulates the highest predicted reward to also serve as a predictor of future actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s comments and for recognizing the significance of the problem statement. The questions are addressed as follows:\n\n> If the action space is rolled out and appended to the input, would this not result in the potential future actions already being included with the input?\n\nThese rollouts are rollouts of imaginary actions performed in a learned world model and thus do not constitute data leakage. This allows action and event predictions to be made L=5 steps before the true future action or event is known. The future action may indeed be included in simulated rollouts of model-based agents, but these agents (including Thinker) won't **necessarily** choose the actions that are simulated in the rollouts due to numerous reasons, such as an agent learning to ignore bad rollouts, adjusting for model prediction error, or continuously refining the plan in future steps. Nonetheless, the simulated actions in rollouts are certainly helpful for predicting future actions—this is expected, and this potential advantage of explicit planning algorithms is one of the core motivations of our work.\n\n> The reviewer is also unsure as to the benefits of the method over, for example, training a world model, rolling out several series of actions and choosing the branch that accumulates the highest predicted reward to also serve as a predictor of future actions.\n\n\nWe interpret the suggestion as using the empirical action or event distribution of the rollout with the highest rollout returns (sum of predicted rewards and the terminal value) as the predicted future action and event distribution.\n\nFor model-based RL agents such as Thinker and MuZero, they do not necessarily select the rollout with the highest rollout return. Figure 19 in the appendix of the Thinker paper shows that a trained Thinker agent selects the next action with the highest rollout return around 90% of the time. In contrast, the same figure for MuZero (shown as MCTS) is around 50%, as MCTS selects the next action based on visit count instead*.\n\nFor model-free RL agents, as they cannot perform rollouts like Thinker or MuZero, their selected actions likely differ from the best action from multiple rollouts. They often exhibit myopic behavior, such as pushing a box to a wall and thus making the level unsolvable. The empirical rollouts (i.e., rollouts using the agent policy) should be more insightful for predicting future actions and events.\n\nAs such, if we had to estimate the future action or event prediction by empirical distribution of a rollout, we suggest using**:\n\n1. The rollout with the highest return for Thinker.\n2. The rollout with the largest visit count for MuZero.\n3. The empirical rollout for model-free RL agents.\n\nHowever, in our paper, we consider inputting these rollouts to a neural network to learn the prediction instead of directly using the empirical distribution. This is because the world model may diverge from the true environment, leading to unreliable rollouts. For example, in the extreme case that the trained world model outputs random noise, the learned neural network can still predict future actions or events based on the current state while ignoring the rollout, resulting in the baseline case shown in Figure 2. This should be considerably better than using the empirical distribution of rollouts, which does not provide any information in this hypothetical example.\n\nAnother advantage of using a learned neural network is that we only need the binary event label instead of the event function when training the predictor. On the other hand, if we use the empirical distribution of rollouts to make event predictions, we also need to handcraft a function that maps predicted states to an event label (e.g., classify whether the agent stands in the blue location from raw predicted pixels in our setting).\n\n*Note that both figures will be lower if we consider the action in L steps (which we are interested in predicting) instead of just the next action.\n\n**The first two are included in the inner-state approach, while the third is included in the simulation-based approach.\n\nWe hope that the above answer clarifies the reviewer’s questions and look forward to further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q3yQQ6K77p", "reviewer": "Reviewer_SSgd", "summary": "The paper presents a comparative analysis of future action and event predictions in Reinforcement Learning settings by comparing inner-step approaches (that predict via inner computations like by generating a plan or via neuron activations) and simulation-based approaches (that simulate a rollout in a learned world model). The results indicate that while the latter are superior in terms of plans generated, the former methods are more robust in future action predictions.", "questions": "Please see the weaknesses section for question.", "limitations": "1) Why do authors write that environments with diverse states do not exist? There are several RL benchmarks used for evaluating diversity in exploration. Is there any other type of 'diversity' that is being talked about here for which designing new environments will be required?\n2) What is meant by 'false alarms' in line 350? Please rephrase the broader impact statement for better clarity.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) The problem statement is cleanly defined, and is easy to understand the rest of the paper.\n2) The paper provides a neat overview of each of the approaches compared via empirical results.", "weaknesses": "1) The authors mention that simulation-based approaches are superior in terms of plans generated, but inner-state approaches are more robust in future action prediction. Why is this result surprising? What was the hypothesis to begin with? Can the contribution of the paper be re-stated to empirically showing an expected result? There is no issue if that is the case, but the fact that there is no mention of an initial hypothesis to draw comparisons between the two types of approaches is misleading.\n2) What is the intuition behind the categorization of planning agents into explicit, implicit and non-planning? What definition of planning do authors have in mind such that these categories are important to talk about?\n3) Line 176: Why are simulation-based approaches expected to predict the same state space as the input to the agent? Works such as Dreamer and Planet, as far as I understand, learn to predict the latent states in the world model. What is the reason if this specific class of approaches are being considered?\n4) Why is Section 5 Related Works in the middle of the paper? Moreover, the bigger issue I find is none of the topics of Safe RL, Opponent Modelling in MA settings, and Predictability for Human-AI interactions are relevant as \"related works\". Indeed, these are important applications to the approaches being compared in this work, but I am not sure how they can be counted as related works. \n5) What are the final takeaways of this work? Which approaches need to be considered for which respective problem settings?\n\nMinor comments:\n1) Why is section 4 titled Methods? There are no method contributions in this work. Please rephrase it as Comparative Study or something like it.\n2) Shift line 159-160 before line 145.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s detailed comments. The questions are addressed as follows:\n\n**W1**: Please see our global response. Briefly, our results largely accord with our hypotheses, and we will add the hypotheses to the introduction following the list of research questions.\n\n**W2**: We define planning as “the process of interacting with an environment simulator or a world model to inform the selection of subsequent actions” (lines 71-72). Our distinction is meant to capture the difference between agents that do this explicitly by construction (explicit planning agent), those that behaviorally seem to do so implicitly by learning (implicit planning agent), and those that behaviorally do not seem to do so at all (non-planning agent). The exact definition of these three types of agents can be found in lines 75-96.\n\nThe type of planning is important when considering predicting future actions or events as different types of information are available. With an explicit planning agent, the rollouts in the world model provide an easily interpretable plan (e.g., we can directly look at the rollouts in Fig. 1 of the Thinker paper). With an implicit planning agent, the hidden activations may contain the same degree of information as the explicit rollouts in explicit planning agents. However, this information is distributed across neurons and likely mixed with other internal computations that are not relevant to its plans, making it harder to extract the agent's plans. Lastly, for a non-planning agent, the lack of planning means that the hidden state may only contain a more compact representation of the state. It is expected that these different agents' inner states have different degrees of information and intractability. This expectation is largely consistent with the results shown in Fig. 2. \n\n**W3**: In the simulation-based approach, we unroll a given agent in a trained world model instead of the environment. Therefore, it is expected that the trained world model predicts the same state space as the input to the agent. In special cases where the action can be computed using only the abstract state (e.g., the actor in Dreamer only requires the latent state to compute the action distribution), we can indeed use an abstract world model (e.g., the same latent world model in Dreamer) to do the simulation. We are thankful that the reviewer pointed this out and will clarify it in the revised paper.\n\nP.S. Latent predicted state may be less effective for event prediction compared to the raw predicted state. We briefly experimented with a latent world model using Thinker and found that the event prediction accuracy dropped significantly, likely because the information in latent representation is harder to extract. However, if we train a decoder to convert the latent predicted state to the raw predicted state, the accuracy restores to the level reported in the paper. As such, we advise decoding the latent predicted state into the raw predicted state even when rollouts in latent space are possible.\n\n**W4**: We will move Section 5 after the introduction to improve clarity. As aforementioned, the paper considers a new problem setting, so we are unable to find any related work that tackles the same problem. As such, we can only list the works done in related areas and/or share similar goals, such as predicting opponent actions in a multi-agent setting or preventing (instead of only detecting) dangerous agent actions in a safe RL setting.\n\n**W5**: Please see our answer to the three research questions in our global response. In particular, we advise using the simulation-based approach when the environment dynamics are easy to model, and using the inner-state approach in other cases. We also advise using explicit planning agents if the inner-state approach is considered, as their explicit plans are generally more informative or easily extractable than the inner states of other agents.\n\n> Why is section 4 titled Methods? There are no method contributions in this work. \n\nWe propose two methods, and both methods are part of the paper’s contributions. As this is a new problem setting, the two methods are constructed to be general and simple to allow future work to build upon them.\n\nWe note that the simulation-based approach has been used in a multi-agent setting to predict opponent movements for training a better policy, but the nature of the problem is very different from the one we consider. For the inner-state approach, we are not aware of any similar methods used in other problem settings.\n\n> Shift line 159-160 before line 145.\n\nThank you. We will update the paper accordingly.\n\n> Why do authors write that environments with diverse states do not exist?\n\nThe diversity here refers to the variety of possible solutions (or optimal policies) as in Sokoban, where there are usually multiple ways of solving the level, leading to high difficulty in predicting agent actions. We did not posit that such environments do not exist, and welcome pointers to alternative environments with this property, as we have not been able to identify any. We briefly experimented with Atari environments, where in some games the state space is very diverse, but we found that the lack of diverse optimal policies reduces the problem of future action prediction to predicting the optimal actions.  \n\n> What is meant by 'false alarms' in line 350? \n\nIt means that when the predictor forecasts that an agent is going to perform an unsafe action, but in fact, the agent won’t be doing it. This may lead to improper responses (such as shutting down the agent) that are not warranted. We will rephrase the broader statement to make it clearer.\n\nWe are thankful for the reviewer’s detailed feedback. We will update the paper as stated above and hope that the reviewer considers revising the score in light of the clarified hypothesis and context. We are also happy to discuss further if the reviewer has any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4l7WVT8iI", "reviewer": "Reviewer_W8D8", "summary": "This paper focuses on predicting future actions and events using policies learned by different types of planning agents. Namely, the authors distinguish between explicit and implicit planning agents, where explicit planning agents learn a model of the environment dynamics/transitions. Given this distinction between agents, the authors investigate how well these agents can reliably predict future actions or reaching a certain states within a finite time frame. They study the agents empirically in the domain of Sokoban and show the findings of their study.", "questions": "See my last point in the previous section.", "limitations": "Yes, but these seem like things the authors should do to make the paper stronger.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality:** This is an investigative study on how the type of agent being trained influences its ability to predict future actions and events. While the prediction of future actions is not a new problem, the novelty arises from the study over different types of agents.\n\n**Significance:** I believe that the problem is interesting and understanding this topic can go along way in how users of AI agents can decide on which agents to train/deploy --- e.g., \"does one learn an environment model or are the gains not worth the increase in complexity?\", etc.\n\n**Clarity:** The paper was clear.\n\n**Quality:** The main motivation behind the paper is simple and well-defined. The problem being studied is an interesting one. The chosen experimental design is reasonable for what the authors are trying to show.", "weaknesses": "- The results are plotting standard deviation instead of standard errors to show confidence intervals.\n\n- Predicting the event of reaching the blue tile seems like it may not be a good evaluation criteria. If the blue tile can be placed, with uniform probability, on any empty tile in the puzzle, then if the puzzle is large, the agent is very unlikely to reach the tile. This prediction task seems very trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s comments and for recognizing the significance of the problem statement. The questions are addressed as follows:\n\n**W1**: As the goal of the figure is to compare the effectiveness of different approaches on different agents instead of the prediction variability, it is indeed better to use a 2-sigma error bar that indicates the confidence interval instead. We will update the paper to use 2-sigma error bar.  Please see the updated figures in our global response for the revised figures.\n\n**W2**: We designed the event to be predicted to have a small probability, so it is closer to real-world applications where we may want to predict an unsafe but rare event. In our trained agents, the agent will step on the blue tile within L=5 steps around 5% of the time. Therefore, a trivial predictor that predicts the future event not occurring will already achieve 95% accuracy. As such, we report F1 score instead of accuracy in event prediction. Such a trivial predictor will only achieve a zero F1 score due to zero recall, making the problem far from trivial. We expect that an even rarer event will make achieving a high F1 score even more challenging due to the scarcity of the data.\n\nWe are thankful for the reviewer’s suggestions and hope that our response adequately addresses the reviewer’s questions. We hope that the reviewer considers revising the score in light of our clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "esKsFdiK0h", "reviewer": "Reviewer_tXgP", "summary": "This paper investigates methods for predicting future actions and events of reinforcement learning agents during deployment. It compares the predictability of explicitly planning, implicitly planning, and non-planning agents using two approaches: an inner state approach that leverages the agent's internal computations, and a simulation-based approach that uses a learned world model. The result from the Sokoban environment shows that plans from explicitly planning agents are more informative for prediction than other types of inner states.", "questions": "- What's the rationality behind the choices in different methods? \n\t- For MuZero, why only use the most visited rollout as auxiliary information? Wouldn't using top-K rollouts provide more information? \n\t- For DRC, have you experimented with using only the final tick or a subset of ticks?\n\t- For IMPALA, have you tried using multiple layers or earlier layers?\n- For simulation-based approach, how do you handle the potential divergence between simulated trajectories and real trajectories over time?\n- How are the training/testing samples generated? Are they from trajectories with fixed length?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper addresses an important and practical problem in RL deployment - predicting future agent behavior.\n- The proposed method is well-motivated, with a clear analysis of its idea.\n- The structure and presentation of the paper are clear and well-organized.", "weaknesses": "See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s comments and for recognizing the significance of the problem statement. The questions are addressed as follows:\n\n> What's the rationality behind the choices in different methods?\n\nThe rationality is mainly based on our intuition of which information is useful. We agree that more comprehensive experiments would benefit the paper. As such, we conducted further experiments to show the impacts of choosing different types of inner-state. Please refer to our global response for the results. We will include these results in the revised paper.\n\n> How do you handle the potential divergence between simulated trajectories and real trajectories over time?\n\nFirst, we note that our problem statement does not permit access to real trajectories, so predictions must be made purely on the basis of simulated trajectories and the current state, which may diverge depending on the quality of the world model. We make no special effort to prevent this divergence, although we note that the world model is trained on transitions the agent encounters during training.\n\nTo cope with the potential divergence when predicting future actions and events, we input the simulated trajectories to a neural network, as the neural network should learn to extract useful information from the trajectories while discarding the non-useful information. In the extreme case of the world model outputting random noise, such that the simulated trajectories contain almost no information, the neural network can still rely on the current state to make a prediction (which is equivalent to the baseline case shown in Fig. 2). We also note that it is due to this possible divergence that we do not just use the empirical distribution of simulated trajectories to manually predict future actions and events.\n\n> How are the training/testing samples generated? Are they from trajectories with fixed length?\n\nWe deploy the trained agents on the training level of Sokoban until 50k transitions are collected. Each episode may last from a few steps to over a hundred steps, depending on whether and how fast the trained agent can solve the level. We perform the same process for generating testing samples, but we deploy the trained agent on testing levels instead of training levels and only collected 10k transitions.\n\nWe are thankful for the reviewer’s suggestions and hope that our response adequately addresses the reviewer’s questions. We hope that the reviewer considers revising the score in light of our new experiments and clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are thankful for the reviewer’s comments. Before addressing the questions, we would like to clarify the context of the paper. In this paper, we consider a problem statement, namely predicting future actions and events for agents trained with different RL algorithms. As far as we are aware, this problem has not been considered in the previous literature, despite most reviewers recognizing its significance. We propose two methods for this problem, namely the inner-state approach and the simulation-based approach. Despite their simplicity, both methods have not been studied before in this context. We then evaluate and compare the two approaches empirically to answer the following three research questions listed in lines 40-48:

- **Q1.** How informative are these inner states for predicting future actions and events?
- **Q2.** How does the predictability of future actions and events vary across different types of RL agents with different inner states?
- **Q3.** How do the performance and robustness of the simulation-based approach compare to the inner state approach in predicting future actions and events across different agent types?

Before performing the experiments, we had some hypotheses about the answers to the above questions. We also summarized the results here to aid understanding.

**Q1 and Q2**

**Hypothesis**: The benefits of the inner state depend on the type of agent. The inner state for the explicit planning agent should be more informative than that of the implicit planning agent, as the plan in the explicit planning agent is more interpretable (the rollouts are human-interpretable) while the plan for the implicit planning agent is stored in the hidden activation. Nonetheless, they both contain indications of the agent’s future actions, so they should hold more information than the inner state of a non-planning agent, which does not contain future plans and may only be a more compact representation of the state.

**Result**: For future action prediction, these hypotheses are consistent with the results shown in Fig. 2, where the inner-state approach performance of the explicit planning agent is better than that of the implicit planning agent, followed by non-planning agents (line 257-270). For future event prediction, the result is more surprising as the inner-state for both implicit planning agents and non-planning agents does not help in event prediction. We conjecture that the agent ignored our defined events in its representation as they do not affect rewards. On the other hand, the inner state of explicit planning agents is still very beneficial, as the world model is explicitly trained to predict all features in the environment (lines 271-277).

**Q3**

**Hypothesis**: If we have an accurate world model, simulation-based approaches should be better. Consider the extreme case of having a perfect world model; then the exact future action and event distribution can be estimated by the empirical distribution of the rollouts. However, we expect the simulation-based approach to be much less robust to model quality than the inner-state approach, as a slight error in the predicted state may confuse the agent and make it act differently due to out-of-distribution (OOD) inputs. For the inner-state approach, in the case of a model-free RL agent, there is no model, so this is not directly comparable. And in the case of a model-based RL agent, the agent is trained along with the model, so even inaccurate model output is not OOD to the agent, possibly resulting in better robustness.

**Result**: Fig. 3 shows that the simulation-based approach indeed performs better than the inner-state approach, as our trained world model is close to perfect. However, when we do ablation on the world model, as shown in Fig. 4, the performance of the simulation-based approach generally drops more for action prediction, which is consistent with our hypothesis (line 314-322). For event prediction, the result is more mixed and surprising, with the simulation-based approach sometimes performing better and sometimes worse. We conjecture that this is due to the trained world model not predicting the defined events well, leading to poor prediction in both approaches (lines 323-327).

We did not put our initial hypotheses in the introduction, as similar discussions are already included in the discussion section. However, we agree with reviewer SSgd that stating the initial hypotheses at the beginning will improve the clarity and flow of the paper, and we will add that to the revised version of the paper.

To improve the robustness of the results, we have repeated the main experiments with 9 seeds and will update the main figures of the paper by plotting the 2-sigma error bar instead of the standard deviation. The updated figures can be found in the attached PDF. In addition, we performed an additional ablation study on the chosen inner state:

- **MuZero**: We considered using the top 3 rollouts ranked by visit counts against only the top rollouts (the default case).
- **DRC**: We considered using the hidden state at all ticks (the default case) against only the hidden state at the last tick.
- **IMPALA**: We considered using the output of all three residual blocks against only the last residual block (the default case).

The results can be found in Fig A3 of the attached PDF. We observe that the results are similar with different chosen inner states, except that (i) using top 3 rollouts in MuZero leads to slightly lower event prediction accuracy, possibly because the top rollout is sufficient to make the prediction, and (ii) using all residual blocks in IMPALA gives slightly better performance in event prediction, likely because lower residual blocks still encode the blue location that is helpful for predicting the chosen event.

We are thankful for the reviewers’ detailed feedbacks and suggestions.

Meta-review: Nearly all reviewers agreed that this work is original, significant, clear, and well-motivated. Any lingering questions or issues were resolved during the rebuttal. In light of this, I am recommending the paper be accepted.
Decision: Accept (poster)