Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning
Authors: Dongsu Lee
Minhae Kwon
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement learning
Episodic future thinking
multi-agent reinforcement learning
Primary area: reinforcement_learning

Abstract: Understanding cognitive processes in multi-agent interactions is a primary goal in cognitive science. It can guide the direction of artificial intelligence (AI) research toward social decision-making in multi-agent systems, which includes uncertainty from character heterogeneity. In this paper, we introduce *episodic future thinking (EFT) mechanism* for a reinforcement learning (RL) agent, inspired by the cognitive processes observed in animals. To enable future thinking functionality, we first develop a *multi-character policy* that captures diverse characters with an ensemble of heterogeneous policies. The *character* of an agent is defined as a different weight combination on reward components, representing distinct behavioral preferences. The future thinking agent collects observation-action trajectories of the target agents and leverages the pre-trained multi-character policy to infer their characters. Once the character is inferred, the agent predicts the upcoming actions of target agents and simulates the potential future scenario. This capability allows the agent to adaptively select the optimal action, considering the predicted future scenario in multi-agent scenarios. To evaluate the proposed mechanism, we consider the multi-agent autonomous driving scenario in which autonomous vehicles with different driving traits are on the road. Simulation results demonstrate that the EFT mechanism with accurate character inference leads to a higher reward than existing multi-agent solutions. We also confirm that the effect of reward improvement remains valid across societies with different levels of character diversity.

Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning
Dongsu Leet and Minhae Kwon Department of Intelligent Semiconductors
"School of Electronic Engineering Soongsil University, Scoul, South Korea
movementvaater&ssonggll ac bc.rr, minhae@ssu. ac.
Abstract
Understanding cognitive processes in multi-agent interactions is primary goal in cognitive science. It can guide the direction of artificial intelligence (AI) research toward social decision making in multi-agent systems, which includes uncertainty from character heterogeneity. In this paper, we introduce episodic future thinking (EFT) mechanism for reinforcement learning (RL) agent, inspired by cognitive processes observed animals. To enable future thinking functionality, we first develop multi-character policy that captures diverse characters with ensemble heterogeneous policies. Here, the character an agent is defined as different weight combination on reward components, representing distinct behavioral pref- erences. The future thinking agent collects observation-acee aattion trajectories of the target agents and uses the pre-trained multi-character policy to infer their characters. Once the character is inferred, the agent predicts the upcoming actions of target agents and simulates the potential future scenario. This capability allows the agent to adaptively select the optimal action, considering the predicted future scenario in multi-agent interactions. To evaluate the proposed mechanism, multi-agent autonomous driving scenario with diverse driving traits and multiple particle environments. Simulation results demonstrate that the EFT mechanism with accurate character inference leads to higher reward than existing multi-agent solutions. We also confirm that the effect of reward improvement remains valid across societies with different levels of character diversity
consider the
1 Introduction
Understanding human seciision-maiin in multi-agent interactions is a significant focus in cognitive science. provides valuable insights into designing interactions among diverse Al agents within multi- agent systems. Research has shown that humans employ counterfactual future scenario simulation 49 While counterfactual thinking, simulating alternative consequences of past events, has been extensively explored in multi-agent RL (MARL) [34 9 52 B),
to enhance decision- makinn [45
episodic future thinking [ 224) the ability to anticipate future events, remains underexplored in literature despite its importance in handling multi-agent interactions.
Human beings strive to anticipate future situations to prevent costly mistakes. One naive approach to incorporate this ability into AI is through future trajectory prediction using model-based RL 40 19 31 54 28, 26)) However, this approach is feasible only if the state transition model is known or easily learnable, which is often not the case in multi-agent systems. The complexity arises from
*Corresponding author: Kwon Project Web: http::///sstt...oooooo////////ts
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the interdependence of state transitions on the actions of both the agent and other agents, making learning the state transition model challenging, Additionally, diverse agent characteristics exacerbate this challenge by introducing a wide range of action combinations and subsequent states. Thus, explicitly integrating character inference functionality regarding other agents into Al more suitable for accurate future state prediction and optimal decision-making.
Our goal is to develop an EFT mechanism for RL agents, enabling them to make adaptive decisions in society where agents have heterogeneous characteristics. We formalize this task as a Multi-agent Partially Observable Markov Decision Process (MA-POMDP), framework tailored to address the RL problem wherein multiple agents operate under partial observation [5]. This study defines a character by reflecting the behavioral preferences of RL agents, which come from different weight combinations on reward components. For instance, in a driving scenario, some drivers prioritize safery, while others prioritize speed, leading to heterogeneous policies and behavioral patterns across agents.
Implementing the EFT mechanism requires two functional modules: multi-character policy and character inference module. The multi-character policy embeds behavioral patterns corresponding characters. It allows the agent observe partial information of the state in continuous space and handles a hybrid action space consisting of discrete and continuous actions. The character inference module leverages the concept of inverse rational control (IRC) 18 25 to infer target agents' characters by maximizing the oogllikeldood of their observation action trajectories. Combining these modules equips the agent with EFT functionality, enabling proactive behavior under heterogeneous multi-agent interactions.
To activate the EFT mechanism, the agent initially acts observer, collecting observation-action trajectories of target agents. Utilizing the character inference module and collected trajectories, the agent infers target agents' characters. With this knowledge and leveraging a multi-character policy, the agent predicts others' actions and simulates future observations with its action fixed as 'no action.' This mental simulation allows the agent estimate the observation at the time point when all target agents have taken actions, but the agent still needs to (i.e., has yet to). It enables the agent to select the best action corresponding the estimated future observation. In summary, the EFT mechanism empowers the agent to behave proactively in heterogeneous multi-agen interactions.
Summary of contributions:
We introduce character diversity in a multi-agent system by parameterizing the reward function. We propose to build the multi-character policy and equip the agent with to infer the character of the target agent (Section 3).
We introduce the EFT mechanism for social decision-making The agent infers the characters of other agents using the multi-characte policy, predicts their future actions based on the inferred characters, simulates the corresponding future observations and selects foresighted actions. This mechanism enables the agent to consider multi-agent interactions in its crtision-making process (Section
We verify the proposed mechanism by increasing character diversity in society. Extensive experiments confirm that the proposed mechanism enhances group rewards matter how high a character diversity level exists in society (Section 5
2 Related Works
Episodic Future Thinking. Cognitive ncuroscience aims to understand how humans use memory in decision-making, Interestingly, the trend of the brain's regional neural activation regarding counterfactual reasoning (i.e., simulating alternative consequences of the last episode) and future thinking (i.e., simulating episodes that may occur in the future) is similar M In [56). the authors study the relationship between future thinking and decision- decision-making making and confirm that humans perform future-orientn decision-makin The decision-making abilities, such as strategy formulation, also significant in scenarios that require multi-agent interactions, e.g., social decision- maaiin..
There are several studies to endow this ability with an Al agent 62 37 61 30
In 30 and [77. the
authors forecast the next state from a macroscopic standpoint without a prediction of each agent's behavior. In [61], the authors predict the behavior of agent through a deep Bayesian network

---

considering the dynamics and the previous surrounding environment information. Even though these studies can infer future information, no strategy formulation incorporated with prediction is suggested. Namely, most existing approaches use future predictions as auxiliary information for the optimization process without incorporating these predictions into the policy explicitly. this study, we propose the ETF mechanism can predict future observations based on the current state and predicted actions of surrounding agents. Consequently, the agent equipped with this mechanism can select a foresighted action corresponding to the anticipated future observation.
Model-based Reinforcement Learning. Model-based RL incorporates an explicit module repre- senting system dynamics, contrasting with model-free RL. Within model-basee RL, two approaches exist: utilizing a known dynamic model and learning it during training [19
32 40
Using the
dynamic model, the model-based RL approaches predict future trajectories, a pivotal step for network Notably, approaches such as Dreamer [55] and Model-Based Policy 14 demonstrate the practical application of these predictions. Dreamer optimizes value function using the return of the predicted future trajectories, and MBPÒ trains the policy using the predicted future trajectories as augmented data samples. Furthermore, to tackle multi- agent problems, and 57 extend these concepts by integrating a global model or communication block.
optimization 1S 16 57 14
Optimization (MBPO) 16
While these methods often exhibit outstanding performance, they assume ideal conditions such as a small number of homogeneous agents and full observability. In reality, agents encounter incomplete and noisy data, and accurately modeling system dynamics is challenging due to complex interactions between multiple agents with unique behavioral characteristics. This work addresses a partially observable agent in a multi-agent environment with heterogeneous characteristics across agents. We allow the agent to infer other agents' characters and make decisions based on predictions of upcoming observations.
Machine Theory of Mind. Human decision-making in social contexts often involves considering multiple perspectives, including the behavioral characteristics of others. This capacity, known as Theory of Mind (ToM) in cognitive science, primarily involves deducing internal models of others and predicting their future actions 2 20). Al research aimed providing machines with this capability has gained attention for enhancing multi-agent system performance, such as machine ToM [42 41).
and Bayesian ToM (6)) These approaches aim to reconstruct the target agent's belief, reward function, or dynamic model based on its trajectories. However, they often operate in simple settings, limiting their applicability to scenarios with small number of agents, a small discrete action space, or minimal character diversity across agents.
inverse learning 43 18
In contrast to previous work, this study explicitly develops character inference module focusing on establishing a link between trajectories and characters. This module allows the target agent's 's behavior to be explained by character, aligning with the researcher's interests. Additionally, it extends the action space from continuous to hybrid.
False Consensus Effect. Psychological research has identified cognitive bias in humans to assume that their character, beliefs, and actions are common among the general population
10 6 7. termed the False Consensus Effect (FCE) [53 12 47 Recent studies suggest that AI may exhibit this false belief [42). To underscore the importance of character inference in heterogeneous multi-agent scenarios, we compare the performance of the EFT mechanism with two types of agents: the proposed agent, equipped with the character inference module, and the FCE-based agent, which assumes that target agents share the same character as the agent.
Roward AGENT
Charac -tor C1
3 Character Inference Using Multi-character Policy
ENVIRONMENT
R.,
Action
(or, C../
C2
MLE
ate Jnuial
C3
We aim to build an agent to make optimal deci-
State
O2, Observation
Cj
C1
CK.
sions under multi-agent interactions. It requires the agent to be able to anticipate the near fu- ture by predicting other agents' actions. The agent should possess the ability to infer the oth- ers' characters, leveraging observation of their behaviors.
Figure 1: A block diagram of an agent i with multi- character policy x (or.,; C)), where C is character space. The agent can infer the character e of others by using the maximum likelihood estimation. Herein, K means the dimension of character vector c.
3

---

Accurate character inference is prerequisite for the EFT mechanism since the character is a crucial clue predicting future action. Therefore, this section proposes two functional modules for character inference: a multi-character policy and character inference. An illustrative explanation of these functionalities is presented in Figure
3.1 Problem Formulations for Multi-agent Decision-making
We consider multi-agent scenarios where RI. agents adaptively behave to each other. All agents have to make decisions and execute actions simultaneously. unlike the extensive-form game (36 which the agents alternate executing the actions.
The multi-agent decision-making problem can be formalized as a MA-POMDP M (,,S, [O:), [A.), T, {{, ( 7) that includes an index set of agents E' = {1,2, ... ,N), continuous states E S, continuous observations Or,i E O, hybrid actions dz { " E Ai, where continuous action ai, E AC and discrete action af,, A {w |w] S W,
L, W
N]. where the size of discrete action space is | = 2W + 1, Z denotes the set integers, and denotes the set of natural numbers. Let A A1 A2 ... x Ax. Subsequently, T S A + the state transition probability; Ω S + O, the observation probability: RG S A, x S - IR denotes the reward function that evaluates the agent's action 0t for given state 8, and the outcome state 8141; E [0, 1) is the temporal discount factor.
An unordered set of the actions of all agents at time / is denoted as
a ( .r...".. ,,,,N) (a ac,-i) A,
where subscript -i represents the indices of all agents in E except i. Thus, a,-- (a.,1, a,,i-1, 01,i+1 a,,N) represents a set of all agents' actions at time t without ag,;. The state transition probability denotes T (s a.). Note that state transition is based on the action combination of all agents a,, not on the action of a single agent ae,i.
Next, C, {c.1 G,2, G.K E C E RK denotes K-dimensional character vector for the agent i. Character c can parameterize the reward function of the agent i, i.e., RA, = (s at 8141: e.). The agent aims to learn the optimal policy that returns the optimal action a;,, x² (-lo,. c,) given observation and character. Specifically, the objective of the agent aims to maximize the expected
discounted cumulative reward J (T) Ex Σ. 7 (s at $141; c.) by building the best policy
т. This defines the state-action value function Q" (s, a; c;) = E. > R4 az,i, c =
a0 = a) In the next section, we discuss the details of the multi-character policy in terms of neural network design and its training.
3.2 Training a Multi-character Policy
The multi-character policy includes inputs in continuous space (e.g., observation o,, and character c,) and outputs in hybrid space (e.g., action or,;). To build the policy generalized over continuous space, the actor-critic architecture used. It approximates the policy (- c,) and Q-function (o az,1; ),), where ø denotes parameters of the actor network and denotes the parameters of the critic network.
The loss functions used to train the actor and critic networks are L(Ø) = Qo To(-)or,s and ((0) = 1/: (o a (- c2, respectively. Herein, yt R (0t+ Tar (0++1, represents the Temporal Difference (TD) target, where 0' and ø' denote the target networks.
Next, we propose post-processor g(.) to handle hybrid action space. Let proto-action at, be the output of the actor network. The post-processor g(-) performs quantization process by discretizing the continuous proto-action a into discrete post-action ag i.e.,
of. = g( W) min (L
2W +
W 2W
(1)
2W
+
where I.] denotes floor function. The derivation of presented in Appendix

---

We summarize the training process in A In the next subsection, we introduce the character in- ference module that infers the characters of other agents.
policy
Algorithm Multi-character policy training
Initialization: Actor network o, critic network 0 Require:
Total episode M, total time steps per episode T, discrete action space W, agent
for episode m = 1, M do Reset $1 and get 01 (2((1s1)
3.3 Inferring Character of Target Agent
Sample character C; ~~ C for timestep / 1, do
After completing the training on the multi- character policy, our next objective is to infer the character Cj of the target agent j E E. The agent first collects observation-action tra- jectories of the target for character inference. Subsequently. it utilizes the multi-character policy to identify the character сj that best explains the collected data. To elaborate, c, can be estimated by maximizing the log- likelihood of observation-action trajectories In P( @1.T,y/,,)) This can be formu- lated as follows.
Get proto-action {aE aे, ~ NH To(
Get post-action
"i,x + g ( W)
Execute o1, = (ai af,;), Update 88+1
Receive Re,, Get 0,+1,1 (.((118++1 Calculate L()), (()), Update 0,
end for end for return 0,
Cj arg max In P(o1:T. (1 = arg max
In [n + In ((,5
(2)
The derivation of 22 can be found Appendix E
Algorithm 2 Character inference module
To efficiently perform the inference task, we use the gradient ascent method. It runs the iteration by changing с toward the direction to increase ((cc)= = In n (a [ c) + In r (af, (ot5; which is sum-
Require: Trained actor network d
length of trajectories T, trajectories 01:T. fai.r, (irgh and initial c ~ C, target agent )
marized in Algorithm 2
repeat
Reset U(c) =0
for t T, do
Foresight Action Selection Based on Episodic Future Thinking Mechanism
Calculate 4c(c) using Eq. 2 end for
Update e + c a (V((() until e converges return čj f
This section presents the proposed EFT mechanism that enables the agent to simulate the subsequent observations and select a foresighted action. The proposed EFT mechanism comprises a future thinking module and action selection module.
The future thinking module includes two steps: ac- tion prediction and the next observation simulation. With these two steps, the agent can foresee the next observation. This process is illustrated in Figure 2 Subsequently, the action selection module enables the agent to decide the current action corresponding the simulated next observation.
at,i
do+11
OL
Action selection
0++1
Action selection
Future thinking
Future thinking
4.1 Future Thinking: Step I Action Prediction
Figure 2:
Diagram of POMDP with EFT mech- anism. The future thinking and action selection modules are included to obtain action from the oh- servation. The solid lines and circles represent the actual event. The dashed event in simulated world the agent
In this step, the agent with the multi-character policy predicts the actions of the neighbor agents by using pre-inferred characters and observations. The agent predict the action of the target agent j (E E,j +
depict the virtual
'By specifying the distribution of x, 2 can be reformulated. In Appendix F an example of the Gaussian distribution of continuous action π7 (off, c) and the Dirac delta distribution of discrete action x <(a,, |o. c) is provided.
5

---

using the trained multi-charade character policy ve and inferred character Ej, ie., des ~ To(-louss Eÿ). Therefore, the predicted action set of others â is as follows.
à = (T (0t,1; C1),
(tt, 0;-), To (0 TT(oN, CN))
4.2 Future Thinking: Step II Next Observation Simulation
In this step, we introduce how the agent simulates its next observation by using the predicted action âr,-i. Note that this prediction the result of the mental simulation of agent i, when dr.i = is satisfied. Herein, @ denotes null action, meaning that action is performed. This is to simulate the observation of the time point when all target agents performed the action, but the agent has not yet.
The simulated next observation 6,-1,3 can be determined based on the predicted ac- tion set ât,-; and the current observation Or,i- The function of the next observa- 1,a D(or,, a Ut = 0). The action selection using the simulated next observation 0 allows the agent to ignore the influence of
Algorithm Episodic future thinking mechanism
Require: Trained actor-network é, discrete action space parameter W, set of inferred characters č_s.
character of agent C₁ initial state $1
tion simulation (.)) is defined
for t 1, T do Get observation ot, 12((8e)
// Start future simulation //
for j = 1, N(j i) do Get observation 01, ~ (2
others' actions. This is because the next state determined solely by its own action dr in the agent's ss mental simulation, 0,+1,; has already applied the others' actions â,,-i-
Predict action of agents j
رzjj =o(-1or.3 Cj)
Store ār in predicted action set a end for
4.3
Action Selection
Simulate future observation of agent
Ô++1,i = D(o,, a Dt,i 0)
Once the agent has simulated the next ob- servation 0,+1,i, the agent can make a fore- sighted decision. The agent uses the multi- character policy Tg with the input of the simulated next observation ô and its own character C,, and finally gets the ac- end for
// End simulation // Get proto-action fat,;; ah,1 ~ 1=(-10++1 Get post-action a + g(ag, W) Execute 04,, {at,a 3 Update 81+1
tion Øz,; = {af,;, = TT(-100+1, c.). In other words, the agent can select an adaptive action with consideration for others' upcoming behaviors. The decision-makee making procedure with the proposed EFT mechanism is summarized in Algorithm 3
5 Experiments
To select suitable task that can verify the effectiveness of the proposed solution, we consider the following requirements. There should be multiple approaches to achieving character diversity, as well as interactions between agents. The agent should have only partial observations of the state, and the action space should be both continuous and discrete.
We chose the autonomous driving task, which has numerous automated vehicles on the road. The task can consider the driving character of the agent based on driving preferences (e.g., one agent prioritizes safety, and the other prioritizes speed) 44 4 50 23 22). Additionally, it is realistic for driver to behave under the partial observation of the road state, and the driver makes decision in a hybrid action space. To implement this task, we use the FLOW framework [38] 8) The scenario includes multiple automated vehicles on the highway. The number of agents E 21, and each agent decides acceleration and lane change control at a given observation. Here, we express the
21
driving character using weights of three reward terms, i.e., C; = [ca,1, 0,,,, C1,3] The target agent is limited to the vehicles located in the observable area.
To confirm the scalability of the proposed solution, we also provide simulation results with multiple particle environment (MPE) [2] and starcraft multi-agent challenge (SMAC) [88. popular MARL testbed. All results in this section are averaged results of over 10 independent experiments. The
'If the agent i cannot observe the entire set of agents, subset of the agent can be the targets of agent i, i.e.
Fo, C F.
Details regarding the experiments are presented in Appendix G

---

markers indicate the average value, and the shaded area represents the confidence interval within one standard deviation.
5.1 Performance Evaluation: Character Inference
To make the EFT mechanism more effective, an accurate character inference should be preceded. In this subsection, we investigate the character inference module with two questions.
How many iterations does it require to achieve an accurate inference (in terms of repetition Algorithm 2
How long should the agent collect the observation-action trajectories of target agents (in terms of trajectory length in Algorithm ?
In Figure 3 the performance of the character inference module is presented. To ignore the effect of the initial point in convergence, the initial point of the character is randomly selected. More results
regarding the initial point are provided in Appendix !
Figure A/ illustrates the convergence of the esti- mated character to the true one. The inaccuracy of inference is evaluated based on the LI-norm between the estimated character and the true one. Thus, a lower L.1 -l-morm implies higher in- ference accuracy. As the number of iterations increases, the Li-norm quickly decreases to ap- proximately zero, meaning that the estimated value quickly converges to the true one. Specif- ically, if the number of iterations is over 500, high accuracy of the character inference can be achieved.
0.35
700
820
Iterations
500
0015
600
00000
0.000
6.055
200
1 50
1000
1250
1500
1750
20000
500
2000
3000
Ierations
Trojectory length (7)
Figure 3: The performance of the character inference module. A. Ll-norm between estimated and true charac- ters over the number of iterations (T 1000). B. The number of required iterations for convergence length the observation action trajectory T.
the
Figure BB shows the trade-off between the length of observation-action trajectory and the number of iterations required for the convergence. The convergence criterion is set to Li-norm & 5 10-4. The results demonstrate that the number iterations for convergence decreases as longer trajectories provided. Thus, the length of trajectories and the number of iterations can be jointly determined by considering system requirements.
5.2 Ablation Study: Character Inference and EFT Modules
We investigate the impact of two main modules (the character inference module and the EFT module) on performance by increasing character diversity levels the heterogeneous society. The following three cases compared.
Proposed: the agent enables the EFT with the inferred character of other agents based on the character inference module.
FCE-EFT: the agent experiences the FCE by assuming that all other agents have equal character to itself (i.e., Cj = c, Vj E E). So, no character inference is required. The agent performs the EFT, but action prediction is performed based on the same character c,,
without EFT (baseline) [II] the agent performs neither character inference nor the EFT mechanism. treats the problem as a single agent RL and selects the best action given observation. The policy is trained based on the TD3.
In Figure 4 the average rewards of entire agents presented over increasing the number of character groups The higher number of character groups means that more diverse characters cocxist in society, and the higher reward implies better performance. Because the number of agents is fixed to EE| 21, the number members per character group is |E/n,, where n denotes the number of groups. The members belonging the same group have the same character Note that character of each group is randomly sampled from character space in every independent experiment.
'Each market point is the average value of 10 independent test experiments. To obtain all results presented in Figure run 7 3 10 210 test experiments.
7

---

Figure 4 highlights the amount of reward enhancement or degradation by equipping the proposed modules. The proposed approach consistently outperforms the baseline (without EFT), and the FCE-EFT is inferior to the base- line when character diversity exists. These results verify that the EFT mechanism with accurate character inference always enhances the reward. However, the naive employ- ment of the EFT mechanism with the incorrect character degrades the reward. This is because incorrect charac- ter inference leads to incorrect action prediction and next observation simulation, which leads improper action selection of the agent, leading to low reward. Therefore, accurate character inference is crucial in the EFT mecha- nism.
Reward difference
400
200
Proposed
ECT
-00
The number character groups (n)
The amount of reward enhance- ment for two EFT approaches by setting wi thout EFT as baseline (i.e. the reward of other approaches the reward of without EFT).
Figure 4:
5.3 Investigating the Effects of Trajectory Noise
To infer the character of the target agent, the EFT agent needs to collect observation-action trajectories of the target agent. Since the observations made by the EFT agent towards the target agent may not be perfect (i.e., they could be a noisy version of the target agent's 's true observations), we further investigate the performance of the proposed EFT framework concerning the accuracy of the collected trajectories. This investigation consists two steps. First, look deeply into the effect of trajectory accuracy on character inference, and thereafter, we examine the EFT performance regarding character inference accuracy.
Character inference with trajectory accuracy. Table shows the character inference accuracy as the noise level for a collected trajectory in- creases. As expected, the character inference ac- curacy decreases the noise variance increases.
Table Character inference accuracy over the standard deviation of trajectory noise. (Accuracy: ACC)
Standard deviation of trajectory noise
0.01
Please be aware that the considered standard de-
viation is not trivial given that our observation range is [-1.1] Specifically, we provide the signal-to-noise ratio (SNR) with quality level (Qual) across each standard deviation. We label the quality of each level based on (13)
ACC
99.6
98.3
91.8
81.1
69.5
#0.01
#0.07
+0.23
+0.52
40.66
SNR[dB] Qual
Excellent
Good
Fair
Poor
Poor
believe that this result provides valuable in-
-
sights into the expected performance of our pro- posed solution, particularly in scenarios where observation prediction technology is deployed.
Reward
1000
EFT performance with character accuracy. In Figure 5 and y axes are the accuracy of character inference and average reward, and n is diversity level. As expected, the result shows that the performance of the EFT agent naturally increases when the accuracy of predicted ob- servation increases. Interestingly, the proposed solution holds up the performance acter inference accuracy of approximately 90% (i.e., the error rate 10%). is also worth mentioning that the performance has similar trend across the diversity levels, which confirms that the proposed method is robust against diversity levels.
1000
70
Accuracy (%)
Figure The average reward for increasing the accuracy of character inference.
a char-
5.4 Assessing Generalizability: Inference on Out-of-Diitrriuution Character
can be impractical and challenging to train all characters within a pre-defined range, and a trained agent can confront out-of-distribution (OOD) character in the deployment phase. This subsection demonstrates the inference performance on the OOD range of pre-trained agents with specific character samples. To this end, we consider the following two cases:

---

1. train on [0.0,0.6] and [0.8, 1.0], thereby inferring on {0.65, 0.7, 0.75),
2. train on [0.2, 0.8), thereby inferring on {0.0, 0.1, 0.9, 1.03.
Figure 0 shows the average of esti- mated characters over true ones. The gray dimmed area is the OOD range. which is an unseen character range in the training phase, and red and blue circles present an OOD and in- distribution estimated character value, respectively. Figure A represents case I, where the model appears to perform well in predicting characters in unseen regions. Figure 6 B repre- sents case 2, which performs inference the points outside of the trained range. It is observed that the inference accuracy is slightly declined compared to case 1, but it can still successfully capture the overall pattern by predicting the extreme values that are close the true ones.
000 range
In-distributon sample
OOD sample
Estimated 4
1.0
Aggo Seongre grrggr
Estimated
0.5
0.0
0.0
True
True
Figure 6: The performance of the character inference module on OOD character range.
5.5 Performance Comparisons
We compare the performance of the proposed solution to the following popular MARL, model- based RL, and agent modeling algorithms: MADDPG [29], MAPPO (63) Q-MIX 44), Dreamer [15], MBPO [66. ToMC2 [59]] and LIAM [38]. In baseline algorithms, we go through independent policy training regarding the diversity level of society"] Note that the proposed method does not need plural
training for different heterogeneity settings. See Appendix D for an additional explanation of the baseline algorithms and standard deviation for Table 2
Table shows the average reward of the entire agents as the number of character groups increases. This re- sult verifies that the proposed solu- tion outperforms all popular MARL algorithms. Note that the MARL al- gorithms assume centralized training, which requires access to the observa- tions and actions of all agents in pol- icy training. In contrast, our solution trains the policy with only local ob- servations and actions, which can be a more practical solution. The Q-MIX has the lowest performance since it operates in a discrete action space, whereas our task is in a hybrid action space.
Table 2: Performance comparison across diversity level.
The number of character groups (rr)
Algorithm
1
2
4
5
Proposed FCE-EFT
2899
3047
2976
2948
3051
2899
2784
2646
2566
2629
MADDPG 29
2763
3006
2800
2933
2856
MAPPO
63
2753
2862
2597
2529
2763
Q-MIX
2199
2310
2288
2118
1861
Dreamer 15
2911
2813
2733
2631
2701
MBPO 16
2089
1964
1523
1893
1633
ToMC2 59
3016
2812
2683
2691
2511
LIAM
1913
1792
1771
1683
1733
Table 2 also demonstrates the performance of popular model-based RL algorithms as the diversity level increases. is obvious that the performance gap between model- based RL and the proposed solution increases the diversity level increases. In addition, the standard deviation of model-based RL algorithms (provided in Table in Appendix[]] DT is much larger than the proposed solution, which shows the difficulty of learning a dynamic model without understanding others in multi-agent systems. Specifically, Dreamer cannot adapt to high diversity levels, and it has a broader variance than other algorithms. Additionally, the result of MBPO exhibits that it is hard trust generated transitions from a dynamic model.
In the case of agent-modelhe algorithms, ToMC2 achieves the best score in the its performance decreases as the diversity level increases; LIAM fails at all diversity levels. On the other hand, the proposed solution is robust to changes in the surrounding agents and maintains high performance across diversity levels. We conjecture why two baselines fail in this setup, as follows.
scenario, but
For each algorithm, five independent trainings are performed since five heterogeneity settings are considered, Le.. 2,, 4,

---

ToMC2 requires retraining or adjusting the ToM module as surrounding agents change. The ToM module is tailored other agents for the prediction of information (e.g.. goals, observations, and actions). Next, LIAM also necessitates a new opponent modeling process for each test environment. In addition, prior works on opponent modeling rarely involve more than four players.
5.6 Additional Evaluation MPE and SMAC
Beyond the autonomous driving task, we run the performance comparison on the MPE and SMAC testbed.
Table 3: Performance comparison with MARL baseline algorithms on MPE tasks. Performance of denoted algorithm is based on (3)
Task
MAPPO*
MADDPG
Q-MIX
Proposed
Multiple Particle Environment. The MPE tasks consider small number of agents (three or four) and groups (one two). Therefore, we the character for each group a single character, that is, the diversity level each task of MPE. Even though our method is specialized for high level of character diversity environment, the results demonstrate that the proposed solution is competitive a simple environment by achieving the best score in two out of three tasks. We provide additional information on the MPE task Appendix K
Spread Adversary
149.26
157.10
-154.70
-149.12
9.61
7.80
8.11
10.01
Tag
13.78
6.65
15.00
14.57
1. Table 3 shows the performance comparison across
StarCraft Multi-agent Challenge. The setup of SMAC tasks is similar to MPE tasks, f.e., the EFT agent does not need to infer the character because they have the same (character diver- sity as n 1). Table 4 exhibits the performance on the SMAC tasks. The proposed solution demonstrates supe- rior performance across SMAC tasks,
Table Performance comparison with MARL baseline algorithms on SMAC tasks. Performance of f denoted algorithm is based on 63
Task
MAPPO
MADDPG
Q-MIX
Proposed
2s3z
100
90.3
95.3
98.8
3s5z_vs_3s66
63.3
18.9
82.8
84.3
6h_vs_88 _z
85.9
68.0
9.4
93.8
particularly excelling in more complex scenarios like shows competitive performance, especially in simpler tasks like 2s3z, the proposed method proves more effective overall in handling both simple and complex multi-agent tasks. Additional information in terms of SMAC tasks can be shown in Appendix L
and 6h_vs vs_ 1s_8z. Although MAPPO
6 Discussion
Conclusion. In this paper, propose the EFT mechanism, which is social decision-makinn approach for a multi-agent scenario. The EFT mechanism enables the agent to behave by considering current and near-future observations. To achieve this functionality, we first build a multi-character policy that is generalized over character space. Then, the agent with the multi-character policy infer others' characters using the observation-action trajectory. Next, the agent predicts the others' behaviors and simulates its future observation based on the proposed EFT mechanism. In the simulation result, confirm that the proposed solution outperforms existing solutions across all diversity levels of the heterogeneous society.
Broader Impacts. The proposed EFT idea paves the way for research on multi-agent scenarios. The proposed method enables the agent to simulate other agents' upcoming actions, which analogous to humans' decision-makin Furthermore, we believe the proposed method can be broadened by combining counterfactual thinking, current information, and future thinking.
Limitations. Even though this work shows promising results with novel method, there are a few limitations tackle. In our experiments, there only one EFT agent, and all other agents do not have the EFT functionality. This is an inevitable setting to make the problem tractable. Additionally, we follow the non-stationary regarding the agent's policy in the training phase and stationary in the execution phase. Since the character is mapped into policy, this stationary property has a connection to the character itself. To improve practicality, we should further investigate how the proposed solution works when the other agent's policy is non-stationary in the execution phase.
10

---

Acknowledgement
This research was supported in part by the National Research Foundation of Korea (NRF) grant (RS-2023-002788, 12), and in part by the Institute of Information & communications Technology Planning & Evaluation (ITTP) grants (No. 2021-0-00739, 2022-2020-0-016100 funded by the Korea government (MSIT). D. Lee is grateful for financial support from Hyundai Motor Chung Mong-Kon Koo Foundation.
References
[1] A. Asayama, M. Nagamine, R. Kainuma, L. Tang, S. Miwa, and M. Toyama. The effect of episodic future thinking on learning intention: Focusing on english learning goal-relevant future thinking in university students Japanese Psychological Research, 2024.
[2] S. Baron, Mindblindness: An essay on autism and theory of mind. MIT press, 1997.
[3] X. Chen, S. Wang, L. Qi, Y. Li, and L. Yao. Intrinsically motivated reinforcement learning based recommendation with counterfactual data augmentation. World Wide Web, 26(5):3253-32742 2023.
[4] L. Eboli, G. Mazzulla, and G. Pungillo. How drivers' characteristics can affect driving style. Transportation Research Procedia, 27:945- 02255,9552220 2017.
[5] V. Egorov and A. Shpilman. Scalable multi-agent model-based reinforcement learning. In International Conference Autonomous Agents and Multiagent Systems, 2022.
[6] D. Engelmann and M. Strobel. The false consensus effect disappears representative informa- tion and monetary incentives given. Experimental Economics, 3(3):241-266 2000.
[7] D. Engelmann and M. Strobel. Deconstruction and reconstruction of an anomaly. Games and Economic Behavior, (6(2))678-689 2012.
[8] C. Eom, D. Lee, and M. Kwon. Selective imitation for efficient online reinforcement learning with pre-collected data. ICT Express, 11(2), 2025.
[9] Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, Counterfactual multi-agent policy gradients. In AAAI Conference Artificial Intelligence, 2018.
[10] D. Folli and I. Wolff. Biases in belief reports. Journal of Economic Psychology, 88:102458, 2022.
[11] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, pages 1587-1596, 2018.
[12] A. Furnas and T. LaPira. The people think what think: False consensus and unelected clite misperception of public opinion. American Journal of Political Science. 2024.
[13] Geier. How to: Define minimum SNR values for signal coverage. Viitatt,, 23:2012, 2008,
[14] A. Gorodetskiy, K. Mironov, and A. Panov. Model-based policy optimization with neural differential equations for robotic arm control. In International Conference on Intelligent Computing and Robotics, 2023.
[15] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: imagination. In International Conference on Learning Representaations, 2020.
Learning behaviors by latent
[16] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Conference Neural Information Processing Systems, 2019.
[17] A. Jern, Lucas, and Kemp. People learn other people's preferences through inverse decision-making, Cognition, 168:46-64, 2017.

---

[18] M. Kwon, S. Daptardar, P. Schrater, and Z. Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. In Conference on Neural Information Processing Systems, 2020.
[19] H. Lai, J. Shen, W. Zhang, and Y. Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning. 2020.
[20] C. Langley, B. Cirstea, F. Cuzzolin, and B. Sahakian. Theory of mind and preference learning the interface of cognitive science, neuroscience, and AI: A review. Frontiers Artificial Intelligence, page 62, 2022.
[21] D. Lee, C. Eom, and M. Kwon. AD4RL: Autonomous driving benchmarks for offline rein- forcement learning with value-based dataset. IEEE International Conference on Robotics and Automation, 2024.
[22] D. Lee and M. Kwon. ADAS-RL: Safety learning approach for stable autonomous driving. ICT Express, 8(3):79--483, 2022.
[23] D. Lee and M. Kwon. Stability analysis in mixed-autonomous traffic with deep reinforcement learning. IEEE Transactions on Vehicular Technology, 72(3):2848-23) 2862, 2022.
[24] D. Lee and M. Kwon. Foresighted decisions for inter-vehicle interactions: An offline rein- forcement learning approach. In IEEE International Conference Intelligent Transportation Systems, pages 1753-1758. IEFE, 2023.
[25] D. Lee and M. Kwon, Instant inverse modeling of stochastic driving behavior with deep reinforcement learning. IEEE Transactions on Consumer Electronics, 2024.
[26] Leroy, P. Morato, J. Pisane, A. Kolios, and D. Ernst. IMP-MARL: a suite of environments for large-scale infrastructure management planning via marl. Conference Neural Information Processing Systems, 2023.
[27] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[28] H. Lin, Y. Sun, J. Zhang, and Y. Yu. Model-based reinforcement learning with multi-step plan value estimation. arXiv preprint arXiv:2209.05530. 2022.
[29] R. Lowe, Y. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and Mordatch. Multi-agent actor- critic for mixed cooperative-competiivv environments. In Conference Neural Information Processing Systems, 2017.
[30] W. Luo, C. Park, A. Cormman, B. Sapp, and D. Anguelov. JFP: Joint future prediction with interactive multi-agent modeling for autonomous driving. In Conference on Robot Learning. 2023.
[31] V. Mchta, B. Paria, J. Schneider, S. Ermon, and W. Neiswanger. An experimental design perspective on model-based reinforcement learning. In International Conference on Learning Representations, 2022.
[32] T. Moerland, Broekens, A. Plaat, and C. Jonker. Model-based reinforcement learning: A survey, Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[33] A. Ng and S. Russell. Algorithms for inverse reinforcement learning. In International Confer- ence on Machine Learning. 2000.
[34] M. Oberst and D. Sontag. Counterfactual off-policy evaluation with Gumbel-max structural causal models. In International Conference on Machine Learning, 2019.
[35] Olichoek. Decentralized POMDPs. In Reinforcement Learning, pages 471-503. Springer, 2012.
[36] G. Owen. Game theory. Emerald Group Publishing, 2013.
12

---

[37] Pan, A. Sumalee, R. Zhong, and N. IndraPayoong. Short-term traffic state prediction based on remporal-spatial correlation. IEEE Transactions on Intelligent Transportation Systems, 14(3):1242-1254, 2013.
[38] G. Papoudakis, Christianos, and S. Albrecht. Agent modelling under partial observability for deep reinforcement learning. Conference on Neural Information Processing Systems, 34: 19:100 19222. 2021.
[39] G. Papoudakis, F. Christianos, L. Schäfer, and S. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Conference on Neural Information Processing Systems, 2020.
[40] P. Parmas, T. Seno, and Y. Aoki. Model- based reinforcement learning with scalable composite policy gradient estimators. In International Conference on Machine Learning, 2023.
[41] M. Patrício and A. Jamssidnejjjd. Dynamic mathematical models of theory of mind for socially assistive robots. IEEE Accexs, 2023.
[42] N. Rabinowitz, F. Perbet, F. Song. C. Zhang, A. Eslami, and M. Botvinick. Machine theory of mind. In International Conference on Machine Learning, 2018.
[43] R. Raju, Z. Li, S. Linderman, and X. Pitkow. Inferring inference. arXiv preprint arXiv:2310.03186, 2023.
[44] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, 2018.
[45] A. Redish and S. Mizumori. Memory and decision making. Neurobiology of Learning and Memory, 117:1, 2015.
[46] S. Rosbach, V. James, S. Großjohann, S. Homoceanu, and S. Roth. Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving. In IEEE International Conference Intelligent Robots and Systems, 2019.
[47] L. Ross, Greene, and House. The false consensus phenomenon: An attributional bias in self-perception and social perception processes. Journal of Experimental Social Psychology, 13(3):279-301, 1977.
[48] M. Samvelyan, T. Rashid, C. S. Witt, G. Farquhar, N. Nardelli, T. Rudner, C. Hung, P. Torr, J. Foerster, and S. Whiteson. The stareraft multi-agent challenge. In International Conference on Autonomous Agents and Multiagent Systems, 2019.
[49] M. Schirner, G. Deco, and P. Ritter. Learning how network structure shapes decision-making for bio-inspired computing. Nature Communications, 14(1):2963, 2023.
[50] M. Schrum, E. Sumner, M. Gombolay, and A. Best. MAVERIC: A data-driven approach to personalized autonomous driving. IEEE Transactions on Robotics, 2024.
[51] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00377 2017.
[52] J. Shao, Y. Qu, C. Chen, H. Zhang, and X. Ji. Counterfactual conservative Q learning for offline multi-agent reinforcement learning. arXiv preprint arXiv:2309. 1...99.6,9...66,, 2023.
[53] T. Spampatti, U. Hahnel, E. Trutnevyte, and Brosch. Psychological inoculation strategies fight climate disinformation across 12 countries. Nature Human Behaviour, 8(2)3380-39, 2024.
[54] R. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 160) 1991.
[55] R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018.
13

---

[56] R. Thorstad and P. Wolff. A big data analysis of the relationship between future thinking and Srcision-making Proceedings of the National Academy of Sciences, 115(8), 2018.
[57] K. Tsunekawa, A. Srinivasan, and M. Spranger. MA-Dreamer: Coordination and communication through shared imagination. arXiv preprint arXiv:2204.0460 2022.
[58] E. Vinitsky, A. Kreidieh, L.. Flem, N. Kheterpal, K. Jang, C. Wu, F. Wu, R. Liaw, E. Liang, and
A. Bayen. Benchmarks for reinforcement learning in sixed-autonomy traffic. In Conference on Robot Learning, 2018.
[59] Y. Wang, F. Zhong, J. Xu, and Wang. ToM2C: Target-oriented multi-agent communication and cooperation with theory of mind. In International Conference on Learning Representations, 2022.
[60] Westby and C. Riedl. Collective intelligence human-ai teams: A Bayesian theory of mind approach. In AAAI Conference on Artificial Intelligence, pages 61 19-6127, 2023.
[61] L. Yang, C. Zhao, C. Lu, L. Wei, and J. Gong. Lateral and longitudinal driving behavior prediction based on improved deep belief network. Sensors, 21(24), 2021.
[62] M. Ye, Y. Kuang, J. Wang. Y. Rui, W. Zhou, H. Li, and F. Wu. State sequences prediction via fourier transform for representation learning. Conference on Neural Information Processing Systems, 2023.
[63] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Conference on Neural Information Processing Systems, 2022.
14

---

Appendix:
Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning
Contents
Summary of Notations
16
System Specification
16
Hyperparameters
16
Algorithm
16
Algorithm 2
16
D Post-processor Function in (1)
17
Derivation of (2)
18
Loss Function for Character Inference
19
Experiments: Autonomous Driving
20
State
20
G.2 Observation
20
G.3 Action
20
Reward
20
H Behavioral Pattern over Character Coefficients)
22
Performance of Character Inference
23
Additional Simulation Results on Autonomous Driving Task
24
Original Plot of Figure|4
24
0.2
Performance Comparison with Confidence Interval
24
K
Multiple Particle Environment
25
Task Description
25
K.2
Performance Comparison with Confidence Interval
25
L StarCraft Multi-Agent Challenge
26
Task Description
26
Performance Comparison with Confidence Interval
26
15

---

Appendix A
Summary of Notations
Notation
Description
Notation
Description
a set of agents a state space
an agent i
S
8t
state at time f an observation of agent at time action of agent i at time t discrete action of agent at time t true action agent at time joint action except agent time an observation transition probabilities of agent i temporal discounted factor
O
an observation space of agent action space
Ot,
A
a continuous action of agent i at time
1
a proto-action of agent at time a joint action at time t a state transition probabilities a reward of agent time
a,
a
T
Ω2
R
?
C
a character space
D
dynamic model
C,
a character vector of agent
Appendix B System Specification
CPU
AMD Ryzen 9 3950X 16-core
GPU
GeForece RTX 2080 Ti
RAM
128 GB
SSD
IT
Appendix C Hyperparameters
C.1 Algorithm 1
Hyperparameter
Value
Hyperparameter
Value
total episodes (K)
3500
total timesteps (T)
3000
policy delay (d)
2
target noise variance (ō)
0.2
replay buffer size ((B)
4 x 10
train batch size (B)
128
discount factor (7)
0.99
soft update rate exploration variance 2 (a2)
1 x 10-3
exploration variance (a)))
0.1
0.6
actor learning rate
10 14
critic learning rate critic hidden node activation function of critic hidden layer activation function of critic output layer
10-4
actor hidden node
[64, 64]
664, 64
activation function of actor hidden layer activation function of actor output layer
ReLU
ReLU
tanh
linear
C.2 Algorithm²
Hyperparameter
Value
Hyperparameter
Values 10-3
optimizer
Adam
learning rate
the number of iterations (L)
200
the number of samples (N)
3000
16

---

Appendix D Post-processor Function in I
To build post-processor function g(-), we first allocate the continuous action space A - W. W into | = 2W + discrete action values. other words, a continuous number lies in the range oz
w
W- 2W SW+++ 2w+1
W
is assigned discrete action value A C. 2, iee,
W < a
W w 2W + 1
af = w. if
w
2W +1
The condition can be written the range of a w,
2W +1
W
2W +1
W
< w
(1)
2W
2W +1
2W
2W +1
and it can be reformulated as
(1
2W +1 2W
W
I.W) )
w wim
min
a
2W +1
where min(-, W) hinders w from being outside of action space [-W, W Here, the floor function is used on the right side of the inequality equation [ But the ceiling function on the left side of the inequality equation can be an alternative with the max function max(;--11
The post-processor function af = g(at W) is finally formulated as follows.
g(ag W) min (1
2W -
W
(.w)
2W
02
2W
17

---

Appendix E
Derivation of
Cj arg max In Plor:Tj,
arg maxin
P(s1: 01
(2)
a.r. d31.T
arg max In
P(sIrloou)) P(surlouT)
(3)
(. x In P(SIT, 01.T, d31.T
arg max
(4)
P(SIr(o1T))
= arg max
(15rr)or15:9: x In 0rs, 0ur + ( (s 9::
(5)
= arg max
((81:))111):::1..)) x In P(s1:r, or..),,..y1
(6)
The equality of 2 and 3) is because of multiplying the same value the numerator and denominator.
The inequality of (3) and (4) is based on Jensen's 's inequality, which means /(E[z]) 2 [x](x)) is satisfied when f(-) concave function (in our case, f(()) is In(-)). Subsequently, we can rewrite -P()) In P(-) as entropy (.)). The inequality of (5) and (6) is because the entropy (-)) is always positive value.
č = arg max / P(sirlouss a:T.)) x In P(sir) (i
T
P(sIrloET) In P(s)) + arg max
Σ t=1
In (n5
Σ
In n (ang)loss;
In T(+4 atj ar. ar. (e.-j)dai:ss) doir
(7)
In t (anllorsc)) x / PT
arg max
(8)
arg max
In n (arlooooosi
arg max
In (a (oggi)) + In Ta
(10)
We can decompose
by the Markov property. Next, we can ignore the S2(-) and T(-) of because these terms are not related to Likewise, we can ignore the P(s 01:T. 41:7,) of Consequently. (9) can be decomposed the probabilities with respect both continuous and discrete action as (1O) because we consider the hybrid action space.
18

---

Appendix Loss Function for Character Inference
If ( Joas c) is the Gaussian distribution and π (e,fi is the Dirac delta distribution, each term of the equation ((c) is defined as follows:
1 In () = In 2TO + 10 mzj
2
In In 1aat # ai,1 - -
*.c where dij and di,j means the indicator function. When the estimated deterministic action of is different to the actual *.d ) indicator function becomes 1; Conversely, when of a indicator action at.j (i.e., of # a..
a.d mean the actual action value sampled by observing the target agent, and 1[-]
+.d
a.d
function becomes O. If inferred character parameter è is similar to the actual character parameter the errors between the action produced by è and the observed actual action would decrease.
19

---

Appendix G Experiments: Autonomous Driving
To deal with a continuous state space, a hybrid action space, and the agents' characters, we consider the autonomous driving simulator.
In the demonstration task, the agents, the autonomous vehicles, drive the L-lane roundabout road. The agents are randomly deployed on the road in every episode. The agents' goal is to drive close to the desired velocity as possible, and the agents should control the acceleration and lane changes to reach the goal. To address this task, set the POMDP. Here, the state includes the velocity and position of all vehicles, and the observation includes information about neighboring vehicles. The action includes acceleration and lane change control in continuous and discrete space, respectively. The reward function comprises three terms: considering the desired velocity, safety distance, and meaningless lane change. We provide the specific POMDP model in the following subsection.
G.1
State
The state st E defined as
[v? Pr T
The state ,, means the total information of all vehicles the road. Here, = = 2 vt,N] represents the velocity of all vehicles, p: = Pr.1, Pt,., ... Pr.N] denotes the positions of the vehicles, and K; [kr,1, 2 kt,N] denotes the lane position of all vehicle at given time
G.2 Observation
The observation og E comprises the partial state information that the agent can observe. We assume that an agent can observe the following and leading vehicles located in the same and next lanes. Thus, we set the observation or, as follows:
o,,i [u, Ave., Apr.s kr.i)
where Ur denotes the velocity of an agent i, Av,, is relative velocity between the agent i and observable vehicles, Apr; is relative position, and ki, denotes the lane number given time I. Here, Av,,i = Av,,II. AUr,s, AU,,R Av.,ft, Av, A and Apr.: Apr,I.. Apr,ss, Apt.RR; A As, RR), where subscripts I and ʃ mean leading and fol- lowing vehicles, and subscripts L, S, and R signify located left, same, and right lane, respectively.
G.3 Action
The action a E A consists of a continuous action of,; E A and a discrete action af E Ad at time t. In this framework, continuous action acceleration control, and discrete action is lane change. Acceleration control space AA' is defined as a space from maximum acceleration to minimum acceleration [amin amar]; Lane change space Ad is defined as {-1, 0, 1}. In Ad, ax, -1 means the agent moves a lane outwards (right side), conversely af = means the agent moves a lane inwards (left side), and of means the agent keeps the same lane.
G.4 Reward
As discussed in section 3.1 the character-based reward function is defined as R (st,,, dr C;). this experiment, the reward function Re, is defined as:
Re, = + C2 + cg + Tfasi.
where {cr, C22 C2,03) e3) denotes vector of the character coefficients and {R,, R2, R₂) denotes vector the reward terms, and Ffast means a penalty for the unfeasible actions (i.e., trial to move a non-existence lane and lane where other vehicles are located.).
We use fas term for punishing unfeasible action, which is designed for safety learning purposes. By introducing this penalty, an agent can learn about unsafe decisions without experiencing an accident. In other words, allows the agent to use the safety assistant system fewer times, such as the ADAS (Advanced Driver Assistance System).
20

---

Subsequently, detailed equations of the reward terms are as follows.
The first reward term is defined as follows:
11+1.1 oj
R1
where of denotes the target velocity of the agent We consider that the agent can drive close the larget velocity. When Ut = v,, the reward term is maximized as the highest value 1; when Ut # oz the reward term lower than 1.
Next, the second reward term is defined as follows:
R2 R2(Ap(4,,1)) min 0,1 (Apt1.1s) (.1-(APET,,5)
8
)
2
where s* denotes the safety distance between the vehicles, and we design this reward term induce the agent to drive with the following vehicle in mind when the agent changes the lane. In this reward term, 8* is defined follows.:
Am++ 1,f
= + max 0, 1+11,
+"+
-Amin x Amaz
where denotes the minimum gap between vehicles, t" denotes the minimum time headway, the minimum time gap between two sequential vehicles required arrive at the location. This safety distance is based on the Intelligent Driving Model (IDM) controller, which one of the adaptive vehicular control systems [1]. If s* Apt+1.fs (i.e., the agent keeps the safety distance with following vehicle when moving the lane), R2 becomes the O; on the other hand, R2 becomes the negative value.
The third term is defined as follows.
R3 App.ss min(0, Apt+1.15 Apr,s
This reward term is related unnecessary lane changes, which is movement to lanes with less driving space than the current lane. When the agent changes the lane |a = and Apt.s < Ap+11.1 or keeps the lane |o!,11 0, this penalty term can be neglected (i.e., R3 0). Conversely, when the agent changes the lane |a and Apr,s 2 APt+I.s, this penalty term becomes the negative value.
21

---

Appendix H Behavioral Pattern over Character Coefficients
A
The number of lane changes
200 88
34
Reletive distance
Velocry
2.8
0.5
0.1
0.4
Figure HI: Behavioral pattern of the agent over character cnefficient Ca- A: Tendency of the average velocity of
the agent over character (C2 = c3 0). B: Tendency of the relative distance to the following vehicle over character cg (c1 cs 0). C: Tendency of lane-changing frequency over eg increases(c) c2 0).
To confirm behavioral differences over the character coefficient, we perform ablation studies on reward function by isolating the independent effect of each character coefficient. It can provide insight into how these characters impact the resulting trajectories. The behavioral differences resulting from character coefficients' changes illustrated in Figure HI The markers and shaded areas represent the average value and confidence interval with two standard deviations, respectively.
As described in Appendix G the reward function is defined Re.; = Rı + c2 + cgR. + ffails where R1, R2, and R. is related to desired velocity, safe distance and, lane-changing, respectively, Therefore, changes in each character coefficient affect average velocity, relative distance, and the number lane changes.
shows the average velocity of the agent as increasing e₁. This result verifies that the autonomous vehicle drives closer the desired velocity (uş 3.5m/s) s). Furthermore, the lower ei widens the dispersion area of velocity.
Figure HIA
Figure HI B represents the relative distance between the autonomous vehicle and the surrounding vehicle over C2. The result confirms that the relative distance increases as 02 grows. This character coefficient is straightforwardly related to safe distance. The agent would pursue safe driving by securing a larger driving space as C2 grows.
Figure HI shows the number of lane changes as c increases. In the reward function, C3 puts weights on the unnecessary lane-changing penalty. The unnecessary lane-changing implies movement to lanes with less driving space than the current lane. As cs decreases, the agent performs lane-chaining action more frequently.
22

---

Appendix I Performance of Character Inference
92
U(e)
90 Low
92
E500men 9,3,
Entimated 53
Estimatte 0
True S.3
Figure II: A. The converging trajectories of the character parameters. black diamond indicates the initial points, a red diamond indicates estimated points, a yellow star means the true point. The estimated character parameters the agent versus true character parameters. The orange line represents the identity line, meaning perfect cstimation; the bluc circles depict the estimated values, and the blue line presents the confidence interval for three standard deviations.
Figure ITA presents the contour plots of the log- likelihooh function for the combination of character parameters C, where & E [1. [1,2,3] 3]. It shows that the true value is well inferred no matter where the initial value located. The yellow star, red and black diamonds in these diagrams represent the true, estimated, and initial points, respectively; the curve line presents the character inference trajectory from an initial point to an estimated point.
Figure II shows the estimated character value by the agent versus the true character value of the target J. Each blue point and bar is the average value and the three-standard deviation considering experiments. The orange line indicates that the estimated and true values are identical. represents that the character inference is successful without large and in particular, Cj,1 and Cj.3 are overall accurate with a small standard deviation. Conversely, the inference about C,2 becomes inaccurate when 2 1.2. We conclude that the character inference module generally infers the agent's 's characters well over the observation-acee actioon trajectory of the target agent.
between the estimated and true value,
23

---

Appendix, J Additional Simulation Results on Autonomous Driving Task
J.1 Original Plot of Figure
Tigure[Tsshovee the original version of Figure i.e. the average reward of entire agents. In sin- gle group scemario (i.e., entire agents have the same characters), the results of both the pro- posed and the FCE-EFT solutions are identical. This is because all agents have homogencous characters, which allows the FCE agent to have the accurate characters of others. The reward of without EFT is lower than two solutions in single group scenario. This confirms that the proposed EFT mechanism can help the agent to consider multi-agent interactions. Next, in two or more group scenarios, the proposed solution consistently achieves the highest reward, and the FCE-EFT consistently achieves the lowest reward.
3200
Propesed FCE-EFT
3000
wo EFT
Reward
2600
2400
The number character groups (n)
Figure J1: Average reward of entire agents in- creasing number character groups.
Performance Comparison with Confidence Interval
Table J1: Table 2 with std confidence interval.
MARL algorithms:
MADDPG [29] It is multi-agent version of Deep Deterministic Policy Gradient (DDPG) (27]. In training, it uses cen- tralized Q-function that uses observations and actions of all agents.
Algorithm
The number character groups (n)
2
3
4
5
2899
3047
2976
2948
3051
Proposed
+217
162
+99
+-11
+00
2899
2784
2646
2566
2629
FFE-EFT
MAPPO 633 It is a multi-agent version Proximal Policy Optimization (PPO) algo- rithm [5T] considers a centralized critic that uses the local observations across all agents.
#217
161
196
1003
125
2763
3006
2800
2933
2856
MADDPG
122
1008
=106
+98
121
2753
2862
2597
2529
2763
MAPPO
+00
+01
114
1131
190
Q-MIX [4]: It uses mixer and individual Q-networks. The mixer network uses the Q-values (output of individual Q-network) of all agents as inputs and calculates a global Qtot handle the discrete action space, we quan- tize the continuous actions.
2199
2310
2288
2118
1861
Q-MIX
156
+39
118
182
132
2911
2813
2733
2631
2701
Dreamer
+312
2
#351
#521
1433
output. Since it can only
2089
1964
1523
1893
1633
MBPO
+8804
773
1948
-72
1821
3016
2812
2683
2691
2511
Model-based RL algorithms:
ToMC2
11109
1.233
L:309
1458
1.397
Dreamer [5]
It trains an agent that solves long-horizon tasks purely through latent imagination. This solution first builds a reward and transition model and then approximates a policy using a value function. This value function is based on leveraging the error between the imagined return and the estimated state value.
LIAM
1913
1792
1771
1683
1733
+:300
1-100
-367
1381
1429
MBPO [1]]: It provides simple data augmentation process of employing short model-synthesized rollouts branched from the actual trajectory. We train a policy using a blend data, comprising synthesized and actual trajectories.
Agent modeling algorithms:
ToMC2 [ By incorporating the theory of mind concept, socially intelligent agents are developed that can determine when and whom they should share their intentions.
LIAM 38 By using autoencoder structures to extract representations from the ego agent's 's local information, it models the behaviors of other agents in partially observable environment.

---

Appendix K
Multiple Particle Environment
K.1
Task Description
We select the three MPE tasks: Spread, Adversary, and Tag.
Spread: In this task, there are three agents, Their objective is to reach three landmarks without collision with each other. A reward function the sum of negative distances from landmarks agents and collision penalty term.
Adversary: This task includes two cooperating agents and third adversary agent; there are true goal and false goal spots. The adversary can observe relative distances without communication about the goal spots. The cooperative agents aim to reach the goal spot while avoiding an adversary. The reward function is sum of the negative distance to the goal spot and the distance from the adversary to the true goal. We use adversary agent controlled by pre-trained [39]
Tag: This task is dubbed predator-pres prey task. The environment includes two types of agents and obstacles: single good agent, three adversary agents, and two obstacle blocks. The adversaries slower than good agent and receive reward when tagging good agent. We employ a pre-trained prey agent from (39
K.2 Performance Comparison with Confidence Interval
Table Performance comparison with MARL baseline algorithms.
Task
MAPPO*
MADDPG
Q-MIX
Proposed
149.29
157.10
154.70
1149.12
Spread
+0.94
+2.30
+4.90
+1.38
9.61
7.80
8.11
10.01
Adversary
10.07
11.43
10.37
10.33
13.78
6.65
15.00
14.37
Tag
14.40
13.90
12.73
12.95
25

---

Appendix L
StarCraft Multi-Agent Challenge
L.1
Task Description
This task includes various scenarios where two armies are controlled by allied agents and the game's AI. Each agent operates under partial observability and can only perceive the environment within its sight range. More precisely, observations include attributes like distance, health, and unit type of nearby allies and enemies. Next, the agents can take actions such moving in direction, attacking specific enemies, or healing allies. The objective is to maximize the win rate across episodes. Agents receive rewards based hit-point damage, enemy kills, and a bonus for winning, while losing results negative reward.
We select the three scenarios of the SMAC task: 2s3z, 3s5z_ vs ss_3s6z, and 6h_ vs 8z.
2s3z: This scenario considers the same number of agents for both alley and enemy. More precisely, each team includes 2 Stalkers and 3 Zealots.
3s5z_vs_3aee vs_3s6z: It deploys the 3 Stalkers and Zealots as allies and Stalkers and Zealots enemies.
6h_ 8z: This combat is performed by 6 Hydralisks against 8 Zealots.
L.2
Performance Comparison with Confidence Interval
Table LI: Performance comparison with MARL baseline algorithms.
Task
MAPPO
MADDPG
Q-MIxt
Proposed
100
90.3
95.3
98.8
2s3z
+1.5
+5.3
+2.5
+2.3
63.3
18.9
82.8
84.3
3s57_vs_3s62
+19.2
14.8
15.3
19.1
85.9
68.0
9.4
93.8
6h_vs_8z
1.0.9
134.7
12.0
16.7
26

---

NeurIPS Paper Checklist
You should answer [Yes], [No], or [NA].
[NA]
means either that the question is Not Applicable for that particular paper the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect
paper's contributions and scope?
Answer:
[Yes]
We have claimed our motivation, scope, and contribution in the section of Introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
Limitations be shown in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
27

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: Mathematical proof can be shwon in Appendix D and Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: We have provided the hyperparameter for the reproducibility of the proposed
solution in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/ cuplic/guides. heegll//guuiees/ooddsubbissssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/nips.cc/puile http:/ubllccggeeeeee https//oooooouoassssooooooooo11e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Task descriptions and experimental settings be shown in the Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use confidence intervals and IQR to confirm the statistical reliability of experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computation resource can be shown in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: Broader impact can be shown in the Appendix 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deg.oyeemmm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "oaFb037aXH", "reviewer": "Reviewer_RgnJ", "summary": "The paper introduces an Episodic Future Thinking (EFT) mechanism for reinforcement learning (RL) agents, inspired by cognitive processes observed in animals, to enhance social decision-making in multi-agent systems with diverse agent characteristics. The EFT mechanism uses a multi-character policy to infer the behavioral preferences of other agents, predicts their actions, and simulates potential future scenarios to select optimal actions. The authors evaluate the EFT mechanism in a multi-agent autonomous driving scenario and demonstrate that it leads to higher rewards and is robust across societies with varying levels of character diversity.", "questions": "Q1: Do you know the concept of **role**  introduced in previous MARL works? is there any difference between the introduced character and role?\n\nQ2: Can you validate the generalization of the agents by training them at a specific level and transferring them to other levels with unseen characters?", "limitations": "The authors discussed the limitations in the discussion.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper introduces an episodic future thinking (EFT) mechanism for RL agents, borrowing from cognitive processes observed in animals, representing an interesting application of biological insights to enhance AI decision-making processes.\n\n+ The multi-character policy handles both continuous and discrete action spaces, expanding the applicability of RL methods to more complex scenarios.\n\n+ The paper demonstrates the effectiveness of the EFT mechanism in a multi-agent autonomous driving scenario. The authors examine the robustness of the EFT mechanism across different levels of character diversity, showing its resilience in various social compositions.", "weaknesses": "- The paper primarily focuses on an autonomous driving scenario. Demonstrating the EFT mechanism's effectiveness across a broader range of multi-agent scenarios, e.g., SMAC and VirtualHome, could strengthen the argument for its generalizability. While the paper mentions the mechanism's effectiveness across different levels of character diversity, a detailed scalability analysis in terms of the number of agents and interactions with human or heterogeneous agents could provide further confidence in the approach.\n- The results in Table. 2 and .3 only report the average performance. It is necessary to report the standard deviation to make the results more confident, as the environments are highly dynamic and varying uncertainty.\n- Lacking baselines. There are some works that incorporate ToM or opponent modeling with MARL[1,2]. It is necessary to compare those methods, e.g., estimate the current observation or hidden state of others instead of the next observation, to demonstrate the advantages of the proposed methods.\n\n\nRef:\n[1] Agent modeling under partial observability for deep reinforcement learning. NeurIPS 2021\n[2] ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind, ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed feedback and valuable suggestions for enhancing our work. In response, we describe how we have revised the paper to address the reviewer's concerns and questions.\n\n- **Additional experiment results to prove generalizability**\n\nTo demonstrate the efficiency of the proposed solution, we would like to **report additional experimental outcomes on the SMAC (StarCraft multi-agent challenge)** [1], which is widely used in the multi-agent RL domain and recommended by the reviewer. The additional result ($4$ seeds) is as below.\n|SMAC Task|EFTM|MAPPO|MADDPG|QMIX|\n|-|-|-|-|-|\n| 2s3z|98.8 $\\pm$ 2.3| **100** $\\pm$ 1.5|90.3 $\\pm$ 5.3|95.3 $\\pm$ 2.5|\n| 3s5z vs 3s6z |**84.3** $\\pm$ 9.1|63.3 $\\pm$ 19.2|18.9 $\\pm$ 4.8|82.8 $\\pm$ 5.3|\n| 6h vs 8z|**93.8** $\\pm$ 6.7|85.9 $\\pm$ 30.9|68.0 $\\pm$ 34.7|9.4 $\\pm$ 2.0|\n| Total|**276.9**|249.2|177.2|187.5|\n\nThe table above shows that EFTM still has surpassing or matching performance with previous solutions. **It means that EFTM is capable of generalizing to solve widely-used MARL tasks, achieving the best total scores.** More precisely, we set a simple setup for the SMAC environment, akin to the MPE task, that is we follow a vanilla setup with a single character diversity $n=1$. The reported scores of MAPPO and QMIX are based on benchmark performance [2]. \n\nWe also checked VirtualHome [3] as the reviewer recommended, but it seemed inappropriate for us due to the requirement of the language model, so we decided not to use this task. Although this response did not cover VirtualHome [3], we believe that our results in three tasks, e.g., autonomous driving, MPE, and SMAC, can alleviate the concern of the reviewer. Thank you for your suggestion, which helps us to prove the generalization of our method.\n\n---\n- **Experiments of additional baselines - theory of mind and agent modeling**\n\nWe thank the reviewer for a valuable suggestion about comparison baselines. As per the reviewer’s suggestion, **we have run experiments on additional baselines, including ToM2C [4] and opponent modeling [5], as follows.**\n|character diversity|n=1|n=2|n=3|n=4|n=5|\n|-|-|-|-|-|-|\n|EFTM|2899 $\\pm$ 217|**3047** $\\pm$ 162|**2976** $\\pm$ 196|**2948** $\\pm$ 91|**3051** $\\pm$ 109|\n|ToMC2 [4]|**3016** $\\pm$ 109|2812 $\\pm$ 273|2683 $\\pm$ 309|2691 $\\pm$ 458|2511 $\\pm$ 397|\n|Opponent Modeling [5]|1913 $\\pm$ 330|1792 $\\pm$ 410|1771 $\\pm$ 367|1683 $\\pm$ 381|1733 $\\pm$ 429|\n\n**This result demonstrates the effectiveness and adaptability of our approach, achieving higher rewards when character diversity exists.** ToMC2 achieves the best score in the $n=1$ scenario, but its performance decreases as the diversity level increases; opponent modeling fails at all diversity levels. On the other hand, the proposed solution is robust to changes in the surrounding agents and maintains high performance across diversity levels. \n\nWe conjecture why two baselines fail in this setup, as follows. **ToMC2 requires retraining or adjusting the ToM module as surrounding agents change**. The ToM module is tailored to other agents for the prediction of information (*e.g.*, goals, observations, and actions). Next, **opponent modeling also necessitates a new opponent modeling process for each test environment**. In addition, prior works on opponent modeling rarely involve more than four players. In contrast, our selective tasks consider $20$ surrounding agents and their policies can be subject to change.\n\n---\n- **Validate the generalizability of the EFT agent**\nWe appreciate the reviewer's valuable feedback about the generalizability of the multi-character policy over unseen characters. As the reviewer pointed out, validating the generalizability of the proposed solution is important for realistic tasks. To verify this, we have run additional experiments under the following two cases:\n\n1) Case 1: Train multi-character policy by experiencing character range $[0.0, 0.6]$ and $[0.8, 1.0]$, then test the accuracy of unseen character inference over unseen characters $\\{0.65, 0.7, 0.75\\}$.\n2) Case 2: Train multi-character policy by experiencing character range $[0.2, 0.8]$, then test the accuracy of unseen character inference over unseen characters $\\{0.0, 0.1, 0.9, 1.0\\}$.\n\nBelow are additional results in terms of character inference ($20$ inference trials).\n|True character|0.65 (case1)|0.7 (case1)|0.75 (case1)|0.0 (case2)|0.1 (case2)|0.9 (case2)|1.0 (case2)|\n|-|-|-|-|-|-|-|-|\n|Inferred character|0.61 $\\pm$ 0.09|0.67 $\\pm$ 0.15|0.76 $\\pm$ 0.08|0.12 $\\pm$ 0.21|0.13 $\\pm$ 0.04|0.85 $\\pm$ 0.13|0.82 $\\pm$ 0.28|\n\nFor case 1, the inferred characters are reasonably close to their true values. This indicates that the policy could have partial generalizability through interpolation, even for values within the gap not explicitly covered by the training ranges. For case 2, as the actual character gets farther away from the experienced value, it loses generalization performance, increasing the standard deviation and gap between the inferred and true one. \n\nTo avoid encountering unseen values as much as possible, we should set a realistic character range and deeply consider the sampling method during the learning process. In our case, we used uniform random sampling so that diverse characters could be experienced evenly within the predefined character range. Additionally, we believe that some few-shot learning and adaptation methods can alleviate these problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRtMxmhuar", "reviewer": "Reviewer_pTth", "summary": "This paper presents Episodic Future Thinking (EFT), an approach for RL in multi-agent environments. EFT involves learning a multi-character policy (where character is a parameter that modifies the reward), and then using this to infer characters of other agents and planning accordingly, using these characters and learned policy to predict others’ trajectories more accurately. The paper demonstrates superior performance on a driving environment and multi-agent particle environments.", "questions": "Given that the proposed method gets to train a multi-character policy, which presumably involves training on a bunch of experience with multiple characters, how are the baselines put on an equal footing in terms of experience in the environment, with these different character objectives?\n\nHow is c varied during multi-character policy training? Is it randomly set each episode?\n\nMinor, and I may have missed this, but what model is used to do forward prediction? It might be helpful to briefly mention that in the main text, if it’s not there.", "limitations": "Seems adequate, if the above are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written throughout. It presents, to my knowledge, an original approach for multi-agent RL with characters. Results are well-described and make sense. The studies of 5.2 and 5.3 are welcome additions that help make sense of how the method works. The cognitive motivation makes a lot of sense, and broadly, modeling diverse other agent motives seems like a promising direction that has not received much attention.\n\n*Edit*: raised score to 6 following rebuttal.", "weaknesses": "My main concern is with the significance of the performance comparisons. For the driving task, my understanding is that the other agents have a range of characters. The proposed method has the opportunity to learn a multi-character policy. First, I have a concern as to how one might put the baselines on an equal footing in terms of experience — see Questions for that. Second, even if the baselines were put on an equal footing in terms of experience, how surprising is the result for the driving experiment? The driver environment has been designed so that the proposed method has precisely the right inductive bias — inferring a latent character vector.\n\nThe MPE testbed is less clearly set up so that the proposed model has the right inductive bias for it — though perhaps it helps to be able to have separate models of the different agent groups — and again (see Questions), it’s really unclear to me how you would put baselines on the same footing in terms of giving them experience modeling both groups.\n\nIt would be very helpful to include confidence intervals for these experiments. Performance on MPE testbed is very close, numerically, to baselines. Are those differences actually statistically significant? I think those environments tend to have pretty high variance.\n\nThe model-based baselines, especially Dreamer, shouldn’t be expected to work well in multi-agent environments like these without significant modifications, I think. Dreamer is not going to handle stochasticity of multi-agent environments well well given how the world model is set up by default. Did you modify it? And why use Dreamer v1 instead of the most recent version?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's thorough review and valuable suggestions about this work. Below, we outline how we have revised the paper to address the reviewer's concerns and questions. \n- **Fairness for experience in character diversity**\n\nWe agree with the reviewer that maintaining a fair experimental setup is crucial for performance comparison. As the reviewer correctly pointed out, we set up MARL agents with a single character during their training. This is because we thought character change could impede MARL training. \n \nTo address the reviewer's comment and confirm our belief, **we provide additional evaluations of baselines with a diverse character experience** - an equal footing in terms of character experience. We report the results (4 seeds) below.\n|Algorithm|n=1|n=2|n=3|n=4|n=5|\n|-|-|-|-|-|-|\n|EFTM|2899 $\\pm$ 217|3047 $\\pm$ 162|2976 $\\pm$ 196|2948 $\\pm$ 91|3051 $\\pm$ 109|\n|**MADDPG (new)**|2368 $\\pm$ 85|2419 $\\pm$ 262|2353 $\\pm$ 56|2310 $\\pm$ 142|2338 $\\pm$ 210|\n|MADDPG (original)|2763 $\\pm$ 126|3006 $\\pm$ 103|2800 $\\pm$ 106|2933 $\\pm$ 98|2856 $\\pm$ 121|\n|**MAPPO (new)**|2192 $\\pm$ 113|2366 $\\pm$ 91|2233 $\\pm$ 250|2180 $\\pm$ 313|2241 $\\pm$ 386|\n|MAPPO (original)|2753 $\\pm$ 206|2862 $\\pm$ 201|2597 $\\pm$ 144|2529 $\\pm$ 131|2763 $\\pm$ 190|\n\nThis result confirms that EFTM outperforms other solutions in a demonstration task. As same with our expectation, **other baselines show a performance degradation compared to the existing setup**. Nevertheless, the performance is maintained rather than decreased as the character diversity level $n$ increases. We conjecture its performance drop is caused by not considering the optimization method for multi-characters. Although we provide the experience variability to agents, they cannot use it efficiently.\n\nHere are explanations of additional procedures for ensuring a fair experience. \n1. Sample the level of character diversity in society. This step allows the agents to experience various levels of society. We establish the set of character diversity levels as $[1, 2, 3, 4, 5]$, considered in the evaluation phase.\n2. Sample the character for each agent. This step allows the agents to experience various characters. Each character component can be sampled from their pre-defined ranges.\n3. Episode runs.\n4. Iterative 1-3 steps.\n\n---\n- **Simulation detail for MPE task**\n\nWe would like to explain the detailed setup for additional tasks. For the MPE task, we consider a simple setup, akin to prior works, with character as a single character, that is, the diversity level $n=1$. For competitive tasks, we deployed the EFT agent on a good agent group and the pre-trained networks on an adversarial one. The EFT agent only predicts teammates' future actions, not an adversarial group. \n\nPlease note that the main body and appendix of the original manuscript include experimental setup information as described below.\n\n**Main body**: We set the character for each group as a single character, that is, the diversity level $n=1$.\n\n**Appendix**: \n1. Spread: In this task, there are three agents. Their objective is to reach three landmarks without collision with each other. A reward function is the sum of negative distances from landmarks to agents and collision penalty term.\n2. Adversary: This task includes two cooperating agents and a third adversary agent; there are true goal and false goal spots. The adversary can observe relative distances without communication about the goal spots. The cooperative agents aim to reach the goal spot while avoiding an adversary. The reward function is a sum of the negative distance to the goal spot and the distance from the adversary to the true goal. We use an adversary agent controlled by a pre-trained [7].\n3. Tag: This task is dubbed a predator-prey task. The environment includes two types of agents and obstacles: a single good agent, three adversary agents, and two obstacle blocks. The adversaries are slower than a good agent and receive a reward when tagging a good agent. We employ a pre-trained prey agent from [7].\n\n---\n- **Version of Dreamer**\n\nThank you for providing key discussion points about the proper baseline selection. We initially considered Dreamer v1 [8], v2 [9], and v3 [10] as candidates for baselines, and we selected Dreamer v1 for the following reasons. Among these, **Dreamer v3 has not been peer-reviewed yet, so we decided not to use it**. Next, we delved into Dreamer v1 and v2 papers, and then we found that they focused on different demonstration tasks. **Dreamer v2 is more focused on discrete control tasks**, for example, the Atari games [11]. Conversely, Dreamer v1 concentrated on continuous control tasks, such as DeepMind control suite [12], DeepMind lab [13], and some continuous Atari games [11].\n\nSubsequently, when considering model-based baselines, we only train a single agent with a world model in the training phase. We then deploy trained agents in driving environments, where there are pre-trained drivers with diversified characters. Note that deployed agents in the training and test phases are the same as EFTM. In summary, we did not over-modify the existing Dreamer model.\n\n---\n- **How does the character $c$ determine?**\n\nAs the reviewer understands, **we randomly sample the character $\\mathbf {c}$ from a character distribution in every episode**. More precisely, we use a uniform sampling method to set the character of an agent during the training process. Character $\\mathbf{c}$ is a vector of character components $[c_1, \\cdots, c_K]$. Each character component $c_k$ randomly sampled from pre-defined ranges, *e.g.*, $[0, 2.5]$. Sampled character is bounded fitting to significant figures, in our experiments we consider one significant figure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "frZB2Awwk4", "reviewer": "Reviewer_Cur7", "summary": "This paper introduces an Episodic Future Thinking (EFT) mechanism for reinforcement learning agents in multi-agent systems with heterogeneous characters. The authors propose a multi-character policy and a character inference module to enable agents to predict other agents' actions and simulate future scenarios. The EFT mechanism allows agents to make adaptive decisions by considering the predicted future state. The approach is evaluated in autonomous driving scenarios and multiple particle environments, demonstrating improved performance compared to existing multi-agent and model-based reinforcement learning algorithms.", "questions": "-  “In contrast, our solution trains the policy with only local observations and actions, which can be a more practical solution.” But you still need to train the character identification model and multi-character policy, which requires access to the other observations too?\n- “In addition, the standard deviation of model-based RL algorithms is much larger than the proposed solution, which shows the difficulty of learning a dynamic model without understanding others in multi-agent systems.“. What standard deviations are the authors referring to?\n- How does the EFT mechanism perform when all agents in the system are equipped with this capability? Does this lead to emergent behaviors or potential instabilities?\n- What is the scalability of the proposed method? How does its performance and computational cost change as the number of agents increases?\n- How robust is the character inference module to noisy or adversarial behaviors from other agents?", "limitations": "- The current study only considers scenarios with a single EFT agent, which doesn't fully capture the dynamics of multiple predictive agents interacting. In scenarios where multiple agents use EFT, there's a potential for feedback loops or cascading effects that could lead to suboptimal or unstable system behavior. This isn't explored in the current work.\n- The paper doesn't address the potential increased computational requirements of the EFT mechanism compared to simpler approaches, which could be a limitation in resource-constrained environments. A table or figure comparing wall-clock time would be insightful.\n- While the method is tested in two different environments, its performance in more complex, dynamic, or partially observable environments remains unexplored.\n\nThis paper presents an interesting approach to multi-agent reinforcement learning by incorporating episodic future thinking. While the idea is novel and shows some promise, the lack of statistically significant increase in performance and lack of comparison with regards to wall-clock time in the current study, raise concerns about its broader applicability and impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  A policy that can handle diverse agent characters is a significant contribution. This addresses a real challenge in multi-agent systems where agents may have different goals or behavioral traits.\n- The authors test their approach across various levels of character diversity and compare it with multiple reasonable baselines. This thorough evaluation strengthens the validity of their claims.\n- The method's effectiveness is demonstrated in both autonomous driving and multiple particle environments, suggesting potential applicability across different domains.\n- The paper is generally well-structured and clearly written.", "weaknesses": "-  The experiments only consider one EFT agent among non-EFT agents.\n- The results in Figure 4 are not statistically significant. There is also no standard deviation for the baseline.\n- No standard deviations provided in Table 2 and 3.\n- The difference between training and execution wasn't clear until it was mentioned in the conclusion.\n- The paper lacks a detailed analysis of the computational costs associated with the EFT mechanism, particularly as the number of agents or environmental complexity increases.\n- While the paper mentions POMDP, it doesn't deeply explore how partial observability affects the performance of the EFT mechanism.\n-  The improvement over baseline methods, while present, is not consistently substantial across all scenarios, particularly in the multiple particle environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s detailed feedback and constructive suggestions for improving our work. In response, we outline the revisions made to address the reviewer’s concerns and questions. We have marked the weakness, question, and limitation numbers associated with each discussion section.\n- **Computational complexity (W5, Q4, L2)**\n\nWe agree that considering computational complexity is crucial for practical solution development. To address the reviewer's concern, we have investigated it using a big $\\mathcal O$ analysis of the proposed solution with our setup. Below are the notations used in this analysis:\n1. $d$: denote the dimension of the input\n2. $|E|$: the number of agents\n3. $|E_{obs}|$: the number of observable agents\n\nBefore looking at the specific analysis, \\textbf{the complexities of EFTM is $\\mathcal O(|E_{obs}|\\times d^2)$ for the execution}, and vanilla policy requires $\\mathcal O(d^2)$. This implies that **the maximum time complexity of EFTM is limited, regardless of the environment's size**, since the maximum number of observable agents is fixed.\n\nWe provide how to calculate the complexity of basic policy operations as shown in the below table. This demonstrates that the complexity of these operations is $\\mathcal{O}(d^2)$. For EFT prediction, our solution requires $|E_{obs}|$-times the computational cost for others' action prediction. Therefore, the complexity of the proposed solution is $\\mathcal O(|E_{obs}|\\times d^2)$ for the execution.\n|Computation|Equation|Matrix Size|Complexity|\n|-|-|-|-|\n|The $1^{\\mathrm{st}}$ policy layer|$\\mathrm{out}_1 = \\sigma_1(W_1\\cdot x_t+b_1)$|$W_1 \\in \\mathbb{R}^{2d\\times d}, x_t \\in \\mathbb{R}^{d}$|$2d^2$|\n|The $2^{\\mathrm{st}}$ policy layer|$\\mathrm{out}_2 = \\sigma_2(W_2\\cdot \\mathrm{out}_1+b_2)$|$W_2 \\in \\mathbb{R}^{4d\\times 2d}, \\mathrm{out}_1 \\in \\mathbb{R}^{2d}$|$8d^2$|\n|The output layer|$a_t = \\tanh(W_3\\cdot\\mathrm{out}_2 + b_3)$|$W_3 \\in \\mathbb{R}^{2 \\times 4d}, \\mathrm{out}_2 \\in \\mathbb{R}^{4d}$|$8d$|\n|**Total of policy**|-|-|$\\mathcal{O(d^2)}$|\n\nOur solution focuses on predicting the actions of agents within the observation area, not the entire state. It means that regardless of the environment's size, the number of observable agents is limited. These computations can be performed in parallel, making our analysis more relevant and effective than simply comparing execution times for single operations.\n\nNext, our solution has advantages in terms of training time. To elaborate, the wall-clock times of EFTM and MARL (averaging MAPPO, MADDPG, and QMIX) are as follows: **for autonomous driving tasks, approximately $3.5$ and $17$ hours, and for MPE tasks, approximately $1.7$ and $2.5$ hours**. This significant gap comes from our objective, training a multi-character policy that can work in any multi-agent interactions. It means that EFTM considers training a single agent, not multi-agents. \n\n---\n- **Accessibility of others' trajectories (W6, Q1, Q5)**\n\nThank you for the thorough review and insightful comments. In this response, we would like to address the accessibility assumption in MARL and alternatives in terms of POMDP setup.\n\n(1) *Accessibility assumption*\n\nWe would like to explain the details of multi-character policy training and our consideration in the character inference process.\n\n**We build a multi-character policy without the accessibility of other's observations**. For training the multi-character policy, we follow this process:\n1. Sample the character for an EFT agent. This step allows the agents to experience various characters. Each character component can be sampled from their pre-defined ranges.\n2. Episode runs and train policy based on the RL process. \n3. Iterative 1-2 steps at every episode.\n\nRegarding the character inference process, **while it requires the other's observation, this does not imply direct access to the other's observation**. The agent can collect by observing other's trajectories. To address such practical concerns, we have studied the performance robustness corresponding to the trajectory noise level in the below discussion.\n\n(2) *Robustness against trajectory noise of POMDP setup*\n\nWe would like to clarify that the **one core aspect of partially observable MDP is the noisy level [14]**. We have provided a study of how robust our solution is as the noise level of the observed trajectory increases in our original manuscript (Section 5.2). In addition, we add the performance of EFTM matching with each standard deviation of additive Gaussian noise, as follows.\n|Std of additive Gaussian noise|0.01|0.05|0.1|0.2|0.3|\n|-|-|-|-|-|-|\n|Inference Accuracy|99.6 $\\pm$ 0.01|98.3 $\\pm$ 0.07|91.8 $\\pm$ 0.23|81.1 $\\pm$ 0.52|69.5 $\\pm$ 0.66|\n|Cumulative reward of EFTM|2833 $\\pm$ 316|2841 $\\pm$ 283|2795 $\\pm$ 613|2237 $\\pm$ 812|1435 $\\pm$ 1023|\n\nWe believe that this result provides valuable insights into the expected performance of the proposed solution, particularly in scenarios where observation prediction technology is deployed. Although we only consider noise level without an adversarial agent scenario, we politely assert that such a scenario is beyond our scope.\n\nPlease be aware that the considered standard deviation is not trivial given that our observation range is $[-1, 1]$. Specifically, we provide the signal-to-noise ratio with a quality level across each standard deviation. We label the quality of each level based on [15].\n|std of trajectory noise|0.01|0.05|0.1|0.2|0.3|\n|-|-|-|-|-|-|\n|signal-to-noise ratio|34.7dB|21.3dB|14.7dB|9.2dB|4.7dB|\n|quality level|Excellent|Good|Fair|Poor|Poor|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mdxFA0Djrk", "reviewer": "Reviewer_6v5k", "summary": "Introduce an episodic future thinking(EFT) mechanism, which, along with the mechanism of counterfactual, is a cognitive activity of human beings.\nThe proposed algorithm predicts future observation transitions and uses them to determine the next steps of action. Although the maximum likelihood method is also used to infer a character c, I do not believe that this paper has made a significant contribution.", "questions": "No", "limitations": "No", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Introduce an episodic future thinking(EFT) mechanism, which, along with the mechanism of counterfactual, is a cognitive activity of human beings.\nThe proposed algorithm predicts future observation transitions and uses them to determine the next steps of action. Although the maximum likelihood method is also used to infer a character c, I do not believe that this paper has made a significant contribution.", "weaknesses": "The proposed algorithm predicts future observation transitions and uses them to determine the next steps of action. Although the maximum likelihood method is also used to infer a character c, I do not believe that this paper has made a significant contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort. Here are our answers to the reviewer's comments.\n\n- **Our contribution and motivation**\n\nWe would like to clarify our contribution is not trivial. To emphasize our contribution, we summarize the novelty as follows. **We introduce a novel social decision-making approach by coupling character inference and upcoming future prediction**. Each independent functionality has been studied partially in different research streams but never jointly considered for social decision-making scenarios. **Moreover, its effectiveness in a scenario where heterogeneous agents coexist has never been experimentally proved**. Combining such functionality in the social decision-making framework is not straightforward or incremental, which is why we put considerable effort into clearly explaining the proposed solution.\n\nAdditionally, all other reviewers acknowledged that our method is novel with detailed points:\n- Reviewer Cur7: A policy that can handle diverse agent characters is a significant contribution. This addresses a real challenge in multi-agent systems where agents may have different goals or behavioral traits.\n- Reviewer ukHS: The integration of episodic future thinking in RL is a significant contribution, providing a new perspective on how agents can predict and simulate future scenarios to improve decision-making.\n- Reviewer RgnJ: The multi-character policy handles both continuous and discrete action spaces, expanding the applicability of RL methods to more complex scenarios.\n- Reviewer pTth: The cognitive motivation makes a lot of sense, and broadly, modeling diverse other agent motives seems like a promising direction that has not received much attention.\n\nThis work addresses a significant topic in the advancement of the MARL domain. As the reviewer cur7 mentioned, handling diverse characters is a substantial challenge in multi-agent systems where agents may have different goals or behavioral traits. \n\nWhile we sincerely want to provide more detailed responses, there is limited information about the discussion points, so we could not elaborate further. Please feel free to ask any additional questions the reviewer may have, and we will be happy to answer them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0YrnTz0WZ", "reviewer": "Reviewer_ukHS", "summary": "This paper introduces an Episodic Future Thinking (EFT) mechanism for reinforcement learning (RL) agents to enhance decision-making in multi-agent scenarios. The EFT mechanism allows an agent to predict the future actions of other agents by inferring their characters from observation-action trajectories. This capability is evaluated in multi-agent autonomous driving scenarios and multiple particle environments, demonstrating that EFT leads to higher rewards compared to existing multi-agent RL solutions.", "questions": "1. How does the performance of the proposed Episodic Future Thinking mechanism scale with an increasing number of agents?", "limitations": "The approach assumes that character traits can be inferred accurately, which might not hold in highly dynamic environments with rapidly changing behaviors. The limitation of having only one EFT-enabled agent in experiments raises questions about the method's effectiveness in scenarios where multiple agents are equipped with EFT capabilities.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The integration of episodic future thinking in RL is a significant contribution, providing a new perspective on how agents can predict and simulate future scenarios to improve decision-making. Besides, The paper is well-structured and clearly explains the methodology, experiments, and results, also provides comprehensive evaluations in diverse experiments, including ablation study, showcasing the robustness of the proposed method.", "weaknesses": "1.\tThe paper does not sufficiently address the computational overhead of implementing the EFT mechanism, especially with varying data sizes.\n2.\tI suggest that authors also implement SOTA in the experiment of investigating the effects of trajectory noise, so that compare the sensitivity of the proposed methods. \n3.\tThe approach assumes that character traits can be inferred accurately, which might not hold in highly dynamic environments with rapidly changing behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive reviews and insightful feedback about this work. Below, we describe how we have revised the paper to address the reviewer's concerns and questions.\n- **Computational complexity**\n\nWe agree that considering computational complexity is crucial for practical solution development. To address the reviewer's concern, we have investigated it using a big $\\mathcal O$ analysis of the proposed solution with our setup. Below are the notations used in this analysis:\n1. $d$: denote the dimension of the input\n2. $|E|$: the number of agents\n3. $|E_{obs}|$: the number of observable agents\n\nBefore looking at the specific analysis, \\textbf{the complexities of EFTM is $\\mathcal O(|E_{obs}|\\times d^2)$ for the execution}, and vanilla policy requires $\\mathcal O(d^2)$. This implies that **the maximum time complexity of EFTM is limited, regardless of the environment's size**, since the maximum number of observable agents is fixed.\n\nWe provide how to calculate the complexity of basic policy operations as shown in the below table. This demonstrates that the complexity of these operations is $\\mathcal{O}(d^2)$. For EFT prediction, our solution requires $|E_{obs}|$-times the computational cost for others' action prediction. Therefore, the complexity of the proposed solution is $\\mathcal O(|E_{obs}|\\times d^2)$ for the execution.\n|Computation|Equation|Matrix Size|Complexity|\n|-|-|-|-|\n|The $1^{\\mathrm{st}}$ policy layer|$\\mathrm{out}_1 = \\sigma_1(W_1\\cdot x_t+b_1)$|$W_1 \\in \\mathbb{R}^{2d\\times d}, x_t \\in \\mathbb{R}^{d}$|$2d^2$|\n|The $2^{\\mathrm{st}}$ policy layer|$\\mathrm{out}_2 = \\sigma_2(W_2\\cdot \\mathrm{out}_1+b_2)$|$W_2 \\in \\mathbb{R}^{4d\\times 2d}, \\mathrm{out}_1 \\in \\mathbb{R}^{2d}$|$8d^2$|\n|The output layer|$a_t = \\tanh(W_3\\cdot\\mathrm{out}_2 + b_3)$|$W_3 \\in \\mathbb{R}^{2 \\times 4d}, \\mathrm{out}_2 \\in \\mathbb{R}^{4d}$|$8d$|\n|**Total of policy**|-|-|$\\mathcal{O(d^2)}$|\n\nOur solution focuses on predicting the actions of agents within the observation area, not the entire state. It means that regardless of the environment's size, the number of observable agents is limited. These computations can be performed in parallel, making our analysis more relevant and effective than simply comparing execution times for single operations.\n\nNext, our solution has advantages in terms of training time. To elaborate, the wall-clock times of EFTM and MARL (averaging MAPPO, MADDPG, and QMIX) are as follows: **for autonomous driving tasks, approximately $3.5$ and $17$ hours, and for MPE tasks, approximately $1.7$ and $2.5$ hours**. This significant gap comes from our objective, training a multi-character policy that can work in any multi-agent interactions. It means that EFTM considers training a single agent, not multi-agents. \n\n---\n- **Comparison of the trajectory noise sensitivity with baseline**\n\nThank you for your comments, which helped us to improve the experimental section. To address the reviewer's concern, we have run additional experiments. Given time constraints, we only performed additional investigation on MADDPG, which showed the second-best performance in our selective demonstration tasks. Below are additional results ($4$ seeds).\n|std of trajectory noise|0.0|0.01|0.05|0.1|0.2|0.3|\n|-|-|-|-|-|-|-|\n|MADDPG (test noise)|2763 $\\pm$ 126|2530 $\\pm$ 439|1891 $\\pm$ 892|1522 $\\pm$ 1039|837 $\\pm$ 711|335 $\\pm$ 693|\n|MADDPG (training noise)|2763 $\\pm$ 126|2891 $\\pm$ 360|2610 $\\pm$ 402|2133 $\\pm$ 519|1258 $\\pm$ 1011|1341 $\\pm$ 955|\n|EFTM|2899 $\\pm$ 217|2833 $\\pm$ 316|2841 $\\pm$ 283|2795 $\\pm$ 613|2437 $\\pm$ 812|1535 $\\pm$ 1023|\n\nThe empirical result confirms that **the proposed solution is more robust than the MADDPG**. We conjecture that the proposed solution might alleviate noise effects through character classification since noise information is not used for direct action computation.  \n\nDetails of two types of baselines are as follows.\n1. Test noise without training noise: This case adds the noise into the MARL agents' observation directly. Direct noise for policy calculations quickly leads to the collapse of the entire multi-agent system. \n2. Training noise without test noise: This case adds the trajectory noise in a training phase, \\textit{i.e.}, building policy and team value function. A small amount of noise during the learning process serves to increase the robustness of the overall system, but as the noise increases, the instability of the learning process increases.\n\nPlease be aware that the considered standard deviation is not trivial given that our observation range is $[-1, 1]$. Specifically, we provide the signal-to-noise ratio with a quality level across each standard deviation. We label the quality level from [15].\n|std of trajectory noise|0.01|0.05|0.1|0.2|0.3|\n|-|-|-|-|-|-|\n|signal-to-noise ratio|34.7dB|21.3dB|14.7dB|9.2dB|4.7dB|\n|quality level|Excellent|Good|Fair|Poor|Poor|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our gratitude to all five reviewers for their insightful feedback. We are pleased to present the updates we have made in response to valuable suggestions, as detailed below. 

- We compared the performance with **two additional baselines** on the research about opponent modeling and theory of mind (Reviewer RgnJ). This result confirmed the effectiveness and adaptability of our approach, outperforming additional baselines.
- We evaluated EFTM and baselines on **SMAC (StarCraft multi-agent challenge)**, which demonstrates the generalizability of EFTM for the MARL domain (Reviewer RgnJ). This result demonstrated that the proposed solution still has comparable performance in a widely used MARL setup.
- We added **time complexity analyses** in the inference phase in terms of big $\mathcal O$ analysis and wall-clock time measure (Reviewer Cur7 and ukHS).

Next, we apologize for not including standard deviations for performance evaluations in the main body of the paper. Due to the page limit, we included **the results with standard deviations in Appendix J2 and K2 of the original manuscript** (Reviewer RgnJ, pTth, and cur7).

Below are the references we use in this response.

[1] S. Mikayel et al., The Starcraft multi-agent challenge. AAMAS 2019.

[2] Y. Chao et al., The surprising effectiveness of PPO in cooperative multi-agent games. NeurIPS 2022.

[3] P. Xavier et al., Virtualhome: Simulating household activities via programs. CVPR 2018.

[4] Y. Wang et al., ToM2C: Target-oriented multi-agent communication and cooperation with theory of mind. ICLR 2022.

[5] P. Georgios et al., Agent modelling under partial observability for deep reinforcement learning. NeurIPS 2021.

[6] Z. Xianghua et al., Effective and stable role-based multi-agent collaboration by structural information principles. AAAI 2023.

[7] G. Papoudakis et al., Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. NeurIPS 2020.

[8] H. Danijar et al., Dream to control: Learning behaviors by latent imagination. ICLR 2020.

[9] H. Danijar et al., Mastering Atari with discrete world models. ICLR 2021.

[10] H. Danijar et al., Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 (2023).

[11] M. Volodymyr et al., Human-level control through deep reinforcement learning. Nature 2015.

[12] T. Yuval et al., Deepmind control suite. arXiv preprint arXiv:1801.00690 (2018).

[13] B. Charles et al., Deepmind lab. arXiv preprint arXiv:1612.03801 (2016).

[14] S. Satinder et al., Learning without state-estimation in partially observable Markovian decision processes. Machine Learning Proceedings 1994.

[15] G. Jim. How to: Define minimum SNR values for signal coverage. Viitattu 23 (2012).

Meta-review: Reviewers unanimous agree on accepting the paper, with scores of (5,6,6,6,6). These scores were reached after the authors provided a detailed rebuttal with a large number of new experiments and results that directly addressed reviewers’ concerns. Reviewers engaged in a discussion with the authors and in several cases raised their scores. 

**Summary of reviewer opinions**

Reviewers praised the following strengths:
- They believed the idea for the method is a good one, and several called it a “significant contribution” (ukHS,Cur7) . “A policy that can handle diverse agent characters is a significant contribution. This addresses a real challenge in multi-agent systems where agents may have different goals or behavioral traits.” (Cur7)
- Inspiration from real cognitive processes in mammals (RgnJ); “The cognitive motivation makes a lot of sense” (pTth)
- Thorough experiments. “multiple reasonable baselines” (Cur7). “The studies of 5.2 and 5.3 are welcome additions that help make sense of how the method works.” (pTth).  Experiments show robustness across different scenarios (RgnJ)
- Clearly written and structured (pTth, Cur7,ukHS)
- “handles both continuous and discrete action spaces” (RgnJ)

Reviewers complained about the following weaknesses:
- “The driver environment has been designed so that the proposed method has precisely the right inductive bias” (pTth) (essentially, the driving experiment is set up so that other drivers have “characters” and this method is the right solution for that scenario)
- Not enough environments (original main paper was autonomous driving and MPE) (RgnJ)
- Complaints about insufficient baselines (pTth, RgnJ) (e.g. lacking baselines that incorporate ToM or opponent modeling with MARL (RgnJ))
- Lack of scalability / computational complexity analysis (RgnJ, Cur7,ukHS)
- Need standard deviations / confidence intervals in the results (RgnJ, pTth, Cur7)

**Effect of rebuttal**

In the rebuttal, authors added experiments which directly addressed the reviewers’ concerns, including comparing to two new baselines, obtaining results on a new environment (SMAC), giving baselines more experience with diverse characters, increasing the proportion of EFT agents, and some new generalization experiments. They also added a big Oh complexity analysis, and measured the wall clock time. Finally, they pointed reviewers to additional results with standard deviations and 3 additional environments provided in the appendix. The rebuttal also had good explanations for why certain decisions were made (e.g. choosing Dreamer v1 as opposed to v2-3 as a baseline). 

**Final recommendation**: Accept
Decision: Accept (poster)