A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
Authors: Anjie Liu
Jianhong Wang
Samuel Kaski
Jun Wang
Mengyue Yang
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent reinforcement learning
targeted intervention
pre-strategy intervention
multi-agent influence diagrams
multi-agent systems
hanabi
Primary area: reinforcement_learning
TL;DR: This paper investigates a principle and develops a method for intervening on a single, targeted agent in a multi-agent reinforcement learning to reach a collective objective consisting of the primary task goal and an additional desired outcome.

Abstract: Steering cooperative multi-agent reinforcement learning (MARL) towards desired outcomes is challenging, particularly when the global guidance from a human on the whole multi-agent system is impractical in a large-scale MARL. On the other hand, designing external mechanisms (e.g., intrinsic rewards and human feedback) to coordinate agents mostly relies on empirical studies, lacking a easy-to-use research tool. In this work, we employ multi-agent influence diagrams (MAIDs) as a graphical framework to address the above issues. First, we introduce the concept of MARL interaction paradigms (orthogonal to MARL learning paradigms), using MAIDs to analyze and visualize both unguided self-organization and global guidance mechanisms in MARL. Then, we design a new MARL interaction paradigm, referred to as the targeted intervention paradigm that is applied to only a single targeted agent, so the problem of global guidance can be mitigated. In implementation, we introduce a causal inference technique—referred to as Pre-Strategy Intervention (PSI)—to realize the targeted intervention paradigm. Since MAIDs can be regarded as a special class of causal diagrams, a composite desired outcome that integrates the primary task goal and an additional desired outcome can be achieved by maximizing the corresponding causal effect through the PSI. Moreover, the bundled relevance graph analysis of MAIDs provides a tool to identify whether an MARL learning paradigm is workable under the design of an MARL interaction paradigm. In experiments, we demonstrate the effectiveness of our proposed targeted intervention, and verify the result of relevance graph analysis.

A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
Anjie Liu"
Jianhong Wang*t INFORMED-AT Hub University of Bristol
Samuel Kaski*
Thrust of Artificial Intelligence HKUST (GZ)
ELLIS Institute Finland
Jun Wang
Mengyue Yang
Centre for Artificial Intelligence University College London
School of Engincering Mathematics and Technology University of Bristol
Abstract
Steering cooperative multi-agent reinforcement learning (MARL) towards desired outcomes is challenging, particularly when the global guidance from human on the whole multi-agent system is impractical in large-scale MARL. On the other hand, designing external mechanisms (e.g.. intrinsic rewards and human feedback) coordinate agents mostly relies empirical studies, lacking easy-to-use search tool. In this work, we employ multi-agent influence diagrams (MAIDs) as a graphical framework address the above issues. First, we introduce the concept of MARL interaction paradigms (orthogonal to MARL learning paradigms), using MAIDs to analyze and visualize both unguided self eefforraanzztion and global guid- ance mechanisms in MARL. Then, we design a new MARL interaction paradigm, referred as the targeted intervention paradigm that is applied only single targeted agent, so the problem of global guidance can be mitigated, In imple- mentation, we introduce causal inference technique-refent referree as Pre-Strategy Intervention (PSI)-to realize the targeted intervention paradigm. Since MAIDs can be regarded as a special class of causal diagrams, a composite desired outcome that integrates the primary task goal and additional desired outcome can be achieved by maximizing the corresponding causal effect through the PSI. More- over, the bundled relevance graph analysis of MAIDs provides tool to identify whether an MARL learning paradigm is workable under the design an MARL interaction paradigm. In experiments, demonstrate the effectiveness of our proposed targeted intervention, and verify the result of relevance graph analysis.
Introduction
Multi-agent reinforcement learning (MARL) provides powerful framework for sequential decision- making in complex interacting systems, applied in autonomous systems including robotics 3 central challenge in MARL is enabling effective coordination among agents that must operate with partial information in dynamic environments. Achieving robust coordination is often hindered by inherent difficulties such as con-stationarity 44, where concurrent learning destabilizes individual agent's 's perspectives, making stable learning difficult. Furthermore, effective coordination often requires guiding a multi-agent system towards desired outcome- an outcome with specific beneficial properties that standard MARL approaches may struggle to achieve consiseenty-esppecillly only maximizing a single, shared team reward (task goal) 6).
especially when
Equal contributions, names listed in alphabetical order. 'Correspondence to Jianhong Wang jjanhong wang@bristol, ac. uk). *Samuel Kaski also with Aalto University and University of Manchester.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address these challenges in MARL, researchers often introduce external mechanisms that guide agent interaction and learning- wwhat we refer to as MARL interaction paradigms- to enhance performance, efficiency, and applicability. Intrinsic rewards, for instance, can improve sample efficiency and direct agent behaviour by encouraging exploration or coordination 7 g 9. Another approach incorporating human feedback, leverages human expertise to accelerate learning. This helps agents tackle objectives that are complex to specify via simple reward functions, and aligns agent behaviour to desired norms or human values [10] [1] [2].
These MARL interaction paradigms go beyond the scope of the conventional MARL learning paradigms (e.g., centralized training and decentralized execution (CTDE) [3][ 9.1.31).
While beneficial, applying global guidance to the entire multi-agent system in MARL is often im- practical due to complexity, costs [14] and safety concerns [15] Consider, for example, autonomous vehicles navigating a complex intersection or merging onto a highway. Ensuring safe and efficient pas- sage requires intricate coordination. Also, providing simultaneous, specific feedback or instructions all vehicles is infeasible, due to challenges such as complex safety validation and lack of universal communication protocols [16]. Yet, effective group coordination remains essential. An intriguing possibility arises if additional desired outcome as a guidance applied single targeted vehicle, e.g., instructing it to adjust its speed or yield at a critical moment. The resulting behaviour can serve as a pattern, enabling surrounding vehicles to coordinate. This scenario highlights a critical challenge where global guidance is essential but prohibitively difficult or costly, leading directly to the central question motivating our work:
Can effective coordination be achieved when assigning an additional desired outcome to a single targeted agent, relying on its influence over the rest of the agents in multi-agent system?
Addressing the challenge of designing effective targeted intervention requires a principled framework to model and analyze how guiding single agent impacts the overall multi-agent system. To this end, we propose leveraging multi-agent influence diagrams (MAIDs) [77, as a formal graphical language to encode complex strategic dependencies and information flow within the MARL learning process. Crucially, their bundled relevance graphs enable the analysis of a key property that we refer to solvability. The identification of solvability guides better understanding and prediction of MARL. interaction paradigms. To answer our research question, we utilize MAIDs to design the targeted intervention paradigm that intervenes entire multi-agent system towards a desired outcome. Formally, we frame this problem as selecting preferred Nash equilibrium (NE) from many possible equilibria (18), where the preferred NE is precisely the one that ensures effective coordination with satisfying an additional desired outcome.
single targeted agent, which reliably guides the
The main contributions of this paper are summarized follows: (1) We introduce the concept of MARL interaction paradigms, using MAIDs to analyze and visualize both unguided self organizzatio and global guidance mechanisms in MARL. Then, we propose new MARL interaction paradigm, referred to as the targeted intervention paradigm, where only a single targeted agent is intervened (2) The solvability of existing MARL learning paradigms under various MARL interaction paradigms can be analyzed, with the help of visualizing the corresponding relevance graphs. (3) Since MAIDs can be viewed as causal diagrams, we draw on the principle of causal inference to implement the targeted intervention paradigm, which we refer to as the Pre-Strategy Intervention (PST). It is provable that the PSI can reach composite desired outcome consisting of the primary task goal for coordination and an additional desired outcome. (4) The PSI is implemented as a pre-policy module that can be integrated into generic MARL algorithms.
The proposed PSI is evaluated in Multi Aalti-Agenttii Particle Environment (MPE) and Hanabi 19 20). Furthermore, the solvability of different MARL learning paradigms concluded from relevance graphs are verified in experiments. Our code is publicly available as an open sourcs repository!"
2 Multi-Agent Influence Diagrams (MAIDs)
We review multi-agent influence diagrams (MAIDs) 17 which is an augmentation of the Bayesian network to describe multi-agent decision making to maximize their utility. In the rest of the paper, the terms variable and node will be used interchangeably. An MAID is usually described as a tuple M = (I, X, D,U, D, where I is a set of agents. X' is a set of chance variables

---

indicating decisions of nature. Each chance variable X E X' is associated with set of parents Pa(X) C X D. The D := U.ez D; a set of all agents' decision variables, where Dj is the set of agent i's decision variables. For a decision variable D D,, Pa(D) the set of variables whose values are informed to agent i when it selects a value of D. U := UiET.4 U, is a set of utility variables, where Ue agent i's utility variable as its utility function. Note that utility variables cannot be parents of other variables. A directed acyclic graph & with variables V X' U DD U is formed. Pr a conditional probability distribution (CPD) defined over chance variables X such as Pr(X and utility variables U E U such as Pr(U/pa), for each pa E dom (a(U))) Note that Pr(U) Pa(U)) is a Dirac function (i.e. U a deterministic function). In other words, for each instantiation pa E dom( Pa(U)), there is a value of U that is assigned probability 1, and probability of other values is O. To simplify the notation, (paa) denotes as the value of U that has probability when Pa(U)- = pa. The total utility that an agent obtained from an instantiation of V is the sum of the values of U4 i.c. Lveu, U Upa) where pa E dom(Pa(U))
Decision Rule and Strategy. An agent makes a decision at variable D depending on its Pa(D),
- A (D). A indicates the probability distribution space over set. An assignment of decision rules to each decision D E D is called a strategy profile. A partial strategy profile dg is an assignment of decision rules to subset of D, as a restriction of to E, and Ø-g denotes the restriction of a to variables D',E. The assignment of og to the MAID .M induces a new MAID denoted by M[or], and each D E & would become a chance variable with the CPD or (D). When a assigned to every decision variable in MAID, the induced MAID becomes Bayesian network with no more decision variables. This Bayesian network defines a joint probability distribution PM|[s over all the variables M.
determined by a decision rule 5 :
Expected Utility and Nash Equilibrium. Given a strategy profile assigned to each decision variable, with the resulting joint probability distribution PM[r) and suppose that U, {U1, .... Um }. we can write the expected utility for an agent i as
EU,(o)
Σ (M1.... N6m (end:...)))) (3.)
PMia] (Ml (UI.... Ums
Uk
(I)
Given Eq. (1) the strategy 0g is optimal for σ, for subset & D, if EU,((O EU,((o_E,0)) oz)) >
[U;((o-E ())), as shown in Definition 2.1 Furthermore, if for all agents E I. OD is opti- mal for the strategy profile a, then is a Nash equilibrium (NE), as shown in Definition 2.2
Definition 2.1 (Optimal Strategy (17). Let E be subset of D. and let a be strategy profie. o' is optimal for the strategy profile a if, in the induced MAID M[o-s), where the only remaining
decisions are those in 3, the strategy og is optimal, for all strategies d'g, such that (U, -, oz))
EU;((O ((o-(( g, dz))).
Definition 2.2 (Nash Equilibrium [77. A strategy profile is a Nash equilibrium for MAID M for all agents E I, OD, is optimal for the strategy profile a.
Remark 2.3. For each MAID there can be multiple NEs (corresponding to multiple strategy pro- files). We denote the random variable describing a possible NE over a set of NEs, {ōi...... }g} as ô. For any ā dom(@), we define the probability for arbitrary NE (0) Pr(ōD....,āā,
where n := [Z] is the number of agents in the MAID. The proba- bility of strategy profile is defined the joint probability that each agent plays some strategy on the agent's 's decision variable Dj.
Definition 2.4 (S-Reachability 17]. node D' s-reachable from a node D in MAID M there some utility node U Up such that if new parent D' were added D', there would be an active path (Example C.1 and Definition C.1 in Appendix C.2) in M from D' U given Pa(D) U (D). where path is active in MAID if is active in the same graph, viewed as Bayesian network (i.e. is not blocked as per the standard d-separation rules). The relevance graph for a MAID M is a directed graph whose nodes are the decision nodes of M, and which contains an edge D' Df and only if D' is s-reachable from D.
Relevance Graph. A relevance graph as shown in Definition 2.4 defines a directed graph induced from an MAID, describing the binary relation between two decision variables. If there exists an edge D' + D, it implies that the decision variable D is strategically relies on another decision variable D'. In other words, the decision rules for D' is required to evaluate the decision rules for D. If there exist both D' + D and D + D', then the relevance graph is cyclic. Furthermore, if D and D'
3

---

D2
D
D
Dr.
D
D,
Da
Da
Da
(a) Self- Suff--gganzztiin.
(b) Global Intervention.
(c) Tageted Intervention.
MAIDs (left part in each figure) and relevance graphs (right part) for one-shot simultaneous- move MARL interaction paradigms: (a) self-organization, where agents coordinate with external mechanism; (b) global intervention, where coordinator provides signals influencing all agents and their utilities; (c) targeted intervention, where pre-strategy intervention applied on a targeted agent. In these diagrams, squares represent decision variables (specifically, red for the targeted agent, blue for other agent, yellow for pre-decision); diamonds represent utility variables; and circles with Z describe signals to guide agents for achieving their desired outcomes represented as utility variables. Relevance graphs depict strategic dependencies between these decision variables.
Figure I:
D
D
D
Do
D.
Da
(a) Self-Organization.
(b) Global Intervention.
(c) Targeted Intervention.
Figure 2: MAIDs and their relevance graphs for one-shot sequential
-ovee move games. In addition to the variables introduced in Figure the orange circles with / indicate the information sets.
belong two agents respectively, their payoffss depend on the decisions at both D and D'. In this case, the optimality of one agent's ' decision rule is intertwined with another agent's 's decision rule [T7]. Therefore, agents seeking decision rules individually to reach NE, analogous to independent learning [13 [Ch. 9.3], is unsolvable, because mismatch between agents' decision rules could happen.
Remark 2.5. One way to make eyclic relevance graphs solvable is making agents' decision rules matched via seeking agents' decision rules collectively [17), analogous to CTDE [Ch. 9.1.3].
3 Multi-Agent Reinforcement Learning Interaction Paradigms
We introduce the concept MARL. interaction paradigms (orthogonal to MARL learning paradigms) describe the ways in which agents' interactions structured-rangee from unguided self- organization to externally guided mechanisms. Specifically, we first formalize two existing MARL interaction seaddiigms-sl--ogggnnili influence diagrams (MAIDs), and then introduce our proposed targeted intervention paradigm.
and global intervention [21] 22 23 using multi-agent
3.1 Formalizing MARL Interaction Paradigms
By the definition of MAIDs in Section 2 we model MARL interaction paradigms based on agent variables (I), decision variables (D), chance variables (A') representing states and information, and utility variables (L() representing objectives. Utility variables capture the desired outcomes that agents pursue, whereas the special chance variables Z X' describe signals to guide agents for achieving their desired outcomes. The formal definition is delineated in Definition 3.1
Definition 3.1. An MARL interaction paradigm can be specified as an MAID (I, D,U). I the of agents. X' is the set of chance variables. D Ulez D; the set of agent decision variables. U are utility variables representing objectives. The set of special chance variables (Z C X)) describe signals guide agents for achieving their desired outcomes.
We now define three MARI. interaction paradigms as follows:

---

(1) Self- rell-Organization. This paradigm refers to the MARL setting where agents coordinate solely via their direct observations of the environment, without any external mechanism to steer them towards their desired outcomes. Each agent must independently elicit a signal (a pattern of behaviours) based on local information to influence the other agents towards its desired outcome. For clarity of illustration, we focus on analyzing guidance to single agent. We visualize this implicit process as a path from decision variables D, to the signal Z, and then to the utility nodes Up. illustrated by MAID structures in Figure Ta and 2a
(2) Global Intervention. This paradigm features an external, centralized coordination providing simultaneous explicit guidance signals to all agents. In the MAID, as shown in Figure [lb and (2b this is represented as information links from Z to agents' decision variables, and Z usually influence their utility nodes (e.g., Ua and U2,) directly, shaping their desired outcomes.
(3) Targeted Intervention. This paradigm involves applying external guidance signal to a targeted agent, as shown in Figure Figure[Tc] 2c] For conciseness, we illustrate this as one of its implementation, referred to as pre-strategy intervention (Section (.3). where Dyre receives a guidance signal Z and outputs filtered information. This intervention modifies the behaviours of the targeted agent, which in turn indirectly influences other agents. As a result, agents coordinate to achieve their desired outcomes provided by the guidance signal Z. While the concept of targeted intervention paradigm could potentially apply to multiple targeted agents with multiple desired outcomes, this paper specifically focuses on the case of single targeted agent.
3.2 Interpreting MARI Interaction Paradigms
We now demonstrate how MAIDs interpret the three MARL interaction paradigms we mentioned above. Each MAID elucidates the structure of a paradigm (through the role of variable 2) and its influence on decisions and utilities. Subsequently, we analyze how these structural differences translate into various strategic dependency patterns, as revealed by their relevance graphs
Analyzing the relevance graph (Definition 2.4) corresponding to each MAID, reveals the key solv- ability property of MARL algorithms. As shown in Figure la and 2a the relevance graphs for the self-organization paradigm are eyclie, implying computational
theoretical intricacies in finding solutions. In contrast, the relevance graphs for both the global intervention paradigm (Figures Tb and 2b) and targeted intervention paradigm (Figures Tcl and 2c) are acyelic, suggesting they may be solvable by a broader class of MARI. algorithms. is possible in global intervention that agents learn independently based solely on the external central signal, in which case the relevance graph would have no edges at all. indicating no policy dependency [24]. While both global intervention and targeted intervention paradigms offer acyclic structures, our proposed method relying on the targeted intervention paradigm provides distinct advantage: it can facilitate sol solvability and effectiveness only by intervention on a single targeted agent, offering more practical approach in contrast the global intervention paradigm.
3.3 Pre-Strategy Intervention: An Implementation of Targeted Intervention Paradigm
This section introduces the pre-strategy intervention-al principled design of the targeted intervention paradigm. By applying pre-strategy intervention, we aim to guide the multi-agent system to a preferred Nash equilibrium (NE), selecting one with an additional desired outcome from the multiple equilibria that satisfy the primary task goal. Our approach first defines the pre-strategy intervention, then details how its influence on reaching the preferred NE is quantified and optimized via learning a pre-policy to generate pre-strategies intervening on single, targeted agent.
3.3.1 Definition of Pre-Strategy Intervention
To regulate the intervened agent to maximize the desired outcome U that captures the preferred NE, we propose to add pre-decision variable Dyre as a new parent to the decision variable D,, of a selected agent E I. In analogy to decision variables in MAIDs, we need to assign a strategy
Tre which we refer to as pre-strategy, and this operation is named pre-strategy intervention. This definition respects the convention of stochastic intervention in causal Bayesian networks [5h 4].
Seeking equilibria according processing decision variables in s-reachability order (influencers first) 17
5

---

A pre-strategy is determined by pre-policy denoted by Spre, which processes the agent's 's information and the guidance signal, as detailed in Definition 3.2
Definition 3.2 (Pre-Strategy Intervention). For a decision variable D D in MAID, a pre- strategy intervention is an operation assigning a pre-strategy prr to a new parent Dpre added to D. referred to as a pre-decision variable. The pre-strategy opre determined by pre-policy ópre dom PP(D)) x Z + A (ddom(ppre)) where Z is the space of guidance signals representing an additional desired outcome.
3.3.2 Causal Effect of Pre-Strategy Intervention
The preferred NE is identified by maximizing a composite desired outcome, formally defined by total utility variable Uto := Utask + Usec comprising the utility Utask of the primary task goal that all agents coordinate to achieve and the utility Usee of an additional desired outcome (a secondary goal) only assigned to single, targeted agent
[ This preferred NE is one of multiple equilibria that all yield the identical, maximum value of the primary task utility Ulakk- We formalize the causal effect of a pre-strategy intervention in Definition 3.3 The causal effect quantifies the total probabilities of Unm = u", where u" is the utility value of the desired composite outcome, under pre-strategy intervention for the additional desired outcome. In many cases, it is difficult to find a pre-strategy seek pre-strategy that induces a probability
intervention that induces the preferred NE. Instead, distribution over a reduced set of NEs covering the preferred NE, denoted as θz. Proposition 3.4 proves pre-strategy intervention maximizing this causal effect is guaranteed to exist.
Definition 3.3 (Causal Effeet of Pre-Strategy Intervention). A pre-strategy intervention is applied to targeted agent's strategy profiles, which influences the total utility Utot = u² associated with a Nash equilibrium (NE) â after the pre-strategy intervention. The set of all possible NEs is denoted by (with no intervention). Given that σ. denotes a reduced set of NEs induced by pre-strategy intervention, covering the preferred NE (aligned to the additional desired outcome), the causal effect measuring this pre-strategy intervention is defined as follows:
A Opre (Urot = u²) J.cr (M u² PP) dō
Parta(Uroo = =o))P(
Aea
(U(oott=u))
Pu(Uem=uu)
In Eq. 2). Paija (Ura = u²) represents the likelihood of the total utility Utat = u² under an arbitrary NE G. (ō) denotes the probability measure on an arbitrary NE ō.
Proposition 3.4. Given MAID M, assume that the function Pr in Eq. ②). representing the probability of observing Utat = u² under a pre-strategy intervention, is upper semicontinuous and defined on a compact domain dom(ayre) an agent that does not decrease the probability of Uzot = u., Furthermore, there exists pre-strategy intervention that maximizes the causal effect.
F. There exists least a pre-strategs intervention on
To practically evaluate possible strategy profiles of the pre-strategies generated by pre-policy, we reshape Pr as the following expression:
P(Urtt = u" | do(apre)) = PMaa(Uh u')Pa(0 do(apre))-
Note that Pv is constant with respect to the intervention, so it can be ignored during optimization. In Eq. (3). Pa (ō do(arre) is a probability distribution whose support includes all possible strategy profiles, which is feasible to implement and induce any possible ōz.
4 MARL for Sequential Decision Making in MAIDs
We now show how the pre-strategy intervention introduced above is implemented in MARL for sequential decision making. We model MARL using team reward Markov games (26), and the team reward is represented as the team utility variable. The team utility variable is defined as the
'The additional desired outcome is constrained to be non-confficting with the primary task goal.

---

total utility variable to identify the preferred Nash equilibrium. The main idea is establishing connection that team reward Markov game can be transformed into a multi-agent influence diagram (MAID), allowing our framework to apply. The formal definition of team reward Markov game and the detailed process of representing as an MAID by matching their respective variables are provided in Appendix G Building on this foundation, the subsequent subsections will extend the MARL interaction paradigms introduced in the previous section to team reward Markov games (Section ). discuss the solvability of MARL learning paradigms under these MARL interaction
paradigms (Section 4.2) and specify how the pre-strategy intervention Eq. is integrated into generic MARL algorithms (Section 4.3)
4.1 Extending MARL Interaction Paradigms to Team Reward Markov Games
DI
D
D
D
D
Ulear
Uleen
(a) Soll-Organzzation.
(b) Global Intervention.
(c) Targeted Intervention.
Figure 3: MAIDs and their relevance graphs for dynamic simultancous move games. For conciseness, we only demonstrate one transition from timesteps t to t + 1. The variables used in one-shot decision making (see Figure [ are decorated with timesteps. The semantics of these variables remain unchanged. The pink circles with 8; and 8,41 indicate states for timesteps and f 1. The green diamonds with Vieam indicate the team utility variable that represents the team reward. The incoming edge from individual utility variable (U, and U to the team utility variable indicates that an agent's individual contribution to the team reward.
D
D
D
0g
D
D
D
D
D
2'
Sears
U1
(a) Self-Organization.
(b) Global Intervention.
(c) Targeted Intervention.
Figure 4: MAIDs and their relevance graphs for dynamic sequential move games. For conciseness, we only demonstrate one transition from timesteps to t + 1. In addition to the variables introduced in Figure the orange circles with I' indicate the information sets.
We now demonstrate how to extend the three MARL interaction paradigms (Definition 3.1 p to Markov games through the lens of MAIDs. Each interaction paradigm under sequential decision making in dynamic environments with simultancous or sequential moves can be extended from one-shot decision making introduced in Section 3.2 by considering timesteps and states. Representing the team reward as the team utility variable D team that is an aggregation of agents' individual utility variables, can derive the MAID graphical representation of these three interaction paradigms in team reward Markov games, which are illustrated in Figures and
It can be observed that the widespread global intervention paradigm implemented for MARL algo- rithms (e.g., applying a centralized coordinator [27 28 29 can be justifiably interpreted with a visualization. More importantly, it opens new door to design novel MARI. interaction paradigms to solve a broad class of problems with the help of MAIDs, such as the targeted intervention paradigm proposed in this paper to solve the team reward Markov game with an additional desired outcome.
7

---

4.2 Solvability of MARL Learning Paradigms under MARL Interaction Paradigms
Independent Learning. Independent learning (IL) [3] 9.3] directly applies single-agent aggen rein- forcement learning algorithms to MARL. If the relevance graph is cyclic, individually optimizing each agent's decision variables using the generalized backward induction algorithm [7] cannot guarantee reaching Nash equilibrium, even when the environmental model (MAIDs) is wwwn-analogous to independent learning in the model-free MARL setting. This aligns with the self-organization paradigm (Figure 3a and 4ab. reflecting the non-stationarity dilemma [T] The global intervention (Figures 3b and 4b) and targeted intervention (Figures [3 and 44c) paradigms can generate acyelic relevance graphs that help mitigate non-stationarity, suggesting that augmenting IL with these two paradigms leads to better performance than its vanilla counterpart, becoming more solvable P
Centralized Training and Decentralized Execution. As discussed in Remark 2.5 one way to address cyclic relevance gruphs enabling two agents' decision making matched. One solution is transforming a cyclic relevance graph to a component graph, where each maximal strongly connected component (SCC) regarded as a supernode [30]. More specifically, a maximal SCC consists of all the decision variables that form cyclie relevance graph at each timestep. Koller and Milch (17) showed that solving the acyclic component grapt using the generalized backward induction algorithm can reach Nash equilibrium when the MAID model is provided. This is associated with the centralized training and decentralized execution (CTDE) [3 9.1.3] applied in the model-free MARL setting. In summary, CTDE can be seen as MARL learning paradigm to address the non-stationarity dilemma of vanilla IL under the self-organization paradigm, rather than switching to other MARL interaction paradigms.
4.3 Pre-Policy Learning in MARL
As we have established the MARL interaction paradigms for team-reward Markov games in MAIDs, the techniques developed in Section 333.2 can be applied to realize the pre-strategy intervention by defining the team utility variable U as the total utility variable U Following Definition 3.2 it requires learning a pre-policy Spre dom (Pa(D) x Z + A(dom(o strategy pre for the possible preferred Nash equilibria. This can be realized by maximizing the causal effect expressed in Eq. (3). with replacing the Utot by the cumulative team utilities over timesteps Σi= Ui as per Eq. ( The resulting algorithm, which can be integrated with generic MARL algorithms, is referred to Pre-Strategy Intervention (PSI).
to generate the pre-
Experiments
5.1 Experimental Setups
We evaluate our proposed Pre-Strategy Intervention (PSI) in two environments: the Multi-Agent Particle Environment (MPE) 19), simultancous- move cooperative navigation game, and the Hanabi card game (20). which challenges coordination within a multi-agent system under partial observability. Hanabi is known for having multiple, distinct equilibria, making it an ideal testbed for analyzing convergence to a specific Nash equilibrium
X For all scenarios, we select a fixed agent to intervene on, which allows for a controlled analysis of the PSI. Full details of these environments, including specific experimental configurations and how additional desired outcomes are defined and implemented, are provided in Appendix H In all environments, intrinsic returns are used for measuring reachability of
'The targeted intervention paradigm can, in principle, be addressed by mixed learning paradigm with asynchronous updates, guided by the dependency of decision variables in the relevance graph: the targeted agent independently updates its policy, while the other agents update theirs under centralized training in the subsequent turn. Since such an approach is not yet a standard MARL. learning paradigm, we leave it for future work.
'A component graph is always acyclic (31.
"By B2). maximizing each agent's 's equal contribution to shared reward is equivalent to maximizing the reward. In the targeted intervention, the team reward is defined as Uf.,. Ulaek + Use.. where Ulack is shared among agents and Ui... is only attributed to the targeted agent. Subsequently, the targeted agent's 's individual utility variable defined as Ut
U. while other agents' are detined as Ufank- Thus, maximizing each agent's 's individual utility leads to the preferred NE among multiple equilibria the primary task utility.
'We choose a convention called "5 Save" for the results in the main paper, while the results of another convention called "The Chop" is reported in Appendix I.2.2
8

---

-1
200 MOSAAAE RETUr
X
bear 48
Nondl MANO
(a) MPE: Main results.
(b) Hanabi: Main results.
HPS
Hanabl
2n4
(c) Hanabi: PST vs. GPSI.
(d) PST vs. Caditional-Outcome--- Approaches.
Figure 5: In (a) and (b), PSI is compared against Intrinsic Reward and Base MARL; these plots show both extrinsic (the primary task completion) and intrinsic (reachability of additional desired outcome) returns. In (c), the targeted intervention paradigm (PST) is compared against the global intervention paradigm (GPSI). In (d), PSI is compared against approaches that only focus on the primary task completion with no consideration of any additional desired outcome.
additional desired outcomes, while extrinsic returns are used for measuring primary task completion. Results from random seeds are reported as means with 95% confidence intervals.
Implementation of Our Method. Our PST is implemented by designing a pre-policy module (a GRU or MLP, matching agent's 's backbones), which takes a concatenated input of environmental observations and measure for an additional desired outcome (formulated an intrinsic reward) It outputs an embedding, which is then fed to Q-value functions for value-based MARL methods or critics for policy-based methods.
Baselines. All baselines are implemented with the same architecture and training setups our method. We evaluate our method against several base MARL algorithms referred to as Base MARL (self-organization paradigms): for MPE, these include IQL [33,, VDN (34] and QMIX (35): for Hanabi, we use IPPO (36), MAPPO 37 and PQN 38 (with PQN-IQL for IL and PQN-VDN for CTDE variants). An ablation study that removes the pre-policy module while retaining the intrinsic reward maximization, referred to as Intrinsic Reward, examines the effectiveness of the pre-policy module. To emphasize the strength of our targeted intervention paradigm, we also compare PSI against the global intervention version of PSI, referred to as GPSI, which applies the pre-policy module to intervene on all agents. Finally, we include baselines of the global intervention paradigm that only focuses on primary task completion, such as the modified versions of LIIR [8] and LAIES 22] as representative policy-based and value-based methods, respectively.
5.2 Results and Analysis
Coordination Achieving When Intervening on a Single, Targeted Agent. As Figure 5d shows, PSI can outperform both LIIR and LAIES the primary task completion in both MPE and Hanabi. This result confirms that coordination can be achieved when assigning an additional desired outcome (c.g. convention) derived from human knowledge, to single targeted agent through the targeted intervention paradigm, answering our motivating research question. Moreover, this may even improve primary task completion, compared with approaches of the global intervention paradigm that does not consider any additional desired outcome, avoiding the issue of miscoordination.
Verification of Relevance Graph Analysis. Our MAID and relevance graph analysis predicts that PSI improves the solvability of MARL learning paradigms (Section 4.2). enabling the vanilla IL learning paradigm more solvable. Experiments verify this across different settings. In simultaneous- move MPE (analysis in Figure [c and results in Figure Saj. IQL equipped with our PSI achieves task completion comparable to VDN, CTDE algorithm. This is notable improvement for IQL
Thii implementation maintains solvability of MARL algorithms with the PSI (Appendix

---

over its baseline performance (without our PSI). Similarly, in sequential-move Hanabi (analysis in Figure 4c and results in Figure (5b). IL algorithms augmented with the PSI also achieve performance comparable to, or even better than CTDE algorithms.
Targeted Intervention Global Intervention. Our targeted intervention paradigm (PSI) consis- lently outperforms the global intervention paradigm (GPSI), as shown in Figures [5c] Although both paradigms achieve comparable performance on the primary task completion, the global intervention paradigm often struggles to reach the additional desired outcome. We attribute this to the inherent practical difficulty of designing (learning) an effective coordination mechanism to globally assign beneficial, non-conflicting goals to multiple agents simultaneously. This highlights the potential application of PSI to safety-critical systems [39], which require strictly controlled operation.
Study on the Pre-Policy Module. To assess our pre-policy module's 's contribution, we conduct ablation study to compare PST against the Intrinsic Reward baseline. Experimental results presented for MPE in Figure 5a and for Hanabi in Figure [5b reveal a key distinction: while both PSI and the Intrinsic Reward baseline effectively lead to the attainment of the additional desired outcome, the PSI (with the pre-policy module) achieves notably superior performance on the primary task completion. This highlights that the pre-policy module is instrumental for retaining the targeted agent's 's adherence to an additional desired outcome while not forgetting the primary task goal.
Ablation
Analysis of Nash Equilibrium Convergence in Hanabi. The Hanabi environment, with its multiple distinct equilibria (20), serves as an ideal testbed for analyzing convergence to a preferred Nash equilibrium (NE). We treat a specific human convention as preferred, high-performing NE and design the intrinsic reward to directly measure the agents' compliance to it. As shown in Figure 5b)] the high and stable intrinsic return achieved by our PSI provides strong evidence that the agents successfully converge to the preferred NE. In contrast, the low intrinsic return of the baselines suggest they fail to establish the convention and are likely stuck in one of Hanabi's many inferior equilibria.
Additional Results. Appendix details further exploration of our method's capabilities and robust- ness. These include: (1) evaluation using varied additional desired outcomes in MPE and Hanabi; and (2) performance assessments under noisy observation conditions.
Conclusion
Summary. We incorporate multi-agent influence diagrams (MAIDs) as principled framework into designing and analyzing the targeted intervention paradigm in MARL. Our Pre-Strategy Intervention (PSI) approach as an implementation of the targeted intervention paradigm, applied to a single targeted agent, maximizes the causal effect on reaching a composite desired outcome that integrates both the primary task goal and an additional desired outcome. Furthermore, the relevance graphs of MAIDs offer theoretical insights into the solvability of MARL learning paradigms under various MARL interaction paradigms. Experiments verify the effectiveness of the PSI.
Limitation. Our principle currently presumes that the underlying structure of MARL interaction paradigms in MAIDs is complete or can be precisely modelled. This structural knowledge is a prerequisite for effective influence propagation via targeted intervention and can be challenging to define a priori in realistic environments or complex systems. Additionally, our analysis primarily focuses on a single targeted intervention, which may limit its applicability to real-world settings.
Future Work. Key future directions of our work include but are not limited to:
40 to reduce reliance on 1. Learning MAID structures from data (e.g., via causal discovery pre-specified models. Further, the MAID can be seen as foundation to describe the world model.
2. Coordinating a multi-agent system under the targeted intervention paradigm on multiple agents. Designing the optimal criteria for selecting the number of targeted agents and appointing proper agents with various types to complete different tasks. The information-thhoreee empowerment
measure of
offers a potential quantitative basis for such criteria.
4. Integrating advanced reasoning modules (e.g. large language models and the broader generative AI) to enhance the capability of the PSI on the targeted agent in online learning and online adaptation to unknown agent teammates and unseen situations [42..
Designing MARL algorithms with asynchronous updates and mixed learning paradigms inspired by the targeted intervention paradigm (Footnote 5)
10

---

Acknowledgment
Jianhong Wang is supported by the Engineering and Physical Sciences Research Council (EPSRC) [Grant Ref: EP/Y028732/1), Samuel Kaski is supported by UKRI Turing AI World-Leading Re- searcher Fellowship, EP/W002973/1.
References
[1] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. survey and critique of multiagent deep reinforcement learning, Autonomous Agents and Multi-Agent Systems, 33(6):750-797, 2019.
[2] Liviu Panait and Sean Luke. Cooperative multi agent learning: The state of the art. Autonomous agents and multi-agent systems, 11:387-434, 2005.
[3] Yongean Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9():4277-438, 2012.
[4] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Deal- ing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv:/906.000 1906.0437, 2019.
[5] Adrian K Agogino and Kagan Tumer. Analyzing and visualizing multiagent rewards dynamic and stochastic domains. Autonomous Agents and Mulli-Agent Systems, 17:320-338, 2008.
[6] Dom Huh and Prasant Mohapatra. Multi-agent reinforcement learning: A comprehensive survey, arXiv preprint arXiv:2312. 10256, 2023.
[7] N. Jaques, A. Lazaridou, Hughes, Gulcehre, A. Ortega, J. Strouse, J.Z. Leibo, and N. de Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. International Conference on Machine Learning (ICML), 2019.
[8] Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Lirr: Learning indi- vidual intrinsic reward in multi-agent reinforcement learning. Advances in neural information processing systems, 32, 2019.
[9] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[11] Hengyuan Hu, David J Wu, Adam Lerer, Jakob Foerster, and Noam Brown. Human-ai coordi- nation via human-regularrzed search and learning. arXiv preprint arXiv:2210.05125,, 2022.
[12] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-555 2015.
[13] Stefano V Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-agent reinforcemens learning: Foundations and modern approaches. MIT Press, 2024.
[14] Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas. Maria Attarian, Maria Bauza. Matthew Bennice, Alex Bewley, Adil Dostmohamed, et al. Learning to learn faster from human feedback with language model predictive control. arXiv preprint wxXv:2442..11500 2024.
[15] Kunal Garg, Songyuan Zhang, Oswin So, Charles Dawson, and Chuchu Fan. Learning safe control for multi-robot systems: Methods, verification, and open challenges. Annual Reviews in Control, 57: 100948, 2024.

---

[16] Araz Taeihagh and Hazel Si Min Lim. Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks. Transport reviews, 39(1):103-128, 2019.
[17] Daphne Koller and Brian Milch. Multi-agent influence diagrams for representing and solving games. Games and economic behavior, 45(1):181-221, 2003.
[18] John Harsanyi, Reinhard Selten, et al. general theory of equilibrium selection in games. MIT Press Books, 1, 1988.
[19] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiivee environments. Neural Information Processing Systems (NIPS), 2017.
[20] Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, March 2020.
[21] DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collabo- rating with humans without human data, 2022.
[22] Eduardo Mosqueira-Rey, Elena Hernández-Pereinn David Monso-Ríos, José and Ángel Femández-Leal. Human-in-the-loog machine learning: a state of the art. Artificial Intelligence Review, 36(4)33005-3054, 2023.
[23] Ziyan Wang. Zhicheng Zhang, Fei Fang, and Yali Du. M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality. arXiv preprint arXiv:2503.00077 2025.
[24] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: short survey, 2022.
[25] Judea Pearl. Causality. Cambridge university press, 2009.
[26] Michael L Littman. Markov games as framework for multi-agent reinforcement learning, In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[27] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. arXiv preprint arXi::2003.08003 2020.
[28] Hyungho Na and Il-chul Moon. Lagma: latent goal-guided multi-agent reinforcement learning. arXiv preprint arXiv:2405. 19998, 2024.
[29] Boyin Liu, Zhiqiang Pu, Yi Pan, Jianqiang Yi, Yanyan Liang, and Zhang. Lazy agents: new perspective on solving sparse reward problem in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 21937-21950. PMLR, 2023.
[30] Maria De Torio, Willem van den Boom, Alexandros Beskos, Ajay Jasra, and Andrea Cre- maschi. Graph of graphs: From nodes to supernodes in graphical models. arXiv preprint arXiv:2310.11 11741, 2023.
[31] Thomas H Cormen, Charles Leiserson, Ronald Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
[32] Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley v-value: A local reward approach to solve global reward games. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7285-7292, 2020.
[33] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[34] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1706005296 2017.
12

---

[35] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[37] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1-24624, 2022.
[38] Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.048/ 2024.
[39] Jianhong Wang. Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent reinforcement learning for active voltage control on power distribution networks, Advances in Neural Information Processing Systems, 34:3271-3284, 2021.
[40] Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. Discovering agents. Artificial Intelligence, page 103963, 2023.
[41] Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 ieee congress on evolutionary computation, volume I, pages 128-135. IEEE, 2005.
[42] Jianhong Wang, Yang Li, Yuan Zhang, Wei Pan, and Samuel Kaski. Open ad hoc teamwork with cooperative game theory. In International Conference on Machine Learning, pages 50902-50930 PMLR, 2024.
[43] Ali Dorri, Salil S Kanhere, and Raja Jurdak. Multi-agent systems: A survey. leee Access, 6:28573 285933, 2018.
[44] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles I. Isbell, and Andrea I. Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. Advances in neural information processing systems, 26, 2013.
[45] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[46] Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neurul information processing systems, 29, 2016.
[47] Riad Akrour, Mare Schoenauer, Michèle Sebag, and -han-Christophe Souplet. Programming by feedback. In International Conference Machine Learning, volume 32, pages 1503-1511. JMLR. org, 2014.
[48] Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A survey of luman-in-the-lope for machine learning. Future Generation Computer Systems, 135:364-381, 2022.
[49] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[50] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-larnnig from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[51] Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordination. In International Conference on Machine Learning, pages 13584-13598. PMLR, 2023.
13

---

[52] Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, and Yan Zheng. Kogun: accelerating deep reinforcement learning via integrating human suboptimal knowledge. arXiv preprini arXiv:2002.074188 2020.
[53] Dingbang Liu, Shohei Kato, Wen Gu, Fenghui Ren, Jun Yan, and Guoxin Su. Integrating sub- optimal human knowledge with hierarchical reinforcement learning for large-scale multiagent systems. Advances in Neural Information Processing Systems, 37: 17:10444--10776 2024.
[54] Zhixun Chen, Zijing Shi, Yaodong Yang, Meng Fang, and Yali Du Hierarchical multi-agent framework for dynamic macroeconomic modeling using large language models. In Proceedings of the 24th International Conference Autonomous Agents and Multiagent Systems, pages 2460 2462, 2025.
[55] Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J Wu, and Jakob Nicolaus Foerster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations, 2023.
[56] Xue Yan, Yan Song, Xidong Feng, Mengyue Yang. Haifeng Zhang, Haitham Bou Ammar, and Jun Wang. Efficient reinforcement learning with large language model priors. arXiv preprint arXiv:2410.07927, 2024.
[57] Natalia Zhang, Xinqi Wang, Qiwen Cui, Runlong Zhou, Sham M Kakade, and Simon S Du. Multi-agent reinforcement learning from human feedback: Data coverage and algorithmic techniques. arXiv preprint 240.02749,00077,, 2024.
[58) Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbecl, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[59] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08200 2022.
[60] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
[61] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint
2019.
[62] Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforcement learning with skill discovery. arXiv preprint /9/2v: 2019.
[63] Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Ldsa: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 35:1698-1710, 2022.
[64] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1- 111111 1999.
[65] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction, 2017.
[66] Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang, Episodic multi-agent reinforcement learning with curiosity- driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
[67] Aya Kayal, Eduardo Pignatelli, and Laura Toni. The impact of intrinsic rewards on exploration in reinforcement learning. arXiv preprint arXiv:2501. /133, 2025.
[68] Ruo Yu Tao, Vincent François-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems, 33:81 8126, 2020.
14

---

[69] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
[70] Haoqi Zhang, Yiling Chen, and David Parkes. A general approach to environment design with one agent. In Proceedings of the 21st International Joint Conference on Artificial Intelligence, pages 2002-2008, 2009.
[71] Daniele Reda, Tianxin Tao, and Michiel van de Panne. Learning to locomote: Understanding how environment design matters for deep reinforcement learning. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1-10, 2020.
[72] Zhan Gao and Amanda Prorok. Constrained environment optimization for prioritized multi- agent navigation, 2023.
[73] Noam Nisan and Amir Ronen. Algorithmic mechanism design. In Proceedings of the thirty-first anmual ACM symposium on Theory of computing, pages 129-140, 1999.
[74] Yang Cai, Constantinos Daskalakis, and S Matthew Weinberg. Understanding incentives: Mechanism design becomes algorithm design. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 618-627. IEEE, 2013.
[75] David Vickrey and Daphne Koller. Multi-agent algorithms for solving graphical games, AAAI/I- AAI, 2:345-351, 2002.
[76] Michael Kearns, Michael Littman, and Satinder Singh. Graphical models for game theory. arXiv preprint arXiv: 1301.22., 2013.
[77] Michael P Wellman. Methods for empirical game-theoretic analysis. In AAAI, volume 980, pages 1552-1556, 2006.
[78] Lewis Hammond, James Fox, Tom Everitt, Ryan Carey, Alessandro Abate, and Michael Wooldridge. Reasoning about causality in games. Artificial Intelligence, 320: 103919, 2023.
[79] Francis Rhys Ward, Tom Everitt, Francesco Belardinelli, and Francesca Toni. Honesty is the best policy: defining and mitigating ai deception. In Advances Neural Information Processing Systems, 2023.
[80] Francis Rhys Ward, Matt MacDernttt, Francesco Belardinelli, Francesca Toni, and Tom Everitt. The reasons that agents act: Intention and instrumental goals, 2024.
[81] Matt MacDermott, James Fox, Francesco Belardinelli, and Tom Everitt. Measuring goal- directedness. arXiv preprint arXiv:2412.04758, 2024.
[82] Judea Pearl. Probubilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014.
[83] Dawei Qiu, Jianhong Wang, Junkai Wang, and Goran Strbac. Multi-agent reinforcement learning for automated peer-to-peer energy trading double-side side auction market. In Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, 2021.
[84] Yuan Zhang, Umashankar Deekshith, Jianhong Wang, and Joschka Boedecker. Improving the efficiency and efficacy of multi-agent reinforcement learning on complex railway networks with local-critic approach. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, pages 698-706, 2024.
[85] Michael L Littman. Value-function reinforcement learning in markov games. Cognitive systems research, 2(1):55-66, 2001.
[86] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar In- gvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktaschel, Chris Lu, and Jakob Nicolaus Foerster. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint arXiv:2311.10090 2023.
15

---

[87] Jonathan Richens and Tom Everitt. Robust agents learn causal world models. In The Twelfth International Conference on Learning Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[88] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 1r 2016.
[89] Arjun Vaithilingam Sudhakar, Hadi Nekoei, Mathieu Reymond, Miao Liu, Janarthanan Rajen- dran, and Sarath Chandar. A generalist hanabi agent. In ICLR, 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The key contributions of this paper are summarized in both the abstract and the final paragraph of the introduction, with each point corresponding to a dedicated section in the subsequent text.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have well discussed the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiattinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification:
The omitted proof of main paper is in Appendix E
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code for reproducing the main experimental results is published on: https://github.
Justificatiin:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeee hutii///guidessoeeSuuiiiss SooodeSubiissi.e ssiiiioiicccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide details of the experimental settings in Appendices H and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results are shown with means and 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclose the computational resources needed in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics. The paper presents algorithmic work in simulated environments without human subjects or potential for imme- diate harmful applications.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on foundational research in multi-agen reinforcement learning and introduces new thcoretical framework for agent coordination. The pro- posed Pre-Strategy Intervention method remains at thcoretical and algorithmic level, with experiments conducted only in simulated environments (MPE and Hanabi).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper does not release models or datasets with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the project we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
All code is extensively commented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve any crowdsoureing or human subjects. All experiments were conducted in simulated environments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects or participants, as all experiments were conducted in simulated reinforcement learning environments,
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used for grammar checking and text formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

A
Notation
Table 1: Summary of key notations used in this paper.
Notation
Description
MAID Core Components
M
Multi-Agent Influence Diagram (MAID). Set of agents in the MAID.
X
Set of chance variables (representing states, information, etc.).
D,D:
Set of decision variables for all agents, and for agent i.
U.U,
Set of utility variables (objectives) for all agents, and for agent i.
Pa(D)
Parent set of a decision variable D.
Directed acyclic graph (DAG) of the MAID.
Pr
Probability distribution.
strategy profile (assignment decision rules all agents). An arbitrary Nash equilibrium (NE).
The set of all possible NEs.
Pre-Strategy
Intervention
2 CCA"
The space of guidance signals representing all possible additional desired outcomes.
ZEZ Z
A specific guidance signal representing an additional desired outcome. Pre-strategy decision variable.
Dpre
Ore
A pre-strategy assigned to the pre-decision variable Dpre- pre-policy, which is function that determines a pre-strategy pre.
6pr
The reduced set of NEs induced by an intervention.
Utility, Probability Distribution and Causal Effect
Uream
Team utility.
Urot
Total utility, representing the overall objective.
Utasky Usec (o)
Utility of the primary task goal and the secondary goal, respectively, Expected utility for agent under strategy profile a.
PM w)
Likelihood of outcome U under strategy profile a.
(o(
Probability distribution over strategy profiles.
P(. P(I
Probability distribution after applying the causal intervention do(opre))
A (U u)
Causal effect of pre-strategy intervention outcome.
MARL
Set of states in a Markov game. Joint actions in Markov game.
A
Transition function in a Markov game.
R
Team reward function in Markov game. Joint policy of agents in a Markov game.
π
B
Related Work
B.1
Multi-agent Team with Human Feedback
Our work connects to the growing field of leveraging human guidance within multi-agent systems
43
A prominent related area is reinforcement learning from human feedback (RLHF), where human input, such as preferences or demonstrations, guides agent policy optimization [44] 45). often proving essential for tasks with complex subjective goals for which explicit reward design is difficult [45] 46
47]. RLHF a specific instance within the broader scope of
multi-agent systems,
which utilize diverse inputs like direct commands [48], demonstrations [49] [50]. policy shaping [46), or language instructions
[51 to steer agent behaviours. Incorporating such human expertise can significantly enhance system performance [44] 52 53 54] interpretability [ 51 [53 [66,, and
alignment [5 [57]. These methods typically treat human feedback as data source to directly train an agent's policy, often in model-free manner.
In contrast, our work introduces structural approach grounded in MARL interaction paradigms. We use multi-agent influence diagrams (MAIDs) formally model the strategic dependencies within

---

a MARL learning process. Based on this model, our targeted intervention paradigm is designed to provably maximize the causal effect on a desired outcome, which can be provided by human feedback. The core advantage of our framework is therefore its ability to provide a principled, analytical method for designing an intervention on a multi-agent system and predicting its system-level impact. This a significant advantage in scenarios where the effects of global, heuristic feedback are hard to foresee.
B.2 Goal-Conditioned MARL
Our work relates coal-conditioned reinforcement learning (GCRL), where agents learn policies
conditioned on achieving diverse goals, such as reaching specific states [58] [5], matching images (60).
or following language instructions [6]]. While concepts similar subgoals like roles [27], skills (62].
or subtasks [63] are used in MARL for task decomposition or coordination. The foundational
approach in this area is the option framework in hierarchical reinforcement learning (HRL) [64]
where " "options" (temporally extended sub-poli policies) are used to achieve subgoals.
Our "additional desired outcome" can serve similar function to a subgoal, though our framework interprets this concept more broadly. A traditional subgoal often refers to a specific intermediate state required for task completion. However, our composite desired outcome can also encompass beneficial patterns of behaviour that, while not directly related to the task completion, significantly improve the MARL learning performance and coordination. For example, in the game Hanabi, team can achieve a score without adhering to a specific convention. However, adopting a convention (an additional desired outcome) makes teammates' actions more predictable, thereby enhancing coordination and improving overall performance.
This broader focus guiding patterns of behaviour, rather than just reaching explicit, predefined subgoals, motivates our distinct approach. Our approach differs from these methods in two principled ways. First, our Pre-Strategy Intervention provides continuous guidance signal that influences an agent's s primitive actions at each timestep, rather than directly invoking a temporally abstract sub-policy (e.g. option) to make decision. Furthermore, approach is designed to be particularly advantageous in scenarios where providing global guidance to all agents is impractical. Our core distinction lies in using a principled framework analyze and apply targeted intervention single agent.
B.3 Intrinsic Reward Method
A common approach to realize guidance in MARL is through maximizing intrinsic rewards during learning. These internally generated signals can encourage diverse behaviours motivated by curios- ity [65] 66), novelty (67 688. social influence 17). empowerment [9], learned reward functions [8] 69)
or utilizing prior knowledge from large language models 23
While our Pre-Strategy Intervention is implemented via an intrinsic reward, our core distinction lies in its purpose and scope. Unlike methods such as those in g 233. which applied intrinsic rewards to all agents to encourage emergent coordination, our Pre- Strategy Intervention strategically directs the signal only to a single targeted agent. Furthermore, standard intrinsic rewards are generally designed to solve local challenges like sparse rewards via using heuristic metrics like novelty. In contrast, our guidance signal explicitly designed to solve system-level problem: reaching desired outcome for the entire multi-agent system.
To achieve this, we leverage Multi-Agent Influence Diagrams (MAIDs) and their corresponding relevance graphs to formally analyze the strategic dependencies between agents. This causal analysis crucial as allows us to move beyond simple heuristics. We then strategically design a guidance signal, which is implemented as an intrinsic reward for single targeted agent. The specific objective of this signal is to maximize its causal effect on the composite desired outcome.
Therefore, our contribution is not merely the use of an intrinsic reward, but rather the principled framework that uses causal analysis inform the design of such a signal.
B.4 Environment and Mechanism Design
Our work relates to concepts from environment and mechanism design. Environment design typically involves structuring modifying environmental configurations to guide agent behaviours towards
25

---

a targeted outcome [70 7 72]. In contrast, our approach does not directly reconfigure the static environment. Rather, focuses on intervening on a targeted agent. Consequently, the behaviour of the intervened agent changes, altering the effective environment dynamics experienced by other agents. This allows us to shape emergent team behaviour by modifying agent interaction paradigms rather than fixed environmental rules.
Our work can also be viewed through the lens of mechanism design, which focuses on designing the "rules of games" such that a desired outcome is attained 73 74]. Our proposed targeted intervention paradigm can be categorized into this context as a mechanism that is applied specific agents designed to steer the MARL learning process towards equilibria that satisfy both the primary task goal and a specified additional desired outcome.
B.5 Probabilistic Graphical Models for Multi-Agent Games
Graphical models based on Bayesian networks were introduced to represent dependencies, relevance, and relationships among variables in multi-agent games by Koller and Milch [17]. This formulation
has since inspired advancements in both algorithmic innovations for solving games [75] [6] and empirical analysis [77]. More recently, Hammond et al. [78] incorporated causal graphical models into multi-agent reasoning, extending multi-agent influence diagrams (MAIDs) to the causal domain. Their work introduced the concept of pre-policy intervention, allowing for queries such as "What if some agents commit to a policy before others make their decisions? ?" This framework has become foundational tool in subsequent research theory of mind (79] 80), reinforcement learning [8], and causal modeling of agents [40]. To maintain consistency with the MAID framework, we rename "pre-policy intervention" "pre-strategy intervention" in this paper.
Our work builds directly on the causal framework proposed by Hammond et al. 77 However, whereas their work introduced pre pre-policy intervention general tool for causal reasoning in game theory, our work is the first to adapt and apply this concept to solve a core challenge in MARL: achieving coordination by guiding multi-agent system towards desired outcome. We leverage MAIDs not just for analysis, but as a practical design tool to construct an MARL interaction paradigm. We design Pre-Strategy Intervention as the core mechanism within the targeted intervention paradigm, using it to guide the MARL learning process towards a desired, high- nigh-performing outcome.
C Extra Background
C.1 An Example of MAIDs
We exemplify MAIDs through a two-agent scenario from
Example. Alice is considering building patio behind her house, which would be more valuable if she could have a clear view of the ocean. However, a tree the yard of her neighbor Bob blocks her view. Alice, being somewhat unscrupulous, contemplates poisoning Bob's tree, which would cost her some effort but might cause the tree to become sick. Bob is unaware of Alice's actions, but can observe if the tree starts deteriorate, and he has the option of hiring tree doctor attention of the tree doctor reduces the chance that the tree will die during winter. Meanwhile, Alice must decide whether to build her patio before the weather turns cold. At the time of her decision, Alice knows whether tree doctor has been hired but cannot directly observe the tree's 's health. An MAID for this scenario shown in Figure
cost). The
In this example, Poison Tree and Tree Doctor are s-reachable to Build Patio, as Alice must take into account whether she has already poisoned the tree and whether Bob has called tree doctor before deciding whether to build the patio.
C.2 Additional Definitions
Definition C.1 (Active Path 82 Let G be Bayesian Network (BN) structure, and let X, X₂ X,, represent an undirected path G. Let be a subset of variables in G (the evidence set). The path X1 -....- Xx, is active given evidence E if the following conditions hold:
26

---

Poison Tree
Tree Doctor
Build Patio
Tree Sick
Cost
Effort
Tree Dead
View
Tree
Figure 6: A MAID for the Tree Killer example; Alice's decision and utility variables are in red, and Bob's are in blue. Decision nodes rectangular, chance nodes are squircular, and utility nodes hexagonal.
1. Whenever a collider is on the path, i.e., structure X - X₃ <- X then either X; or one of its descendants is in E;
No other variables along the path is in E.
D Illustrative Example: Pre-strategy Intervention in a Logisties Game
Background Two logistics companies, Company A and Company B, share warehouse and use robots to manage inventory. Each company has two options:
Optimize space usage: Focus on efficient organization.
Prioritize speed: Focus on moving items quickly.
Both companies' choices affect each other's performance, and they aim to achieve the best outcome for their operations.
Utility Table
Company A Company B Optimize Space Usage (A)
Optimize Space Usage (B)
Prioritize Speed (B)
(9, 9)
(3, 6)
Prioritize Speed (A)
(6,3)
(5,5)
D.1 Pre-Strategy Intervention
Our Pre-Strategy Intervention is applied to Company A (the targeted agent) to guide the system towards the more desired (9,9) equilibrium. The pre-policy module learns to provide a guidance signal to Company A that effectively incentivizes it to choose "Optimize Space Usage." For instance, this guidance might be implemented as an additional input signal that favorably alters Company A's perceived utility Q-values for choosing "Optimize Space Usage," making it the optimal choice for Company A given the intervention.
By influencing Company A to select "Optimize Space Usage," and assuming Company B best responds to this action (or also converges "Optimize Space Usage" due to the system dynamics and its objectives), the system is steered towards the (9,9) outcome. This targeted intervention
27

---

on Company A thus fosters cooperative outcome and prevents convergence to the less efficient (5,5) equilibrium, demonstrating how Pre-Strategy Intervention can be used for equilibrium selection in MARL setting.
E
Theoretical Proof of Proposition 3.4
The proposition relies the assumption that the probability function Pr upper semicontinuous. This is reasonable assumption because the best-response function in game is not always continuous. An intuitive example support this assumption is the game paper, rock, scissors, where the best response is conducting each action uniformly. If we consider pre-strategy that shifts one player towards slightly less likely playing rock, then the probability of the opponent playing paper would experience a "jump" to O, which can be seen as a discontinuity in the function. A detailed example solution of pre-strategy intervention can be found in Appendix D
Proof. Proposition| 3.4 claims that an intervention exists: (a) it does not decrease the probability of the desired outcome and (b) a maximum- effeect intervention exists.
(a) A trivial case exists where a pre-strategy that equals the marginal conditional probability of U = be achieved by doing empty intervention.
(b) prove that there exists pre-strategy maximizing the causal effect, we observe that the second term the right-hand side of Eq. (2) is constant. Therefore, maximizing the first term is equivalent to maximizing the causal effect.
To analyze how an agent's 's strategy influences the final outcome, we can decompose the probability of achieving the desired outcome Utot u² Using the law of total probability and the Markov property of the Bayesian network (17), we can marginalize out the decision variable D and its parents DD. This decomposition explicitly isolates the agent's decision rule, (M pap). which is the component that our intervention targets:
[ pa Edom(Pa(D)) (Mio[(Pap) dpap
PMi = u²) =
x dEdom(()
PM pap) dd
x (M u² d, pap)
(4)
The function f(ppre), representing the expected probability of U = under the pre-strategy, is defined as:
f(ppre) Pr(Usst =u") = Joeas PMi u² Pa dō
Assuming f is an upper semicontinuous function defined on compact domain dom (amee) C Rm aim demonstrate that f has maximum this domain. This follows from the Extreme Value Theorem. We may replace the notation Ore with a in the following steps for simplicity, with a slight abuse of notation.
Suppose, for contradiction, that unbounded above. For each E N, there exists σg E dom((a) such that f((xx) > k. Since dom((a) is compact, the sequence { contains convergent subsequence {ok,} converging to some dg E dom(ar.
Boundedness Above:
The property of upper semicontinuity implies lim supj-tee f(ok,) < f(av), which contradicts the assumption because it suggests lim supt f(a sc. This shows f is bounded above. Then we can define:
suup sup(f(o)
dom(a))
Since the set (f(o) : E dom(a)) is nonempty and bounded above, 7 E
Existence of Maximum: Let {ot} be a sequence in dom(a such that {f (ax)} converges to ?- By the compactness of the domain, the sequence {ak} has a convergent subsequence {or,} that converges to some ō E dom(a). Then
lim f (ax,) = lim sup f (axz) S f(ō) S
=
E-400
t4000
28

---

Conclusion: The equality 7 f establishes that y is the maximum value of f on dom(o), and thus f(o) S f() for all a in the domain dom(o).
Multi-Agent Influence Diagrams and Relevance Graphs in Implementation
gr++1
D
9
D
D
D
g
D
D
D
U
D
U
U
D
D
Z
Z'
UL
team
U
(a) Simultancous Moves.
(b) Sequential Moves.
Figure 7: MAIDs for pre-strategy intervention approach implemented in experiments, in sequential- move sequential decision makings, their corresponding Markov games and the relevance graphs for the Markov games. The figures illustrate the structure for (a) Simultaneous moves and (b) Sequential moves. The variable "Zt" here indicates intrinsic reward to implement an additional desired outcome, attributed to the the single targeted agent's individual utility variable U/-- This means the reachability of the additional desired outcome be observed by the single targeted agent during learning.
In this section, we provide structure that represents the practical implementation of our proposed Pre-Strategy Intervention. This shows how the intervention, designed to guide single targeted agent towards an additional desired outcome, is integrated into that agent's 's decision-making process within the MAID framework. We present these implementtatoonsseeiif MAIDs and their corresponding relevance graphs for sequential decision-making scenarios.
detailed illustration of the multi-agent influence diagram (MAID)
Figure 7 illustrates the MAID structures and its implementation for applying Pre-Strategy Inter- vention for sequential decision making with simultancous moves. Similarly, Figure (b) shows these for sequential-move scenarios. These diagrams illustrate how the pre-decision variable (Drre) and an additional desired outcome variable (Z*) (represented an intrinsic reward) directly inform the targeted agent's ' decision (D)) at each relevant timestep. Furthermore, an edge from a variable representing the additional desired outcome Z' the targeted agent's individual utility variable Uf indicates that it can observe the reachability of the additional desired outcome during learning.
G Representing Team Reward Markov Games as Multi-Agent Influence Diagrams
This appendix details how a team reward Markov game can be equivalently represented as a Multi- Agent Influence Diagram (MAID).
Markov games (26 is a popular mathematical model to describe the multi-agent sequential decision For succinet description, we only consider
process across various real-works applications 3
39
84
the team reward Markov game with a finite episode length, as described in Definition G.T
Definition G.1 (Team Reward Markov Game 85 ). A team reward Markov game can be described as tuple S, R, L). I is set of agents; S is a set of states; A XiezA, A; is set of joint actions and A; is agent i's action set; T A - describes the transition function that maps state E S at timestep t to 8141 E S at timestep t 1; R S x A I2 is a team reward
29

---

function evaluating the immediate joint action E A at some state E S. In a team reward Markov game with an episode length of timesteps, agents aim to learn a joint policy πT (Tz)ezz where Ti S - A; is agent i's stationary policy, to solve the following optimization problem such that max, E,,r IE R(st, a.)].
A team reward Markov game can be represented as a MAID, because we can match variables between these two models. In more details, both models' agent sets are Z; S is associated with chance variables X;; AA is associated with decision variables D; T is associated with conditional probability distributions Pr; л is associated with decision rules å; and E,, E R(s,, az)] is associated with the expected utility shown in Eq. (.. with representing R(ss, ae) as the team utility variable Uteam-
For the self- orfaazaaaiz,tion, each agent's individual utility variable in the MAID can represent the equal contribution of the team utility variable Vieam that implies team reward. According to [32] this is equivalent to the case where each agent's 's individual utility variable directly represents the shared team reward Rs,,, at). Therefore, maximizing the objective of team reward Markov game max, E2 R(s, at) is equivalent to reaching a Nash equilibrium (NE) (Definition 2.2). For the targeted intervention, the team utility variable is defined as the total utility variable: Uteam Ulot = Utask + sec. where Upass is shared among agents and U is only attributed to the targeted agent. Thus, the targeted agent's ss individual utility variable is defined agents' are defined as Utask- Therefore, maximizing each agent's 's individual utility leads to the preferred NE among multiple equilibria of the primary task utility. For the global intervention with an additional outcome, both Utack and U are shared among agents. According to (32), each agent's 's individual utility variable is equally defined additional outcome, the situation is the same as the self-organization.
U task + U while other
Jaskk + U See For the global intervention without an
H Implementation Details
The implementation is mainly based on JaxMARL [6.. Our experiments were run on NVIDIA RTX 4090 and A100 GPUs. A run of experiments for the Hanabi environment typically completes in about one hour on these systems, while a run of experiments for the MPE environment only requires 5 minutes. The following paragraphs detail the architecture of our proposed method and the baselines used for comparison. Subsequently, we describe how the "additional desired outcome" is defined and implemented within the MPE and Hanabi environments respectively.
H.1 Implementation of Our Method
Our method, Pre-Strategy Intervention, is realized by augmenting base MARL agent's s architecture with our novel pre-policy module. The full architecture is composed of three key components that work in concert: standard network backbone, a Graph Neural Network (GNN) module for relational reasoning, and the pre-policy module for processing the intervention signal.
Agent Architecture. At each timestep, the agent's 's raw observation is processed in parallel by two components. First, the GNN module constructs a graph representation of the observation to capture relational features between entities (e.g., agents, landmarks). which aligns with the principles of MAIDs (full implementation details are provided in Appendix H.3).. Second, the pre-policy module takes the same observation along with the intrinsic reward from the previous step to produce guidance embedding. Finally, the outputs from the standard backbone, the GNN embedding, and the pre-policy guidance embedding are concatenated and fed into the agent's decision-making head (e.g.. to compute Q-values or inform the critic).
This entire architecture, including the agent's 's policy, the GNN module and the pre-policy module, is trained jointly end-to-end to maximize a composite shaping reward. The composite shaping reward is the sum of the extrinsic task reward and the intrinsic reward derived from the additional desired outcome. As established in Section 4.3 this training process is theoretically sound, as maximizing the intervention's 's causal effect requires observing the total utility variable (Utoz), which directly aligns with the use of a shared team reward in many previous MARL works.
Intrinsic Reward Function. The intrinsic reward quantifies the agent's adherence to the specified "additional desired outcome" at each timestep. For example, in MPE this the negative distance to a target landmark, while in Hanabi it is value indicating compliance with a convention like "5 Save".
30

---

Algorithm 1 Graph Embedding using GNNs
procedure GRAPIEEMEDINN(OOhrrraal
Input: Observation vector
Represent the observation as an influence diagram with semantic features Apply graph convolution using GNN with an adjacency matrix (either learned or manually crafted)
Return:
Graph embedding vector
end procedure
This intrinsic reward serves two purposes: (1) it is direct input to the pre-policy module, and (2) it added to the extrinsic task reward to form a composite shaping reward that guides the learning of the entire system.
Architectural Choice (Nrammtrre Sharing). Since our method focuses on the effect of intervening single targeted agent, other agents' parameters update should be isolated from the targeted agent. For this reason, it is most naturally implemented with a non-parameter sharing (NP) architecture. The Pre-Strategy Intervention is designed to influence only the targeted agent, and an NP setup allows for this specialized learning without forcing homogeneity across all agents' policies.
Targeted Agent Selection. For the experiments presented in this paper, we manually select single targeted agent each environment. This experimental design allows for a clear and controlled analysis of the mechanism's 's impact and mirrors practical scenarios where a human operator might manually choose a specific agent to intervene on.
H.2 Implementation of Baselines
This refers to the standard, unmodified ("vanilla") version of the underlying MARL algorithm. For MPE, these include IQL
Base MARL.
For Hanabi, we specifically PQN-IQL and PQN-VDN. All Base MARL variants use the same network architecture and hyperparameters our method but exclude both the pre-policy module and the additional intrinsic reward to maximize.
33 VDN
34 and QMIX 35
IPPO 36. MAPPO (7). and variants of PQN
38
Intrinsic Reward. This approach serves as a direct ablation of our method to assess the contribution of the pre-policy module. In this variant, the pre-policy module is removed, while the additional desired outcome is conveyed the single targeted agent implicitly via maximizing intrinsic reward along with maximizing the extrinsic task reward.
This approach represents category of algorithms implementing the global intervention paradigm. our experiments, we follow the specific implementation of two repre- sentative methods for this approach: LIIR [8] for policy-based algorithms, and LAIES (2) for value-based algorithms. Both methods principle allocate an additional intrinsic reward to each agent. This contrasts the Intrinsic Reward approach above, where only one targeted agent is provided with an intrinsic reward to maximize. For LAIES, the intrinsic reward is implemented by a function calculating the causal effect of their actions external states. The LAIES framework proposes two types of intrinsic motivation: Individual Diligence (IDI) and Collaborative Diligence (CDI). Notably, the training process for the CDI component relies access to the joint actions of all agents. As proposed method and the other baselines in our experiments operate without leveraging joint action information during training. implemented the LAIES baseline using solely its IDI component to maintain consistency and ensure a fair comparison.
Global Intervention.
H.3
GNN Implementation
The foundation of our targeted intervention method (Pre-Stree (Pre-Strategy Intervention) is formed by the s-reachability graph criterion, introduced identify the relevant policies of other agents for decision- making [17]. However, considering only policies is insufficient, as an agent's policy may depend on other elements within the game 778.. Algorithm [ provides an implementation how we can build connection with a causal graph structure for pre-policy learning. By leveraging this graph structure, we incorporate prior knowledge about the game to help guide agents' policy-making in practical. The fcasibility of learning the causal graph during training has been demonstrated by
31

---

Richens and Everitt [87]. where agents can learn the causal model implicitly during interaction with the environment.
Architecture. If the adjacency matrix is not predefined, the GNN processes the observation vectors by first encoding them into logits, which used generate soft adjacency matrix via the gumbel- softmax technique [88]. This matrix defines the relationship between features in the observations. Once the adjacency matrix formed, graph convolutional layer applies message passing to update the features of each node based on its neighbors [82]. The output node features are then aggregated using a mean- peea-pooling operation produce graph embedding. This embedding used for further processing or decision-making.
H.4 MPE
In the MPE Simple Spread task 19, our setup features agents in 2D continuous space coopera- tively navigating to landmarks while avoiding collisions. We apply our targeted intervention to one agent, using intrinsic reward defined by the negative distance to its selected landmark. This is simultaneous-move environment.
H.4.1 Intrinsic reward
In the environment, define two scenarios for providing intrinsic rewards to the targeted agent.
Scenario (Fixed Target Landmark). The targeted agent receives an intrinsic reward for approach- ing specific, predetermined landmark. This is implemented by treating one landmark (e.g., the first one listed in the observation, regardless of its random initial position) as the target. The agent receives an additional reward inversely proportional to its distance to this fixed target landmark.
Scenario (Dynamic Target Landmark Farthest from Teammates). The targeted agent receives an intrinsic reward for approaching the landmark currently farthest from its other two teammates. This encourages strategic positioning relative to teammates. The farthest landmark I' is determined dynamically at each step as follows:
Definition of the Farthest Landmark. For each landmark 1 L, compute the Fuclidean distances to teammates L1 and p2: (1)) = ||--plla. for i 1,2.
Next, determine the minimum distance for each landmark:
dmin (I) min min (1)
Identify the farthest landmark 1" with the maximum minimum distance:
1
arg max dmin (I). lef
The intrinsic reward for the targeted agent then calculated as:
additional rew = - - -1" 12
where Po is the targeted agent's position.
H.S Hanabi
Team success in the Hanabi card game heavily relies on shared conventions-implic rules that enhance predictability and coordination 200 Our experiments study the 2-agent version of the game. We investigate single agent, can effectively promote adherence to such conventions and thereby improve overall team coordination. Specifically, we guide the targeted agent towards one of two common Hanabi conventions additional desired outcome: "5 Save" (prioritizing hints for unique, high-value cards to prevent accidental discards) or "The Chop" (making discards predictable by targeting the newest, unhinted card).
targeted intervention method (Pre-Strategy Intervention), applied to
32

---

H.5.1 Hanabi Environment and Experimental Setup Details
Hanabi is played by a team of 2-5 players (agents). The standard deck consists of cards in five different suits (colors), with cards in each suit numbered to There are three "1"s, two each of "2"s, "3"s, and "4"s, and one "5" for each suit, totaling 50 cards. The objective is for the team to collaboratively play cards in ascending order (1 through 5) for each of the five suits onto shared "firework" piles. Players hold a hand of cards (typically 4 or 5 cards, depending the number of players), but crucially, they cannot see the cards their own hand; they can only see the hands of their teammates.
On their turn, an agent must perform one of three actions:
Give a Hint: The agent can give a hint to teammate about the cards in that teammate's hand. To do so, they expend one of the team's 's shared "hint tokens". no hint tokens are available, this action cannot be performed.
Discard a Card: The agent selects a card from their own (unknown) hand to discard to common discard pile. This action recover one hint token. The discarded card revealed to all players and is then out of play. After discarding, the agent draws a new card from the deck.
Play& a Card: The agent selects a card from their own hand and attempts to play it onto one of the firework piles. If the card is playable (i.e., it is "1" of a suit for which card has been played, or it is the next number in sequence for a suit that has already been started), it is successfully added to the corresponding firework. If the card is not playable, it is discarded, and the team loses of shared "lives". If all lives are lost, the game ends immediately, and the team scores O. After playing (successfully or not), the agent draws new card.
The team's 's score is the sum of the highest card values successfully played for each of the five suits.
The main difficulty in the Hanabi card game comes from the fact that players cannot see their own cards, a major limitation on what they know. Also, communication between players is very restricted and has cost, as hints are limited. This requires players to make complicated deductions and makes working together effectively as a team extremely hard. For example, players need to discard cards to get more hint tokens. However, this is risky because they might accidentally remove unique and essential cards from the game, which could lower the team's possible score.
Consequently, high-performing Hanabi teams, both human and AI, heavily rely on shared conventions. These are implicit behavioural rules or patterns that enhance the predictability and informativeness of actions, particularly for hints and discards, beyond their literal meaning. However, establishing consistent and effective conventions across all agents is a known and significant hurdle for standard multi-agent reinforcement learning (MARL) approaches, which may struggle to ensure all agents converge to and adhere to the same implicit rules.
H.5.2 Intrinsic Reward
In the environment, we implement two conventions for defining intrinsic rewards.
Convention 1 (5 Save). This convention is erucial for maximizing potential scores, as rank 5 cards are unique for each suit and essential for completing a firework. It aims to enhance team coordination by guiding agents to provide timely and clear hints about rank cards held by teammates. This is especially important if those cards are currently unhinted and therefore at high risk of being accidentally discarded by a teammate unaware of their value. The objective to ensure these vital cards are identified, preserved, and eventually played successfully. The specific logic for calculating the intrinsic reward, which encourages the targeted agent's 's comvention-adherent hinting actions (c.g.. when a teammate holds an unhinted rank 5 card), is detailed in Algorithm 2
Convention 2 (The Chop). This convention promotes predictability in discarding actions, which in turn helps improve team efficiency and reduce miscoordination. It establishes a clear default discard target, often targeted as the newest card in a player's hand that has not yet received any hint (the "chop" card). By adhering to this rule, especially when no other plays or hints are clearly beneficial, an agent provides predictable pattern for discards. This allows teammates to make more informed inferences about the game state and their own hands. The intrinsic reward encourages the targeted agent to follow this specific discard protocol. The detailed logic for identifying the "chop" card and calculating the reward based on the agent's 's discard action is provided in Algorithm
33

---

Algorithm 2 "5 Save" Convention
1: procedure CALCULATE SAVE REWARD(a;, T, hands, hints)
2: Input: action at, of teammates T, hands hands, hint info hints.
3: Output: Intrinsic reward 75sane-
4: T5save +
5: if' any teammate p E T has unhinted rank 5 card in hand then
6:
if af is hint action and a, does NOT hint rank "5" or the color of unhinted then b Penalty: Failed to prioritize hinting the
7:
Sssaee
-1
8: end
9: end if
Return: PSscve
11: end procedure
Algorithm 3 "The Chop" Convention
1: procedure CALCULATE THE CHOP REWARD(O,, hand, hints)
2: Input: current action (₁, agent's 's hand hand, hint info hints.
3: Output: Intrinsic reward rchop-
4: Tekop +-
5: if' a, is a discard action then
6:
Identify catt__ard__nee using hand and hints (rightmost unhinted card index, or None
if all hinted).
7:
Let discorded disccrded_iider be the index discarded by az-
8:
if chop_ shop_card__ae perddidder is None then Tchop +-1
9:
Discouraged: Discarding when all cards are hinted
else discarded inder + chop card inder then
11:
Yrhop < -2
Penalty: Discarded non-chop card Else (discarded the chop card): reward remains
end end if
13:
14:
Return: Tchop
15:
end procedure
I Additional Experiments
L.1
MPE
L.11
Detailed Results of the Scenario shown in the Main Paper
We present the detailed results for the MPE scenario discussed in the main paper, compar- ing our targeted intervention approach (Pre-Strategy Intervention) against base MARL methods. These comparisons are conducted across various MARL algorithms, with results shown in Figure The baseline methods are implemented in two common settings: parameter sharing (PS) and non- parameter sharing (NP). Our Pre-Straat Pre-Strategy Intervention is implemented using non- non-parameter sharing architecture, reflecting more realistic scenarios where agents may not be able to directly exchange parameters during training.
I.1.2 Detailed Results of an Additional Scenario
In this scenario, the targeted agent receive additional feedback in intrinsic rewards, when it is close to the landmarks that are farthest from the another two teammates, detailed Appendix [H.]] Figure presents a detailed comparison of our targeted intervention approach (Pre-Strategy Intervention) against baseline methods, evaluated across various underlying MARL algorithms. Similarly, we consider baselines implemented in two popular implementation settings: parameter sharing (PS) and non-parameter sharing (NP). Our method, using non-parameter sharing, is observed to generally outperform the NP baselines by significant margin in both primary task completion (extrinsic return) and attainment of the additional desired outcome (intrinsic return). Even compared with baselines implemented in PS, method can still manifest comparable performance.
34

---

-10
Intrinale Retuann
-70
PSI
HARI (PS)
Base MARL (NP)
Base MARL INP)
200
Intrinsic Reward (PS)
-17
Intrinsic Reward (PS)
Reward
Timesteps
268
(a) IQL (extrinsic).
(b) IQL (intrinsic).
-40
Intrinsic Return
20
-20
PSI
-70
Base MAIL, IPS)
MARL
-40
Intrinsic Reward (PS)
Intrinsic Reward (PS)
Intrinsic Reward (NP)
10-
Intrinsic Reward (NP)
Timestaps
Timestaps
(c) VDN (extrinsic).
(d) VDN (intrinsic).
Intrinsic Return
-20
-20
-20
Base MARL (P5)
40-
Base MARL (PS)
Base MAIL
-150
Intrinsic Reward (PS)
-10
Intrinsic Reward (NP)
Intrinsic Roward (NP)
-10
5.8
Timestaps
Timestaps
(c) QMIX (extrinsic).
(f) QMIX (intrinsic).
Figure 8: Detailed results of the scenario of MPE shown in the main paper. For each base MARL algorithm (IQL, VDN, and QMIX), extrinsic and intrinsic returns are displayed side by side. The algorithm implemented in parameter sharing is denoted as PS. while that implemented in non- parameter sharing is denoted as NP.
1.2 Hanabi
L.2.1
Detailed Results of the Scenario shown in the Main Paper
We now present detailed results for the Hanabi scenario discussed in the main paper (using the "5 Save" convention as the additional desired outcome), comparing our targeted intervention approach (Pre-Strategy Intervention) against base MARL methods. These comparisons are conducted across various underlying MARL algorithms, with results shown Figure 10 The baseline methods are implemented in two common settings: parameter sharing (PS) and con-parameter sharing (NP). Our Pre-Strategy Intervention is implemented using parameeeetrr sharing architecture, reflecting more realistic scenarios where agents may operate with decentralized training and not directly exchange parameters during training.
1.2.2 Detailed Results of an Additional Scenario
We now consider the scenario with new convention called "The Chop." Similar to above, we present detailed results comparing our targeted intervention approach (Pre-Strategy Intervention) against baselines, across various underlying MARI algorithms, in Figure [1] The baselines again include implementations with parameter sharing (PS) and non-parameter sharing (NP). The results indicate that method generally outperforms the non-parameter sharing (NP) baselines by significant
35

---

Intrinaie Returnn
-70
-22
-.0
PSI
HARI (PS)
Base MAGL (NP)
Base MARL INP)
100
Intrinsic Reward (PS)
Intrinsic Roward (PS)
40
Reward
-110
0.2
Timesteps
268
(a) IQL (extrinsic).
(b) IQL (intrinsic).
-22
50-
Intrinsic Return
40-
25-
PSI
-70
Base MAIL, IPS)
20
MARL
Intrinsic Reward (PS)
Intrinsic Reward (PS)
Intrinsic Reward (NP)
Intrinsic Reward (NP)
Timestaps
Timestaps
(c) VDN (extrinsic).
(d) VDN (intrinsic).
-10
-20
Intrinsic Returr
-20
-100
-120
Base MARL (P5)
Base MARL (PS)
-200
Base MAIL
Intrinsic Reward (PS)
Intrinsic Reward (NP)
Intrinsic Roward (NP)
5.8
Timestaps
Timestaps
(c) QMIX (extrinsic).
(f) QMIX (intrinsic).
Figure 9: Detailed results of additional scenario of MPE, For each base MARL algorithm (IQL, VDN, and QMIX), extrinsic and intrinsic returns are displayed side by side. The algorithm implemented in parameter sharing denoted as PS, while that implemented in non- paraaarmettrr sharing is denoted as NP. Our method is implemented in non-parameter sharing.
margin both primary task completion (extrinsic return) and adherence to "The Chop" convention (intrinsic return). An exception is observed with the IPPO backbone, where the performance difference less pronounced. Compared to parameter sharing (PS) baselines, our Pre-Strategy Intervention often demonstrates comparable performance.
I.3 Additional Results Analysis
Generalizability Across Different Additional Desired Outcomes and Base MARI Algorithms. As shown in Figures g 10 and II] our targeted intervention approach (Pre-Strategy Intervention) consistently outperforms baseline approaches across various base MARL algorithms. Additionally, the results across different experimental scenarios (cach with distinct additional desired outcomes, such specific landmark targeting in MPE or convention adherence in Hanabi) suggest that incor- porating well-defined additional desired outcomes via the Pre-Strategy Intervention can effectively promote the agent learning process towards improved task completion performance.
Relationship Between Primary Task Goals and Additional Desired Outcomes. We find that guiding the targeted agent (via the additional desired outcome) to move towards the landmark farthest from the other two teammates aligns with effective task completion the MPE environment. As shown in Figure the baselines exhibit increasing trend in intrinsic reward compared to Figure 8
36

---

Intrinsic Return
15.0
12.5
MARA y2
Mal
2.5
rotnnd soveld
0.
01
0.4
0.5
0.9
Timestaps
Timestaps
(a) IPPO (extrinsic).
(b) IPPO (intrinsic).
17.5
Intrinsic Return
12.5
Heroii fonord
5.0
3.3
0.2
6.3
0.5
0.5
0.5
0.7
30
0.9
0.1
0.2
0.3
0.5
0.6
0.7
0.9
14
Timastaps
Timestaps
160
160
(c) MAPPO (extrinsic).
(d) MAPPO (intrinsic).
Extrinsic Return
Intrinsic Return
loce
MARI
Irti Ronard
Inbmacc Rovard
wnree: Beaant (87)
Inttted Rewar
0.
02
3.3
25
0.5
03
5.8
1.0
54
0.
03
0.4
0.2
0.5
07
0.8
0.9
Timestep tmmestess
Timesteps
(c) PQN-IQI (extrinsic).
(f) PQN-IQI. (intrinsic).
Intrinsic Return
Extrinaic
Inao toat 050
Dace Honk (S)
Enso (NF1
barval fovert
03
32
03
05
0.5
0.7
85
03
10
02
63
8.3
0.6
5.7
0.
00
Timesteps
10
(g) PIPO-NDI VDN (extrinsic).
(h) PQN-VDN (intrinsic).
Figure 10: Detailed results of the scenario of Hanabi shown in the main paper (with the convention
as "5 Save"). For each base MARL algorithm (IPPO, MAPPO, PQN-IQL and PQN-VDN), extrinsic and intrinsic returns displayed side by side. The algorithm implemented in parameter sharing is denoted as PS, while that implemented in non-parameter sharing is denoted as NP. Our method is implemented in non-parameter sharing.
suggesting that moving towards the farthest landmark may share some overlap with the solution to optimality of the task goal in MPE. However, in scenarios where targeted agent's ss additional desired outcome is randomly pre-assigned (fixed) landmark, the baselines do not show similar spontaneous increase in intrinsic reward for that specific fixed landmark, as observed Figure This may be because moving to the farthest landmark is not inherently required to accomplish the primary
37

---

Intrineic Return
Pise (5)
MAILINS
cum Hoard
Timesteps
Timestaps
(a) IPPO (extrinsic).
(b) IPPO (intrinsic).
25.0
17.5
Intrinsic Return
15.0
-20
Rewe MAXt y5
MANk LnS
2.5
Doaak sosrd (81
0.2
0.4
0.7
98
0.3
0.1
0.2
0,3
0.5
0.7
0.
Timesteps
Timesteps
(c) MAPPO (extrinsic).
(d) MAPPO (intrinsic).
20
Extrinsic Raturn
Intrinsic Return
-10
FSI
VARi, (N7)
Inrind Repand
255
5.8
10
0.3
Timestaps
Timestaps
(e) PQN IQL (extrinsic).
(I) PQN-IQL (intrinsic).
Extrinnic Return
Intrinsic Return
1.2
-20
10
Susi MARL (75)
Faon MAR (NP)
WAR),
wnreed: Iesant|PO byz roi Ronard (97)
25
strrok: fawurd wernet: Saward (M
2.5 Timesteps
Timesteps
1d
(g) PQN-VDN (extrinsic).
(h) PON-VDN (intrinsic).
Figure 11: Detailed results of the scenario Hanabi shown main paper (with convention as "The Chop"). For each base MARL algorithm (IPPO, MAPPO, PQN-IQL and PQN-VDN), extrinsic and intrinsic returns are displayed side by side. The algorithm implemented in parameter sharing denoted as PS, while that implemented in non-parameter sharing is denoted as NP. Our method is implemented in non-paramete sharing.
task goal. Instead, the predictable movement pattern the targeted agent consistently favouring certain landmarks helps coordinate the team's movement, implicitly facilitating task completion.
38

---

1.4 Additional Comparison with Outtoeio Approaches in Hanabi
20-
Extrinsie Returns
1.5
20-
PSI (PQN-IQL)
(PQN-VDN)
LAIES (PQN-IQL)
LAIES (PQN-VDN)
0.1
0.2
0.4
0.5 Timesteps
0.6
0.7
08
0.
1.0
Figure 12: Comparison of extrinsic returns in Hanabi, evaluating Pre-Strategy Intervention (PSI) implementing the targeted intervention paradigm against LAIES (2]) across various MARL algorithm backbones.
Figure 12 provides further evidence in the Hanabi environment, specifically utilizing value- based MARL algorithm backbones (PQN-VDN and PQN-IQL). for the comparison between our Pre- Strategy Intervention (PSI) implementing the targeted intervention paradigm and the LAIES [29] implementing the global intervention paradigm that only considers task completion. The results for extrinsic returns (task completion) are consistent with the findings presented in the main paper (e.g.. Figure 5d] which used different backbones). When applied with either PQN-VDN or PQN-IQL, PSI generally leads to higher performance than LAIES.
1,5 Experiment with Noisy Observation
To evaluate the robustness of our targeted intervention approach (Pre-Strategy Intervention) under imperfect information that is common in many real-world MARL settings, the agents' observations. Specifically, we perturb the belief distributions agents form about cards in Hanabi, simulating scenarios with sensor noise or imperfect state estimation.
we introduce noise into
The experiment is conducted in Hanabi, using "The Chop" convention the additional desired outcome guided by the intervention. For conciseness, we evaluate performance using the PQN-VDN backbone. We test the following three noise scenarios assess robustness under different train-test. conditions:
(L) Noise during Training and Testing, This simulates deploying the system in an environment with persistent observation noise, where the agent can potentially adapt during training.
(2) Noise during Training Only. This simulates training under noisy conditions but testing in clearer environment. This tests robustness to a decrease in noise level compared training conditions.
(3) Noise during Testing Only.
This simulates training in relatively clean conditions but deploying in noisy environment. This tests robustness to unexpected or higher levels of observation noise not seen during training.
I.5.1 Both Training and Testing Containing Noise
In this scenario, the uniform noise (scaled by 0.2) perturbs the card belief distributions during both training and testing, simulating persistent observation uncertainty. As shown in Figures [3a and [3b our method still generally outperforms the base MARI algorithms in primary task completion (extrinsic return). Furthermore, the adherence to the additional desired outcome (intrinsic return) remains effective, indicating robustness when agents can adapt to consistent noise during training.
39

---

Extrinsic Return
Extrinsic Return
10
10
Bove MAAL (PS)
Bace MARI (SS)
Boss MARLOWT
Buss MARLIWI
5.5 Timesteps
6.1
0.3
5.5
Timesteps
1e9
Noise for both training and test (extrinsic).
(b) Noise for both training and test (intrinsic).
Baso MAR. (75)
Bass MAA 1991
Extrinsic Return
Intrinsic Return
10.
10
Bass NANL PSS (NP)
-16
2.3
Timestaps
1c9
1c9
(c) Noise for only training (extrinsic).
(d) Noise for only training (intrinsic).
Ruse MALL (55) Mos.
Extrinsic Return
Intrinsic Return
10-
-b0
Timesteps
Timesteps
1e9
1e9
(c) Noise for only test (extrinsic).
(f) Noise for only test (intrinsic).
Figure 13: Comparison of performance across scenarios with different noise conditions imposed on training and testing. The first row shows results when both training and testing environments contain noise, the second row shows results when only the training environment contains noise, and the third row shows results when only the testing environment contains noise. For each scenario, extrinsic and intrinsic returns are displayed side by side. The algorithm implemented in parameter sharing is denoted as PS, while that implemented in non-parameter sharing is denoted as NP. Our method is implemented in con-paraetter sharing. The base MARL algorithm we select is PON-VDN.
I,5.2 Only Training Environments Containing Noise
In this scenario, the noise (scaled by 0.05) is present only during training, simulating a scenario where the deployment environment has less noise than the training conditions. Figures 13c and [3d show that our method maintains an advantage in task completion over the baselines. However, the performance regarding the additional desired outcome (intrinsic return) is notably impaired compared to the noise-free or consistently noisy cases, performing similarly to the parametee sharing baseline.
1.5.3 Only Testing Environments Containing Noise
This scenario introduces the noise (scaled by 0.02) only during testing, simulating deployment in an environment with unexpected observation noise. As seen in Figures 13c and 13f the performance
40

---

of our method aligns closely with the parameter-sharing baseline in terms of extrinsic return (with overlapping confidence intervals). Similarly, there is no significant advantage in achieving the additional desired outcome (intrinsic return) compared to the non-parameter sharing baseline.
1.6
Hanabi 4 Players
Extrinsic Returne
17.5
Intrinsic Returns
12.5
7.5
PSI
PSI
Intrinsic Reward
2.5
Base MARL
-5
Base MARL
0.2
0.3
0.4
0.6
0.7
0.9
1.0
0.0
0.1
0.2
0.3
0.6
0.7
0.8
0.9
1.0
Timesteps
1e9
Timesteps
le9
(a) 4-player Hanabi (extrinsic).
(b) 4-player Hanabi (intrinsic).
Figure 14: Performance comparison in 4-player Hanabi, with PQN-VDN the base algorithm for all methods. Our Pre- Fre-Strategy Intervention (PSI) is compared against the Base MARL, the GPSI and the Intrinsic Reward baselines.
To further validate the scalability of our Pre-Strategy Intervention, we conducted new experiments on the 4-player version of the Hanabi challenge. This setting serves for coordination, as the strategic complexity and communication burden are known to increase dramatically with the number of players [89]. The results presented in Figure 14 demonstrate that our method achieves significantly higher scores than both the base MARL algorithms and the approach of the global intervention paradigm. This strong performance in larger-scale setting provides robust evidence of our method's 's ability to effectively handle more complex coordination challenges.
demanding benchmark
1.7 Heterogeneous MPE
-40
-50
Intrinsic Returrn
-00
-70
-100
-22
-30
Intervene on Agent (Sprintar)
Intervene on Agent 0 (Sprinter) Intervene Agent (Normal) MARL
Intervene Agent (Normal)
Base
-10
02
5.6
1.0
-20
Timestaps
Timestaps
2e6
Ze6
(a) Heterogeneous MPE (extrinsic).
(b) Heterogeneous MPE (intrinsic).
Figure 15: Performance evaluation in Heterogeneous MPE scenario. We compare the effect of applying our Pre-Strategy Intervention to a fast agent versus normal-speed agent, with IQL as the base algorithm, against baseline without intervention (Base MARL). The heterogeneous environment consists of one fast and two normal-speed agents.
To evaluate the performance our targeted intervention approach (Pre-Strategy Intervention) in settings with diverse agent capabilities, we designed a heterogeneous MPE scenario. This environment features three agents with asymmetric speeds: one "sprinter" agent with five times the normal speed and two normal-speed agents, with all methods built upon the IQL algorithm. The results, shown Figure 15, reveal a consistent trend: intervening on a normal-speed agent leads to noticeable improvement in final team performance (extrinsic return) compared to intervening on the uniquely capable sprinter, although the magnitude of this difference modest in this specific environment. Our analysis suggests this is because the system's 'is primary limitation: the coordination bottleneck
41

---

between the two normal-speed agents, when intervening on the sprinter agent. Even subtle increase in the predictability of one normal agent's behaviour through our intervention can alleviate this core challenge, leading to more effective overall team strategy and demonstrating our approach's s potential in heterogeneous teams where targeted coordination adjustments can yield benefits.
L.8 Extending Pre-Strategy Intervention to Global Intervention Paradigm
Intrinsic Return
Return
15-
10
FS Sano saoe)
crsi
11n2 Chae
(PS)
0.5
0.1
0.7
0.3
04
0.5
Timestops
168
Timesteps
349
(a) IPPO (extrinsic).
(b) IPPO (intrinsic).
20.0
Intrinaic Return
12.5
7.5
de
5.0
Sevel
FSI Save
2.5
(The
Chopl
6P% (The chopo
(P95) (The Chan)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.B
0.9 1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Environment Steps
1s0
Environment Steps
(c) MАPPO (extrinsic).
(d) MAPPO (intrinsic).
225
Returns
20.0
Intrinsic Returns
15.0
-10.
10.0
Save)
-12.
GPSI Chap)
5.0
-17.
Timestaps
1eg
Timestaps
leb
PQN-IQL (extrinsic).
(f) PQN-IQL (intrinsic).
Intrinsii Returns
17.5
22.5
-20
7.5
(The Chopl
Savel
PSi ITne Chop)
Savel
CPS) (The Chop)
GPSi (The chop)
-14
0.
0.3
0.2
0.3
2.4
0.6
0.7
D.
04
1.0
0.1
0.2
0.3
1.A
U.S
0.7
0.01
QM
10
Timesteps
Timesteps
(g) P.N-VDN (extrinsic).
(h) PQN-VDN (intrinsic).
Figure 16: Impact of intervention scope on performance in Hanabi. While extrinsic returns (task completion) are similar, Targeted Intervention (on single agent) leads to higher per-agent intrinsic returns (convention adherence) compared to a Global Intervention baseline where all agents are guided towards the convention. Intrinsic returns are averaged per intervened agent.
42

---

To understand the impact of intervention scope. Figure compares our standard Targeted Pre-strategy Intervention (PSI) with the Global Pre-Strategy Intervention variant (GPSI) where the pre-policy module is applied to guide all agents simultaneously towards the convention. While the performance on primary task completion (extrinsic returns) is broadly similar between these two intervention scopes across the tested MARL algorithms, a notable difference appears in the intrinsic rewards (e.g., as shown in subfigures like [6b] 1ff, and [6 specific backbones). When PSI is targeted to single agent, that agent typically achieves higher intrinsic return (representing convention adherence) than the average per-agent intrinsic return obtained by GPSI. This suggests that targeted intervention may lead to more focused and efficient convention learning by the targeted agent, potentially avoiding redundancies or conflicting convention- based actions that can arise when all agents are simultaneously guided by the same pre-strategy.
Broader Impacts
Our work on Pre-Strategy Intervention (implementing the targeted intervention paradigm) in multi- agent reinforcement learning (MARL) introduces theoretical framework for agent coordination. As foundational research conducted entirely in simulated environments, the immediate socictal impact limited. However, acknowledge that theoretical advances often lay groundwork for future applications. If extended to practical domains, this research could potentially enable more efficient coordination in multi-agent systems by focusing guidance on specific agents rather than requiring direct control of an entire system. Our targeted intervention paradigm could reduce computational and communication overhead in domains like robotic teams, traffic management, or resource allocation. At the same time, we recognize that techniques for influencing multi-agent systems through targeted intervention could potentially be misapplied if deployed in socially sensitive contexts without appropriate oversight. Future work that moves this theoretical foundation toward real- world applications will carefully consider application-spee speciifc ethical implications and incorporate appropriate safeguards against potential misuse.
K Hyperparameters
We show the hyperparameters for all experiments we conduct for the ease of reproducing results.
Table 2: Hyperparameters for MPE in IQL across two scenarios.
Hyperparameter
Scenario I
Scenario 2
Total Timesteps
2 x 10t
2 x 10"
Number of Environments
8
8
Number of Steps
26
26
Buffer Size
5000
5000
Butfer Batch Size
32
32
Network Hidden Size
64
64
Epsilon Start
0.8
0.6
Epsilon Finish
0.02
0.1
Epsilon Decay
0.1
0.1
Learning Rate (LR)
0.0035
0.0045
Learning Starts
10000 timesteps
10000 timesteps
Gamma (r)
0.9
0.9
Max Gradient Norm
5
5
Target Update Interval
200
200
Tau (r)
1.0
1.0
Number of Epochs
Intrinsic Reward Ratio
0.2
Pre- Policy Module Output Size
16
32
Pre- Pre-Policy Module Hidden Size
64
64
GNN Node Embedding Size
16
16
43

---

Table 3: Hyperparameters for MPE in VDN across two scenarios.
Hyperparameter
Scenario 1
Scenario 2
Total Timesteps
2 x 10%
2 x 10"
Number of Environments
8
8
Number of Steps
26
26
Buffer Size
5000
5000
Buffer Batch Size
32
32
Network Hidden Size
64
64
Epsilon Start
0.8
0.6
Epsilon Finish
0.1
0.1
Epsilon Decay
0.1
0.1
Learning Rate (LR)
0.007
0.0045
Learning Starts
10000 timesteps
10000 timesteps
Gamma (y)
0.9
0.9
Max Gradient Norm
5
5
Target Update Interval
200
200
Tau (T)
1.0
1.0
Number of Epochs
Intrinsic Reward Ratio
0.6
0.2
Pre- Policolicy Module Output Size
32
32
Pre-Policy Module Hidden Size
64
64
GNN Node Embedding Size
16
16
Table 4: Hyperparameters for MPE in QMIX.
Hyperparameter
Value
Total Timesteps
2 x 10"
Number of Environments
8
Number of Steps
26
Buffer Size
5000
Buffer Batch Size
32
Network Hidden Size
64
Epsilon Start
0.6
Epsilon Finish
0.1
Epsilon Decay
0.1
Learning Rate (LR)
0.004
Learning Starts
10000 timesteps
Gamma (7)
0.9
Max Gradient Norm
Target Update Interval
200
Tau (T)
1.0
Number Epochs
Intrinsic Reward Ratio
0.2
Pre-Policy Module Output Size
32
Pre-Policy Module Hidden Size
64
GNN Node Embedding Size
8
Mixer Embedding Size
32
Mixer Hypernetwork Hidden Size
128
Mixer Initialization Scale
0.001
44

---

Table 5: Hyperparameters for Independent PPO in MPE.
Hyperparameter
Value
Total Timesteps
1 1 107
Number of Environments
16
Number of Steps
128
Learning Rate
0.00055
Hidden Size
64
Update Epochs
4
Number of Minibatches
8
Gamma
0.99
GAE Lambda
0.85
Clip Epsilon
0.2
Scale Clip Epsilon
False
Entropy Coefficient
0.015
Value Function Coefficient
0.95
Max Gradient Norm
10
Activation
Tanh
Learning Rate Annealing
True
Intrinsic Reward Ratio
0.2
Pre-Policy Module Hidden Size
64
Pre-Policy Module Output Size
16
GNN Node Emhedding Size
8
Table 6: Hyperparameters for PQN in Hanabi.
Hyperparameter
Value
Total Timesteps
1 x 109
Number of Environments
1024
Number of Steps
Hidden Dimension
512
Number of Layers
Normalization Type
Layer Normalization
Normalize Inputs
False
Dueling
True
Epsilon Start
0.03
Epsilon Finish
0.005
Epsilon Decay
0.05
Max Gradient Norm
0.5
Number of Minibatches
1
Number of Epochs
Learning Rate
0.0002
Learning Rate Linear Decay Lambda
True
0.0
Gamma
0.99
Intrinsic Reward Ratio
0.02
Pre-Policy Module Hidden Size
512
Pre-Policy Module Output Size
128
GNN Node Embedding Size
8
GNN Observation Encoder Hidden Size
64
GNN Adjacency Matrix Sampling Temperature
1.0
45

---

Table 7: Hyperparameters for IPPO in Hanabi.
Hyperparameter
Value
Total Timesteps
1 x 10
Number of Environments
1024
Number of Steps
128
Learning Rate
0,0006
Anneal Learning Rate
False
Update Epochs
8
Number of Minibatches
8
Gamma
0.99
GAE Lambda
0.9
Clipping Epsilon
0.1
Entropy Coefficient
0.02
Value Function Coefficient
0.5
Max Gradient Norm
0.5
Intrinsic Reward Ratio
0.035
Pre-Policy Module Hidden Size
256
Pre-Policy Module Output Size
128
GNN Node Embedding Size
8
GNN Observation Encoder Hidden Size
128
GNN Adjacency Matrix Sampling Temperature
1.0
Table 8: Hyperparameters for MAPPO in Hanabi.
Hyperparameter
Value
Total Timesteps
1 x 10
Number of Environments
1024
Number of Steps
128
Learning Rate
0.0065
Anneal Learning Rate
True
Number of Update Epochs
4
Number of Minibatches
4
Gamma
0.99
GAE Lambda
0.95
Clipping Epsilon
0.1
Entropy Coefficient
0.01
Value Function Coefficient
0.5
Max Gradient Norm
0.5
Activation Function
ReL.U
Intrinsic Reward Ratio
0.035
Pre-Policy Module Hidden Size
128
Pre-Policy Module Output Size
256
GNN Node Embedding Size
8
GNN Observation Encoder Hidden Size
128
GNN Adjacency Matrix Sampling Temperature
1.0
46


Reviews:
[{"review_id": "Eifw9zYOvY", "reviewer": "Reviewer_rNPo", "summary": "This paper proposes a novel method to improve zero-shot generalization in multi-agent reinforcement learning (MARL) via targeted intervention. The approach focuses on selectively modifying the observations or dynamics experienced by a specific agent during training, with the goal of improving coordination and generalization across a broader set of unseen scenarios. The method is motivated by the observation that intervening on certain \"key agents\" can have disproportionately positive effects on group performance. The paper provides theoretical motivation, proposes a framework for identifying candidate agents for intervention, and demonstrates empirical gains across multiple MARL environments.", "questions": "How is the targeted intervention actually implemented in the system? Please add an explicit algorithm or example to clarify.\n\nHow is the key agent identified? More formalization of this step would improve reproducibility and transparency.\n\nWhat is the full training loop, including how and when interventions are applied? For example, are they applied during every episode? Is the policy conditioned on the presence of interventions?\n\nWhat happens if the selected agent for intervention is trivial or uninformative?", "limitations": "While the paper proposes an innovative framework, several limitations remain:\n\nThe effectiveness of the method may be highly sensitive to the choice of target agent for intervention. However, the paper does not provide an empirical analysis of what happens when a non-optimal or trivial agent is selected. This raises concerns about the method's robustness and reliability in more diverse or less structured environments.\n\nThe proposed approach is only tested on relatively small-scale MARL environments. It remains unclear whether the method can scale to systems with dozens or hundreds of agents, where the identification and intervention over a \"key\" agent becomes less straightforward.\n\nWhile the core idea is intuitively compelling, the paper lacks theoretical analysis explaining under what conditions targeted interventions are effective. \n\nAlthough the paper argues for improved generalization, it is not clear how broadly the proposed method can transfer to tasks with heterogeneous agent types, varying dynamics, or different coordination protocols.\n\nThe paper does not clearly explain how the intervention strategy itself is learned or tuned. If interventions are manually specified or heuristically chosen, this could limit the method’s adaptability and scalability to new domains.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The idea of targeted intervention is novel and addresses a major limitation of MARL: generalization to new agent configurations or dynamics. This could be impactful in settings such as swarm robotics or decentralized decision-making.\n\nThe paper offers compelling intuition that not all agents contribute equally to the system’s generalization ability, and that targeted manipulation of key agents can help bootstrap stronger coordination patterns.\n\nSections 3 and 4 are difficult to follow and feel disjointed. It is unclear how the proposed intervention mechanism is actually implemented and how it interfaces with the learning algorithm. A simple walkthrough or toy example would greatly help.\n\nThe learning pipeline remains vague. How is the intervention learned? Is it hardcoded, stochastic, or learned through a meta-objective? This lack of detail makes reproduction difficult.\n\nWhile the idea is conceptually appealing, the theoretical justification for why and when targeted intervention leads to generalizable policies is weakly supported. The paper would benefit from more rigorous analysis or at least ablations across different agent heterogeneity levels.", "was_revised": true, "final_justification": "Many of the issues I raised have been resolved through the authors' new experiments.", "rebuttal": "# Response to Reviewer rNPo\n\nThank you for the detailed feedback and insightful questions. We are encouraged that you found our core idea \"novel\" and \"impactful.\" The primary concerns regarding the clarity of our methodology, the need for a more thorough empirical analysis, and the strength of our theoretical justification are all valid points.\n\n\n### **Re: How is the targeted intervention actually implemented in the system? Please add an explicit algorithm or example to clarify.**\n\nAs stated in Line 305, the intervention is realized as a learned pre-policy module (an MLP/GRU). As detailed in Appendix H, this module takes the agent's observation and a measure of the additional goal as input, and its output embedding is used to influence the agent’s Q-value or critic function. For example, in the MPE environment, the pre-policy module inputs the targeted agent's observation and its distance to the selected landmark as the additional goal, and outputs an embedding to the agent's Q-value function. We prove in Appendix F that this practical implementation retains the same solvability properties as the original theoretical pre-strategy intervention.\n\n\n### **Re: What is the full training loop, including how and when interventions are applied? ...  Is the policy conditioned on the presence of interventions?**\n\n**Our pre-policy module is learned end-to-end, not hardcoded.** The training loop operates as follows:\n\nAt every timestep, our pre-policy module generates an embedding that is fed into the agent's critic or Q-value function. The entire system, both the agent policies and our intervention module, is then trained jointly using standard MARL algorithms to maximize a composite reward (Section 4.3). This reward combines the standard extrinsic (task) reward with an intrinsic reward derived from the additional goal.\n\nThe additional goal itself is provided by a heuristic function, such as the distance to a landmark in MPE or the \"Save 5\" convention in Hanabi (detailed in Appendices H.2.1 and H.3.2).\n\nWe agree that this reliance on human-defined heuristics is a limitation. However, as we stated in our future work section, we see great potential in using Large Language Models (LLMs) to automate the generation of these secondary goals. This would significantly enhance the autonomy of our framework.\n\n\n### **Re: How is the key agent identified? More formalization of this step would improve reproducibility and transparency.**\nThe key agent is identified based on its strategic influence, which we formalize using a **relevance graph** that maps the dependencies between strategies of agents. Our method requires selecting an agent that possesses outgoing edges in this graph, as this indicates its strategy directly influences the behavior of other agents, achieving coordination. \n\nIn our current experiments, we select a fixed agent to intervene. This allows us to easily analyze the effectiveness of the intervention mechanism itself. This setup also covers many practical applications where a human operator would manually choose a specific agent to guide, confirming the applicability of our method in real-world systems.\n \n ### **Re: What happens if the selected agent for intervention is trivial or uninformative?**\n \nThis is an excellent question that directly addresses the importance of the \"targeted\" aspect of our method.\n\nIn our framework, a \"trivial\" or \"uninformative\" agent is the one whose strategy has no direct influence on the rest of the team. This corresponds to a node in the **relevance graph** with no outgoing edges, as no other agent is **s-reachable** (Definition C.3) from it by definition.\n\nIf such an agent was selected, the intervention would be ineffective. Any changes to its behavior would be isolated and would not propagate throughout the system to improve coordination.\n\n\n### **Re: Sections 3 and 4 are difficult to follow and feel disjointed...A simple walkthrough or toy example would greatly help.**\nWe are sorry for the lack of clarity. We have revised Sections 3 and 4 to improve the flow and better articulate the connection between our theory and its application.\n\nOur paper is structured to first establish a theoretical foundation and then show its practical implementation.\n\n* **Section 3** establishes a unified theoretical lens by showing how various MARL **coordination paradigms can be represented** using Multi-Agent Influence Diagrams (MAIDs). From this formal representation, we derive the corresponding relevance graph, which is the basis for our intervention mechanism.\n\n* **Section 4** then demonstrates how this framework extends to **sequential decision-making by proving the formal equivalence between MAIDs and Markov Games** (detailed in Appendix G), which justifies our method's application to MARL.\n\nIn the revised manuscript, we have added a clearer narrative bridge to ensure this logical progression from theory to practice is easier to follow.\n\n### **Re: It is not clear how broadly the proposed method can transfer to tasks with heterogeneous agent types**\n\nThank you for the insightful question. To directly address the transferability of our method to heterogeneous agents, we added a new experiment in a modified MPE environment with asymmetric agent capabilities: one **\"sprinter\" agent** (5x speed) and two **normal-speed** agents. All methods, including our interventions and the baseline, are built upon the IQL algorithm. Other settings remain consistent with the experiments described in the main paper.\n\nWe compared the team performance when intervening on the uniquely fast sprinter versus a normal-speed agent. The results are summarized below.\n\n\n **Heterogeneous MPE (Extrinsic Returns)**\n\n| Timesteps| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline** | -68.74 | -63.19 | -66.18 | -62.42 | -57.25 | -53.24 | -49.92 | -46.89 | -44.24 | -41.87 |\n| **Intervene on Sprinter** | -61.35 | -59.70 | -59.03 | -54.49 | -49.00 | -46.20 | -40.34 | -37.72 | -36.18 | -35.05 |\n| **Intervene on Normal Agent** | -65.71 | -62.44 | -60.86 | -55.85 | -50.43 | -44.79 | -40.87 | -36.59 | -35.16 | -34.05 |\n\n **Heterogeneous MPE (Intrinsic Returns)**\n\n| Timesteps| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline** | -41.90 | -25.28 | -67.59 | -47.68 | -39.75 | -29.30 | -27.79 | -27.10 | -27.20 | -23.94 |\n| **Intervene on Sprinter** | -18.06 | -18.23 | -17.48 | -14.88 | -13.06 | -12.64 | -10.91 | -10.24 | -9.77 | -9.51 |\n| **Intervene on Normal Agent** | -20.83 | -19.47 | -19.28 | -17.71 | -15.52 | -13.27 | -12.29 | -11.46 | -11.30 | -10.96 |\n\n\nThese results reveal a compelling result: intervening on a normal-speed agent leads to superior team performance (extrinsic return) over intervening on the uniquely-fast \"sprinter.\"\n\nWhile the sprinter easily achieved its individual goal (as shown by higher intrinsic rewards), the team's success was limited by the **coordination bottleneck** between the two slower agents. Our result shows that making one normal agent's behavior highly predictable allows the other to coordinate with it more effectively, directly addressing this primary bottleneck.\n\nThis experiment provides strong evidence that our method works effectively in **heterogeneous teams**, and reveals that success is attributed to targeting the agent that is most critical to the team's coordination bottleneck, rather than the most individually capable one.\n\n\n\n\n### **Re: The proposed approach is only tested on relatively small-scale MARL environments.**\n\nTo address the valid concern about scalability, we have also added new experiments on the 4-player version of Hanabi based on PQN-VDN. All other experimental settings and hyperparameters were held constant. We believe this is a significant extension, because as highlighted by recent work, the complexity of Hanabi's coordination problem increases dramatically with the number of players[1]. \n#### **4-player version Hanabi Performance Results (Extrinsic Returns)** \n\n| Timesteps      | 10%      | 20%      | 30%      | 40%      | 50%      | 60%      | 70%      | 80%      | 90%      | 100%     |\n| :----------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| Baseline MARL | 8.58     | 11.19    | 11.96    | 11.90    | 15.33    | 14.01    | 14.90    | 15.51    | 15.98    | 17.70    |\n| Global Intervention| 8.03     | 10.73    | 11.49    | 13.67    | 13.68    | 14.27    | 16.57    | 15.48    | 15.81    | 17.64    |\n| Our Method | 12.08| 14.70| 17.29| 18.26| 18.96| 19.44| 19.78| 20.24| 20.43| 20.18|\n\nOur method's strong performance in the 4-player setting, achieving a final score of **20.18** compared to the baseline's **17.70**, provides a strong evidence to show its ability to scale and handle more complex coordination challenges. We will include these results in our revised manuscript.\n\n[1] Sudhakar A V, Nekoei H, Reymond M, et al. A Generalist Hanabi Agent[C]//ICLR. 2025.\n\n\n### **Re: The theoretical justification for why and when targeted intervention leads to generalizable policies is weakly supported.**\nOur method is most effective when there are strong strategic dependencies, a condition we analyze using **relevance graphs (Section 4.2)**. These dependencies manifest as cycles in the graph that impede independent learners. Our intervention **breaks these cycles** by making a targeted agent's behavior predictable, transforming the problem into a more solvable, acyclic one.\n\nThis works because the intervention is explicitly optimized to maximize its **causal effect** on the team's outcome—a property we formally guarantee in Proposition 3.5. We verify this empirically by showing our method allows a decentralized learner (IQL) to match the performance of a centralized one (VDN)."}, {"review_id": "zXqU6CIyAD", "reviewer": "Reviewer_CWb2", "summary": "The paper introduces a framework for guiding cooperative multi-agent reinforcement learning (MARL) systems using targeted pre-strategy intervention on a single agent within a coalition.  The authors use MAIDs to analyse the strategic dependencies within a multi-agent system and then based on this, they propose a “Targeted Intervention” paradigm which uses a pre-strategy intervention mechanism to maximise the causal effect on a desired system-wide outcome or goal. The key contributions of this paper are a theoretical analysis of different MARL paradigms using MAID-based relevance graphs and their solvability under different learning methods and a practical application of target pre-intervention in Hanabi and MPE which makes difficult problems more feasible for simple MARL algorithms.", "questions": "- The paper states that the experimental details and hyperparameter settings are in the appendix. For the sake of transparency and reproducibility, could you provide a more detailed breakdown of the results in the main paper, particularly for the MPE environment? Showing per-scenario performance would offer deeper insight into where the method excels or struggles, which is more informative than an aggregated curve.\n- A central question left open by the \"Targeted Intervention\" framework is the selection criteria for the target agent. This seems especially critical in heterogeneous environments where agents may possess unique information or abilities, making the choice of whom to guide a non-trivial design decision. I would be interested in the authors' perspective on this:\n  - How would you expect the method's performance to vary if a suboptimal agent were chosen as the target? An empirical analysis of this would greatly strengthen the paper.\n  - Could the relevance graph be used to formalise a notion of \"agent influence\" to guide this selection?\n  - Do you have any heuristics or preliminary principles for choosing the most effective agent to target?", "limitations": "Yes, the authors have adequately addressed the limitations of their work in Section 6. They correctly identify that the framework currently assumes a known MAID structure and that their analysis focuses only on a single targeted intervention. They appropriately position these as key directions for future work, including learning the MAID structure from data and investigating the optimal criteria for selecting the target agent(s). The discussion is transparent and constructive.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Originality:** The paper introduces a novel framework that synthesises concepts for MAID, causal inference and MARL. This provides a principled justification for the new methods introduced by the authors and provides a new lens through which to view solving the challenge of guidance in MARL.\n- **Significance:** Directing a complex system where universal intervention is unreasonable is addressed in this work. The method presented by the authors is more scalable and offers both theoretical and practical analysis of their contributions.\n- **Quality and Clarity:** The paper is well written and easy to read. It also presents a well-founded theoretical framework to back its experiments. The use of MAID diagrams is also a clear and effective method to illustrate the coordination paradigms being studied. \n\n## Weaknesses\n\n- Agent selection criteria are undefined: The paper does not propose a method for selecting the target agent. This is limiting as the success of this method relies on effective agent selection.\n- Lack of experiment details in the main paper. Key details for how the experiments are done have been deferred to the appendix and all reported performance is aggregate data. Presenting aggregate performance only may obscure performance nuances between different MPE scenarios.\n- Experiments are only on homogeneous settings. Agent selection is less important in these settings.", "was_revised": true, "final_justification": "The authors have provided new experimental results that address many of the concerns raised by the reviewers and have clearly explained how their method would be implemented in practice. \n\nAlthough the authors do not introduce MAIDs,  it is important that we develop newer paradigms to train MARL agents, which, through future iteration, may lead to more powerful algorithms. I view the use of MAIDs as a retooling of existing methods to support the development of new types of algorithms by the authors for the paradigm of targeted-intervention, which does not seem as heavily explored as other MARL methods.", "rebuttal": "# Response to Reviewer CWb2\n\nWe sincerely thank you for your thorough, insightful, and encouraging review. We are delighted that you found our framework to be novel, significant, and clear. Your constructive feedback has been invaluable, and we have conducted new experiments to directly address the weaknesses you identified.\n\n### **Re: Showing per-scenario performance would offer deeper insight into where the method excels or struggles**\n\nWe thank you for this excellent suggestion. You are correct that the main paper only presents the results for a single MPE scenario (Scenario 1) due to space constraints, and we agree that a per-scenario breakdown offers more valuable insight. As we cannot include images in the rebuttal, the table below summarizes the returns at each 10% of the training process.\n\n\n#### **Per-Scenario MPE Results (Extrinsic Returns)**\n\n| Timesteps | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline** | -67.88 | -61.74 | -62.36 | -61.46 | -59.92 | -55.80 | -53.91 | -50.99 | -48.56 | -47.23 |\n| **Our Method (Scenario 1)** | -63.34 | -54.90 | -52.34 | -49.46 | -45.50 | -42.30 | -40.09 | -38.30 | -37.50 | -36.35 |\n| **Our Method (Scenario 2)** | -68.26 | -61.77 | -63.55 | -60.73 | -56.75 | -53.17 | -49.63 | -47.00 | -44.98 | -43.46 |\n\n#### **Per-Scenario MPE Results (Intrinsic Returns)**\n\n| Timesteps | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline (Scenario 1)** | -24.47 | -22.00 | -24.41 | -23.56 | -24.56 | -23.67 | -23.46 | -23.35 | -22.99 | -22.66 |\n| **Our Method (Scenario 1)** | -15.68 | -11.21 | -10.32 | -10.14 | -10.17 | -9.96 | -9.84 | -9.44 | -9.50 | -9.39 |\n| **Baseline (Scenario 2)** | -25.72 | -21.47 | -22.38 | -21.64 | -20.59 | -19.26 | -17.85 | -16.94 | -15.92 | -15.40 |\n| **Our Method (Scenario 2)** | -24.15 | -20.98 | -21.64 | -20.43 | -18.50 | -16.92 | -15.55 | -14.57 | -13.94 | -13.46 |\n\n\n#### **Analysis**\nAs a brief reminder of the two MPE scenarios: Scenario 1 provides the designated agent with an intrinsic reward for approaching a specific, predetermined landmark. In contrast, Scenario 2 provides an intrinsic reward for approaching the landmark that is farthest from its teammates.\n\n* In Scenario 1, we guide the target agent towards a fixed, predetermined landmark. The results show that this leads to a substantial and consistent improvement in the team's extrinsic return, significantly outperforming the baseline. This demonstrates that making one agent's behavior predictable can benefit the rest of the team to build a more effective coordinated strategy.\n\n* In Scenario 2, the additional goal is more complex, requiring the agent to reason about the target. While the absolute performance is lower than in the simpler Scenario 1, the results show that this \"farthest landmark\" strategy aligns well with an efficient team solution, even when the baseline agents begin to learn the solution emergently. Nevertheless, our intervention still provides a clear benefit by accelerating and refining the learning of this effective coordination strategy. \n\nThese per-scenario results provide a clearer picture of our method's robustness, showing it can improve team performance. We also acknowledge that the more complex goal in Scenario 2 proved more challenging for the agents to learn, as reflected in the overall performance.\n\n\n\n\n\n\n### **Re:  Selection Criteria for the Target Agent.**\n\nThis is a central and very important question. We agree that providing an analysis of agent selection criteria, especially for heterogeneous environments, is crucial. Based on this feedback, we have developed a more formal perspective on agent influence and provide both a principled method and an intuitive case study below.\n\n#### **1. Formalizing \"Agent Influence\" using the Relevance Graph**\n\nAs you suggest, the relevance graph is a very promising tool for formalizing a notion of \"agent influence.\" A key agent can be identified by analyzing the graph's structural properties. While a simple heuristic is to measure an agent's direct influence via the out-degree of its decision nodes, a more concrete measure is its **total influence**, which considers all downstream decisions that are affected by its actions.\n\nFormally, let $\\mathcal{D_i}$ be the set of decision variables for agent $i$, and let $Desc(D_k)$ be the set of all descendant nodes reachable from $D_k$ in the relevance graph. The total influence of agent $i$, denoted $I_{total}(i)$, can be quantified as the size of the union of the descendant sets of its decision nodes:\n\n$$I_{total}(i) = \\left| \\bigcup_{D_k \\in \\mathcal{D}_i} Desc(D_k) \\right|$$\n\nThis metric captures the entire chain of strategic dependencies that originate from agent $i$'s decisions. A higher $I_{total}(i)$ suggests that intervening on agent $i$ would have the broadest possible causal impact on the system.\n\n#### **2. Case Study: An Intuitive Example**\n\nTo make our framework intuitive, imagine four friends planning a weekend trip:\n\n* **Alex (The Planner)** decides the **destination ($D_A$)**: \"Beach\" or \"Mountains.\"\n* **Ben (The Driver)** decides the **vehicle ($D_B$)**: \"SUV\" or \"Convertible.\"\n* **Chris (The Packer)** must choose the **shared gear ($D_C$)**, a decision that depends on both the destination ($D_A$) and the vehicle ($D_B$).\n* **Dana (The Booker)** must book **accommodation ($D_D$)**, which depends only on the destination ($D_A$).\n\n\nThis planning process creates a relevance graph defined by the dependencies: $D_A \\to D_C$, $D_A \\to D_D$, and $D_B \\to D_C$. Applying our influence metric:\n\n* **Alex's Influence (A):** The descendants of $D_A$ are $\\{D_C, D_D\\}$, so $I_{total}(A) = 2$.\n* **Ben's Influence (B):** The only descendant of $D_B$ is $D_C$, so $I_{total}(B) = 1$.\n\nThis formal analysis reveals that intervening on **Alex (The Planner)** is the optimal choice, as their decision has the broadest impact on the team's plan. This case study demonstrates how our framework provides a principled, quantitative method for choosing between intervention targets.\n\n#### **3. Other principles and Future Work**\n\nIn addition to this graph-theoretic approach, another formal principle we are exploring for future work is **empowerment** [1, 2], an information-theoretic measure of an agent's control, which may also correlate strongly with its ability to influence the system.\n\n[1] Klyubin A S, Polani D, Nehaniv C L. Empowerment: A universal agent-centric measure of control[C]//2005 ieee congress on evolutionary computation. IEEE, 2005, 1: 128-135.\n\n[2] Jaques N, Lazaridou A, Hughes E, et al. Social influence as intrinsic motivation for multi-agent deep reinforcement learning[C]//International conference on machine learning. PMLR, 2019: 3040-3049.\n\n### **Re: Experiments are only on homogeneous settings.**\n\nWe added new experiments in a heterogeneous MPE environment where agents have asymmetric speeds. Specifically, we created a scenario with one \"sprinter\" agent given 5x the normal speed, and two normal-speed agents. We then compared the team's performance when intervening on the sprinter versus intervening on one of the normal-speed agents. \n\n **Heterogeneous MPE (Extrinsic Returns)**\n\n| Timesteps| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline** | -68.74 | -63.19 | -66.18 | -62.42 | -57.25 | -53.24 | -49.92 | -46.89 | -44.24 | -41.87 |\n| **Intervene on Agent 0 (Sprinter)** | -61.35 | -59.70 | -59.03 | -54.49 | -49.00 | -46.20 | -40.34 | -37.72 | -36.18 | -35.05 |\n| **Intervene on Agent 1 (Normal)** | -65.71 | -62.44 | -60.86 | -55.85 | -50.43 | -44.79 | -40.87 | -36.59 | -35.16 | -34.05 |\n\n **Heterogeneous MPE (Intrinsic Returns)**\n\n| Timesteps| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline** | -41.90 | -25.28 | -67.59 | -47.68 | -39.75 | -29.30 | -27.79 | -27.10 | -27.20 | -23.94 |\n| **Intervene on Agent 0 (Sprinter)** | -18.06 | -18.23 | -17.48 | -14.88 | -13.06 | -12.64 | -10.91 | -10.24 | -9.77 | -9.51 |\n| **Intervene on Agent 1 (Normal)** | -20.83 | -19.47 | -19.28 | -17.71 | -15.52 | -13.27 | -12.29 | -11.46 | -11.30 | -10.96 |\n\nThese results reveal a compelling insight: intervening on a normal-speed agent leads to superior team performance (extrinsic return) over intervening on the uniquely-fast \"sprinter.\"\n\nWhile the sprinter easily achieved its individual goal (as shown by higher intrinsic rewards), the team's success was limited by the **coordination bottleneck** between the two slower agents. Our analysis shows that making one normal agent's behavior highly predictable allows the other to coordinate with it more effectively, directly addressing this primary bottleneck.\n\nThis experiment provides strong evidence that our method works effectively in **heterogeneous teams**, and reveals that success comes from targeting the agent most critical to the team's coordination bottleneck, not necessarily the most individually capable one. While our current work demonstrates this principle, we agree that automatically identifying this key agent is an important direction for future work, for which information-theoretic metrics like **empowerment** are a promising approach."}, {"review_id": "9USthas12S", "reviewer": "Reviewer_4sVj", "summary": "The authors present MAIDs, a graphical tool to visualize existing coordination mechanisms among multi agents. Moreover, they design Target Intervention, a new coordination mechanism applied to a single agent. This algorithm is their answer to the challenge of achieving coordination through the stimulation of a single agent. \nThe paper formalizes MAIDs and the way MARL are represented using MAIDs. It also describes the pre-strategy intervention and demonstrates the causality of this representation.\nExperiments are then conducted to prove the efficiency of the targeted intervention algorithm.", "questions": "In addition to the general comments, please find below some questions:\n- In Fig 5, wouldn't it be expected to have MAPPO being outperformed by IPPO as MAPPO can be seen as a global intervention and IPPO as a direct one? Can you comment?\n- In Section 5, wouldn't it be interesting to provide an analysis on the Nash Equilibrium (especially as it is detailed in the theoretical section)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written, and is very clear to read. The contributions are clearly listed, as well as the challenge the authors wish to tackle. Background and related work are very clearly described. \nThe principles of MAIDs and targeted intervention are extensively described, both in textual explanation and in mathematical forms. \nThe experimental setup is clearly described, making it doable for a reader to repeat the experiments. However, some of the methods mentioned in the baselines are not reported in the graphs. \nIn terms of results, the connection between the Z variable mentioned in the methodology and the results is not explicit. It would be worth elaborating on this aspect.\nConclusions and future work are very clear and include limitations.", "was_revised": true, "final_justification": "Authors have addressed most of my concerns. I maintain the score.", "rebuttal": "# Response to Reviewer 4sVj\nWe sincerely thank you for your thorough review and positive assessment of our work.\n\nWe are particularly encouraged that you found the paper to be \"well written, and is very clear to read,\" and appreciated that our \"contributions are clearly listed.\" We are also grateful for your recognition of the clear descriptions of our background, methodology, and experimental setup, which are crucial for reproducibility.\n\nThis positive feedback is very valuable, and we also appreciate the insightful questions which we address below.\n\n\n### **Re: In Fig 5, wouldn't it be expected to have MAPPO being outperformed by IPPO as MAPPO can be seen as a global intervention and IPPO as a direct one?**\n\nThis is an excellent question that gets to the core of the practical challenges in MARL. The reviewer's intuition is correct: it is not always a given that a method with access to more \"global\" information like MAPPO will outperform a local information method like IPPO, especially in complex environments.\n\nWhile MAPPO's centralized critic is designed to mitigate the non-stationarity problem, this benefit comes at a significant cost in solving a game as complex as Hanabi: the **Curse of Dimensionality**.  The global observation spaces in Hanabi are enormous (over 1000 dimensions). MAPPO's centralized critic must learn a value function via exploring this massive joint space, which can be incredibly sample-inefficient and difficult to optimize. While a good solution may be guaranteed to exist in theory, the difficulty of searching this larger space means it is not guaranteed to be found in practice.\n\nAt the same time, a naive direct interaction approach like IPPO also struggles. As our results in Figure 5 demonstrate, the baseline IPPO fails to converge to a good solution on its own, performing significantly worse than methods with guidance, such as the intrinsic reward approach or our own method.\n\nThis highlights the core challenge that motivates our work: both complex centralization (MAPPO) and simple decentralization (IPPO) can fail to find a desirable equilibrium in challenging coordination games. This underscores the need for a more targeted and efficient guidance signal, as provided by our Targeted Intervention, which can be a more practical and effective approach to achieving coordination.\n\n\n\n### **Re: In Section 5, wouldn't it be interesting to provide an analysis on the Nash Equilibrium.**\n\nWe thank you for this excellent suggestion. We agree that a direct analysis of the learned equilibrium strengthens the connection between our theory and results. The Hanabi experiments provide a perfect case study for this.\n\nAs established in the literature (e.g., \"The Hanabi Challenge\"[1]), the primary difficulty in Hanabi is its multiple, non-interchangeable equilibria. The challenge for MARL agents is to avoid converging to \"inferior equilibria.\" For example, a team can get stuck in a stable but low-scoring \"babbling equilibrium\" where no agent gives meaningful hints because they expect to be ignored, and no agent listens because hints are uninformative. The goal is to find a high-performing, coordinated equilibrium instead.\n\nA specific human convention, such as \"5 Save,\" represents exactly such a desirable, high-performing Nash Equilibrium [1, 2]. It is a shared strategy where no player has an incentive to unilaterally deviate, as doing so would break the team's coordination and lower the expected score.\n\nIn our experiments, the intrinsic reward is explicitly designed to measure the agents' adherence to the target convention. Therefore, the intrinsic reward curves in our plots can be interpreted as a **direct, quantitative measure of the agents' convergence to this desired convention-based equilibrium.**\n\nAs shown in Figure 5c (right plot), our Targeted Intervention method achieves a consistently high and stable intrinsic reward. This provides strong evidence that the agents have successfully learned and converged to the \"5 Save\" convention equilibrium. In contrast, the low and volatile intrinsic rewards for the baseline methods demonstrate their failure to converge to this desirable equilibrium, suggesting they fell into one of Hanabi's many inferior equilibria and were unable to establish a consistent, high-performing convention.\n\n[1] Bard N, Foerster J N, Chandar S, et al. The hanabi challenge: A new frontier for ai research[J]. Artificial Intelligence, 2020, 280: 103216.\n\n[2] H-Group Conventions https://hanabi.github.io/beginner\n\n\n\n### **Re: Some of the methods mentioned in the baselines are not reported in the graphs.** \n\nWe thank you for this comment and apologize for the lack of clarity. The plots in Figure 5 of the main paper were intended as a high-level summary due to space constraints, and we see now that this was unclear.\n\nTo clarify, the curves labeled **\"Global Intervention\"** in Figure 5 directly represent the performance of the specific baseline methods we mentioned for that paradigm: **LIIR** and **LAIES**.\n\nFor a complete view of our method's performance against every baseline algorithm mentioned in the text, please refer to the appendix, where we provide detailed plots for each individual comparison. For not reported  experiments in the main paper, the information can be found in the Appendix I, where we provide detailed plots comparing our method against each individual baseline algorithm mentioned in the text. Specifically:\n\n * Figures 9 and 10 in the appendix show the per-algorithm results for MPE (vs. IQL, VDN, QMIX).\n\n* Figures 11 and 12 in the appendix show the per-algorithm results for Hanabi (vs. IPPO, MAPPO, PQN-IQL and PQN-VDN).\n\n\n\n### **Re: The connection between the Z variable mentioned in the methodology and the results is not explicit.**\n\nRegarding the $Z$ variable, its practical implementation in our experiments is the intrinsic reward that guides the agent(s) toward the additional desired outcome. Specifically:\n\n* In our MPE experiments, $Z$ is the goal of reaching a specific landmark, and the intrinsic reward is the negative distance to it.\n\n* In our Hanabi experiments, $Z$ is the goal of adhering to a convention (like \"5 Save\"), and the intrinsic reward is a value indicating compliance with that convention.\n\nAfter $Z$ is determined, the measure of the goal (intirnsic reward) would be an input to pre-policy module at each step, as detailed in Appendix F."}, {"review_id": "zmF53FyWTx", "reviewer": "Reviewer_djxC", "summary": "The authors introduce a Targeted Intervention methodology for improving system performance in multi-agent reinforcement learning (MARL) scenarios. In the Targeted Intervention methodology, a single agent is identified which is augmented with additional capabilities in order to facilitate more effective learning by all agents in the system. The authors compare their methodology against numerous baselines, including a simple intrinsic reward methodology, and show small but measurable improvements in system behavior.", "questions": "How many agents were simultaneously interacting in each of the evaluation scenarios (MPE and Hanabi)?\n\nGiven the additional perceptual abilities granted to agents, why is this paper focused on achieving a Nash equilibrium rather than seeking a far more desirable Pareto-optimal equilibrium?\n\nWas a different set of baseline algorithms selected for each of the evaluation tasks (MPE and Hanabi)? The results would be far more illuminating if the same set of baselines was used in both sets of experiments.", "limitations": "This paper does not discuss the limitations of the work or identify potential areas of weakness. The authors are encouraged to consider the bounds within which their work is likely to be useful, and identify those boundaries for the edification of other researchers. It may also be worth identifying the limitations implied by seeking Nash equilibria rather than Pareto-optimal equilibria, and discussing the likely applicability (or lack thereof) of this methodology in real-world contexts where many aspects of the system cannot be known or observed.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The authors provide complete technical detail and a formal description of the proposed methodology\n- The authors compare their work against multiple baselines using both intrinsic and extrinsic reward measurements\n- The authors are tackling an important problem with many real-world applications\n\nWeaknesses\n\n- The writing style over-emphasizes technical detail to the point that the main objectives of the work are difficult to understand. The authors are encouraged to provide simple layman's explanations of their methodology, while still maintaining the rigor of their formal descriptions\n- The core contribution is minimal, and results in only minor performance improvements\n- The Conclusion includes an ambiguous statement that gives the impression that the authors are the originators of the MAID framework, when in reality the authors are instead applying this framework within their specific use case", "was_revised": true, "final_justification": "I thank the authors for their responses. \n\nWhile there are clearly interesting ideas explored in this paper, the limited scope of the experiments (2 agents in one instance, 3 in another) and the relatively small gains reported in the results section leave me unsatisfied with the work as a whole. I would be interested to see the application of these ideas in more complex scenarios.\n\nAs a stylistic note: I disliked the authors' repeated appeals to the opinions of other reviewers as a justification for the work. Strong work should be able to stand on its own, without appeals to external sources of validation.", "rebuttal": "# Response to Reviwer djxC\n\nWe thank you for your detailed feedback. We appreciate them recognizing the technical completeness of our formal description, the thoroughness of our baseline comparisons, and the importance of the problem we are tackling.\n\n### **Re: How many agents were interacting?**\nFor the MPE experiments, we used 3 agents. For the Hanabi experiments, we used the standard 2-agent version, as detailed in Appendices H.2 and H.3.\n\n\n### **Re:  Why is this paper focused on achieving a Nash equilibrium rather than seeking a far more desirable Pareto-optimal equilibrium?**\n\nOur work focuses on achieving a **specific, desirable system-level outcome** that is defined by a human designer. This outcome is a composite of the primary task goal and an additional desired property (e.g., adhering to a convention).\n\nThe concept of Pareto optimality is centered on the individual utilities of the agents—a state is Pareto-optimal if no single agent can be made better off without making another agent worse off. **While this is a crucial concept for social welfare, it is not our direct objective.** The human-defined goal we aim for may not be a social optimum from the agents' perspective, and it's possible that not all agents individually benefit from the intervention in terms of their local rewards. Our evaluation is based on whether the system achieves the human-specified goal. Therefore, Nash Equilibrium (NE) is the more relevant solution concept for our work.\n\n\n### **Re: Why were different sets of baseline algorithms selected?**\n\nWe chose baselines that are considered standard and strong performers for each specific environment. This approach is common practice in the MARL community, as different classes of algorithms are often better suited to different types of coordination challenges. This is reflected in benchmark suites like JaxMARL [1], which often evaluate distinct sets of algorithms on different environments.\n\nTo ensure a thorough and fair comparison, we evaluated our method against a wide range of algorithms. For instance, in **MPE**, in addition to the value-based methods reported in the paper (IQL, VDN, QMIX), we also added new experiments with the policy-based **IPPO**. The results below show a clear and significant performance improvement when our Targeted Intervention is applied.\n\n#### **IPPO MPE Results (Extrinsic Returns)**\n\n| Timesteps | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline IPPO** | -64.84 | -54.03 | -50.41 | -48.74 | -46.60 | -45.17 | -43.57 | -42.85 | -42.88 | -43.32 |\n| **Our Method (IPPO)** | -39.28 | -36.24 | -33.25 | -32.24 | -31.29 | -31.77 | -30.98 | -30.91 | -29.82 | -29.85 |\n\n#### **IPPO MPE Results (Intrinsic Returns)**\n\n| Timesteps | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Baseline IPPO** | -23.48 | -21.70 | -21.49 | -21.22 | -21.37 | -21.39 | -21.29 | -20.19 | -20.81 | -20.80 |\n| **Our Method (IPPO)** | -10.15 | -9.59 | -9.12 | -8.98 | -8.27 | -8.66 | -8.64 | -8.56 | -8.92 | -8.54 |\n\nSimilarly, for **Hanabi**, while the main paper highlights results against policy-based methods (IPPO, MAPPO), our appendix provides a full evaluation against strong value-based methods as well, specifically **PQN-IQL** and **PQN-VDN**. PQN is a popular and powerful baseline for Hanabi [2].\n\nBy evaluating against both value-based and policy-based algorithms across our environments, we have ensured our comparisons are robust and tested against strong, modern implementations from both major classes of MARL algorithms.\n\n\n[1] Rutherford A, Ellis B, Gallici M, et al. Jaxmarl: Multi-agent rl environments and algorithms in jax[J]. Advances in Neural Information Processing Systems, 2024, 37: 50925-50951.\n\n[2] Gallici M, Fellows M, Ellis B, et al. Simplifying deep temporal difference learning[J]. arXiv preprint arXiv:2407.04811, 2024.\n\n\n### **Re: This paper does not discuss the limitations of the work or identify potential areas of weakness.** \n\nWe **respectfully disagree with assessment that the paper does not discuss limitations.** We would like to gently point the you to **Section 6, \"Limitation and Future Work,\"** where we explicitly address the primary limitations of our current framework.\n\nIn this section, we directly address the exact point you raises about the \"likely applicability (or lack thereof) of this methodology in real-world contexts where many aspects of the system cannot be known or observed.\" We state that:\n\n> \"assumes that the underlying structure ... is known or can be accurately modeled. [which] can be challenging to define a priori in complex systems.\" [line: 354-357]\n\nWe then propose \"learning MAID structures directly from data\" as a key direction for future work to overcome this precise limitation. We were encouraged that **Reviewer CWb2** found this discussion to be **\"transparent and constructive,\"** and we hope this clarification is helpful.\n\n### **Re: The authors are encouraged to provide simple layman's explanations of their methodology.**\n\nThank you for this suggestion. As advised, here is a simple, layman's explanation of our motivation, research question, and methodology, drawing from the example in our introduction.\n\n#### **Motivation and Research Question**\nAs we state in our introduction, guiding a team of agents is often impractical when you can't give instructions to everyone.\n\n> Consider autonomous vehicles at a complex intersection. Providing specific feedback to all vehicles at once is infeasible due to safety validation and communication challenges. Yet, effective coordination is essential.\n>\n> This leads directly to our central research question: **What if providing an additional guidance signal to just one designated vehicle—perhaps to predictively adjust its speed—allowed other vehicles, by observing this targeted behavior, to adapt accordingly and foster smoother, safer passage for all?** Can we achieve effective team-wide coordination by assigning a targeted goal to just one agent?\n\n\n\nOur method, **Targeted Intervention**, provides a principled way to answer this question. It works in three steps:\n\n1.  **Map the Team's Influence:** First, we use a graphical tool (a Multi-Agent Influence Diagram) to create a \"flowchart\" of the team's strategic dependencies. This map shows us which agent's decisions have the biggest ripple effect on the actions of others.\n\n2.  **Give a Special Goal to One Agent:** Based on this map, we select a key agent and give it an additional goal on top of the main task. For the autonomous vehicle, this would be the \"predictively adjust speed\" instruction. For the card game Hanabi, it might be \"always signal information about the most valuable cards.\"\n\n3.  **Use a \"Pre-Policy\" to Guide the Agent:** We implement this with a learned component we call a **pre-policy module**. This module takes the special goal and translates it into a guidance signal that directly **influences the agent's decision-making process**, encouraging it to follow the instruction while still pursuing the main team objective.\n\n\n### **Re: The core contribution is minimal, and results in only minor performance improvements.**\n\nWe take the your concern about the significance of our contribution seriously. We **respectfully disagree with the assessment that the contribution is minimal**, and we would like to clarify the full scope of our work, which we believe is substantial.\n\nOur contribution is a complete, theory-to-practice framework:\n1.  First, we **formalize** different multi-agent coordination paradigms using the principled graphical language of Multi-Agent Influence Diagrams (MAIDs).\n2.  Second, based on this framework, we introduce our novel **Targeted Intervention** paradigm and a practical implementation via **pre-strategy intervention**, which is theoretically guaranteed to maximize the causal effect on a desirable system outcome.\n3.  Third, we show how this theoretical concept can be **practically applied to MARL**, and our solvability analysis using relevance graphs provides a new way to predict the effectiveness of different learning approaches.\n4.  Finally, we **verify** these theoretical claims with extensive experiments, showing that our method is not only effective but often superior to common alternatives.\n\nWe are encouraged that other reviewers recognized the significance of this complete contribution. Specifically, Reviewer CWb2 highlighted that our method addresses the important challenge of \"directing a complex system where universal intervention is unreasonable\" and is \"more scalable\" while offering \"both theoretical and practical analysis.\" We believe this consensus among other reviewers underscores the significance of our work.\n\n\n\n### **Re: The Conclusion includes an ambiguous statement that gives the impression that the authors are the originators of the MAID framework.**\n\nWe thank you for your careful reading, **but this could be a misunderstanding**. You are correct that MAIDs were invented by Koller and Milch (2003). **Our use of the word \"introduced\" was meant in the context of introducing MAIDs as a framework for our specific problem of  Targeted Intervention, not to claim their invention.**"}]

Author rebuttal:

Meta-review: This paper presents a method using targeted interventions on single key agent in a MARL setting to improve performance. The approach builds on the Multi-Agent Influence Diagram (MAID) framework, formalizing MARL in this context along with describing the approach, and demonstrating performance. 

There are some concerns about the presentation and the novelty of using MAIDs but the author response and discussion was helpful in terms of presentation questions and while MAIDs isn't a contribution, its use in MARL to build the proposed intervention method is. The paper should be updated to clarify some of the assumptions about the intervention (e.g., selecting the target agent and heuristics), solution concept, and implementation details.
Decision: Accept (poster)