Causal Mean Field Multi-Agent Reinforcement Learning
Authors: Hao Ma
Zhiqiang Pu
Yi Pan
Boyin Liu
Min Chen
Shijie Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement mearning
causal inference

Abstract: Scalability remains a challenge in multi-agent reinforcement learning and is currently under active research. However, existing works lack the ability to identify the essential interaction under the non-stationary environment. We propose causal mean field Q-learning (CMFQ) to address this problem. It has the advantage of MFQ, which can compress the space size dramatically. Besides, it is ever more robust toward the non-stationary caused by increasing agents. We enable agents to identify which ally or opponent is more crucial by asking "what if" with the help of the structural causal model (SCM), then pay more attention to more crucial ones. We test CMFQ in mixed cooperative-competitive and cooperative games, which verify our method's scalability performance.

Under review as a conference paper at ICLR 2023
CAUSAL MEAN FIELD MULTI-AGENT REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Scalability remains challenge in multi-agent reinforcement learning and is cur- rently under active research. A framework named mean-field reinforcement learn- ing (MFRI. could alleviate the scalability problem by employing Mean Field Theory to turn a many-agent problem into a two-agent problem. However, this framework lacks the ability to identify essential interactions under non-stationary environments. Causality contains relatively invariant mechanisms behind inter- actions, though environments are non-stationary. Therefore, propose algo- rithm called causal mean-field Q-learning (CMFQ) to address the scalability prob- lem. CMFQ ever more robust toward the change of the number of agents though inheriting the compressed representation of MFRL's action-statee space. Firstly, we model the causality behind the decision mrtisiionmakinng process MFRL into structural causal model (SCM). Then the essential degree of each interaction is quantified via intervening on the SCM. Furthermore, we design the causality-aware com- pact representation for behavioral information of agents the weighted sum of all behavioral information according to their causal effects. We test CMFQ in a mixed cooperative-comper that our method has excellent scalability performance in both training in environ- ments containing large number of agents and testing in environments containing much more agents.
game and cooperative game. The result shows
INTRODUCTION
multi-agent reinforcement learning (MARL) has achieved remarkable success in some challenging tasks. e.g., video games(Vinyals al. 2019 Wu 2019). However, training large number of agents remains a challenge in MARL. The main reasons are 1) the dimensionality of joint state-action space increases exponentially as agent number increases, and 2) during the training for single agent, the policies of other agents keep changing, causing the non-stationarity problem, whose severity in- creases as agent number increases. increases.(Sycara) 1998. Zhang et al. 2019 Gronauer & Diepold 2021). Existing works generally use the centralized training and decentralized execution paradigm to miti- gate the scalability problem via mitigating the non-stationarity problem(Rashid (Rahhid et al. 2018 8 Foerster cet al. 2018 Lowe et al. 2017 Sunchag et al. 2017). Curriculum learning and attention techniques are also used to improve the scalability performance(Long et al., 2020 Iqbal & Sha 2019). How- ever, above methods focus mostly tens of agents. For targe-scale multi-agent system (MAS) contains hundreds of agents, studies in game theory(Blume) 1993) and mean-field theory(Stanley
1971 Yang et al. 2018)) offers feasible framework to mitigate the scalability problem. Under this framework, Yang et al. (2088) propose a algorithm called mean-field Q-learning (MFQ), which replaces joint action joint Q-function with average action, assuming that the entire aise interactions could be simplified into the mean of local pairwise interactions. That is. MFQ reduces the dimensionality of joint state-action space with merged agent. However, this approach ignores the importance differences of the pairwise interactions, resulting in the poor robustness. Neverthe- less, one of the drawbacks to mean field theory is that does not properly account for fluctuations when few interactions exist(Uzunov 1993) (e.g., the average action may change drastically if there are only two adjacent agents). Wang et al. (022) attempt to improve the representational ability of the merged agent by assign weight to each pairwise interaction by its attention score. However, the observations of other agents are needed as input, making this method not practical enough in the real world. In addition, the attention score is essentially correlation in feature space, which seems

---

Under review as a conference paper at ICLR 2023
unconvincing. On the one hand, agent pays more attention to another agent not simply because of the higher correlation. On the other hand, it may be inevitable that the proximal agents will be assigned high weight just because of the high similarity of their observation.
In this paper, we want to discuss a better way to represent the merged agent. We propose algorithm named causal mean-field Q-learning (CMFQ) address the shorteoming of MFQ in robustness via causal inference. Research in psychology reveals that humans have sense of the logic of interven- tion and will employ it in poeiision-makin context(Slomaal & Lagnado 2015). This suggests that by allowing agents to intervene in the framework of mean-field reinforcement learning (MFRL), they could have the capacity identify more essential interactions humans do. Inspired by this insight, we assume that different pairwise interactions should be assigned different weights, and the weights could be obtained via intervening. We introduce structural causal model (SCM) that rep- resents the invariant causal structure of decision-makinn in MFRL.. We intervene on the SCM such that the corresponding effect of specific pairwise interaction can be presented by comparing the dif- ference before and after the intervention. Intuitively, the intervening enable agents to ask "what if the merged agent was replaced with an adjacent agent" as illustrated in Fig 1 In practice, the pair- wise interactions could be embodied actions taken between two agents, therefore the intervention also performs on the action in this case.
CMFQ based on the assumption that the joint Q-function could be factorized into local pairwise O-functions, which mitigates the dimension curse in the scalability problem. Moreover, CMFQ alleviates another challenge in the scalability problem, namely non-stationarity. by focusing on cru- cial pairwise interactions. Identifying crucial interactions is based on causal inference instead of attention mechanism. Surprisingly,
scalability performance CMFQ is much better than the iitention-based method Wang et al.) (022). The reasons will be discussed in experiments section. As causal inference only needs local pairwise Q-functions, CMFQ is practical real-world appli- cations, which are usually partially observable. We evaluate CMFQ in the cooperative predator-pre prey game and mixed cooperative compeeitvcei battle game. The results illustrate that the scalability of CMFQ significantly outperforms all the baselines. Furthermore, results show that agents controlled by CMFQ emerge with more advanced collective intelligence.
The Imaginary of agent
Wellor perged of REUT
repHon Banqua OQUIT POSES
Whstt
My woald charge comnutiiily & th perreed oyor nun deadd
Blue agents and orange agents belong to different teams. The purple agent denote a merged agent that simply average all agents in agent i's neighborhood. The diagram on the left shows a scenario in which the central agent i interacts with many agents, is denotes the h agent in the observation of agent In the framework of MFRL, the scenario is transferred to the diagram in the middle, in which an merged agent is used to characterize all the agents in the central agent's s observation. Our method further enables the central agent to learn to ask "what if". When it asks this question, it can imagine the scenario illustrated in the right diagram. The central agent can hypothetically replace the action of the merged agent in MFRL with the action of a neighborhood agent, and if this replacement will cause dramatic changes in policy, it means this neighborhood agent is potentially important. Thus central agent should pay more attention to the interaction with this neighborhood agent.
Figure I:
RELATED WORK
The scalability problem has been widely investigated in current literatures. Yang al. 2018) pro- pose the framework of MFRL that increases scalability by reducing the action-state space. Several works in related named mean-field game also proves that using compact representation to
2

---

A N-player stochastic game could be formalized G =< S,A, P,,, in which N agents in the environment take action a E A = N to interact with other agents and the environment. Environment will transfer according to the transition probability P(s' s, a) : A x [0, 1], then every agent obtains its reward (s, a²) S x A' + R and y E [0, 1] is the discount factor. probability Under review as a conference paper at ICLR 2023
characterize population information helps solve scalability problem(Gur et al. 2019 Perrin al. 2021).
Several works were proposed to improve MFQ Wu et al. (222) proposed weighted mean- mean-field assigning different weights to neighbor actions according to the correlations the hand-craft agent
attribute set, which is difficult generalize different environments. [Wang al. (2022) calculate
the weights with attention score. The observations of other agents are needed to calculate the
attention scores, making its practicality not satisfactory.
Our work is also closely related to recent development in causal inference. Researches indicate that once the SCM, which implicitly contains the causal relationships between variables, is constructed, we can obtain the causal effect by intervening. The causal inference has already been exploited for communication pruning(Ding et al., 2020), solving credit assignment problem(Foerseae
al., 2018 Omidshafiei et al. 2019). demonstrating the potential of causal inference in reinforcement learning. (Pearl 2019 2001: Peters et al., 2017). Xia et al. (202I) and Zečević et al. (202I) further proved that SCM could be equally replaced with NCM under certain constraints, enabling us to ask "what if" by directly intervening on neural network.
I's nelborhood
LG
Coiel Mude
Network
Softraxs
Netoe
7
Coail
Block
(a)
(b)
Figure 2: (a) is CMFQ's architecture. Each neighborhood agent is assigned a weight according to its causal effect to the policy of the central agent. (b) is the causal module. It calculate the KL divergence between the two policies that the merged agent is represented by the average action and the h neighborhood agent action respectively. A large K L. divergence means the 1 neighborhood agent might ignored in the merged agent represented by the average action, hence it should be assigned a higher weight to form better merged agent.
PRELIMINARY
This section discusses the concepts of the stochastic game, mean-field reinforcement learning, and causal inference.
3.1 STOCHASTIC GAME
Agent makes decision according to its policy x (s) S 2(A*). where (2(A*) distribution over agent i''s action space A'.
The joint Q-function of agent is parameterized by 0, and takes s and a. It is updated as
(,,,,) = Es...ss'
[Q* (s, as,) v) -))
y = r max Q² (s)));0,)
where 8,5 is updated by with 0 every C steps and set fixed until the next C' steps finish. where predict network parameters update at every epoch. Target network parameters 0 denotes parameters be updated with
3

---

Under review as a conference paper at ICLR 2023
3.2 MEAN FIELD REINFORCEMENT LEARNING
Mean field approximation turns a many-agent problem into a two-agent problem by mapping the joint action space a single action space. The joint action Q function is firstly factorized considering only local pairwise interactions, then pairwise interactions are approximated using the mean-field theory
Qf (s,a", a N
1 NO
Σ
Q' (s. aa) = Q' (s,
(2)
AEN())
where N = [i) (i) the set of agent i's neighboring agents. Interactions between central agent i and its neighbors are reduced to the interaction between the central agent and an abstract agent, which is presented by average behavior information of agents in the neighborhood of agent Finally, the policy of the central agent is determined by pairwise Q-function
exp (BQ) (se, o4. aj-1))
के (at 8, a.)
(3)
Σ Ennt exp (3Q) (sc. a² ..-1))
It is proven that के will converge eventually Yang et al. 2018
3.3 CAUSAL INFERENCE
The data-driven statistical learning method lacks the identification of causality which is quite vital part of composing human acknowledge. The SCM established with human knowledge is needed to represent the causality among all the variables we consider. An SCM is a 4-tuple M =< U, V, F, P(U) >. U = (U1, U2. Um is the set of exogenous variables which are determined by factors outside the model.
(V1, ...... Vn is the set endogenous variables that are determined by other variables. F is set of functions {ʃv. fV., fv,, J such that fV; maps Pay, U UU,, to Vj. where Uv, S U is all the exogenous variables directly point to V, and Pay, S VV, is all the endogenous variables directly point to VG. That is, V, fv, (Pav,. Uv,) for J = 0, n. P(U) is the probability distribution function over the domain of U. The causal mechanism in SCM M induced acyelic graph G, which uses direct arrow to present direct effect between variables as shown Fig 3 Intervention is performed through an operator called do(x), which directly deletes fx and replaces with constant X I, while the rest of the model keeps unchanged. The equation defines the post-intervention distribution
Pm((ylo(x)) & PM,
where M₁ is the SCM after performing do(x). Once we obtain the post-intervention distribution, one may measure the causal effect by comparing it with the pre- pre-intervention distribution. A common measure is the average causal effect.
[do(do()))] F[[(do(za))
(5)
where z/ and To are two different interventions. The causal effect may also be measured by the experimental Risk Ratio(Pear) 2010)
E [[{(d((b)]
(6)
E do()))
METHOD
4.1
CONSTRUCTION OF SCM
The first step to obtain the causal effects behind the interactions among agents is constructing an SCM, which reveals the causal relations among all the variables. In the setting of MFRL, mean action /i_ and state determine the policy T]( st, af_,) ) of agent i. As the key causal effect we concern how important an interaction for decision making, that is, how the interaction contains in aل_, affects тद. we construct the SCM center on T/ as illustrated Fig 3(b) The importance of the interaction with adjacent agent on the central agent i could be estimated by replacing ai-
t. with a and quantified by the change of Tे. Formally, the causal effect of acting at on лi is
T TF DIST( 8t, tt,,t, at a nt(. 8t, at, do(at_ 1.k -((1)))
4

---

Under review as a conference paper at ICLR 2023
i.k is the action of the k* agent in the neighborhood of agent The causal effects in Eq. where d, 1 and 6 are quantified using the difference in statistics before and after the intervention. As the utilize the difference in policy to quantify causal effects. DIST measures the difference between pre-intervention policy and post-intervention policy. We use IT divergence as the DTST function in practice. As n parameterized using neural network, the do(-) calculation is performed by directly changing the input of πÿ. It is worth noting that not all neural networks capable of causal inference(Xia al. 2021). As a neural network learned by interacting with the environment, zi lies on the second layer of Pearl Causal Hierarchy (Bareinboim et al. 2022), wese wise interaction and the causality between
policy are known,
and naturally contains both the causality between agent- agent-environment interaction. It is sufficient for estimating the causal effect of certain interaction.
sp-P (t-a.
2 = f2(U2)
af-nt( (s.
(x(2.U))
Y : fxX,
X
Y
de
U2 Ug (0,1)
(x(x,)
1 ( 1.
Ao(
(a)
(b)
Figure 3: (a) is a canonical SCM, when do(zo) operation is performed on X, all causes of X will be broken and keep all variable constant but only change X to zig. (b) is the SCM MFRL, the do(-) operation on ج:_ة follows the same procedure.
4.2 IMPROVING MFQ WITH CAUSAL EFFECT
In MFRL, assume that different pairwise Q-functions should be assigned different weights de- pending on their potential influences on the policy of central agent. Hence, the factorization of Eq.22 2 should be revised to
Q' (s, a a. = E wit (s,a'
(8)
kEN())
where (t) is the set of agent i's adjacent agents. Then Q'(ss, a² a',,a,,,,, a,a , is approximated using oean-field theory and considering the causality-awan weights
(s, a² .... ,,,) = Σ
* (s,a',ath)
KEN())
Σ witk Q' (s((s,')a)) Va (s,af, a²) -aai batt Qz (e,dd,ait Sa KEN())
Qssaaa "a'aa,,")) V Q' (s,a","")
Σ wiikk Satk
+ = wiik
(2 20 Siai..Va... Q's,at -8at
kEN())
kEN())
Q² a')
toikp! Ri (atit) xo Q² (s)
kEN(1)
where Sa = a ot and 0² EkEN() aotikathe hence En witk = 0. In the second-order term, ãid ă² ci eik (0, 1). H Higg (aiik) denotes the first-order Taylor expansion's Lagrange remainder which is bounded by [-L, L] in the condition that the Q' (s. a', a²*²) function I--smotthed. To self-contained, we put the derivation Appendix B The remainder a value fluctuating around zero. As Yang et al. (2018) discussed in their work, under the assumption that fluctuations caused by adjacent agents tend to cancel each other, the remainder could be neglected. Once causal effects of pairwise interactions are known, the next question is how to to improve the representational capacity of the merged agent. Both linear methods, e.g., weighted sum, or ensure
nonlinear methods, e.g., encoding with neural network, might be useful. However, the merged agent's 's reasonability, we prefer a representation in the linear space formed by adjacent agents' action vectors. An intuitive method that can induce reasonable output is weighted sum. In
5

---

Under review as a conference paper at ICLR 2023
practice, we find that weighted sum using respective causal effects as weight is enough to effectively improve the representational capacity of average action
exp (3Q) (st, aj a;-1))
oं (a) 82, 011 Enres exp (3Q) (st, a", ")-1))
Σ
ikk i.k Wi
a
(10)
kEN(s)
TE +E (TE +e)
1.k EKEN(O)
W
(11)
where subscripts are used to denote time steps. TE; is calculated according to Eq. Each d is encoded in one hot vector. Hence the weighted sum returns reasonable representation in the linear space formed by the actions of neighborhoods. Moreover, the representation is close to essential actions, emphasizing high-potential impact interactions. A term was introduced smooth the weight distribution across all adjacent agents, avoiding additional non-stationarity during training. Besides, the naive mean-field approximation could be achieved when c + 0G. The Q-function Q² update using the following loss function similar with Eq.
[(0 a';0.) (r+ (++7 7-xxmax max Q' at: aft0 ))))
(o:):) = Bs,a,r,o'
(12)
a²
EXPERIMENTS
We evaluate CMFQ in two tasks: a mixed cooperative-competine battle game and a coopera- tive predator-prey game. In the battle task, we compare CMFQ with IQI-Tampuu al.) (017). MFQ et al. 2018), and Attention-MFO/Wang et al. (022) to investigate the effectiveness and scaling capacity of CMFQ. We further verify the effectiveness of CMFQ in another task. In the predator-prey task, we compare CMFQ with MFQ and Attention-MFO. Our experiment environ- ment is MAgent(Zheng al. 2018).
5.1 MIXED COOPERATIVE-COMPETTII
GAME
Task Setting. In this task, agents separated into groups, each containing N agents. Every agent tries to survive and annihilate the other group. Ultimately the team with more agents surviving wins. Each agent ob-
tains partial observation of the environment and knows the last actions other agents took. Agents will be pun- ished when moving and attacking to lead agents to act efficiently. Agents are punished when dead and only rewarded when killing the enemy. The reward setting requires the agent to cooperate efficiently with team- mates to annihilate enemies. In the experiments, we train CMFQ, IQL, MFQ, and Attention- MFQ in the setting of N = 64, then we change N from 64 to 400 to investigate the scalability of CMFQ. The concrete reward values are
Tood Rewadd
200
Eptoade
set as follow:
Tattack = -0.1, I move = -0.005, Idead
Figure 4: Total reward during training.
-ksil 5. We train every algorithm in self-play paradigm.
Quantitative Results and Analysis. illustrated in Fig 5(a) we compare CMFQ with Attention- MFQ, MFQ, and IQL. We do not choose Wu et al. (2022) as a baseline because it is correlation based algorithm identical to Attention-MFQ. We assume that the attention- based method a more challenging baseline. Moreover, in addition to these algorithms, we also set ablation algorithms named Random to verify that the performance improvement of CMFQ is not caused by random- ization. Random follows the interaction. FigMshoww the learning curve of all algorithms. We can see that the total rewards of all algorithms converge stable value, empirically demonstrating the training scalability of our
pipeline CMFQ but returns random causal effect for each

---

Under review as a conference paper at ICLR 2023
algorithm.
To compare the performance of each algorithm, we put trained algorithms in the test environment that N 64, and let them battle against each other. Fig shows that MFQ performs better than IQL but worse than Attention-MFQ. indicating that the mean-field approximation mitigates the scalability problem in this task. However, the simply averaging MFQ is not good repre- sentation of the population behavioral information. In order to improve its representational ability for large-scale scenarios, it is necessary assign different weights to different agents. Moreover, CMFQ outperforms Attention-MFQ during the test, verifying the correctness of our hypothesis that correlation- based weighting insufficient to catch the essential interactions properly, while the tervention fills this gap by giving agents the ability to ask "what if"
1402 Airk-wiing COR2
603
L-6-2L
(a) Performance comparisons.
(b) Test Scalability curve.
(c) Ablation experiments of e.
Figure 5: Win rate during execution. (a) shows the results that algorithms battle against each other.
the horizontal axis divided into five groups by algorithms, and within each group there are five bars representing the win rate of the algorithm on the horizontal axis. (b) shows win rates of algorithms in the label against MFQ algorithms which are on the horizontal axis. (c) shows the win rate of CMFQ with different against other algorithms.
We further investigate the test scalability of CMFQ, MFQ, and Attention-MFO Firstly, we train these three algorithms in 64 vs. 64 scenario with self-play, denoted as CMFQs4. MFQ Attention-MFQea. respectively, and further train the MFQ algorithm in 100 vs. 100 and 400 vs. 400 scenarios, denoted as MFQ100 and MFQ Then, allow CMFQ MFQ and Attention-MFQes to battle against MFQ MFQ and MFQ400 in environments 64 vs. 64, 100 vs. 100, 400 vs. 400 respectively, that is, letting CMFQ, MFQ, and Attention-MFQ control more agents than they were trained, to reveal the test scalability of the algorithms. As shown in Fig]5(6) the test scalability of MFQ is the worst, which means that we need to retrain MFQ when the number of agents increases and the test scalability of Attention-MFQ is slightly better. The test scalability of CMFQ is significantly better than both of them. Furthermore, CMFQ achieves win rates of nearly 100% against MFQ100 and 32% against MFQ100-
Ablations. We set two ablation experiments. The first one to ablate the effectiveness of causal effects in CMFQ. As illustrated 5(g the performance of Random is inferior to MFQ, verifying the validity of causal effect in CMFQ. The other one is ablation for e. As we analyze in 4.2 is adjustable parameter in the interval [0, +oo). As increases, the effect of each interaction becomes smoother and eventually CMFQ equal MFQ when e - +00. From the Fig we can see that as we adjust e from 0.001 to 1, the learning curve of CMFQ always converges, and in the test environment, win rates of CMFQ always outperform other baselines. When e is relatively large, the win rate close to that of MFQ.
(a)
(b)
Figure 6: Visualization of CMFQ vs MFQ in 64 vs 64 en- vironment. Red squares denote CMFQ, and blue squares denote MFQ, the vertical bar on the left side of the square indicates its health point, and the surrounding circular area indicates its attack range. When agent attacks, an arrow will be extended to point at the attack target.

---

Under review as a conference paper at ICLR 2023
Visualization Analysis. As illustrated in 6(a) MFQ tends to confront frontally. The results in 6bg indicate the tricky issue in mixed couperative-compettivee game: agents need to cooperate with their teammates to kill enemies, whereas only the agent who hits a fatal attack gets the biggest reward risnt, driving agents hesitating to attack first. When there are few agents, the policies of MFQ and CMFQ tend be conservative. However, CMFQ presents more advanced tactics: agents learn the trick of teaming up in the mixed cooperatiee-compeliivee game. When an agent chooses to attack, the adjacent teammates will arrive to help, achieving the maximum reward with the smallest cost of health. Moreover, Fig bbb also shows that attacks of CMFQ are more focused than baselines. CMFQ can discriminate key interactions and have a more accurate timing of attacks, while MFQ lacks this discrininatory ability and thus keeps attacking.
CMFQ learns the tactic of besieging, while
5.2 COOPERATIVE GAME
Task Setting. In this task, agents are divided into predator and prey. Prey move 1.5 times faster than predators, and their task is to avoid predators as much possible. Predators are four times larger than prey and can attack but not yield any damage. Predators only get re- warded when they are close to prey. Therefore, to gain the reward, they must cooperate with other predators and try to surround prey with their size advantage. In our experiments, to test the scalability of the CMFQ, we first train employing
2006
(a) tor.
Total reward of Preda- (b) Total reward of Prey.
MFQ, CMFQ, and Attention the self-play paradigm in scenario involving 20 predators and 40 prey, and then test them in environments involving (20 predators, 40 prey), (80 predators, 160 prey). (180 predators, 360 prey)
Figure 7: Total reward during training.
respectively. The reward are set as follow: Tattack = -0.2, Tsurround = 1, I'be, surrounded
1
035
0.37
015
(a) MFQ controls prey.
(b) Attention-MPQ controls prey.
(c) CMFQ controls prey.
Figure& 8: Total reward of predators during execution changes when the number of agents increases. 1x denotes Npredator=200 N0r 4x demotes Npreddaoo==80, N1r and so on. All algo- rithms are trained in the 1x environment.
Quantitative Results and Analysis. We compare CMFQ with MFQ and Attention-MFO First, we investigate their training scalability in (20 predators, 40 prey), as shown in Fig 7(a) and Fig 7 stable reward total reward, verifying their training scalability. Then, we
all of them converge enlarge the number of agents during execution to investigate their test scalability. To demonstrate the scalability gap of different algorithms, we allow the algorithms to execute in an adversarial form, which means that one algorithm controls the predator and another controls the prey. For the environment, we change the number agents to Ix, 4x, and 9x of the number in the training environment.
Because the reward Pbessurrounded of prey and the reward Fearround of predator are zero-sum and cooperation exists mainly among predators, we use the total reward of predators to indicate each algorithm's 's performance. The results are shown in Fig[IO] Total rewards in specific environment indicate the train scalability, since higher total reward means agents learn better policy during
8

---

Under review as a conference paper at ICLR 2023
training. Trends of lines related to test scalability, and more flat line indicates the better test scalability of the algorithm. We can see that the total reward of Attention-MFO is higher than that of MFQ, and the trend is similar to that of MFQ. In comparison, the total reward of CMFQ is higher than that of both MFQ and Attention- MFQ, and the trend is ever more flat, indicating that CMFQ has better scalability.
Figure 9: Visualization of cooperative predator prey game. The first row results of CMFQ. the second row is results of Attention-MFQ. 2yre for the left column, for the middle column, 118 for the last column. Red squares are predators while blue squares are prey, the grey squares obstacles. All images ob- tained 400 steps after the game begin.
40..prr
Visualization Analysis. The results that the trained CMFQ and Attention-MFQ controls predators are shown in F In the the environment that N Npredator=222 =20, N 440e both CMFQ and Attention- MFQ perform similarly. Predators learn two strategies: four predators cooperating to surround the prey in an open area; two three predators surrounding the prey with the help of obstacles. In the environment that 400 N8v when number of agents increases, predators con- trolled by Attention-MFO are more dispersed than predators controlled by CMFQ. Besides, MFQ has more predators idle than CMFQ. Predators controlled by CMFQ gather on map edges, because it is more efficient surround prey with the help of map edges. In addition, predators controlled by CMFQ learn an advanced strategy to drive prey to map edges then take advantage surround them. In the environment that N Npredator=1188 presented. Moreover, predators controlled by CMFQ master the skill to utilize the bodies of still teammates who have captured prey obstacles. Thus, predators controlled by CMFQ present high degree of aggregation and environmental adaptability.
the terrain to N =360, the advanced strategy is also
CONCLUSIONS
This paper aims at scalability problem in large-scale MAS. Firstly, We inherit the framework of MFRL which significantly reduce the dimensionality of joint state-action space. To further handle the intractable non-stationarity when the number of agent is large, propose SCM model the decision-making process, and enable agents to identify the more crucial interactions via intervening on the SCM. Finally a causality- obtained by the weighted sum of the action of each agent according to its causal effect. Experiments two tasks reveal the excellent scalability of CMFQ.
representation of population behavioral information could be
REFERENCES
Elias Bareinboim, Juan Correa, Duligur lbeling, and Thomas Ieard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea

---

Under review as a conference paper at ICLR 2023
Pearl, pp. 507-556. 2022.
Lawrence E. Blume. The statistical mechanics of strategic interaction. Games and economic behav- ior, 5(3):387-44))) 424, 1993.
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33:22069-22079, 2020.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey. Artificial Intelligence Review, April 2021. ISSN 1573-7462. doi: 10.1007/s10462--21--99999
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. Advances in Neural Information Processing Systems, 32, 2019.
Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning, In Interna- tional conference on machine learning, pp. 2961-2970. PMLR, 2019.
Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang. Yi Wu, and Xiaolong Wang. Evolu- tionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003. /0423, 2020.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettiie environments. Advances in neural informa- fion processing systems, 30, 2017.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 770300490, 2017.
Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P How. Learning teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 6128 6136, 2019.
Judea Pearl. Direct and indirect effects. Probabilistic and Causal Inference The Works of Judea Pearl, pp. 373, 2001.
Judea Pearl. Causal inference. In Isabelle Guyon, Dominik Janzing, and Bernhard Schölkopf (eds.), Proceedings of Workshop on Causality: Objectives and Assessment at NIPS 2008, volume 6 of Proceedings of Machine Learning Research, pp. 39 58, Whistler, Canada, 12 Dec 2010. PMLR. URL
mir www.srrssspee..... wtt.:e.peeess/eeeeaeeeeec html
Judea Pearl. The seven tools of causal inference, with reflections on machine learning. Communi-
cations of the ACM, 62(3):54-60, 2019.
Sarah Perrin, Mathieu Laurière, Julien Pérolat, Matthieu Geist, Romuald Élie, and Olivier Pietquin. Mean field games flock! the reinforcement learning way. In Proceedings of the Thirti- eth International Joint Conference on Artificial Intelligence, California, August 2021. Interna- tional Joint Conferences Artificial Intelligence Organization. ISBN 9780999241196 doi: 10.249633/jcai.222./50
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International conference on machine learning, pp. 4295-4304. PMLR, 2018.
Steven Sloman and David Lagnado. Causality in thought. Annual review of psychology, 66: 223-247, 2015.
10

---

Under review as a conference paper at ICLR 2023
H Eugene Stanley. Phase transitions and critical phenomena, volume 7. Clarendon Press, Oxford, 1971.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint
2017.
Katia P Sycara. Multiagent systems. Al magazine, 19(2):79-79, 1998.
Ardi Tampuu, Tambet Matüisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))e0172395, 2017.
Dimo Uzunov. Introduction the theory of critical phenomena: mean field, fluctuations and renormalization. World Scientific, 1993.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-334 2019,
Baolai Wang, Shengang Li, Xianzhong Gao, and Tao Xie. Weighted mean field reinforcement learning for large-scale uav swarm confrontation. Applied Intelligence, pp. 1-16, 2022.
Bin Wu. Hierarchical macro strategy model for moba game ai. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 1206-1213, 2019.
Tingyu Wu, Wenhao Li, Bo Jin, Wei Zhang, and Xiangfeng Wang. Weighted mean-field multi- agent reinforcement learning via reward attribution decomposition. In International Conference on Database Systems for Advanced Applications, pp. 301-316. Springer, 2022.
Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823-10836, 2021.
Yaodong Yang. Rui Luo, Minne Li. Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. International conference on machine learning, pp. 5571-5580. PMLR, 2018.
Matej Zežević, Devendra Singh Dhami, Petar Veličković, and Kristian Kersting, Relating graph neural networks to structural causal models. arXiv preprint 1rXiv:2109.04173 2021.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-Agent reinforcement learning: A selective overview of theories and algorithms. November 2019.
Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
The pseudocode of CMFQ is listed below.
Algorithm Causal Mean Field Q-learning
Input: Initialize state So: Qe, Q. 0 for all agent i E {1,2, ... N trajectory length M: while in the training loop do
for f = 0, 1....... .M do for i
... .N do Calculate policy n.( St, - with average merged agent; Calculate causal effect for every neighborhood agent by Eq Obtain new merged agent ū;-1 and new policy T&(- st, a;-1) by Eq. 10: end for
Sample joint action a = [a a² ,a I from [T]] o x obtain the next state St+1 and the reward (r', .... Mo lo 1.. 1 1]:
and merged agent
Store transition < 81: a, 81+1, replay buffer;
end for for i = 1,2,....,, do
Sample minibatch transition from replay buffer; Calculate L; and update 0g by Eq. 12):
Updata target network by 0,- 6, after every C' updates of 0;; end for end while
B DERIVATION FOR THE BOUND OF THE LAGRANGE REMAINDER
As 8, a' in ('(s, a², ahl)) are fixed parameter in the derivation of Fq. for simplicity, the pairwise Q-function Q' a², azik) can be rewrite as (a* in the following. We assume that a² a one-hot encoding for actions, to make Q(a²) more general, we replace the discrete a (at R²) by continuous x (Ir E IRV) which don't violate the domain of the parameterized Q-function. Given the
Q (x) is L-soohth then for any points x, y dom (Q) C R. there exists Lipschitz constant E [0, +00) that
(Q(()) VQ( < L|I vll2
(13)
By the first order Taylor expansion with Lagrange remainder, we have
VQ (v) = VQ (x) + V (x) u u- (u))
(14)
where u = y x, lim
R(w) u
0. Assume + y, then we can reform the first order Taylor
expansion
Q²)(2) ull2
(VQ(() vQ(x) R(u)l2
|u/|2
|ull2
(QQ(2) VQ(x)lla [R(l)
(15)
|
||l||
[[(u)]l2
<
L +
| Vz.y d
u could be the eigenvalue of V2Q (z), then Eq. 15 can convert to
[V2(()) ull2
Duul2
R(u)ll
=
X
++
(16)
|u||2
||u||2
1u
12

---

Under review as a conference paper at ICLR 2023
Obviously, obtain the bound of X, A [-L, (=) is real symmetric matrix, so there exist an orthogonal matrix U diagonalize V2Q (a)) such that U² [ViQ (x)) U diag [21, .aa...,AN:: Then the bound of R aaaa((a-A) can be derived as follow
A -
RC (a<hA) =
1 -.attk²20 -2Q (at)
Sath
1 U batt""A UU- saat]
Σ n=1
An
U 8a ta
2
(17)
LII 8a 00a**112 S
Σ An U dati1 LIU 80".*112
(18)
where [ . 5a**], refers to the th element of vector U Sati
IIU da 80"0"112 ||6a** (atth a²) ('a" a') a t ath a T ath a'a =2(1 - -((1-a)) 2
(19)
where a a is one-hot encoding action, at denotes the nthh element in ä'. Finally, according to Eq. 17 Eq. 18) Eq.19, 19). the bound of R (a.ahaa is L-, L].
C
VISUALIZATION FOR THE WEIGHTS OF CMFQ AND ATTENTION-MFQ
To further analyze the reasons why CMFQ is more effective than Attention-MFQ empirically, we randomly select an agent in the mixed cooperative-compelie
game task and visualize its weight. Some interesting observations can be made from Fig To(a) First of all, it makes sense that the agents on the front line will be given high weights because they are battling. Secondly, the weights of agents at the edge of the front line are relatively small, possibly because these agents can cooperate with nearby teammates to attack an enemy due to their position advantages, so they are relatively dominant state. In addition, agents at the very edge of the front line are given higher weights, even they are out of combat. This is because they are a position to flank their opponents and work with their teammates to surround the opponents. In Fig [0(b] observe a result consistent with the analysis in our paper. That is, the attention- based method uses the attributes of other agents to calculate the attention scores, and observation is important part of the attributes, so it tends to give high weight to the agents nearby because their observations are similar.
35
25-
:00::
15
12-
(a) The weights obtained by CMFQ.
(b) The weights obtained by Attention-MFO.
Figure 10: The two figures visualize the mixed cooperative-compettiie task, where each agent in the blue team in (a) is controlled by CMFQ and each agent in the blue team in (b) is controlled by Attention-MFQ. Each agent in the red team is controlled by MFQ. We label the agents in the blue team whose weights are visualized in green. The number above the blue agent represents the normalized weight given by the green agent to the pairwise interaction between them. Due to space constraints, the integer bits of all weights are omitted
13

---

Under review as a conference paper at ICLR 2023
D SUPPLEMENTAL EXPERIMENT ON MPE
To further investigate the applicability of CMFQ. we perform an experiment on another environment named multi-agent particle environment (MPE)(Moratee)
Abbeel, 2017). As the dimentionality of action-state space will change as the initial number of agent changes, making it difficult to verify scalability, but we believe that CMFQ's scalability performance has been adequately validated in previous experiments. For MPE, we tested the predator prey task in MPE when the number of agents was the same that in the training environment, and compared it with 5.2 see whether the same conclusions could be drawn in the two environments.
Task Setting, There are 20 predators, 40 preys, and 20 obstacles. Predator gets Tooftide 10 if it collide with prey. Prey gets lbe.collided -10 it collided with predator. The speed of prey is 1.3 times of that of predator. In order to make preys learn to leverage obstacles in- stead of running infinity, We manually draws an area. If preys go beyond this area, they will get penalty Tbound which will be aggravate as the distance preys go beyond this area increase, until Pbound 10. We trained MFQ. CMFQ and Attention-MFQ in the self-play paradigm. The training curve is shown in Fig 11
(a) Average reward of (b) Average reward of Predator.
Prey.
Figure 11: Average reward during training.
Quantitative Results and AnalysisIn the test phase, controlled 20 Predators and 40 prey with different algorithms respectively, test 10 times and calculated the average reward of each algorithm, shown in Table T First, the average reward of MFQ is lower than CMFQ and Attention-MFO regardless of whether it controls predators or preys. This indicates that the representational ability of average merged agent is insufficient. Secondly, when MFQ controls prey, the average predator reward of CMFQ is higher than Attention-MFQ. indicating that the weight obtained by CMFQ was more representational. Finally, in the comparison between CMFQ and Attention-MFO. CMFQ out- performs Attention-MFF MFQ in both predator reward and prey reward, further confirms the superiority of CMFQ. In the task that the number of agents testing was the same as that in the training, We compare the performance of MFQ. CMFQ, and Attention-MFO and come to the same conclusion consistent with 5.2 empirically certify the applicability of CMFQ.
Predator
Predator reward
Prey
Prey reward
MFQ
4.23
CMFQ
-8.64
CMFQ
6.68
MFQ
-13.05
MFQ
4.01
Attention-MFF
-8.47
Attention-MFQQ
6.05
MFQ
-12.46
CMFQ
3.15
Attention-MFQ
-11.07
Attention-MFO
3.02
CMFQ
-4.23
Table 1: Results that let two different algorithm control predators and preys respectively. Predator reward is the average reward a predator obtain every step. Prey reward is the average reward a prey obtain every step.
14


Reviews:
[{"review_id": "yTsI_EjLPm", "reviewer": "Reviewer_bK4T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes causal mean-field Q-learning (CMFQ) to address the scalability problems in non-stationary environments. Results show better scalability in mixed cooperative-competitive game and a cooperative game. ", "summary_of_the_review": "The paper builds on mean field q-learning hence the contribution may be limited. More experimental scenarios can be considered to showcase the superiority of the proposed approach. ", "strength_and_weaknesses": "strengths:\n-the paper address the limitation of ignoring pairwise interactions in mean field Q-learning using by using a weighted combination through causal inference\n- factorizing joint-q function into local pairwise Q-functions is the basis to address scalability issues \n\nweakness:\n-more experimental scenarios need to be evaluated \n-the figures are hard to read and need to be explained more clearly especially 5a\n-how the proposed method improves scalability needs to be demonstrated in the experimental results when compared to the other approaches\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g8gmt9ViIim", "reviewer": "Reviewer_3dJB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the scalability problem in MARL, scaling to environments containing more agents during testing, and proposes a causal mean-field Q-learning (CMFQ) algorithm. This paper solves the non-stationarity challenge by focusing on crucial pairwise interactions identified by Causal inference. And Mean Field Theory is beneficial to scalability since it turns a many-agent problem into a two-agent problem. They evaluate CMFQ in the cooperative predator-prey game and mixed cooperative-competitive battle game. The results illustrate that the scalability of CMFQ significantly outperforms all the baselines. Furthermore, results show that agents controlled by CMFQ emerge with more advanced collective intelligence than those controlled by other baselines.\n", "summary_of_the_review": "This work proposes a new causal mean-field Q-learning algorithm, it is original to apply causal inference in mean-field Q-learning. This paper addresses the scalability problem,  experiments results are sufficient to prove that it can scale to environments containing more agents. The writing of this paper is easy to follow, and the structure is clear.\n", "strength_and_weaknesses": "Strength\n\n1. Adequate experiments show better performance on reward and scalability than other baselines and surprising visualization analysis shows the CMFQ has better ability to cooperate.\n\n2. This work employs the causal model to infer the weight of pairwise Q functions replacing previous average-based mean-field and attention mean-field methods. I think this approach is original.\n \nWeaknesses:\n1. In the introduction section, the potential shortcoming of previous mean-field Q-learning (MFQ) is discussed, however, the discussion about attention-MFQ, e.g. the limitation of attention-MFQ seems missing.\n2. equation (9) seems to need more explanation, and what the $Q^j$ and $Q^i$ indicate respectively?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_mfk7JMUsyz", "reviewer": "Reviewer_v11b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Multi-agent interactions result in non-stationary environment for reinforcement learning algorithms, which harms the performance of mean field method, which reduces high number agent's problem to pairwise interactions approximations. \n\nTo make the algorithm more robust, the authors propose learning counterfactual effects to generalized representation of merged agents. \n\n\n\nMean field method reduces the heterogeneous agents interactions to be centered to a  common agent interaction with its merged neighbors. Concretely, the value-action function involves the state of the agent in question and all its neighbors' interaction, which is decomposed to be the sum of pair wise value-action and approximated to be interaction with merged agent state-value function. \n\n\nTo quantify the importance of one specific neighbor to the current agent in question, the author proposes to quantify using the counterfactual policy distance by replacing the merged agent with that specific neighbor. \nThis importance weight is then used to average the contributions of the pairwise state-value function to the state value function with respect to all its neighbors. \n\nThe author further use Taylor expansion and assume state-value function is L-smooth thus the Lagrange remainder could be eliminated. The policy is conditioned on a weighted sum of previous step actions. \n\n\nTo evaluate the performance empirically,   IQL, MFQ(mean field) and attention MFQ is used as baselines.  Test environment has more agents than training environments. ", "summary_of_the_review": "I think the paper addresses an important issue in multi-agent reinforcement learning, where the number of agents are big and existing mean field method tend to ignore the different contributions of neighbors to the fate of the agent in question. The proposed method is well explained, though lack more in depth analysis. ", "strength_and_weaknesses": "Strength:\n\n1. I think the proposed method is quite novel and addresses an important issue\n2. Empirical studies are good. \n\n\n\nWeakness:\n1. I think Figure 2need more explanation, I could understand the mechanism intuitively, but are there some more insightful explanation? Fig 2(b) is essentially comparing the KL divergence between two policies by using a concrete agent or a merged agent, the result in used in Fig 2(a) to generate the weight, why would this be a good mechanism? \n2. Could you add more explanation to the L-smooth justification of the Q function with respect to action? I thought in your game, action is discrete? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject