Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning
Authors: Wei Duan
Jie Lu
Junyu Xuan
Venue: NeurIPS 2025 poster ()
Keywords: networked multi-agent reinforcement learning
Multi-Agent Reinforcement Learning
Decentralized Reinforcement Learning
Actor–Critic Methods
Primary area: reinforcement_learning
TL;DR: We propose BayesG, a decentralized actor–critic framework that jointly learns context-aware interaction graphs and policies via variational Bayesian inference over ego-graphs in large-scale networked MARL.

Abstract: In networked multi-agent reinforcement learning (Networked-MARL), decentralized agents must act autonomously under local observability and constrained communication over fixed physical graphs. Existing methods often assume static neighborhoods, limiting adaptability to dynamic or heterogeneous environments. While centralized frameworks can learn dynamic graphs, their reliance on global state access and centralized infrastructure is impractical in real-world decentralized systems. We propose a stochastic graph-based policy for Networked-MARL, where each agent conditions its decision on a sampled subgraph over its local physical neighborhood. Building on this formulation, we introduce \textbf{BayesG}, a decentralized actor–critic framework that learns sparse, context-aware interaction structures via Bayesian variational inference. Each agent operates over an ego-graph and samples a latent communication mask to guide message passing and policy computation. The variational distribution is trained end-to-end alongside the policy using an evidence lower bound (ELBO) objective, enabling agents to jointly learn both interaction topology and decision-making strategies.
BayesG outperforms strong MARL baselines on large-scale traffic control tasks with up to 167 agents, demonstrating superior scalability, efficiency, and performance.

Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning
Wei Duan
Jie Lu Australian Artificial Intelligence Institute University of Technology Sydney Sydney, Australia jie lufuts..uu....m edu.
Australian Artificial Intelligence Institute
University Technology Sydney
Sydney, Australia wei duantttnttt.. uts. edu.
Junyu Xuan
Australian Artificial Intelligence Institute University of Technology Sydney
Sydney, Australia junyu. xuan@uts edu. au
Abstract
In networked multi-agent reinforcement learning (Networked-MARL) decentral- ized agents must act autonomously under local observability and constrained communication over fixed physical graphs. Existing methods often assume static neighborhoods, limiting adaptability to dynamic or heterogeneous environments. While centralized frameworks learn dynamic graphs, their reliance global state access and centralized infrastructure is impractical in real-world decentralized systems. We propose stochastic graph-based policy for Networked-MMRRL. where each agent conditions its decision on a sampled subgraph over its local physical introduce BayesG, decentral-
neighborhood. Building on this formulation, ized actor-critic framework that learns sparse, context-aware interaction structures via Bayesian variational inference. Each agent operates over an ego-graph and samples a latent communication mask to guide message passing and policy com- putation. The variational distribution is trained end-to-end alongside the policy using an evidence lower bound (ELBO) objective. enabling agents to jointly learn both interaction topology and Decision-making strategies. BayesG outperforms strong MARL baselines large-scale traffic control tasks with up to 167 agents, demonstrating superior scalability, efficiency, and performance.
Introduction
Multi-agent reinforcement learning (MARL) has emerged powerful framework for sequential decision-making in distributed systems, enabling applications in autonomous driving [1,2), wircless
communication 3 4] multiplayer games [5 and urban traffic control 7 g popular training paradigm is centralized training with decentralized execution (CTDE), where centralized critic
leverages global state information to train decentralized
12
While CTDE methods demonstrate strong empirical performance in simulation benchmarks [3] 15], they often rely access to global observations and centralized learning infrastructure, shown in Figure aa) This assumption rarely holds in real-world applications, where agents are geographically distributed and subject to local sensing and communication constraints. Instead, many practical domains-from urban mobility to smart grids are better modeled as networked MARI. [1]
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

where agents interact over fixed communication graph and can only observe or exchange information with nearby neighbors.
major challenge in networked MARL is the use of static communication graphs, where agents are hardwired exchange information with all local neighbours regardless of contextual relevance [16] [77. This can lead to inefficient coordination, unnecessary message exchange, or even performance degradation, particularly in dynamic settings like traffic control, where congestion varies over time and not all neighbours are equally informative. This raises a fundamental question:
Can decentralized agents learn to dynamically adapt their interaction structure using local observations and task feedbbcckk
While recent CTDE methods have explored learning dynamic interaction graphs 20 applicability limited in decentralized settings, where agents must reason over local observability and adhere to the physical structure of the environment.
1S
21 22 their
To address this, we propose BayesG, decentralized graph- based criti-critic framework for net- worked MARL that learns latent interaction structures via Bayesian inference. As illustrated in Figure [b), we begin by introducing graph-based policy, where each agent conditions its decisions on a stochastic subgraph sampled from a learned distribution over its physical neighbourhood. Each agent operates over an ego-graph, localized subgraph capturing its immediate neighborhood, en- abling context-aware decisions under topological constraints. We formulate this process as Bayesian variational inference of latent graphs, where each agent i infers a binary mask Ze over its local neighbourhood. The mask is treated as latent variable with posterior p( Z, G D.), conditioned on the physical subgraph CRy and agent-specific data D; (c.g.. neighbour states, trajectories and polices). The posterior approximated by variational distribution (Z, Φ:), optimized end-to-end via the evidence lower bound (ELBO).
BayesG integrates latent graph inference into policy learning, enabling agents to prioritize critical communication links within their local ego-graphs and prune irrelevant ones-all without requiring global supervision. This leads task-adaptive, uncertainty-aware, and communication-effiiie. coor- dination under topological constraints. Experiments on both synthetic and real-world traffic control benchmarks show that BayesG outperforms state-of-the-arl MARL baselines in both performance and interpretability.
Our main contributions are:
We propose a stochastic graph-based policy for networked MARL, where each agent conditions decisions on a sampled subgraph over its physical neighbourhood.
We formulate latent graph learning as Bayesian variational inference, treating edge masks as posterior distributions constrained by the environment topology and agent-local data.
We develop an end-to-end training algorithm that integrates variational graph inference with actor-critic learning via an ELBO objective.
2
Related work
Networked MARL. Networked MARL focuses on decentralized learning over fixed topologies, often without explicitly adapting the communication structure. Consensus-based methods [6 17 synchronize local value functions via neighborhood averaging, typically assuming partial or global ob- servability. Other works incorporate network-aware priors such as distance-based decay 23] or local reward aggregation [19 to promote spatially coherent coordination. Model-based frameworks 24 25 exchange predicted trajectories but still rely static interaction graphs. Communication- methods like NeurComm 77] enable information sharing across neighbors, yet use fixed topologies throughout training. Recent work [26]] in networked systems, but these methods sample from predefined or uniform distributions at the environment agent level. Our method differs fundamentally by introducing Bayesian latent graph inference mechanism that allows each agent to adaptively select task-relevant neighbors from its physical graph, enabling
23 27 has explored sampling strategies to improve scalability
caamuunnicaiioeffffcieeeee locally grounded coordination.
Graph- MARL and Latent Structure Learning. Recent advances in cooperative MARL. leverage latent interaction graphs to enhance coordination and scalability. Methods such as DGN 28

---

De (ey, MAG.) G CV. O Z
IA
Sample
(tes Cip kid
KL/p Cy. DJ dazza 6.))
Gerao
Dj (sy ux. Gj== GV 2g
AA
Rerthe Cuy lus)
Sample
Global Sate
goooph Criticritic
KLíp(Z, GV, DJ zZ
Caras,
(a)
(b)
Figure 1: (a) In CTDE, the global state is available for learning both the centralized critic and the interaction graph. (b) Overview of BayesG. In networked MARL, each agent's state and action are influenced by its neighbors, forming local data D; {sv; Wi UN, We formulate latent graph learning as Bayesian variational inference, where each agent infers binary mask Z; over its neighborhood from the environment graph G and local data D;. The posterior p(Z; G D:) is approximated by a variational distribution q ZZ @.), from which a sparse subgraph is sampled and used for graph-condttioned policy learning via an ELBO objective.
DICG [29], and HGAP 30) employ attention mechanisms graph neural networks (GNNs) B1 32
to infer inter-agent dependencies and guide information exchange. Variational approaches 20 22
further treat interaction graphs latent variables, jointly optimizing communication structure and policy learning. However, most existing methods, including Dec-POMDP-basee coordination graph 14 21], assume centralized training with access to global state and reward signals, and allow unrestricted graph rewiring, which are impractical in real-world decentralized settings. In contrast, we address the more realistic networked MARL scenario by learning dynamic latent interaction masks on the environment graph using only local observations.
approaches [3 29
34
Preliminaries
We consider networked MARL, over fixed interaction graph G = (V,E), where each agent i E V interacts locally with neighbors N₁ = {j (i.j) E E} and defines closed neighborhood V := Nr U {i). At each step, agent receives a local observation 8, f(vv). which may include its own state and aggregated information from neighbors, and selects an action u, E Ut according to local policy mM(M, 5t). The encoder f(-) captures neighhourhood-levee context using neural architectures (e.g., MLPs (7): specific choices are detailed Section 1.2 We begin with the spatiotemporal MDP formulation, followed by the decentralized actor- (A2C) training objective.
Definition 1 (Spatiotemporal MDP(7). A spatiotemporal MDP is defined as the tuple (8, (4') Diev, R, y, where S is the global state space, Ut the action space of agent i, and Ç the initial state distribution. The local transition model of agent i is:
(
p((s sV,, sV,, u/) = =Σ
III
Tu [(ov.)) p(s) sv,, ui uN.): JEN,
(I)
UN;
where the transition depends on local neighborhood states and actions. The local reward is R(sv, (v,) E R. and the objective is to maximize expected return: E, 1Et r R((v,, uyy,)),, with discount factor 7 E (0, 1].
Definition 2 (Networked MARL with decentralized A2C Let frohivv and {Vu, hev be decentralized actor and critic networks, respectively. Given an on-policy minibatch B ( Ui Ti,,, uN..)): the losses are:
C(0,)
1
Σ
log Te, (wir 1i. A + Σ
To, (w4 3i.r log Te, (wi Bl,r)
(2)
TEB
ElL
[(wi) =
PM G.r
Vo, uN.x))
(3)
|B
TEB
3

---

where Bi, = f(vv.+) is the encoded local observation. The AT, R -ir is advantage estimate, where the reward is RT,,- = EK= 7ht Ejev, a'urj + y Uir and K denotes the rollout horizon. The Vi,r (N,,) is the target critic output. The a E (0, 1] adjusts influence from distant neighbors, and B3 controls the entropy regularization.
Method
We propose BayesG, a decentralized actor-critic framework for networked MARL that learns sparse, context-aware interaction graphs via variational inference. We begin by formulating graph-based policy, where each agent conditions its decision on a sampled subgraph over its physical neighborhood. This formulation enables agents to adaptively modulate their coordination structure based on local observations while respecting communication constraints. We then extend this framework to optimize both the policy and graph sampling distribution jointly using a variational learning objective.
4.1 Graph-based Policy with Latent Interaction Structures
In conventional A2C for networked MARL (Definition 2). agents condition their policy on local observations ši,r (sy,,r), which implicitly encode neighbor influence but treat all neighbors as equally informative. This uniform treatment limits the agent's ability to adapt to dynamic envi- ronments or prioritize critical interactions. To overcome this, we introduce graph-based policy in which each agent samples subgraph from learned distribution and conditions its decision on the sampled structure. (See Appendix Oopendix[A.]foo for details).
Definition (Graph-based based Policy). The policy of agent i is defined as:
Te(Us, Gv, Syy; 8,, (P:) = p(Gv, I sv.; R,(us) fi(sv,, Gv.);
(4)
where Gy, E {0, 1M is a sampled binary adjucency matrix drawn from distribution pl: Pp) over agent i's closed neighborhood. The graph is restricted to the environment topology: Gv. E Gon == [G E {0, 131V.xID. | G 1 G.r)), }. ensuring only physically permitted connections. The function [((.) is a graph-conditioned encoder, and R.(. 110;; 0.) is the action- policy.
This formulation enables agents to stochastically select contextually relevant neighbors, driven by local observations. By formulating coordination approach provides a general framework for inferring context-dependent interaction struc- tures-alee allowwng agents to explore diverse coordination patterns and adapt to non-stationary dynamics. In traffic signal control, this allows dynamically emphasizing congested junctions while suppressing irrelevant neighbors.
distribution p(-) over subgraphs, this
We now extend the A2C training objective to accommodate graph-conditioned policies. This graph- based loss serves the actor component of our training framework and jointly optimizes the policy and interaction structure. (See Appendix [A.2 for details). Definition 4 (Graph-based A2C Objective).
EGV [- log ( fe((v,,Gvi)) Gvi)) Ar + H(m. 11a)).
Lo =
(5)
B
TEB
where the entropy term is defined as H(tt() 1 fi)) = = - R.(u; | fi) log (,s I fi), and B denotes an on-policy trajectory batch, and A the estimated advantage. Gradients are backpropagated not only through the policy ã, but also through the sampling distribution p(Gv, sv.. enabling joint optimization of action selection and interaction structure.
4.2 BayesG: Variational Latent Ego-graph Inference for Policy Optimisation
Recall that the graph distribution p(Gv, sy,). introduced in Definition 2 governs which neighbors agent i attends to. To infer this stochastic distribution over the fixed physical topology, we model via a binary mask Z, E (0, 1 | applied over the physical neighborhood graph G where each entry zij = 1 indicates that agent communicates with neighbor j. This yields the effective subgraph:
G = Zi 2 G.Y
(6)

---

We formalise the learning of the latent edge mask Z; as a Bayesian inference problem. This is natural choice Bayes' theorem offers principled way to infer latent distribution conditioned on the directly observable physical graph G.,, and agent-local data D, Treating edge masks as latent variables enables agents to capture uncertainty over coordination structures- paaieessaaatllllrlly valuable when noisy or non-stationary dynamics make deterministic graph pruning unreliable.
Let Dr denote agent-local data collected during training, such as neighbor states, policy outputs, and trajectory embeddings. The posterior distribution over edge masks is given by Bayes' theorem:
p Ze Gy, D:) x pD;; Zi GV.) p ((.)),
where p(D; Z, G.,) is the likelihood, measuring how well the masked graph explains observed behavior, and p(Z)) is prior (e.g., Bernoulli with bias).
To enable tractable optimization, we approximate the posterior with a variational distribution:
(Zz;0,)) = II B
JEA
where 6; are Iearnable logits and (-) is the sigmoid function. We apply the Gumbel-Softmax reparameterization 33| 36 to allow gradient-based learning through discrete edge sampling.
The variational parameters o, are learned by minimizing the Kullback-Leibler (KL) divergence between the approximate posterior (Z4 05) and the true Bayesian posterior p( G Dt):
[Liq(Z @.)lp((Z GV, D:)] (Z,) [log ((z) log p(
GV, D.)]
This approximation enables each agent to learn a structured distribution over interaction graphs that reflects the most informative subgraphs for policy optimization. Following the standard variational inference framework, minimizing this KL divergence equivalent (up to an additive constant) to maximizing the evidence lower bound (ELBO), or equivalently, minimizing the negative ELBO:
LELBO = E.((2;pb)) log Di Z,, G., ) + log 2(2.) log (Z,;; Φ.) + const
(10)
= E.(2..d.) L + log p Z.) - log q (Z;; ox)] + const.
where the likelihood term logp( (D; Zi, G. is instantiated by the graph-conditioned actor loss Ca.e defined in Definition 4 which provides task feedback to guide the learning of latent structures. (See Appendix A.3 for detarls.)
Connection to maximum entropy RL. Our formulation parallels the probabilistic interpretation of RL in Soft Actor-Critic (SAC) (37] which casts policy optimization as untroy--reullarin inference. In SAC, policy entropy H(((-+s)) regularizes action selection; in our framework, the term - log q( Z5; Φb.) Equation (10) acts as mask entropy H(q(Zz Z; ot)), regularizing the variational distribution over communication structures. This prevents premature collapse to deterministic graphs and enables incertaity--aaan neighbor selection. The dual entropy regularization over both actions (embedded in La,p) and edges (the log q term)-is key to our method's 's robustness in dynamic environments where action diversity and adaptive communication are both critical.
Expanding the ELBO in Equation (10). we obtain:
Definition 5 (BayesG-ELa ELBO Objective). The BayesG objective integrates policy learning and graph inference in unified variational framework. For agent the ELBO is:
LELBO Egg(2;:0)
- - Σ JEA:
a(0;)) log -2--
(1)
where La.s denotes the graph-conditioned policy loss defined Definition and ( is the Bernoulli parameter for each candidate edge. This objective jointly optimizes the policy and the structure of the latent subgraph via variational inference.
We complement the actor-based ELBO with a standard A2C critic loss:
LCiitic
1
Σ
(Ba, (5,, R
2
(12)
B
TEB
5

---

where 8t, is the graph-conditioned input for the critic, and R,,-- is the estimated return. The final training objective combines both components across agents: Luxal = Eigy (-LEO + Lenie) This formulation enables decentralized agents to learn both communication structure and policy behavior from local interactions, while adhering to the physical structure of the environment. The ego-graph terminology reflects that each agent independently infers its own local communication neighborhood. Full derivations are provided in Appendix A.3
5
Experiments
5.1 Experimental Setup
We evaluate BayesG on five benchmark scenarios for adaptive traffic signal control (ATSC), imple- mented using the SUMO microscopic traffic simulator [38]. Each environment simulates peak-hour traffic, with one MDP step corresponding via induction-loop detectors (ILDs), including vehicle density, queue lengths, and waiting times on incoming lanes, and control local traffic signals. The reward is the negative number of halted vehicles, normalized by a fixed scale. These environments naturally satisfy Definition [T: state transitions depend on local and neighbor actions, rewards are localized per agent, and neighborhood structure is fixed by the road network. Detailed MDP mappings are in Appendix
fixed control interval. Agents observe traffic conditions
ATSC_ Grid and Monaco. These two medium-scale scenarios are widely used in prior ATSC benchmarks (e.g., NeurComm (7]. ATSC_ (rrdrrriii a synthetic x network with regular connectivity, while Monaco replicates real-world 28-intersection layout. Both use a 5-second control interval and run for 720 steps per episode, totaling 3600 seconds simulated time. States include lane-level traffic conditions and neighbor observations. Actions correspond to local phase switches, and 2-second yellow phase enforced for safety.
New York. To assess scalability, we introduce three large-scale scenariss-NewYrrkee NewYork33, NewYork51, and wwwfoork67--oopiise 33, 51, and 167 signalized intersections, respectively. These networks are derived from real-world Manhattan layouts (see Appendix C for more details). We use a 20- second control interval (increased to 40 seconds for NewYork167) and simulate 500 steps per episode, Intersections feature heterogeneous phase designs and ILDs configurations. States include normalized lane-level metrics and neighborhood-awwad features. These environments pose significant challenges for coordination and generalization in decentralized MARL.
5.1.1 Baselines
We compare BayesG against six representative multi-agent actor-criti baselines, all implemented using a unified A2C backbone with consistent neighbor access and decentralized execution:
IA2C 13
Independent actor-critic training where each agent optimizes its policy based local observations and a critic that observes neighboring actions. No inter-agent communication is used.
ConseNet [[6]
A consensus-based method where critics synchronize neighbours' parameters via local averaging to promote stability and coordination, without exchanging message content.
FPrint 39
Mitigates non-stationarity by attaching policy fingerprints to local transitions, enabling agents to condition their updates on the behavior of neighbors over time.
LToS 19 Introduces a hierarchical reward- wwward-sharing scheme where agents learn to shape local rewards via neighborhood-level value estimates. No explicit communication involved.
CommNet 40 A communication-hasse architecture where each agent receives the average of all encoded neighbor messages, limiting expressiveness but allowing simple message integration.
NeurComm 7i: A more flexible communication protocol that encodes and aggregates messages from neighbors without averaging, supporting richer interaction modeling. It generalizes earlier approaches such as CommNet and DIAL
We categorize IA2C, ConseNet, FPrint, and LToS as non-communicative baselines, as they do not involve direct message exchange. In contrast, CommNet, NeurComm, and our method BayesG communication-bate
Code and data available at httppp://gghtuu..o//////////////

---

IA2C
Consensus
FingerPrint
Ulo
CommNet
NeurComm
BayesG
ATSC Grid
Monaco
4400
-300
350
Reward
-400
-600
-55
-700
NewYork33
New York51
NewYork167
-600
-800
Reward
Reward
-900
880
950
-550
9300
1100
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Step (k)
Step (k)
Step (k)
Figure 2: Training reward curves of BayesG and baselines across five ATSC environments. BayesG consistently achieves higher returns and faster convergence, particularly in large-scale settings, demonstrating the benefit of learning task-adaptive communication graphs.
5.1.2 Implementation Details
For all environments, we use fixed control interval of 5 seconds. Policy and critic networks share similar architectures across all baselines. Each experiment is averaged over 5 random seeds.
To highlight the differences of communication--asce methods, we describe how each method con- structs the agent-specific input representation 3; which is input to the policy and critic networks. All methods encode combinations of the following components: (1) State features (5;, 8AG, ): cap- turing local and neighboring state; (2) Policy features (TA;): ): representing the action distributions (fingerprints) of neighboring agents. (3) Trajectory features( h represented as the hidden states of RNNs that reflect recent behavior. Each channel is encoded via multilayer perceptron (MLP), unless otherwise noted.
CommNet. Each agent encodes its own and neighboring observations with an MLP, and aggregates neighbors' trajectory features via mean pooling followed by another MIP. The final input the sum of these two components: 3; = .LPP state ([s,, sx.]) + MLP MLPtraj (mean((())))))
NeurComm. NeurComm encodes each of the three information types separately and concate- nates their embeddings before passing the result into an LSTM for temporal reasoning: 8; = MLP staPe ([s sAx.)) MLP potacy (Txx) MLP wy (hwr).
BayesG (Ours). BayesG replaces fixed communication with learned latent subgraph. Let A, {0, 1 | denote the local physical graph and Z, {0, || sampled binary mask; the effective graph is defined as: A; = Z, 3 A, The input is then computed via masked message passing over A using GNNs for each input channel:
6; GN AT) GNNpolicy (Il, A&) GNN At).
(13)
Here, Sy. Hly, and Hy, represent the observation, policy, and trajectory features of agent i's neighborhood. While any GNN can be employed, we use graph convolutional networks (GCNs) [22 for efficiency in our implementation. This formulation allows each agent to learn sparse and context-aware communication structures tailored to local dynamics. (See pseudocode in Appendix D)
5.2 Performance Comparison
Training Performance. Figure presents five ATSC environments. Each curve shows the smoothed average episode return (defined as the
presents the training curves of BayesG and baseline methods across
7

---

|A2C
CommNet
NeurComm
BayesG
Figure 3: Visualization of traffic congestion on the Grid map at 3500 simulation seconds. Road thickness and color represent vehicle density (thicker and redder indicates more congestion). BayesG results in significantly lighter and sparser traffic Additiona qualitative comparisons across different simulation times provided in Figure 10 (Appendix P)
negative of total queue length) over random seeds. Several key trends emerge: (1) Consistent supe- riority of BayesG: Across all environments, BayesG steadily outperforms both conmmmmuiiiaaivee baselines (IA2C, ConseNet, FPrint, LToS) and explicit communication protocols (CommNet, Neur- Comm). This highlights the advantage of learning task-adaptive latent subgraphs that promote efficient and stable coordination. (2) Faster convergence: BayesG exhibits faster convergence in early training, particularly on ATSC_ Grid and Monaco. This indicates that selective interaction via learned masks accelerates policy learning by reducing noisy or redundant communication. (3) Scala- bility in large networks; On the large-scale NewYork33, NewYork51, and NewYork167 scenarios, BayesG achieves higher asymptotic returns and more stable learning curves, while baselines often suffer from plateauing instability. This validates BayesG's ability scale to high-dimensional decentralized settings by suppressing uninformative edges. These results collectively demonstrate that BayesG not only enhances learning efficiency but also improves final policy quality by discovering sparse, context-aware coordination patterns.
To further understand the optimization dynamics of BayesG, we report the evolution of key training loss components Appendix for ATSC_Grid, Monaco, and NewYork33. Specifically, we track the policy loss, value (critic) loss, ELBO loss, prior loss, mask entropy regularization, and the total objective. These loss curves offer deeper insight into how BayesG balances policy optimization with variational graph inference, and demonstrate the stability of the joint training process across different network scales.
Qualitative Comparison on Grid Environment. Figure 3 presents visual comparison of traf- fic congestion for IA2C, CommNet, NeurComm, and BayesG on the TTSC_Grid map at 3,500 simulation seconds. Road segment thickness and color indicate vehicle density, with thicker and redder lines denoting heavier congestion. IA2C and CommNet exhibit severe congestion across many intersections, while NeurComm reduces some bottlenecks through explicit communication. BayesG, in contrast, achieves the smoothest traffic flow, with thinner and less congested segments. These results highlight BayesG's ability to adaptively focus critical interactions via its learned latent graph, enabling more effective and scalable coordination dense traffic scenarios. Additional qualitative comparisons across different simulation times provided in Appendix F
5.3 Case Study: Interpreting Learned Interaction Structures
To further understand the behavior BayesG, we conduct a case study on the ATSC_Grid map at time step 1,400. Figure 4 illlustrate:: (a) Left: The latent interaction matrix, where each row corresponds to an agent's inferred ego-graph-icce i.... a probabilistic mask over its physical neighborhood. The aggregated matrix represents the likelihood of interaction between all agent pairs.(a) Right: The physical road network layout with edges colored and weighted according to the same communication probabilities, providing interpretable spatial view. (b): The vehicle density snapshot from SUMO simulation at the same time step. Thicker and redder road segments indicate higher congestion levels.
This visualization shows that BayesG learns to prioritize communication from low-congestion intersections toward more congested regions. For example, when congestion builds up in a particular area (e.g.. tip-right and bottom right clusters), left-stream intersections increase their coordination

---

Network IEtpe withh catos probaiclity!
Whore Groph Adjacercy Motik
Figure 4: Case study on the ATSC_ Grid map time step 1400. (a) The latent interaction graph inferred by BayesG. Each agent samples probabilistic binary mask over its ego-graph and the global latent graph is formed by aggregating these per-agent ego-graph masks. Edge thickness reflects the inferred likelihood of communication between intersections, (b) The vehicle density snapshot from the SUMO simulation. Thicker red/orange lines denote higher vehicle accumulation on road segments, indicating local congestion.
Monaco
NewYork51
Monaco
NewYoris1
300
-00
300
-350
350
-400
Dewardd
Reward
-800
-900
-500
-550
-550
200
400
650
800
2002
100
200 BayesG
300
400
500
250
400
800
500
1200
100
200
300
400
Stoo
Notlask
RandWark
Pollcy
Trojectory
BayesG
Figure 5: Ablation study on the Monaco and NewYork51 environments. Left: Performance comparison of different graph masking strategies. Right: Effect of different feature types used to generate the variational mask.
weights, effectively regulating inflow and providing more green phases to help alleviate downstream pressure. Unlike approaches that treat all neighbors equally, BayesG captures directional, task- driven cooperation: encourages agents outside a congested zone to adjust proactively, thereby relieving bottlenecks before they worsen. This reflects real-world traffic control principles, where strategic upstream coordination plays a critical role minimizing congestion spread. These results demonstrate BayesG's ability to infer decentralized, context-aware communication structures that adapt dynamically to evolving traffic conditions. (See more cases Appendix G
5.4
Ablation Studies
To better understand the impact of BayesG's components, we conduct two sets ablation studies on the Monaco and NewYork51 environments, focusing (i) how the graph mask is generated, and (ii) what input features are used learn the mask. Results are summarized in Figure S
Graph Masking Strategies. We compare three strategies for subgraph construction: (1) No Masking:
Uses the raw environment topology A, without any pruning- (2) Random Masking: Applies randomly sampled binary mask Z;, simulating naive edge dropout. (3) Learned Mask (BayesG): Applies the full variational inference pipeline to infer Z,, resulting in adaptive and task-specific subgraphs. As shown in Figure (left), random masking severely degrades performance, while the learned mask significantly outperforms both baselines by suppressing irrelevant links and preserving only informative neighbor connections.

---

Mask Input Features. We further examine how different information types affect the learned mask. Specifically, we use:(1) State: Mask is generated based on neighbor observations sy, (2) Trajectory: Uses LSTM hidden states by,, capturing temporal behavior. (3) Policy: Based on neighbor action distributions n. Results in 3 (right) indicate that trajectory- performance that state-only, as the iccrporate richer temporal context. Policy-based masks also provide meaningful signals, reflecting agents' behavioral intentions. The best results are achieved when the learned mask uses combination of these features, as implemented in BayesG.
features lead to stronger
6
Discussion
6.1 Comparison with Sampling-baser Networked MARL
Recent work has explored sampling strategies for scalability in networked MARL, but differs fundamentally in what is sampled, how distributions determined, and where adaptivity occurs.
Sampling targets and distributions. Qu et [26] establish convergence results for networked systems with local dependencies on fixed graphs. Lin et al. 233 sample active links from predefined distribution D for j--dcaay analysis. Anand and Qu (27 use uniform agent subsampling (k out of n) for global controller. In contrast, our method learns context-dependent, per-edge distributions (Z26 @s) optimized jointly with the policy via an ELBO.
Locus of adaptivity. Lin et al. [23] achieve dynamics-level stochasticity through time-varying link sets. Anand and Qu (27) vary agent subsets for a global controller. Our approach operates at the policy level: each agent samples task-adaptive I-hop subgraph for local decision -aaiiin.
First-hop inclusion probabilities. Lin al. 23 use the marginal of 'D (predefined, not state- adaptive). Anand and Qu (27) use uniform inclusion (k/n). Our method learns edge-specific probabilities Pr(zij 1] (.))()) that are state and trajectory-dependent.
These approaches provide valuable convergence guarantees under structural assumptions. Our contribution is complementary: learning which neighbors to attend to, optimized end-to-end with the policy. A detailed comparison is in Table T (Appendix
6.2 Distinction from Dec-POMDP-based Coordination Graphs
Our method addresses the Spatiotemporal-MDD setting (Definition 1), where agent transitions p:(s[]sy,, dk, uA;) and rewards R(sv,, Ny,) are localized over physically connected neighbors deter- mined by Genv. This reflects real-world systems (traffic networks, power grids, sensor fields) where agents only observe and affect local neighborhoods through fully decentralized learning.
In contrast, Pec-POMDP-based methods such as CASEC [5), DCG (33,, DICG [29], SOP-CG 34 and GACG (22) assume global state global reward R(s, u), joint transitions p(s' (s, u). and centralized training with decentralized execution (CTDE). They allow unrestricted graph rewiring unconstrained by physical topology, making them incompatible with physically grounded networked systems. Our Daselines-Comene CommNet [40], NeurComm 77 and LToS [3] ---ommniccctee only with physical neighbors, respect fixed topology, and perform fully decentralized learning. While Dec- POMDP coordination graphs are important contributions, we exclude them due to incompatibility with Spatiotemporal-MDDPss decentralized, physically constrained nature.
7 Conclusion
We presented BayesG, variational framework for learning latent interaction graphs in networked multi-agent reinforcement learning. By operating over grrggh---ocalizze subgraphs centered on each agent-BayesG enables decentralized agents infer task-adaptive communication structures using local observations and Bayesian inference. Our approach integrates graph inference with actor-critic training via an ELBO objective, jointly optimizing both the interaction topology and policy. Experiments on five adaptive traffic signal control benchmarks demonstrate that BayesG consistently outperforms strong baselines, achieving superior coordination and reduced congestion. Case studies and ablation results further highlight the efficiency and interpretability of the learned graphs. BayesG offers principled approach to structure-awe aware cooperation in decentralized settings and opens up new opportunitics for scalable learning in networked systems.
10

---

Acknowledgements
This work supported by the Australian Research Council under Australian Laureate Fellowships FL190100149.
References
[1] Zhou, M., Luo, J. Villela, et al. SMARTS: an open-source scalable multi-agent RL training school for autonomous driving. 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, vol. 155 of Proceedings of Machine Learning Research, pages 264-285. PMLR, 2020.
[2] Yeh, J., V. Soo. Toward socially friendly autonomous driving using multi-agent deep reinforce- ment learning. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, (AAMAS 2024), Auckland, New Zealand, May 6-10, pages 2573-2575. 2024.
[3] Naderializadeh, N., Sydir, M. Simsek, et al. Resource management in wireless networks via multi-agent deep reinforcement learning. IEEE Trans. Wirel. Commun., 2016):3507-3523, 2021.
[4] Lv, Z., L.. Xiao, Y. Du, et al. Efficient communications in multi-agent reinforcement learning for mobile applications. IEEE Trans. Wirel. Commun., (240))12440-22444 2024.
[5] Samvelyan, M., T. Rashid, C. S. de Witt, et al. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MulriAgent Systems, (AAMAS 2019), Montreal, QC, Canada, May 13-17, pages 2186-2188. 2019.
[6] Shao, J., 7. Lou, H. Zhang, et al. Self-organized group for cooperative multi-agent reinforcement learning. In Advances Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems, (NIPS 2022), November 28- December 9, New Orleans, LA, USA. 2022.
[7] Chu, T., S. Chinchali, Katti. Multi-agent reinforcement learning for networked system control. In 8th International Conference on Learning Representations, (ICLR 2020), Addis Ababa, Ethiopia, April 26-30. 2020.
[8] Zhang, Y., Y. Zhou, H. Fujita. Distributed multi-agent reinforcement learning for cooperative low-carbon control of traffic network flow using cloud-based parallel optimization. IEEE Trans. Intell. Transp. Syst., 25 25(12):20715-2077
2024.
[9] Rashid, T., M. Samvelyan, C. S. de Witr, et al. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings the 35th International Conference on Machine Learning (ICMI. 2018), Stockholmsmülssan, Stockholm, Sweden, vol. 80, pages 4292-4301. 2018.
[10] Wang, T., H. Dong, V. R. Lesser, et al. ROMA: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference Machine Learning (ICMI. 2020), Virtual Event, vol. 119 of Proceedings of Machine Learning Research, pages 9876 9886. 2020.
[11) Wang, J., 7. Ren, T. Liu, et al. QPLEX: duplex dueling multi-agent q-learning. 9th International Conference on Learning Representations (ICLR 2021), Virtual Event, Austria. 2021.
[12] Duan, J. Lu, F. Yu, et al. Bandwidth--oosstranee variational message encoding for coopera- tive multi-agent reinforcement learning, 2025.
[13] Lowe, R., Wu, A. Tamar, et al. Multi-agent actor-critic for mixed cooperative--ooppttive environments. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, pages 6379-6390. 2017.

---

[14] Liu, Y., W. Wang. Y. Hu, et al. Multi-agent game abstraction via graph attention neural network. In The Thirty-Four Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, NY, USA,, pages 7211-7218. AAAI Press, 2020.
[15] Wang, T., L.. Zeng, W. Dong, et al. Context-aware sparse deep coordination graphs. In The Tenth International Conference on Learning Representations (ICLR 2022), Virtual Event. OpenReview.net 2022.
[16] Zhang, K., 7. Yang. H. Liu, al. Fully decentralized multi-agent reinforcement learning with networked agents. In Proceedings of the 35th International Conference on Machine Learn- ing.(ICML 2018), Stockholmsmässan, Stockholm, Sweden, July 10-15, vol. 80 Proceedings of Machine Learning Research, pages 5867-5876. PMLR, 2018
[17] Qu, G., A. Wierman, N. Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control (L4DC 2020). Online Event, Berkeley, CA, USA, 11-12 June, vol. 120, pages 256-266 PMLR, 2020.
[18] Chu, T., J. Wang, L. Codecà, et al. Multi-agent deep reinforcement learning for large-scale traffic signal control. JEEE Trans. Intell. Transp. Syst., 21(3):1086-100 1095, 2020.
[19] Yi, Y., G. Li, Y. Wang, et al. Learning to share in networked multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, (NIPS 2022), New Orleans, LA, USA, November 28 December 9. 2022.
[20] Du, Y., B. Liu, V. Moens, et al. Learning correlated communication topology in multi-agent reinforcement learning. In 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021), May 3-7,, Virtual Event, United Kingdom, pages 456-464. 2021.
[21] Duan, W. Lu, J. Xuan. Inferring latent temporal sparse coordination graph for multiagent reinforcement learning. IEEE Trans. Neural Networks Learn. Syst., pages 1-13, 2024.
[22] Duan, W., J. Lu, J. Xuan. Group-aware coordination graph for multi-agent reinforcement learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, (IJCAI 2024), Jeju, South Korea, August 3-9, 2024, pages 1926-39344 2024.
[23] Lin, Y., G. Qu, L. Huang. et al. Multi-agent reinforcement learning in stochastic networked systems. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS 2021), December 6-14, virtual, pages 7825-7837 2021.
[24] Du, Y., C. Ma, Y. Liu, et al. Scalable model-based policy optimization for decentralized networked systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems. (IROS 2022), Kyolo, Japan, October 23-27, pages 9019-9026. IEEE, 2022.
[25] Ma, C., A. Li, Y. Du, et al. Efficient and scalable reinforvement learning for large-scale network control. Nature Machine Intelligence, 609):1006-1020 2024.
[26] Qu, G., Y. Lin, A. Wierman, et al. Scalable multi-agent reinforcement learning for networked systems with average reward. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 2020.
[27] Anand, E., G. Qu Efficient reinforcement learning for global decision making in the presence of local agents at scale. CoRR, abs/2403.00022,, 2024.
[28] Jiang, J., C. Dun, T. Huang, et al. Graph convolutional reinforcement learning. In 8th Interna- tional Conference Learning Representations, (ICLR 2020), Addis Ababa, Ethiopia, April 26-30. OpenReview.net, 2020.
[29] Li, S., J. K. Gupta, P. Morales, et al. Deep implicit coordination graphs for multi-agent reinforcement learning. In AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 202/), Virtual Event, United Kingdom, pages 764-772. ACM, 2021.
12

---

[30] Lin. B., C. Lee. HGAP: boosting permutation invariant and permutation equivariant in multi- agent reinforcement learning via graph attention network. In forty-first International Conference on Machine Learning (ICMI. 2024), Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[31] Duan, W. J. Xuan, M. Qiao, et al. Graph convolutional neural networks with diverse negative samples via decomposed determinant point processes. IEEE Transactions on Neural Networks and Learning Systems,
2024.
[32] Duan, J. Lu, Y. G. Wang, al. Layer-diverse negative sampling for graph neural networks. Trans. Mach. Learn. Res., 2024, 2024.
[33] Boehmer, W., V. Kurin, S. Whiteson. Deep coordination graphs. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event, vol. 119 of Proceedings of Machine Learning Research, pages 980-991. PMLR, 2020.
[34] Yang, Q. W. Dong, Z. Ren, et al. Self-organized polynomial-time coordination graphs. In International Conference on Machine Learning (ICMI. 2022), Baltimore, Maryland, USA, vol. 2022.
162, pages
[35] Jang. E., S. Gu, B. Poole. Categorical reparameterizztion with gumbel-softmax. In the Sth International Conference on Learning Representations (ICLR 2017), Toulon, France. 2017.
[36] Maddison, C. J., A. Mnih, Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In the 5th International Conference on Learning Represenlations (ICLR 2017), Toulon, France. 2017.
[37] Haarnoja, T., A. Zhou, P. Abbeel, et al. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. In Proceedings of the 35th International Conference Muchine Leurning, (ICMI 2018), Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, vol. 80 Proceedings of Machine Learning Research, pages 1856-1865.
[38] Lopez, P. A., M. Behrisch, L. Bieker- Walz, et al. Microscopic traffic simulation using sumo. In The 21 IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018.
[39] Foerster, J. N., N. Nardelli, G. Farquhar, et al. Stabilising experience replay for deep multi- agent reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, (ICML 2017), Sydney, NSW, Australia, 6-11 August, vol. 70, pages 1146-1155. PMLR, 2017.
[40] Sukhbaatar, S., A. Szlam, R. Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NIPS 2016), December 5-10, Barcelona, Spain, pages 2244 2252. 2016.
[41] Foerster, J. N., Y.M. Assael, N. de Freitas, et Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, pages 2137-2145. 2016.
[42] Kipf, T. N., M. Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (ICLR 2017), Toulon, France, April 24-26. 2017.
13

---

Appendix
A Detailed Derivation
A.1 Derivation of Graph-based Policy
We expand on Definition [ which defines each agent's policy two-stage process: (i) sampling binary subgraph Gy, from a variational distribution p. and (ii) conditioning the action policy on the graph- srph--iltered encoding fivv, Gv.).
In conventional MARL, each agent conditions its policy fixed observation 3; f(v,)) where Vi is the agent's 's closed neighborhood. our formulation, the observation additionally depends on latent subgraph Gy; drawn from learned distribution:
Gy, p(G sv.1(;),
(14)
where p parameterized by variational parameters fp:- To ensure feasibility, we constrain sampled subgraphs to lie within the physical topology:
Gy. 3 GV
(15)
ie., agents only sample from edges permitted by the environment graph.
The graph-conditioned encoder (sv,, Gv,) is implemented using GNNs over the masked subgraph. The resulting policy is defined as:
(u sv.) EGG, ~p [us fiss,, Gv.)
(16)
where म; is the action policy given graph-filtered input.
This formulation enables each agent to dynamically adapt its local observation space via sampled interaction subgraphs, supporting sparse and context-aware reasoning under topological constraints.
A.2 Derivation of Graph-base based A2C Objective
We derive the actor loss introduced in Definition 4 which integrates latent subgraph sampling into A2C learning.
Recall the standard A2C actor loss:
(A))( - log To, (u: 52) AT,
(17)
where &; is the observation, u4 the action, and Åş the estimated advantage.
Under the graph-based based policy framework, the policy depends sampled subgraph Gy.:
Te (u, sv,) = Eay, ~pt (
(18)
The expected actor loss becomes:
[- log (u, fiss, Gv.)) AT]
Ce.p EGG, mp
(19)
We add entropy regularization to encourage exploration:
H(#x(-1 (.)) - Σ #c(m (1) log (lu f.).
(20)
Over batch B, the full objective is:
Co.p
EGv, up [- (a,,, f( Gv.) A + B H( (.a..111))].
(21)
B
This defines the graph based actor loss used in main training objective.
14

---

A.3 Derivation of Variational Objective and ELBO for BayesG
We present the detailed derivation the ELBO objective used in BayesG. beginning with Bayes' theorem and ending with a tractable training objective for policy and graph optimization.
Bayesian Inference Latent Graphs
In networked MARL, the environment graph CV, represents the physical connectivity between agents (e.g., traffic lights connected via roads), which imposes hard topological constraints on interaction. We model the stochastic interaction graph for agent as a binary mask Z; E {0, 1 x sampled from learned distribution the physical neighborhood graph G The resulting posterior distribution is:
p(D;) Ze. GV. p((.))
p(Zz GV, D,)
(22)
p(Da GV.
where:
Dj denotes the agent-specific data (e.g., neighbor states, trajectories, and policies),
p(Z,) is a prior over edge masks,
p(D; Ze, (TT)) ) is the likelihood of the data under the masked graph,
p(D; G is the marginal likelihood.
Variational Approximation and KL Divergence
Since the exact posterior in Eq. 22 is intractable, introduce variational distribution 9(Z;; o;) to approximate The variational parameters Φ; are learned by minimizing the Kullback-Leibe (KL) divergence between the approximate posterior and the true posterior:
KL[g(Z;; ob ())l
GV. D:)] (gz). [log (Zz)) - log pog GV., D:)]
(23)
Applying Bayes' rule to the log posterior, we rewrite:
log G. D2) log p(D; Z; G..) + p(Z)) log pDD; G..
(24)
Substituting Eq.
24 into Eq. 23
yields:
KL Ey(,,)
log (Z.)
log p(D;
Zi G..)) logp(Z;) log log G
(25)
B,(
[log (Z,)
P(D;
Z;, G., log)(Z,))
log p(D; GV,)
(26)
= -Ey(Z,)
[log pDD;
Zi GV,
+ log p( Z2) - log (Z,) + log p(D;
G
(27)
= -- -yy(2,) [log D;
Zi, GV,)
+ logp(22) - log g( [+)
+ const,
(28)
where we have used the fact that log p( D; G does not depend on Z, or Φ;. Therefore, minimizing the KL. divergence is equivalent (up to an additive constant) to maximizing the evidence lower bound (ELBO):
LELBO = Eq(2...) [(o Z:) + (ogp((2)) log a( Z;; dx) + const.
(29)
Term-by-Term Breakdown
1. Likelihood Term: log DoD:((D:) Z,)
This is modeled using the graph- conditoond policy loss. i.e., log (D; Z:) As Logpp
(30)
where Co.p is the actor loss under sampled subgraph Z, @ Gan.
Prior Term: log p (2))
We define the prior as an element-wise Bernoulli with retention bias X:
p(Z,)
II
x (1 2
(31)
JEN;
15

---

Then:
log ((2)) =
Σ
Zig log A + (1 21)) log(1 - 2).
(32)
jEN;
Taking expectation under (Zz)):
Egllog p( [,)]
Σ
(o(d))) log A (1 a(di))) log(1 A)].
(33)
JeN,
3. Entropy Term: - log g (i)
Since (Z,)) is factorized Bernoulli:
H(g(Z)))) -o(()) log logg)))o)))) (
(34)
Then:
Byglog ((i))
Σ H(g(Z)))).
(35)
jEN;
Final Objective
Combining all terms:
LELBO = By(2,,0)) |- Co +
Σ
[ (lloo + (1 X) log(1 o(bb))) Σ H(o(Zz)))
JEN:
JEN,
= Eg(2rsoh)
[l +
Σ ( + X) 1og(1-ooddsp)) - -(d))))
JEN.
Σ
(a(0s)) - (1 a(oi;)) log(1 ((b)))
JEN.
Lo +
Σ
o(1r)))
JEN.
( (og))))ogo((0 logo + ) log( log(1 ((06)))
= (gg(z
- -Loup +
Σ (000 22- ( log(1 σ (0iz))
JEN.
(36)
This is the BayesG training objective as formalized in Definition
5
B ATSC Environment: MDP Component Mapping and Spatiotemporal-MDPP Justification
We provide detailed mappings from the adaptive traffic signal control (ATSC) domain to the Spatiotemporal-MDD framework (Definition 1).
B.1 MDP Components
States (s;). Each agent (a signalized intersection) observes its local traffic state via induction loop detectors (ILDs) placed on incoming lanes. The state includes:
Vehicle density: Number of vehicles per lane
Queue length: Number of stopped vehicles per lane
Waiting time: Average waiting time of vehicles per lane
Current phase: Active traffic signal phase (c.g., north-south green)
16

---

NewYork33
NewYork51
NewYork167
Figure 6: Visualization of the NewYork33, NewYork51, and NewYork167 environments. Top: SUMO network with signalized intersections. Bottom: extracted graph structure used networked MARL, including traffic light nodes and their physical neighbors.
Phase duration: Time elapsed in current phase
For neighborhood-awware coordination, agents also receive aggregated statistics from immediate neighbors N2 (e.g., neighbor queue lengths, active phases).
Actions (u:). Each agent selects a traffic signal phase from a discrete action space, Actions correspond to:
Phase switching: Transition to different signal phase (e.g., from north-south green to east-west green)
Phase holding: Maintain the current phase for another control interval
mandatory yellow phase (2 seconds) is enforced between conflicting green phases for safety.
Transition Probabilities (p,(s))s,; u4 )),)). Transitions are not analytically available but are governed by the SUMO microscopic traffic simulator [38). The key property is locality: agent i's next state sf depends primarily on:
Its own action u; (phase decision)
Local state s; (current congestion)
Neighbor actions uus (upsttream/downstteamm traffic release)
This satisfies the Spatiotemppral-MDD assumption that transitions are localized over the physical neighborhood V,
Rewards (R(sy, )). The reward for agent i is:
Σ num h
(37)
C
éElanes;
where num halted(E) is the number of stopped vehicles on lane e, and C is normalization constant. This encourages local queue minimization. The reward is localized agent i's intersection, consistent with the Spatiotemporal-MDD framework.
B.2
Why Spatiotemporal-MDD Fits ATSC
The ATSC domain is a canonical example of Spatiotemporal-MDD due to:
17

---

1. Localized dynamics. Traffic flow is governed by physical proximity: upstream intersections release vehicles that propagate downstream intersections. Each agent's state evolution depends on its immediate neighbors' actions, not the global joint action of all agents.
2. Fixed physical topology. The road network structure is fixed and sparse, with agents (intersections) only interacting with directly connected neighbors via shared road segments.
3. Decentralized execution requirement. In real-world deployments, traffic signals operate inde- pendently with limited communication bandwidth. Centralized control is impractical due to:
Scalability: City-scale networks have hundreds of intersections; centralized joint action spaces grow exponentially
Communication constraints: Real-time global state aggregation infeasible under latency and bandwidth limits
Robustness: Centralized systems are vulnerable to single points of failure
Local observability. Each intersection has sensors only for its incoming lanes, consistent with the partial observability assumption in Spatiotemporal
These properties make ATSC fundamentally different from cooperative benchmarks (e.g.. Star- Craft) that assume global rewards, unrestricted communication, and arbitrary coordination graphs. Our method exploits this structure by learning sparse, physically grounded communication masks, enabling scalable and deployable traffic control.
C New Scenario Visualization and Graph Statistics
To support the evaluation of BayesG on large-scale environments, we include visualizations and statisties for the three real-world maps: NewYork33, NewYork51, and NewYork167. These maps are derived from SUMO simulations based on real intersections in Manhattan, and are visualized in Figure
Each row in Figure 6 shows:
Top: The SUMO map used for microscopic traffic simulation, where green nodes represent
traffic light-controlled intersections.
Bottom: The corresponding environment graph used for MARL training. Blue circles represent agents (traffic lights), and red edges indicate neighbor connections used for policy input.
The graph used in our MARL environment is a subgraph of the physical road network. Specifically, we include only intersections that are signal-controlled (i.e., managed by traffic lights), and define edges based on direct traffic flow connections between these nodes. Road segments that connect to unsignalized intersections or that skip over intermediate traffic lights are excluded from the environment graph. This design reflects the decentralized setting which agents can only communicate and coordinate with neighboring controlled intersections.
Note:
Scenario
# Nodes
# Edges
Avg. Degree
Max Degree
New York33
33
56
1.70
New York51
51
125
2.45
4
NewYork York167
167
384
2.30
Table 1: Graph statistics for the New York traffic signal control environments. Nodes represent
signalized intersections; edges represent physical connectivity between controlled agents.
D Algorithms
We provide pseudocode for BayesG training and decentralized execution. Algorithm D Noutlines asynchronous multi-agent training with latent graph inference, following the procedures in Sections
18

---

and 4. The training loop includes: (i) edge sampling and message propagation, (ii) policy and trajectory updates, (ii) value estimation and environment simulation, and (iv) gradient-basee updates of actor, critic, and variational graph parameters.
Algorithm 2 describes decentralized execution: each agent samples subgraph, encodes its neighbor- hood, and selects actions in real time without global information.
Algorithm BayesG: Multi-agent A2C Training with Variational Graph Inference
1: Parameter: a, 8, 7, 7, T, [B], 7],, Tlo f
2: Result: {0,, wi; 91 Jev
3: Initialize 80, T-1, h-1, < 0, k < O.
4: repeat
5: for i E V do
6: Sample Zi ~ q (Zzz Φi), set A t 2, 3 6%
7:
Encode 84,1 +- fi(sv, Ai)
8:
Update h4 -- gv, (hi,1-1, S1,1)
9:
Sample ai,t no, (. hi,t)
10:
Compute riA <- Vus (his, UN.)))
11:
Execute ai,t
12:
end for
Simulate (
14:
B B U 1 aies Tia Uiee) hiv
L 1+ 1, k+1
16:
ift = T then
Reset so. T- 1, +
18:
end if
if &
[B] then for E do
20:
Estimate R Ag, VT EB
22:
Update critic:
Tlw VLritic
23:
Update actor & graph: 0;, 0i &- 0; ti 170 Vo. (-LEBRO) end for Reset 0, +0
24:
25:
26:
end if
27:
until Stop condition is reached
Algorithm 2 BayesG: Multi-agent Execution with Latent Graph Sampling
Parameter: [0,, diliev, Atcom:: Alconmol
2: for E V do
3: Initialize h; 0, ग. 0, fsy, TyEEN, +0
4: repeat
5: Observe s;
6: Sample mask 7; ~ 4( (s),, compute AT + 7, ② G
7:
Send 8,, क, to neighbors
8:
for j E NN; do
9:
Receive and update 8,, ny within Atcomm
10:
end for
11:
Construct graph-conditioned imput: 5; +- fe(sv, Update h; + Iv, (h,, 5;), compute policy m, < no. (. ha) Execute action a; ~ π.
12:
13:
14:
Sleep for duration Atcontro until Stop condition is reached
15:
16:
end for
19

---

C prior
Linask_entropp
Epolicy
Run
Run 2
0-
-0
Run
Run
Run 5
Loos
51
-15
-250
-54
Run Run
302-
Run
Run
0.0
0.2
0.4
0.6
0.8
1.0
5.0
0.2
044
0.5
0.
1.0
0.0
0.2
2.4
0.5
0.8
1.0
Training Steps
Training Steps
Training Steps
Celbo
Cvalue
Etotal
1.00
Run
2000
Run
2000
Run
Ran
50
Run
Run
Run
1500
Ran
1300
Run
Ran
LO08
-50
1000
1000
-150
500
500
Run
-250
0.0
0.4
0.6
0.8
0.0
0.2
0.4
0.5
1.0
0.0
0.2
0.5
0.8
Trining Staps
Trining Steps
Trairing Steps
Figure 7: Training loss curves of BayesG on ATSC_Grid.
Cpiir
mamk entropy
Ppliic
Run
Run
Run
Run
35.5
Run
Run
Run
Run4
100
Run
Run
36.5
33.0
18
-20
Run
-37.5
Run
Run
-38.0
14-
-300
Run
Run 5
-38.5
-150
15
25
35
10
30
10
26 (x10f)
30
Training (x10f)
Training (x10%)
Cel
Lvalue
Ceotal
Run
Run
Run
3103
Run
Run
Run
-1.0
-50
25:00
Run
Run
2300
Run 4
Run
20000
Run
Run
1300
1500
-200
Run
1000
-250
Run
Run
500
15
25
35
10
30
10
20
30
Steps (x1)f))
(x10%)
(x10f)
Figure Training loss curves of BayesG Monaco.
E Training Loss Analysis
and g illustrate the evolution of training losses for BayesG three representative environments: ATSc_Grid, Monaco, and NewYork33. We report the component-wiss, losses across five random seeds.
Figures
Policy loss Lo.g. The policy loss reflects the negative log-probability of selected actions, weighted by the estimated advantage (see Definition During training, temporary increases in this loss may occur due distributional shift in the graph-conditioned state as the variational mask distribution qi Ze Φ.) evolves. As the latent subgraphs adapt, agents may explore new action patterns that
20

---

Cpror
Lmask_entropp entropy
Epolicy
-30
Ran
Run
Rus
Ran
Run
Run
Ran 3
Run
Ryhh
Rin 4
Rund
Rurs
-2
Run
32
Run
Purh
34-
23
28
24
-36
22
40
10
26
05
10
20
46
(x10*)
(x10*)
(x10*)
Lelbo
Lvalue
Crotal
72
Run
Run
Run
Run 2
20103
Run
20300
Run
70
Ban 3
17500
Bun
17500
Run
Run 4
Run
Run
Run 5
Run
Run 5
LaK5
66
22300
22100
62
10000
10000
7500
58
5000
2b
10
20
40
10
20
50
Training Steps (x109)
Training Steps (X10
Trakiing Steps (X101)
Figure 9: Training loss curves of BayesG on NewYork33.
momentarily reduce alignment with advantage estimates, leading to transient spikes. However, as both the policy and graph inference converge, the loss gradually stabilizes, indicating improved policy learning under the inferred interaction structures.
ELBO loss LELBO- The ELBO combines the actor loss with KI. regularization terms (see Defini tion 5). On Grid and Monaco, we observe an increasing trend in the ELBO, which stems from the increasing policy confidence (lower entropy) and the corresponding rise in the KL penalty due to deviation from the Bernoulli prior. Notably, the ELBO remains smooth and stable across seeds.
On NewYork33, a different pattern emerges: the ELBO decreases in early training as the agent learns useful sparse subgraphs and aligns with the prior. Later, however, the policy sharpens and latent graphs become more deterministic, the KL term increases, causing the ELBO to rise again. This U-shaped behavior highlights natural trade-off between policy certainty and exploration through stochastic subgraph sampling.
Other losses. The mask entropy term gradually decreases, indicating transition from exploratory communication structures to more deterministic ones. The prior loss stabilizes, confirming conver- gence toward a learned sparsity level. The value loss steadily decreases, showing reliable value function learning.
These results confirm that BayesGi effectively balances exploration and exploitation during latent graph learning and demonstrates stable convergence behavior across varying scales of networked environments.
F Additional Visualizations on Grid Environment
To complement the qualitative analysis in Figure 3 of the main paper, we provide additional visu- alizations of traffic conditions and coordination patterns for IA2C, CommNet, NeurComm, and BayesG on the ATSC_Gr map. These snapshots are captured at multiple simulation times: 1000, 1500, 2000, 3000, and 3599 seconds.
Figure displays the evolving traffic state and learned communication structure for each method. In these visualizations, the thickness and color intensity of each road segment reflect real-time vehicle thickkr and redder segments indicate higher congestion levels.
Key observations:
21

---

IA2C
CommNet
NeurComm
BayesG
1000
1500
2000
3000
3599
Figure 10: Qualitative comparison different simulation times (1000-3599s) on ATSC_Grit Gr_dd. Road thickness and color represent vehicle density (thicker and redder indicates more congestion). BayesG consistently learns to avoid congestion by adaptively shaping communication, in contrast fixed or overly dense schemes used by baselines.
IA2C struggles consistently across all time points due to the absence of inter-agent commu- nication, resulting persistent congestion.
CommNet shows slightly improved flow but lacks adaptivity, often over-communicating with irrleevnn neighbors.
NeurComm mitigates congestion more effectively by selectively encoding neighbor mes- sages, though its structure remains relatively fixed.
22

---

BayesG consistently exhibits the most adaptive behavior, dynamically adjusting its latent communication graph over time emphasize critical neighbors and suppress redundant links, leading to the smoothest overall flow.
These rime-resolved results further highlight the importance of dynamic, ago-graph-based communi- cation in complex, real-world coordination tasks.
G Extended Case Study: Dynamic Graph Adaptation Across Time and Runs
To further validate BayesGi' ability to infer dynamic and context-aware communication structures, we present additional case studies across two independent training (see Figure 1T).
the Grid environment
Each row in Figure I corresponds to different simulation time step and training run: Run-l shows the system behavior at step 600 and 1600, while Run-2 presents steps 1000 and 1400. For each time step, we visualize:
(a) Left: The aggregated latent interaction matrix, where each row corresponds to an agent's inferred ego ego-graph-a probabilistic mask over physical neighborhood. The overall matrix reflects the likelihood of communication between all agent pairs.
(a) Right: The corresponding spatial network visualization of the inferred graph, where edge width encodes communication probability. This provides an interpretable view of where information is likely to flow across the physical grid.
(b): The corresponding traffic density snapshot from the SUMO simulator. Redder and thicker lines denote higher congestion levels road segments.
These results highlight how BayesG dynamically adjusts interaction structures based evolving traffic conditions. In both runs, we observe that as local congestion increases, nearby agents increase their communication probabilities effectively directing This behavior illustrates the emergence of adaptive, directional cooperation: upstream or adjacent intersections proactively modulate traffic signals to help alleviate pressure downstream. Such behavior aligns with human-designed traffic heuristics, where context-sensitive cooperation is essential for preventing cascading congestion.
coordination toward problematic areas.
23

---

Whole Graph zation Step
Whole Grid Network
Whole Graph Adjatancy Manrs
[Eoge indicatis probabiliy)
Contict can Push
Run-1 Step 600
Visualization
Whore
Whole Graph Natrix
(Eoge indiccates probaolityl
Connect tan Rusboblllyy
10
10
Run-1 Step 1600
Whole Grid Network
Whole Graph
Run-2 Step 1000
Whole Grazh Visuaizariin at Step 1400 Wholee Gnd Network
Whole Graph Adjacency Matrik
(Eoge width indicaliss probabiltty
Coreacts Wobbi
Run-2
Step 1400
(a)
(b)
Figure Extended case study visualizations for BayesG on ATSC_Gris Grid. Each row shows a different simulation step and training run. (a) Left: Latent interaction matrix (ego-graph probabilities). (a)
Right: Physical network view with edge thickness indicating communication strength.
24

---

Table 2: Comparison of sampling strategies in networked MARL. Our method learns context- dependent, per-edge distributions optimized jointly with the policy, in contrast to predefined or uniform sampling approaches.
Aspect
Lin et al.
23
Anand & Qu 27
BayesG (Ours)
Distribution for sampling
fixed distribution D over
Exogenous,
Fixed, uniform distri- bution over size-k sub- sets of local agents for value estimation and action
Learned, per-agent variational posterior q( Z4; db) over 1-hop edges; state/trajectory- dependent, trained end-to-end (ELBO)
active link sets
(T,2, [i',, sampled i.i.d.; assumes distance truncation/decay
Sampling target
Global edges that are "active" for transitions (L") and rewards (L")
Agents (a subset of locals) for the single global agent to aggre- gate over (not edges)
Binary edge mask on cach agent's 1-hop ego-graph (which neighbors to "listen to" for pol- icy/communication)
each step
Locus of dynamic topology
Dynamics-leve stochasticity: new ac- tive link sets each step; multi-hop influence via chains across time
Agent-level subsam- pling: the global agent's view changes different locals sampled; topology effectively star-like, not edge-level
Policyleereseenationn level adaptivity: each agent samples task-adaptive 1-hop subgraph each step: no multi-hop rewiring (multi-hop only via
repeated local passes)
1-hop inclusion probability
Inclusion of (j + +) the marginal of D (dxedd/ooellassuumel) not learned or state- adaptive)
No edge-level 1-hop probability: agent- level inclusion k/n (uniform, fixed) per step for the global agent
For neighbor inclu- of agent sion Pr[zy==]]
( (-)) is learned
and context- deeeexttee (from
H Detailed Comparison with Sampling-based Networked MARL
Table 2 provides a detailed side-by-side comparison of our approach with recent sampling- seaseing-based based methods for networked MARL 23 27 The comparison highlights key differences in sampling targets, distribution types, locus of adaptivity, and first-hop inclusion probabilities.
These distinctions highlight that while all three methods employ sampling for scalability, they differ fundamentally in whether the sampling distribution is learned, what structural elements are sampled, and where adaptivity occurs in the system architecture.
I Limitations
While BayesG demonstrates strong empirical performance and scalability on traffic signal control benchmarks, several limitations remain:
Fixed physical topology. Our approach assumes a predefined, static environment graph that restricts the set of possible neighbor interactions. While this models many real-world systems such as traffic networks, it may not generalize to domains with dynamic or learned topology.
Local observability. Each agent infers its latent communication graph based solely on local observations within its ego-graph. This limits the model's ability to reason over long-ram rangeragee dependencies that may require more global context.
Hyperparameter sensitivity. The performance of the learned mask depends on the Gumbel- softmax temperature and sparsity prior. Improper tuning may lead to either under- or over-pruning of edges, potentially degrading coordination.
25

---

Computational cost. Compared to con-oommuiiattiee methods, BayesG introduces ad- ditional overhead due variational sampling and masked GNN computation. While still efficient in practice, training cost may increase with neighhorhood size or message-passing depth.
We believe these limitations open avenues for future work on dynamic graph adaptation, broader task domains, and further efficiency improvements.
26

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1-2 sentence) justificctin right after your answer (even for NA).
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The checklist answers
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd a proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" " or "we were unable to find the license for the dataset we used"). ) general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the contributions and scope the paper. We propose a decentralized actor-critic framework, BayesG, that learns sparse, context-aware interaction graphs over ego-graphs via Bayesian inference. The main claims regarding the formulation of a stochastic graph-based policy, integration with variational inference, and empirical performance large-scale adaptive traffic control tasks are supported by theoretical formulation and extensive experimental results. Assumptions (e.g., fixed physical topology. decentralized observability) and limitations appropriately stated. The results are presented in a way that reflects both their practical relevance and generalizability to other networked MARL scenarios.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
27

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include dedicated "L "Limitations" section in the appendix, where discuss several aspects of our approach, including assumptions on fixed topology, potential sensitivity Gumbel-softman
temperature, and the applicability of our method other con-graph-constraig MARL settings, We also acknowledge that our empirical validation is focused on traffic control benchmarks, and further testing in other networked domains (e.g., swarm robotics, power grids) remains future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does present formal theorems or propositions that require full theoretical proofs. Instead, it provides well-defined mathematical formulations (e.g.. the graph-based policy, actor-critic objectives, and ELBO) grounded in variational inference and reinforcement learning. These are derived with standard assumptions and are supported by detailed derivations Appendix A. Appendix A.33 While these formulations are essential to the proposed method, they are not presented as novel theoretical results in the form of formal theorems requiring assumptions and proof structures.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
28

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides detailed descriptions of the experimental setup, includ- ing environment configurations (e.g., map layouts, control intervals), reward definitions, state/action representations, and the architectures used for both policy and critic networks. Implementation details for all baselines are unified under the same A2C backbone and described clearly in Section 5.1.2] Hyperparameters such as training steps, batch size, and random seed settings also specified. Moreover, we include full algorithm pseudocode in Appendix D and derivations in Appendices A.I A.I A.3 which provide further clarity for reproducing our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: We provide full source code and configuration files for all experiments in the supplementary material, including detailed instructions for installation, environment setup, and execution. The traffic control benchmarks are built on SUMO, and we include the necessary network files, route generation scripts, and map preprocessing tools to recon- struct the environments. Hyperparameters and implementation details are documented to ensure faithful replication of results. All code and data will also be released publicly upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /pipssccc/ublic//uiideee http://guiiis/e/eeeee http://uddeeeeoeebbiiiiiin ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides comprehensive implementation details in Section 5.1.2 including the number of training timesteps, control intervals, input feature construction, network architectures, and the of consistent settings across baselines. Hyperparameters such as learning rates, rollout horizon, optimizer type (RMSprop), and entropy coefficients are documented. Additionally, environment-specific simulation details-such as episode lengths, control intervals, and traffic statistics-ar are described in the Experimental Setup section. Further details are included in the supplementary material ensure transparency and reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
30

---

Justification: The experimental results report mean episode returns across 5 random seeds, with shaded error regions representing one standard deviation. This provides clear vi- sualization of the variance and stability of different methods. All performance curves are smoothed using moving average for readability while preserving variability trends. These practices follow common standards for statistical reporting in reinforcement learning research.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experimental results report mean episode returns across 5 random seeds, with shaded error regions representing one standard deviation. This provides a clear vi- sualization of the variance and stability of different methods. All performance curves are smoothed using a moving average for readability while preserving variability trends. These practices follow common standards for statistical reporting in reinforcement learning research.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neurippe httppp//ww......apttcccccccc.ce.
Answer: [Yes]
Justification: The research does not involve human subjects, sensitive data, or deployable systems. It focuses solely algorithmic development and empirical evaluation in simulated traffic control environments. All experiments were conducted accordance with NeurIPS ethical guidelines, with no foresceable risk of harm, privacy concerns, misuse.
31

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper includes a discussion of societal impacts in the Limitations section. On the positive side, BayesG has the potential to improve urban traffic efficiency, reduce congestion, and lower carbon emissions. On the negative side, if deployed unfairly, such systems might unintentionally prioritize certain routes or neighborhoods, raising fairness concerns. Additionally, reliance traffic data may introduce biases if the input data is unbalanced or misrepresentative.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed method, BayesG, is designed for decentralized coordination in networked multi-agent systems, particularly adaptive traffic signal control. It does not involve the release of models or data with high risk for misuse, such as generative models or large-scale pretrained models. Therefore, this question is not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
32

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All third-party assets used in the paper, including simulation environments (e.g., SUMO), baseline implementations, and datasets, are properly cited in the main text. The terms of use and licenses for these assets (e.g., SUMO under Eclipse Public License) have been reviewed and respected. Any reused code or models are referenced appropriately in the implementation section.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, www.ssppppesttaeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces decentralized MARL. framework (BayesG) and associated implementation. Full documentation, including code structure, environment setup, and reproducibilityy instructions, is provided in the supplementary material and will be made publicly available upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
33

---

Answer: [NA]
Justification:
This paper does not involve any crowdsourcing experiments or human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects participant praseeipanttbased studies and therefore does not require IRB approval or related disclosures.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Answer: [NA]
Justification: This paper does not incorporate large language models (LLMs) as part of its core methods. Any usage of LLMs limited to minor writing support and did not affect the scientific contributions or methodology of the work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non sta-stad components.
Please refer to LLM policy httpee/r/baarrrpeccc///ooeeeeeep for what should or should not be described.
34


Reviews:
[{"review_id": "ZJjeG5Lx5E", "reviewer": "Reviewer_6Gbd", "summary": "In this paper, Networked-MARL, decentralized agents struggle with local observability and fixed-graph communication, hindered by static methods or centralized demands. We propose a stochastic graph policy with sampled local subgraphs and introduce BayesG, a decentralized framework learning sparse interactions via Bayesian inference. BayesG excels in large-scale traffic control (167 agents), outperforming baselines in scalability and efficiency.", "questions": "Why not use direct communication but instead employ variational latent variables as the medium for information propagation? And could you explain the specific mechanisms behind why using variational latent variables as communication leads to better policy performance?", "limitations": "No", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \nUsing a variational approach with latent variables as carriers for information propagation.", "was_revised": false, "final_justification": "", "rebuttal": ">**Why not use direct communication? Why use variational latent variables for message masking?**\n\nWe appreciate this insightful question. There are two layers to unpack here: \n\n(1) Why not direct communication over the full environment graph? \n\n While the environment-provided physical graph (e.g., road topology in traffic networks) provides a grounded prior for interactions, communicating over the entire neighborhood indiscriminately can lead to redundant or noisy information aggregation. Our ablation in Section 5.4 (Figure 5) compares no masking (i.e., direct message passing on the environment graph) with BayesG’s learned mask, and shows that direct communication performs consistently worse. Therefore, selectively filtering communication is essential to avoid distraction and enhance relevance. \n\n(2) Why use a variational latent variable instead of learning a deterministic mask? \n\n Rather than directly learning a fixed mask, we model the mask as a stochastic latent variable sampled from a variational distribution $q(Z_i; \\phi)$. This design brings several key advantages: \n\n  - Captures uncertainty in graph structure, which is particularly important under partial observability or sparse feedback in MARL. \n\n  - Encourages exploration in structural space, enabling agents to discover better interaction topologies during training. \n\n - Incorporates structured priors, such as a Bernoulli sparsity prior, to promote minimal and efficient graphs. \n\n- Supports entropy-based regularization, which smooths optimization and improves training stability. \n\nIn summary, direct communication is ineffective due to a lack of relevance filtering; deterministic masking fails to model structural uncertainty. By contrast, our variational latent mask provides a principled, adaptive, and uncertainty-aware mechanism that leads to improved decentralized decision-making, as validated by ablation results. \n\n >**Mechanism for improved policy performance and empirical evidence (Section 5.4)** \n\n The learning signal for $Z_t$ is provided via the graph-conditioned actor loss $\\mathcal{L}_{\\theta,\\phi}$  (see Definition 4), which appears in the ELBO's likelihood term. This loss connects interaction topology learning to downstream task performance: **if a sampled graph leads to better policy outcomes, it receives a higher likelihood.** This tight coupling ensures that the learned graphs are task-driven and performance-aligned. \n\nIn our ablation study on different graph masking strategies (Section 5.4), we compare: \n\n- No masking (i.e., using the full environment graph), \n\n- Random masking, and \n\n- Our learned Bayesian mask (BayesG). \n\n The learned mask consistently outperforms the baselines, showing that task-aware variational latent variables yield more effective communication graphs, ultimately leading to better decentralized policy performance."}, {"review_id": "Yrprj4Zj6u", "reviewer": "Reviewer_2Jg6", "summary": "This paper introduces BayesG, a MARL method for cases when an individual agent can pass messages to others in its neighborhood. The novel technique at the core of BayesG is the use of Bayesian variational inference to learn sparse, context-aware interaction graphs. Each agent operates on a localized ego-graph and learns to sample a latent binary mask over its neighbors. This mask dynamically prunes communication links, allowing the agent to condition its policy on a task-relevant subgraph. The distribution for this mask is trained end-to-end alongside the agent's policy by maximizing an Evidence Lower Bound (ELBO) objective, which enables agents to jointly optimize their interaction topology and decision-making strategy using only local information.\n\nEmpirically, BayesG was evaluated on five adaptive traffic signal control benchmarks, including large-scale scenarios with up to 167 agents based on real-world New York City layouts. The results demonstrate that BayesG consistently and significantly outperforms a selection of MARL baselines comprising communication-based and non-communication-based methods.  BayesG achieves significantly higher coordination returns, faster convergence, and better scalability. Furthermore, case studies and ablation experiments validate the effectiveness of the learned graphs, showing they are interpretable and crucial for achieving efficient coordination by dynamically adapting to changing traffic conditions.", "questions": "1. What is the reasoning for selecting CommNet and NeuroComm as representative communication-based baselines for the experiments?\n\n2. How does the proposed method perform compared to other methods that also learns a graph structure for communication, such as CASEC?\n\n3. What is the probabilistic interpretation of using the usual policy gradient objective (also added with the entropy term) as the log likelihood function within the ELBO objective?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n**(Major Strength - Quality - Experiment Design).** Aside from some small details that can be improved (see the weaknesses section below), the authors did a good job of designing and conducting experiments that demonstrate the paper's main claims. Aside from demonstrating the proposed methods' significantly higher returns compared to other methods, the ablation study and empirical analysis of the interpretability of the graph structure discovered by the method were strong aspects of the paper that I hope will remain in the final version.\n\n**(Major Strength - Clarity).** Overall, the paper was quite well-written. It was easy to identify the motivation underlying the research, the main claims, and the intuition behind the method.\n\n**(Major Strength - Clarity - Experiment Findings & Difference Between Compared Methods).** I believe the authors also did a great job highlighting the purpose of different experiments and what findings were discovered from each of them. At the same time, a sufficient description of the differences between the proposed method and other baseline methods/ablations was provided, making it easy for a reader to appreciate the significance of the comparisons made by the authors.\n\n**(Minor Strength - Significance). Despite the proposed idea being relatively incremental, I believe the well-designed and well-executed experiments in this paper have demonstrated that the proposed method is a performant method, which, according to the experiments, also scales to settings with hundreds of agents. At the very least, I am confident that the MARL community, specifically those working on learning inter-agent communication to perform well in collaborative tasks, would view this as a competitive baseline for future work.** \n\n**Weaknesses**\n\n**(Minor Weakness - Originality - Incremental Method).** The idea of optimising the evidence lower bound (ELBO) to infer the appropriate graph structure for some GNN is something that has been explored in prior work [1,2]. At the same time, in the context of MARL, the idea of learning a local graph structure to focus on attending more relevant messages from a subset of neighbours has also been explored in prior work [3]. Since these prior work was not cited in the paper, I expect that the paper's final version will contain adequate references to these papers. That being said, the combination of these two ideas, along with the specific use of the usual RL objective as the log likelihood term in the ELBO evaluation, is something newly introduced in this paper.\n\n**(Minor Weakness - Quality - Communication-based Baseline Selection).** It is also unclear to me why more recent communication-based methods, which also learn an appropriate graph structure for action selection, such as CASEC [3], are not selected as a baseline for this method. Furthermore, it is unclear why CommNet and NeuroComm, which are relatively older MARL methods, are instead selected as the representative baselines of the communication-based MARL methods.\n\n**(Minor Weakness - Clarity - Likelihood Function Selection in ELBO).** I would also suggest providing stronger justification for the use of the usual policy gradient objective (i.e., the product between action log likelihood and returns added by the entropy term) as the log likelihood function. In particular, I believe the authors could draw a parallel between this log likelihood function and the probabilistic interpretations underlying soft policy iteration [4] (and more generally, maximum entropy reinforcement learning).\n\nReferences:\n\n[1] Kipf et al. 2018. Neural Relational Inference for Interacting Systems. ICML 2018.\n\n[2] Löwe et al. 2022. Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data. CLeaR 2022.\n\n[3] Wang et al. 2022. Context-Aware Sparse Deep Coordination Graphs. ICLR 2022.\n\n[4] Haarnoja et al. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018.", "was_revised": true, "final_justification": "The authors addressed all the minor concerns I outlined except for my point regarding the method's novelty. Given the extensive work in similar problem settings, I believe it is crucial to develop a novel graph structure learning method to improve my scores. That being said, I am confident this is a borderline paper that is otherwise ready for acceptance.", "rebuttal": ">**On Baseline Selection and Comparison with Graph-Structured Methods like CASEC (Q1 & Q2)** \n\nWe thank the reviewer for the thoughtful questions regarding our choice of baselines and the omission of recent graph-based methods like CASEC from our experimental comparisons. \n\nOur baseline selection is grounded in the theoretical setting of **Spatiotemporal Markov Decision Processes (Spatiotemporal-MDP)**, as formalized in Definition 1 of our paper. In Spatiotemporal-MDP, each agent’s transition dynamics and reward function are explicitly localized over its physically connected neighbors, meaning that the environment inherently imposes a structured graph topology (e.g., road networks in traffic signal control). \n\nSpecifically, the local transition function `p_i(s_i' | s_{V_i}, u_i, u_{N_i})` depends only on the state-action tuples of agent $i$ and its neighbors $\\mathcal{N}_i$, and the reward\n`R(s_{V_i}, u_{V_i})`\nis likewise defined over the local subgraph. The resulting learning setting is fully decentralized, grounded in real-world physical constraints, and lacks any centralized critic, global reward, or global coordination mechanism.\n\n\nIn contrast, many recent graph-based MARL methods such as CASEC, DCG [1], DICG [2], SOP-CG [3], and GACG [4] are built on the Decentralized Partially Observable MDP (Dec-POMDP) framework. These methods operate under significantly different assumptions, including: \n\n- Access to or estimation of a **global state** $s \\in S$, \n\n- A **shared global reward signal** $R(s, \\mathbf{a})$ based on the joint action, \n\n- **Joint action-based transitions** $P(s^{\\prime} \\mid s, \\mathbf{a})$, \n\n- And often, centralized training schemes with **global observability** or coordination. \n\nThese assumptions make Dec-POMDP-based algorithms fundamentally incompatible with real-world physical systems governed by local dynamics, such as traffic networks, power grids, or sensor fields, where each agent only observes and affects its **local neighborhood**. \n\nTherefore, to ensure a fair and meaningful comparison aligned with our problem setting, we selected baselines that **respect the assumptions and structure of the Spatiotemporal-MDP framework**. Specifically, methods like CommNet, NeuroComm, and LToS are well-suited because: \n\n- They communicate **only with physically connected neighbors**. \n\n- They respect the **fixed topology** imposed by the environment graph. \n\n- They perform **local message passing** and **decentralized policy learning**, without relying on a global reward or state. \n\n- These baselines thus share the same structural and semantic assumptions as our proposed method. \n\nFinally, while CASEC and other Dec-POMDP-based methods represent important contributions to multi-agent coordination under different settings, we intentionally excluded them from our main comparisons due to their incompatibility with the physical and decentralized nature of ST-MDP. **We will add a clarifying discussion in the final version and cite these works in the related work section to clearly distinguish between the modeling assumptions and ensure transparency in our baseline choices.** \n\nReferences: \n\n [1] Boehmer et al. Deep Coordination Graphs. ICML 2020. \n\n [2] Liu et al. Deep Implicit Coordination Graphs. AAMAS 2021. \n\n [3] Yan et al. Self-Organized Polynomial-Time Coordination Graphs. ICML 2022. \n\n [4] Duan et al. Group-Aware Coordination Graph. IJCAI 2024. \n\n---------------\t\n>**The probabilistic interpretation of using the usual policy gradient objective (Q3)** \n\nWe thank the reviewer for this insightful question. \n\nIn our framework, we adopt a probabilistic perspective on reinforcement learning. The trajectory data $D_i$  generated by each agent is treated as being sampled from an underlying distribution governed jointly by the agent's policy and the latent communication structure $Z_i$. \n\n**Why Use the Policy Objective as the Likelihood in the ELBO** \n\nIn variational inference, the log-likelihood term `log p(D_i | Z_i, G_env_{V_i})` measures how well the latent structure $Z_i$  explains the data. We instantiate this term using the graph-conditioned actor loss $\\mathcal{L}_{\\theta,\\varphi}$ , defined in Definition 4. \n\nSpecifically:  $log \\tilde{\\pi}_i(a_i \\mid \\tilde{f}_i) \\cdot \\hat{A}_i$ \nis treated as the data log-likelihood. This quantity provides task-aligned feedback on how suitable the sampled graph $Z_i$  is for policy optimization: **if a sampled structure leads to high advantage (good performance), it is assigned higher likelihood in the ELBO objective.** This ties the learning of latent graph structures directly to downstream task performance. \n\nThis formulation follows the same probabilistic treatment as entropy-regularized RL, particularly the Soft Actor-Critic (SAC) framework [Haarnoja et al., 2018], which casts RL as probabilistic inference. In SAC, the optimal policy is derived by maximizing the expected log-likelihood of actions under a soft Q-function, plus an entropy bonus to ensure sufficient stochasticity and exploration. This regularization is critical for: \n\n- Improving exploration in high-dimensional action spaces. \n\n- Avoiding premature convergence to suboptimal deterministic policies. \n\n- Encouraging robustness to model misspecification and variance in gradients. \n\nOur actor loss mirrors this design, embedding stochasticity into policy optimization while preserving alignment with the Bayesian structure learning objective. \n\n**Why Include Mask Entropy $-\\log q(Z_i; \\phi_i)$**\n\nAnalogous to policy entropy in SAC, we include a mask entropy term in our ELBO objective, defined over the variational distribution $q(Z_i; \\phi_i)$ of edge masks. This term serves several key purposes: \n\n- Uncertainty Modeling: The entropy of the variational mask distribution captures uncertainty in the learned communication structure. In real-world settings with noisy and partially observable interactions, such uncertainty is inevitable and informative. \n\n- Regularization: Including mask entropy discourages premature collapse of the variational distribution to a deterministic mask. This is particularly important in early training when structural estimates are still evolving. \n\n- Diversity and Exploration: High entropy promotes diversity in sampled graphs, allowing agents to explore multiple plausible communication topologies. Over time, the posterior sharpens around those that consistently lead to higher policy performance. \n\n- Analytical Tractability: Since $q(Z_i)$ is a factorized Bernoulli distribution, its entropy can be computed in closed form using the sigmoid outputs $\\sigma(\\phi_{ij})$. This allows for efficient optimization. \n\nThis design is aligned with the maximum entropy principle in SAC and variational inference theory, where entropy promotes robustness and avoids overfitting to noise or spurious correlations in structure learning. \n\nBy treating the graph-conditioned policy objective as a proxy for the log-likelihood and incorporating entropy terms for both policy and structure, our ELBO formulation provides a principled probabilistic mechanism for joint optimization of communication and action strategies. We will clarify this probabilistic interpretation and its relation to Soft Actor-Critic in the final version."}, {"review_id": "F67BmBV4lz", "reviewer": "Reviewer_ZRqv", "summary": "This paper studies networked multi-agent reinforcement learning (Networked-MARL), where each agent’s transition probability only depends on local neighborhood states and actions. The authors propose a new policy class for each agent that samples a subgraph over its local neighborhood. The policy decides local actions based on the information within this masked subgraph. The authors formulate the problem of learning latent edge masks as a Bayesian inference problem. As a result, the variational distribution can be trained end-to-end using an evidence lower bound objective. Intuitively, this policy class allows each agent to emphasize high-impact neighbors and suppress irrelevant ones. The authors test their methods on a traffic signal control task.", "questions": "In the rebuttal, I hope the authors can answer my questions from the previous section and clarify the missing content.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: It’s interesting to see that a more sophisticated design of the RL policy class can improve the performance of Networked-MARL. Emphasizing more relevant neighbors makes sense, especially when the network graph is dense, and most neighbors/information may be irrelevant. It’s also good to see that the latent edge masks can be learned end-to-end. In the ablation studies, the authors make the point that such graph masking is critical.\n\nWeakness: Section 3 about the preliminaries is too concise. The authors should make the section self-contained rather than relying on references. I also suggest dividing this section into two subsections. The first one can focus on the MDP environment, i.e., what is special about the transition probability, rewards, and available information. The second one discusses the existing algorithm and its limitations.\n\nSection 5.1 about the experiment setup also misses important details. Specifically, I encourage the authors to explain what the states, actions, transition probabilities, and rewards correspond to in the application of adaptive traffic signal control. And, most importantly, why does this application (approximately) satisfy Definition 1?\n\nSince a key novelty of this work is about graph masking strategies, I think the first ablation study in Section 5.4 should be strengthened. The reader might ask when the graph masking strategies enable the most significant improvements over no masking. From Figure 5, the improvement on the dataset ‘NewYork 51’ is not as significant as ‘Monaco.’ I think the authors should do this ablation study on all ATSC environments and discuss the effectiveness of graph masking strategies. For example, does it improve more on dense networks?", "was_revised": true, "final_justification": "Since the authors did not provide additional ablation studies to support the intuition behind graph masking strategies, I would like to keep my score unchanged at the borderline accept.", "rebuttal": "> **Conciseness of Section 3 and suggestions to restructure it.**\n\nThank you for the suggestion. Due to space limitations in the submission version, we provided a condensed version of the background. However, we acknowledge the importance of making the preliminaries more self-contained and structured. In the final version, we will: \n\nRestructure Section 3 into two subsections: \n- One focusing on the spatiotemporal MDP setting, clearly explaining how state transitions, rewards, and local observability are defined under neighborhood-based interactions. \n- The second summarizes key prior approaches and their limitations in the context of dynamic, decentralized interaction modeling. \n\nWe will also move some essential formal definitions from the appendix to the main text in the final version to enhance readability and self-containment. \n\n-----------------------\n> **Details of adaptive traffic signal control (ATSC) in Section 5.1 and justification for Definition 1**\n\nWe appreciate this question and the opportunity to clarify. In Section 5.1, we briefly described the mapping from the real-world traffic environment to the MDP components. To elaborate: \n\n- **States**: Each agent observes its local traffic state via sensors such as induction loop detectors (ILDs), which provide vehicle density, queue length, and waiting times on incoming lanes. \n\n- **Actions**: The action space of each agent consists of phase control decisions (e.g., switching between green and red for traffic directions). \n\n- **Transition Probabilities**: These are not analytically available but are implicitly defined via the microscopic traffic simulator (SUMO). The transition of each agent’s local traffic state depends primarily on its own action and the actions of neighboring intersections (e.g., upstream traffic release). \n\n- **Rewards**: Defined as the (negative) number of halted vehicles at the intersection, normalized by a fixed scale. This encourages reducing traffic congestion locally. \n\nThese components are naturally aligned with the **Spatiotemporal MDP** framework (Definition 1) because: \n\n- The transition dynamics are **localized and depend on neighborhood interactions**, consistent with the assumption that `p_i(s_i'| s_{V_i}, u_i, u_{N_i})`.\n\n- The reward function is **decomposed per agent**, depending only on local state-action pairs. \n\n- The neighborhood structure is explicitly defined by the **physical road network graph**, which is fixed and sparse. \n\nMoreover, **centralized training methods are often impractical or suboptimal** in ATSC due to several real-world constraints: \n- **Scalability**: As the size of the traffic network grows (e.g., in city-scale scenarios), centralized models become computationally expensive and suffer from poor generalization due to the curse of dimensionality in joint state-action spaces. \n- **Communication constraints**: Centralized training assumes full observability or global message aggregation, which is unrealistic in large distributed traffic systems with bandwidth and latency limitations. \n\nThus, our formulation — learning decentralized policies that respect the fixed physical connectivity and local dynamics — not only satisfies the assumptions of the spatiotemporal MDP, but also reflects the operational needs of real-world traffic control systems. \n\nWe will revise Section 5.1 to clarify these mappings and add this justification. We also plan to release our code and ATSC simulation environments upon publication. \n\n-----------------------\n> **Strengthening the ablation study of graph masking strategies**\n\nWe agree that understanding when graph masking yields the most benefit is valuable. While our current ablation study includes two maps (Monaco and NewYork51) with different characteristics, we did not include all five due to time constraints during the rebuttal phase. We will extend this study across all ATSC maps in the final version. \n\n**Regarding the effectiveness of masking**: \n\n- According to our analysis, the benefit of graph masking is more pronounced in **denser physical networks**, where agents are connected to more neighbors. In such settings, the probability of receiving **redundant or irrelevant information increases**, which can negatively impact policy learning. Our learned masks help filter out such noisy edges, promoting more focused and effective communication. \n\n- In sparser environments, where each agent has fewer neighbors by design, the amount of incoming noise is naturally limited, and thus the marginal benefit of masking is expected to be smaller. \n\n**On the apparent difference between Monaco and NewYork51 in Figure 5:**\n\nThe seemingly smaller improvement on NewYork51 is partially due to scale compression, caused by the relatively poor performance of the RandMask baseline. In absolute terms: The performance gap between BayesG and NoMask is ~100 in Monaco and ~60–70 in NewYork51, both of which are substantial. \n\nWe will clarify this interpretation in the revised version and present both raw and normalized performance gains in the final plots."}, {"review_id": "YzNnlPHyQn", "reviewer": "Reviewer_GrDU", "summary": "This paper presents BayesG, a novel variational Bayesian inference framework designed for networked multi-agent reinforcement learning (Networked-MARL). BayesG enables each agent to make decisions based on a sampled subgraph of its local physical neighborhood, thereby achieving sparse and context-aware interaction structures. The method integrates variational Bayesian inference with A2C, employing the evidence lower bound (ELBO) objective function for joint optimization. This allows for dynamic adjustment of the interaction structures among agents, thereby reducing unnecessary communication overheads. Experiments conducted in five adaptive traffic signal control (ATSC) benchmark scenarios demonstrate that BayesG outperforms existing MARL baselines in terms of performance, convergence speed, and scalability, highlighting its potential and applicability in complex networked systems.", "questions": "Please refer to the section \"Weaknesses\" for detailed information.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n﻿\n1. The study holds a certain degree of value.\n2. The associated experiments are relatively comprehensive.\n﻿\n**Weaknesses**\n1. There are quite a few grammatical and formatting errors in this paper. For instance, the legend of Fig 2 should be placed at the top rather than the bottom; there are missing punctuation marks in equation (12), etc. It is suggested that the authors review paper carefully.\n﻿\n2. Section 4.2 appears to be poorly written, which I find very difficult to comprehend. I would recommend expanding this section to clarify the rationale behind the use of the Bayesian approach, as well as the specific implementation details. This would enhance the readability significantly.\n﻿\n3. Does this work provide any theoretical guarantees? As is well-known, policy iteration under DTDE typically encounters the issue of non-stationarity, yet I have not seen how this paper addresses this problem.\n﻿\n4. Still regarding the issue of non-stationarity, employing a learning-based method for sampling is an intuitive idea. However, does such sampling have any impact on the stability of policy iteration?\n﻿\nIf you are willing to provide clear answers to these questions, I would consider raising my score.", "was_revised": true, "final_justification": "**Final Decision**: I have decided to raise my score to 4.\n\n**Justification**: The paper mainly considers BayesG, enables each agent to make decisions based on a sampled subgraph of its local physical neighborhood, thereby achieving sparse and context-aware interaction structures. Overall, the writing is clear and the motivation is relatively clear. During the rebuttal phase, the issues that have been resolved through active interaction with the authors are: clarification of the second paragraph; theoretical guarantees, discussion on non-stationarity, and distinction from related work; the issues not addressed due to content and space limitations are modifications of minor errors. Combined with relevant comments from other rebuttals, I now understand the contributions of this paper, and I have decided to raise my score to 4.", "rebuttal": "> **Grammatical and formatting issues (e.g., Fig. 2 legend, punctuation in Eq. (12))**\n\nWe appreciate this comment and will thoroughly revise the manuscript in the final version, including: \n\n- Moving all legends (e.g., Fig. 2) to consistent and appropriate positions. \n\n- Carefully proofread all equations, especially Eq. (12), to ensure correct punctuation and presentation. \n\n- Conducting a full grammatical and typographic pass. \n\n> **Clarity and completeness of Section 4.2 (Bayesian Graph Inference)** \n\nThank you for pointing this out. We agree that Section 4.2 is central to the paper and could benefit from expanded exposition. In the final version: \n\n- We will revise this section to clarify the motivation for using Bayesian inference, specifically, to enable uncertainty-aware edge masking and regularized graph structure learning. \n\n- We will better explain how the variational distribution over binary masks is amortized using graph neural networks and trained end-to-end with the policy. \n\n- We will also clarify the interaction between the ELBO objective and policy optimization. \n\nWe also note that a more detailed explanation of this mechanism and architecture is currently provided in the Appendix, but we agree that bringing some of this detail into the main text will improve accessibility and understanding. \n\n\n> **Theoretical guarantees and handling of non-stationarity in DTDE settings**\n\nWe appreciate the reviewer’s insightful question regarding theoretical guarantees and non-stationarity in decentralized MARL. \n\n1. **Non-stationarity in DTDE**\n\nIn DTDE settings, each agent learns its policy independently while treating the environment as stationary. However, in multi-agent scenarios, the environment includes other agents whose policies are continuously evolving during training. This results in **non-stationary learning signals**, which often degrade stability and convergence in policy iteration. This issue is well-documented in prior works (e.g., Hernandez-Leal et al., 2019 [survey on MARL]). \n\n2. **How CTDE alleviates non-stationarity**\n\nCentralized Training with Decentralized Execution (CTDE) mitigates this issue by providing each agent access to **global information or centralized critics** during training. This stabilizes learning by accounting for co-adaptation among agents. Prior works such as QMIX (ICML 2018) and MADDPG (ICML 2017) have shown that centralized critics can **eliminate or significantly reduce non-stationarity during training**, leading to more stable and sample-efficient learning. \n\n3. **Our relationship to CTDE via Spatiotemporal MDP**\n\nOur work lies **between DTDE and CTDE** in terms of structural assumptions. Specifically, we operate under the Spatiotemporal MDP framework (Definition 1), in which: \n\n- Each agent's **transition** and **reward** are conditioned on its **physical neighborhood**. \n\n- This imposes a partial coupling across agents via a fixed environment graph (e.g., road topology), ensuring that agents only adapt to **localized dynamics** rather than the entire system. \n\n- Consequently, the variability perceived by each agent is constrained to its **stable local neighborhood**, substantially reducing global non-stationarity. \n\nThrough this **structural prior** and **ego-centric masking**, our approach preserves decentralized execution while benefiting from the stability typically associated with CTDE. While we do not offer formal convergence guarantees, our design choices are empirically validated in terms of training stability and sample efficiency (Section 5.4, Fig. 5). \n\n**On the lack of formal analysis:** \nWe also note that many recent NeurIPS publications proposing novel methods for MARL similarly do not provide theoretical guarantees, focusing instead on empirical performance and structural insights. These include: \n\n- Learning to Share in Multi-Agent Reinforcement Learning (NeurIPS 2022) \n\n- Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning (NeurIPS 2023) \n\n- Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2023) \n\nWe will expand our discussion in the final version to better articulate the structural mitigation of non-stationarity via spatiotemporal MDPs and position our approach relative to CTDE and DTDE. \n\n> **Whether the learned graph sampling process impacts the stability of policy iteration**\n\nThis is an insightful concern. Our response has two parts: \n\n- Learning a variational distribution over edge masks allows us to capture epistemic uncertainty over which neighbors are most informative. This stochasticity can improve robustness in dynamic or partially observable environments, which reflects real-world scenarios (e.g., noisy or fluctuating traffic patterns). \n\n- The sampled communication graph $Z_i$ influences the actor loss via the graph-conditioned policy $\\tilde{\\pi}_i$, as shown in Definition 4. This actor loss is embedded in the likelihood term of our ELBO formulation: \n`L_ELBO = E_{q(Z_i; φ_i)}[ -L_{θ, φ} + log p(Z_i) - log q(Z_i; φ_i) ]`.This setup ensures that mask sampling is directly driven by task performance—**graphs that yield better policies are more likely under the learned posterior.** \n\nTherefore, our sampling-based approach is not only **grounded probabilistically** but also aligned with policy improvement. Empirically, as shown in our ablation studies (e.g., Fig. 5), this coupling leads to more **stable and performant learning** than alternatives like deterministic graph construction or random masking."}]

Author rebuttal:

Meta-review: This paper presents BayesG, a decentralized actor–critic approach for networked multi-agent reinforcement learning (MARL), that learns sparse, context-aware communication graphs in spatiotemporal MDPs using Bayesian variational inference. The paper is mostly well-written and has strong performance in the traffic signal control tasks. 

There are some concerns about the novelty of the approach and the presentation, but the approach is interesting and performs well. As noted in the reviews, the paper should be updated to more clearly discuss the assumptions needed, motivation for algorithmic choices, relationship with other methods, and to fix typos and other grammatical issues. 

The author response and discussion was helpful in addressing these points but the paper should be updated further to address these issues.
Decision: Accept (poster)