Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems
Authors: Hao Liang
Shuqing Shi
Yudi Zhang
Biwei Huang
Yali Du
Venue: NeurIPS 2025 spotlight ()
Keywords: Multi-Agent Reinforcement Learning
Networked Systems
Domain Generalization
Domain Adaptation
Causal Structure Learning
Primary area: reinforcement_learning
TL;DR: We propose a generalizable multi-agent reinforcement learning framework exploiting causal structures, enabling efficient adaptation across networked systems under domain shifts, validated by theoretical guarantees and numerical experiments

Abstract: Large‑scale networked systems, such as traffic, power, and wireless grids, challenge reinforcement‑learning agents with both scale and environment shifts. To address these challenges, we propose \texttt{GSAC} (\textbf{G}eneralizable and \textbf{S}calable \textbf{A}ctor‑\textbf{C}ritic), a framework that couples  causal representation learning with meta actor‑critic learning to achieve both scalability and domain generalization. Each agent first learns a sparse local causal mask that provably identifies the minimal neighborhood variables influencing its dynamics, yielding exponentially tight approximately compact representations (ACRs) of state and domain factors. These ACRs bound the error of truncating value functions to $\kappa$-hop neighborhoods, enabling efficient learning on graphs. A meta actor‑critic then trains a shared policy across multiple source domains while conditioning on the compact domain factors; at test time, a few trajectories suffice to estimate the new domain factor and deploy the adapted policy. We establish finite‑sample guarantees on causal recovery, actor-critic convergence, and adaptation gap, and show that \texttt{GSAC} adapts rapidly and significantly outperforms learning-from-scratch and conventional adaptation baselines.

Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems
Hao Liangh, Shuqing Shi'; Yudi Zhang", Biwei Huang', Yali Du
King's College London Eindhoven University of Technology "The Alan Turing Institute
University of California, San Diego
Abstract
Large-scale networked systems, such as traffic, power, and wireless grids, challenge reinforcement-leamiie agents with both scale and environment shifts. To address these challenges, we propose GSAC (Generalizable and Scalable Actor- (ritr-Ciitic),, framework that couples causal representation learning with meta actor-critic learning to achieve both scalability and domain generalization. Each agent first learns sparse local causal mask that provably identifies the minimal neighbor- hood variables influencing its dynamics, yielding exponentially tight approximately compact representations (ACRs) of state and domain factors. These ACRs bound the error of truncating value functions to k-hop neighborhoods, enabling efficient learning on graphs. meta actor-critic then trains a shared policy across multiple source domains while conditioning the compact domain factors; at test time, few trajectories suffice to estimate the new domain factor and deploy the adapted policy. We establish finite-sample guarantees causal recovery, actor-critic con- vergence, and adaptation gap, and show that GSAC adapts rapidly and significantly outperforms learning-frood
and conventional adaptation baselines.
Introduction
Large-scale networked systems, such as traffic networks 41. power grids 3. and wireless commu- nication systems 44, present significant challenges for reinforcement learning (RI.) due to their massive scale, sparse local interactions, and structural heterogeneity. These characteristics impose two fundamental difficulties: scalability and generalizability. On one hand, the joint state-action space grows exponentially with the number of agents, making conventional RL algorithms [29] computationally intractable. On the other hand, real-world networked systems often experience environment shifts and structural changes, necessitating algorithms that can generalize and adapt efficiently across different environments. Therefore, natural question arises:
it feasible to design a provably generalizable and scalable MARL algorithm for networked system?
While this question has attracted increasing attention in single-agent RI. domaín generalization literature [39 10 6). its resolution remains open in the multi-agent reinforcement learning (MARL) context, especially for learning in networked system. In this work, we provide an affirmative answer by developing munsaallty-nsspire framework, GSAC (Generalizable and Scalable Actor- ector-Critic), which couples causal representation learning with meta actor-critic learning to achieve both scalability and generalizability. We summarize main contributions follows.
We establish structural identifiability in networked MARL, providing the first sample complexity results of causal mask recovery and domain factor estimation.
"qqual contribution.
Correspondence to: Hao Liang (hao. liang@kcl..... ac. uk).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We introduce efficient algorithms to construct approximately compact representations (ACRs) of states and domain factors. ACR improves both scalability, by significantly reducing the input dimensionality required for learning and computation, and generalizability, by isolating the minimal and most informative components. This approach may be of independent interest.
We propose a meta actor-critic algorithm, which jointly learns scalable localized policies across multiple source domains, conditioning on the compact domain factors to generalize effectively.
We provide rigorous theoretical guarantees finite-sample convergence and adaptation gap, and empirically validate our method on two benchmarks demonstrating rapid adaptation and superior performance over dearning-from from-scratch and conventional adaptation baselines.
1.1 Related works
Networked MARL with guarantees. Our work is closely related to the line of research on MARL in networked systems [23 17 24). These studies leverage local interactions to enable scalability, proposing decentralized policy optimization algorithms that learn local policies for each agent with finite-time convergence guarantees. In particular, [24] provides the first provably efficient MARL framework for networked systems under the discounted reward setting. The works in (23 and
[7] extend this framework to the average-reward setting and to stochastic and non-local network structures, respectively. However, to the best of our knowledge, both the design and theoretical analysis of policies that are simultaneously generalizable across domains and scalable in networked MARL. Our work fills this gap by introducing principled framework that achieves provable generalization and scalability via causal representation learning and domaain-onniitooned policy optimization.
existing methodology addresses
Domain generalization and adaptation in RL. RL agents often encounter environmental shifts between training and deployment, prompting recent efforts to improve generalization and adaptation. [10 proposes learning factored representations along with individual change factors across domains, while 7 extends this approach to handle non-stationary environments. [22] 33 aim to enhance generalization unseen states by eliminating redundant dependencies between state and action variables in causal dynamics models. Beyond causal approaches, a substantial body of work focuses on learning domain-invariant representations without causal modeling, such as bisimulation metrics relevant structure while filtering out nuisance features. Causal represen- tation learning has also been applied to model goal-conditioned transitions (6), offering theoretical guarantees via -anccure-aware meta-lcarning (10), and credit assignment 442 32). However, these works focus solely on agggee settings or multi-agent settings with a limited number of agents. In contrast, we provide the first sample complexity guarantees for structural identifiability in networked MARL and introduce ACRs, a novel mechanism that enables both scalable learning and provable generalization across domains.
39 that preserve
detailed discussion of related work is provided in Appendix A
Preliminaries
For clarity, we provide a complete table of notation in Appendix B
Networked MARL.. We consider networked MARL represented as a graph & ( where M = =,, ................................ n denotes the set of agents and & C M x V encodes the interaction edges. Each agent i N observes local state s; S; and selects an action a; E A,, forming the global state s = (s),..., ) E == Si x .... x S,, and the joint action a (a),.... an) E A Ai x x An. At each time step f, the system evolves according to the following decentralized transition dynamies:
P(s(t ((((++)){((),a)))) a())
II P.(s:((+1) sx)
where N, S A denotes the set of neighbors of agent in the interaction graph, including i itself, and sx; (1) collects the states of agents in M;; at time I. Each agent adopts a localized policy n parameterized by 0; E O; which specifies a distribution over local actions conditioned on its local neighborhood state: ny (a; sx;). Agents act independently according to their respective policies. We write 0 (01..... ,,,) to denote the tuple of all local policy parameters, and define the joint

---

policy as (a s) := -T ". (a, sx.). We only consider localized policy and use 0 and π interchangeably throughout the paper when referring policies.
Each agent receives a local reward ra(S; a;) depending on its local state and action, and the global reward is defined the average across all agents: r(s, a) = n Ei= r:(8;, a.). The goal is to learn a set of localized policies 0 that maximize the expected discounted sum of global rewards, starting from an initial state distribution po:
max (()) := Es-pp Ea (()),, s(t++) (((()(()))))) r'r(s(t). = lE
(2)
For clarity, we assume the reward function known; however, our framework and analysis readily extend to the setting with unknown rewards.
Truncation efficient approximation in networked MARL A central challenge in applying reinforcement learning to networked systems is the curse of dimensionality: while each agent's s local state and action spaces are relatively small, the global state and action spaces grow exponentially with the number of agents n. This renders standard RL methods computationally intractable at scale. 244 exploit the local interaction structure and demonstrate that the Q-function exhibits exponential decay with respect to graph distance. This property enables a principled truncation of the Q-function for scalable approximation. Specifically, for state-action pair (s, a) and a policy
Q (s. a) = Ea(t) (+(()))
rr(s(t), a(t) s(0) = s, a()) = a
(3)
Bn(t) (+())=(((()())))
(1) a; (1)) s(0) s, a(0) = a = 72
O (s, a).
72
For an integer x 2 O, NV," denote the k-hop neighborhood of agent i, and define N := M W as the set of agents outside this neighborhood. We write the global state and action as s (SNy= Swz, and a (aw;,aw*)) awa,), respectively. (24] show that for any π,, agent N, and any tuples Syre SNE, Sy aNy avs, a've, O satisfies the exponential decay property:
Q (SAY, SAC awy awz.) O (SNY, swn, aNY alya
(4)
where F is the upper bound the reward functions. Thus the influence of distant agents QT diminishes rapidly with graph distance. This motivates the use of a truncated Q Q-function:
Q (SNY, aw:) == Q7 (SNY, BNC, aNy aN=
(5)
where Sys, and aw*, are fixed (and arbitrary) placeholders for the unobserved components. Due to exponential decay, the truncated Q-function approximates the true Q-function with bounded error:
(A QT (SNr. aw:) QT (s. a) I ا Furthermore, this truncated approxima- tion can be directly used in actor-critic updates, enabling scalable policy gradient estimation. The approximation error in the truncated policy gradient is also exponentially smal [] Crucially, the truncation technique has significantly reduced input dimensionality, making it much more efficient to compute, store, and optimize in large-scale networked settings.
Networked MARL under domain generalization The standard formulation of networked MARL, as reviewed above, assumes a fixed environment or domain. We extend this framework to incorporate domain generalization, where the environment may shift between training and deployment. Each environment E (ww) is characterized by its own transition dynamics (see Equation (6. and is uniquely specified by latent domain factor dynamics for agent i. To model structured variation across environments, we define the local generative process for each agent i. Let s; (si,1,...,, 88,,;) where dł is its dimensionality. The state and reward for agent i evolve as:
(Wi)ie_r Ωt where w; encodes the domain-specific
$i. + = fe, (c773 @ (t), of (t), c o wi (., (t),
(6)
"eee Appendix C detailed review of key concepts in networked MARL.
3

---

where @ denotes element- wise multiplication, and denotes i.id. stochastic noise. The binary vectors are causal masks, indicating structural dependencies between variables. Crucially, the functions fijj and the causal masks c c' are shared across all environments, and only w; varies. This decomposition allows us to disentangle invariant causal structure from domain- specific variations.
We consider training setup with M source domains (Ei,..., ΣM) and a target domain M each drawn i.i.d. from an unknown domain distribution wm D for m -,.... M + The goal is to learn networked MARL policies from the source domains that generalize effectively to new, unseen target domains, using as little adaptation data as possible.
3 Approximately compact representation
In networked MARL, each agent's truncated value function and localized policy are given by OF : :Sy, x AvT + R and T. : SN, - A(A), respectively. The input dimensionality of the value function is dim( (Sx; x AvT) = EjEN= - (dz + ) and that of the policy is dim(Sx;) = = EEE,, dj. While truncation to the k-hop neighborhood significantly reduces the input size compared to the global space dim(S x A) = high. To further mitigate this issue, we propose constructing ACRs by leveraging the identifiable causal masks c; for each agent i, as defined in Equation 6 Specifically, we use the causal structure to extract reduced set of relevant variables from s,y, denoted Sx C s,y- which preserves the predictive information for the value function. This leads to a strictly smaller input space dim (x²) < dim(sN ). with approximation error that decays exponentially in k.
when node degrees -Eje.v jejer// (d) + ddf), the input dimensionality can still be large, especially
Moreover, we extend this ACR framework to include the policy inputs and domain-specite factors, enabling efficient transfer across domains. The resulting ACR framework improves both scalability, by significantly reducing the input dimensionality required for learning and computation, and gener- alizability, by isolating the minimal and most informative domain speciiii components relevant to each agent. In the following subsections, we detail the algorithmic construction of ACRs, beginning with the eeviomenvronmmet setting. For clarity, we assume the causal masks and domain factors are given; their identification and the estimation of domain factors are addressed in Section 5
3.1 ACR for fixed environment
We begin by formalizing the notion of an ACR for truncated Q-functions.
Definition (Value ACR). Given the graph y encoded by the binary masks c, for each agent i and its k-hop neighborhood Ag", we recursively define the k-hop ACR siz (t) as the set of state components
S satisfying least one of the following:
has direct link agent i's reward ri i.e. c7. 1, and Wy = U3et7r=115.5
* for 1 S k² K: "Na' (t) has a link to s (t+
+ 1) such that s "ae' E so App
Figure Jillutaatsss how the proposed ACR framework identifies the key state components influencing an agent's reward through local causal structure. At each time step, the agent's reward r; depends directly on a small subset of neighboring state variables at the pre- vious time step. These relevant components are recursively traced backward along causal edges, yielding a compact representation sy; within the x-hop neighborhood. The figure also visualizes how the cumulative discounted rewards r; (t), (t,(tt+).......... rr,,, (t++)) are aggregated, emphasizing that only the identified influential components contribute significantly to the value function. This provides the structural foundation for the exponentially tight ap- proximation guarantees in Proposition I] Based on this, we define the approximately compact Q-function as:
111(t 1)
4/t 1)
Sui(t- 1)
Y (
v5 + x)
Qu (S aNF = Q7 SNT .
(7) Figure 1: Illustration of ACR.
where S / denotes fixed, arbitrary values for the irrelevant components. The approximation error of Q compared to the full Q-function is exponentially small in as shown below.

---

Proposition (Approximation error of value ACR), For any agent é, policy x, and Ai 2 0. the approximation error between O and Q satisfies:
2F
Q7 si aN::) O (s & 1 mx+
sup (
The proof is deferred to Appendix We now define ACRs for policies.
Definition 2 (Policy ACR). Given the graph & encoded by binary causal masks c, the ACR for agent i's policy over its neighborhood No is defined recursively using Algorithm (see Appendix [
Algorithm takes as input the causal masks CNy, and the local states SN,, and outputs a compact representation Sy, C S&G such that dim(s); < dimmss;)). We then define the approximately compact policy At S + AA: by # (-1 (x)) == T; (.1 , /xN.) where /x;/s8; are fixed values for the non-influential components. By combining the value and policy ACRs, we define
the doubly compact Q-function Q ($pp awx) which reduces input complexity while retaining performance guarantees.
Proposition 2 (Approximation error). For any policy let म be the corresponding approximately compact policy constructed using Algorithm for any i, and i 2 0, we have
S 3 F x+1
sup SSx)
Q7 (sig; aNT)
Q, (s, a)
The proof of Proposition 2 is deferred to Appendix These results show that identifying ACRs enables dramatic reductions in the state-space dimensionality for each agent, making learning and computation tractable even in large-scale networks. particular, the effective input dimension satisfies v | I. In the next subsection, we extend ACR construction to domain factors,
enabling generalization across multiple environments.
3.2
ACR for efficient domain generalization
We now extend the ACR framework to support generalization across environments characterized by latent domain-specife factors. Given policy æ and environment w = (Wi)se.r, the environment- conditioned Q-function is defined as
Q7 (s. a, w) = ((0)))/(((((()))a()())))))<))))) a()).(())
(sr,(s s()) = s, =
We show (Lemmal) 7 that this w-conditioned Q-function retains the exponential decay property.This motivates defining a truncated w-conditioned Q-function by fixing irrelevant variables:
Q (sNg, aNT wxx-) Q (SNG Sw aNT: aN", WNT W
(8)
We consider w w-conditioned policies ग = (TI,..., Th) such that each π; : SN x ΩNZ + A i... (t) xz( (-Iss, (t), w. To reduce the input space further, we define a domain-factor ACR w (Definition 5) which identifies the minimal subset of WAY influencing the reward. Using both the value ACR and domain ACR, we define the approximately compact w-conditioned Q-function QF ( aNy, WN; (Definition Similarly, define policy ACRs for w w-conditioned policies (Definition] 77.. and let स denote the corresponding approximately compact policy. We then obtain a fully compact representation O of Q-function, which provably approximates the true Q-function. Proposition 3. For any policy æ, and any i, we have
3F 11-7"
Q ( aNy wxx) Q7 (s, a. w)
x+1
sup (Sx
The proof of Proposition 3 is deferred to Appendix E Proposition 3 shows that it suffices to operate on the key components the r-hopp state and latent domain factors, which substantially reduces input dimensionality. As result, both training and test-time inference become significantly more scalable-without sacrificing theoretical guarantees. In the next section, we present our main algorithm, which leverages these ACRs to achieve generalizable and scalable networked policy learning. For simplicity, we will omit "approximately compact" when referring to the functions and policies Q, knowing that all components operate the identified ACRs.
5

---

Generalizable and scalable actor-critic
4.1 Roadmap
We propose GSAC (Generalizable and Scalable Actor-Critic) a principled framework for scalable and generalizable networked MARL. Our GSAC framework (Algorithm [ integrates causal discov- ery, representation learning, and meta actor-critic optimization into unified pipeline, with each phase supported by theoretical results. Phase 1 (causal discovery and domain factor estimation) is underpinned by Theorem | and Propositions 45 which establish structural identifiability and sample complexity guarantees for recovering causal masks and latent domain factors. Phase 2 (construction of ACRs) leverages the causal structure build compact representations of value functions and policies, with bounded approximation errors rigorously characterized by Propositions 3 Phase (meta actor-critic learning) performs scalable policy optimization across multiple source domains. with convergence of the critic and actor updates guaranteed by Theorem 2 (critic error bound) and Theorem 3 (policy gradient convergence). Finally, Phase (fast adaptation new domains) exploits the learned meta-policy and compact domain factors to achieve rapid adaptation, where the adaptation performance gap is formally controlled by Theorem[] Together, these results demonstrate that each algorithmic component is theoretically justified and collectively leads to provable scalability and generalization in networked MARL. Figure 3 Visually illustrates the GSAC pipeline.
4.2 Algorithm overview
GSAC (Algorithm
D consists of four sequential phases:
Phase 1: Causal discovery and domain factor estimation. In each source environment, agents estimate their local causal masks and latent domain factors to disentangle invariant structure from domain-specific variations; details are deferred to Section
Phase Constructing ACRs. Using the recovered causal masks, each agent constructs ACRs for value functions and policies as described in Section 3 These ACRs significantly reduce the input dimensionality while preserving decision-relevant information for the following phases.
Phase 3: Meta-learning via actor-critic optimization. Agents are trained on across M source environments by optimizing policies through local actor-critic updates using ACR-based inputs. We provide a detailed description of this procedure in the following Section
Phase 4: Fast adaptation to target domain. In new, unseen environment, each agent collects a few trajectories and estimates its domain factor WM+,, The learned meta-policy ., conditioned on the adapted ACR input ( M+1) for immediate deployment. This process allows efficient generalization without requiring further policy training from scratch.
((K) then
4.3
Key idea: meta-learning via actor-critic optimization
At each outer iteration a source domain m(k) is sampled and each agent roll out trajectories using their current policies n; 0(k) which are conditioned on the compact ACR inputs (s ( These interactions are used to update both the critic Q and the actor e in a decentralized yet coordinated manner, enabling policy generalization across domains.
Critic update. Each agent maintains local tabular critic Q, over its the ACR input space Si X AuF x pe.. At every inner iteration t, the critic is updated using cemporal-differeee (Line 16): the Q-value for the current late-action-domall triple is updated toward a bootstrap target composed of the received reward and the next-step value. All other Q-values remain unchanged. This TD update leads an estimate of truncated Q-function for current domain.
(TD) learning
Actor update. After completing an episode, each agent aggregates the Q-values of agents within its и-hop neighborhood and weights them by the gradient of the log-policy at each timestep. The actor parameters 0 are then updated using stochastic gradient ascent with stepsize 7) =n/ n I.
We present the finite-time convergence and adaptation error bounds of GSAC in Section 6 Prior to
that, we discuss causal discovery and domain factor estimation in Phase / in Section 5

---

Algorithm GENERALIZABLE AND SCALABLE ACTOR-CRITIC
1: Input: (0): parameter K; T, length of each episode; stepsize parameters h, to. 7).
2: for source domain index m = 1, ........ M do Pl: causal recovery and domain estimation
3: Sample w D, each agent estimate the causal mask e; and domain factor a
4: end for
5: for each agent i do
P2: approximately compact representation
6: Identify Size f- (C, and sy < ACR (c,
7: Identify WNr + ACR@(C, i, i) and W.."" ACR, (c, i) for each m 1,2, ,M
8: end for
>> P3: meta-learning Sample domain index mk) {l...., K}. set do am(()) sample s(0) Po 11: Each agent takes action a, (0) ~ 0.(k) (- Sx; (0), 3, and receive reward r;
9:
for k= 0.......... 1
Initialize critic Q R Axy xfxx to be all zeros
12:
13:
for = to T
14:
Get state (t).. take action a, (t)
0.(k) (: sir R;)), get reward rt())
Update Q-function with stepsize 04-1 + 1-1+1
(5 (1-1), aN? (1- (N))), + (1 a- 1) (5 (1- aNY (1-1),
+ a (ri(t-1) + + - (s); (4), av. (4) (x:))
( aNs wx²) < Q:- (s aNr Rgt), , for (sygs aw;) + (si: (t- 1), aN, (t-1))
16:
end for Each agent estimates policy gradient:
(k) Σ xt. Σ
(s3y (t), (x), (t) Vo. log t (:(M) (a;(t) (t),
n
JENT
Update policy: (,(k 1) (k) + (k))((k with stepsize Tes
vk+I
19: end for
20: Collect few trajectories ((s(t, (t), + 1) in the domain
P4: generalization
21: Each agent estimates new domain factor as M+1 and deploy policy π. (118, (
((K)
Causal recovery and domain factor estimation
In this section, first discuss the computational overhead of causal discovery and ACR construc- tion, and then provide the theoretical guarantees for causal recovery and domain factor estimation in Theorem TT and Proposition 5 Phase 1 (causal discovery) and Phase 2 (ACR construction) introduce upfront costs that are only one-time, local, and amortized. In particular, they are one-time preprocessing steps for each source domain and do not need be repeated during meta-training or adaptation. Both steps are local per agent and over small neighborhoods, parallel across agents, and the results are re-used for the entire meta-training horizon and for adaptation.
Theorem 1 (Structural identifiability in networked MARL). Under the standard faithfulness assump- tion, the structural matrices e, in are are identifiable from the observed data.
Theorem T guarantees that the underlying structure, that encodes how neighboring states, local actions, and latent domain factors affect local transitions, be uniquely recovered from trajectories under standard causal discovery assumptions. The proof is deferred to Appendix(F Furthermore, Proposition 4 provides a sintle guarantee for recovering the local structural dependencies.
Proposition 4 (Informal), Under standard assumptions, including faithfulness, minimum mutual in- formation for true causal links, bounded in-degree dmaxx sub-Grab Gaussian noise, and Lipschitz continuity of the transition function, the sample complexity to recover the causal masks satisfies
dim(sN.) dmax log (di(iisss))) n2/8)
X2

---

with probability at least - 8, where À quantifies signal strength.
The required sample size scales almost linearly with the number of observed variables dim(sx,. the sparsity level dmax. and decays quadratically with the strength of causal signals The formal statement and proof of Proposition and discussions on the imposed assumptions are deferred to Appendix F
Proposition 5 (Informal). Suppose the causal masks are recovered and the domain -eeeenden transition dynamics are identifiable. Assume that distinct domain factors induce distinguishable state transitions in total variation, and that Ω2 is compact with diameter Dn. Then, with probability at least 1- 6, the estimated factor w given a trajectory of length T, generated from true factor w' satisfies
|| w" 122 S (Te) O (1 Do log(nTT/60)
(10)
Te
The estimation error decays as D(./VT.) with high probability, and depends logarithmically on the number of agents. The result highlights that domain generalization can be performed efficiently with only few samples. The formal statement and proof of Proposition are deferred to Appendix [F]
Convergence results and adaptation gap
For clarity of analysis, first establish convergence and adaptation guarantees for ACR-free variant of GSAC (Algorithm detailed in Appendix [D.. Theoretical results for GSAC with ACR follow corollaries. We define the expected return of a policy parameterized by O using estimated domain factor w' in a true environment w:
:) = Eampo Ea(1)~ ((td))))"))(((t)_PPPPd000... J[0, w':
Σ 2rr((((), s(0)
(1))
where TT (als, I x. (a/ss,, WNr) is the joint domain-coodiiionn conditioned policy. For notational simplicity, we write [(, w) = [(, w: w) when the estimated and true domain factors coincide. To this end, for domain generalization objective under domain distribution D is:
max J(0) = Ew- [ w)]
6.1 Convergence
We begin by introducing standard assumptions (Assumption 1 used in networked MARL. 24 as well as additional ones (Assumptionf (] for the domain generalization setting.
Assumption 1. Rewards are bounded: 0 S r,(s,, a:) S F for all i, S; a,. The local state and action spaces satisfy |S,] < S and |A.1| < A.
Assumption 2. There exist E N and E (0, such that, for any 0 and w. the local transition probabilities satisfy P. ((SN:- an; (r)) (s', a') (s(1), a(1) (s, a)) 2 for all i and all (s'' (s, in the appropriate product spaces.
Assumption 3. For all i, SAC WNi. a, and 0,, VV, log 7. [s,s wx)il < L,. |V, (als w)]l < L VEL L3. and VJ((I) is L' L'-iisschitz continuous in A.
Assumption 4. Each agent's parameter space O C R24 compact with diameter bounded by De- Assumption 5. For all i: (i) + (t), a, (f), wi) is Lp-Lisschitz in wy; (ii) Q4 (s. a w) Lo-Lipschitz in w; (iii) Vo, [0, w) is Ly-Lipschitz w. Assumption 6. The domain factor space Ω2 is compact with diameter bounded by Dn-
Discussion assumptions. Assumption 1 are standard for proving convergence of networked
MARL algorithms, without consideration of domain generalizationn/daattiio [23 24 To ac- count for generalizability across domains in networked systems, we introduce additional Assumption 6 regarding the latent domain factor. Assumption 5 is similar with Assumption 3 which imposes the smoothness w.r.f. the domain factor, while Assumption 3 imposes the smoothness w.r.t. the actor parameter Assumption 6 is a regularity assumption to ensure the compactness of domain factor space. We provide detailed discussion on these assumption in Appendix G

---

Critic error bound. We first analyze the inner-loop critic update. Fixing any outer iteration k. and omitting & from the notation, we establish the following result. Theorem shows that the inner loop converges to an estimate of Q; with steady-state error decaying with 1/v and exponentially in The proof and formal statement of Theorem are deferred to Appendix
Theorem 2 (Critic error bound, informal). Under Assumptions
I 6 and for any 8 E (0, 1), if the
critic stepsize is set as a, = (1 + (n) and the domain factor & is estimated from Ta trajectories, then with probability least 1 8, the critic estimate satisfies:
Ca t to
Co
2cpr+1 (1-7)2
Iog(nTT/0)
(Q.(s,a,w) w) Of (SNTe avy WN
+
+ +C
Te
where Ca. C. and Cw are constants. These will be further characterized and discussed in Section 6.3 where we analyze the additional benefits of incorporating ACR.
Actor convergence. Based on the critic bound, we derive the bound on policy gradient updates. The first term, of order (11/VK + 1), vanishes as the number of outer iterations increases. The remaining three terms correspond to different sources of error: the second arises from neighborhood truncation and decays exponentially with k; the third stems from estimation of the domain factor w, with error decreasing 1/ 1/TTe: and the fourth reflects the approximation of the domain distribution
DD using only M sampled source domains, decaying with 1 1/VM. M. The formal statement and proof of Theorem are deferred to Appendix G
Theorem 3 (Policy gradient convergence, informal). Under Assumptions 16 for K 2 and sufficiently large T, if the actor and critic stepsizes are chosen appropriately and domain factors are estimated from Te samples, then with probability at least 8:
k== Nk (
(VX++ x+1 VK+1
S Ô
+V
M
k=
6.2 Generalization
In Phase 4, for a new domain w M+1 D. ACR-free GSAC collects To trajectories, estimates M+1, and deploys the policy (-K)s as M+)). The expected return is:
J(B(K), w M+ = (0)~p) Ba(()
Σ x'r
[==
Theorem (Adaptation guarantee). Under Assumptions 16 with probability at least 8:
log(n/8)
E [J(((K,, & M+ ( [(K]] 2 ((((K) - Ia Cw
To
Theorem establishes that the adaptation gap decreases a rate of O(IvVT.) relative to the return of the meta-trained policy. The proof is deferred to Appendix G
6.3 Additional gains from ACR
Notably, ACR introduces only a constant-factor increase in approximation error over truncation (multiplicative factor 3), and thus all ACR-free convergence and generalization results naturally extend to GSAC. Beyond computational gains discussed in Section
3 ACR also improves sample efficiency.
In Theorem 2 the explicit constants defined C. with ē stepsize a r4:0
Gz Ih [log( 3 f(x) log SA] 2F. C² T- max( 16Ehr 2F (T to)). and Cl LQCwv Dr. and the A satisfying h max(2, -- to max(2h, 4oh, r). By leveraging ACR. the effective dimensionality of the k-hop state space is dramatically reduced, leading to smaller 7 and larger in Assumption 2 This reduces Ca. C² and l.. enabling faster convergence and reducing the required inner-loop iterations to achieve given accuracy Q-value estimation of within critic, thereby decreasing the total sample complexity.

---

(a) grid size 3 (16 agents)
(b) grid size 4 (25 agents)
(c) grid size (36 agents)
Figure 2: Adaptation comparison for different grid sizes in wireless communication benchmarks.
7 Numerical experiments
We evaluate GSAC on two standard benchmarks for networked MARL algorithms: wireless commu- nications [44] 24 36 and traffic control [30 24] (the latter deferred to Appendix (..2). A detailed description of the experimental setup and comprehensive results are provided in Appendix [H] We consider M 3 source domains, each with domain factors E {0.2, 0.5, 0.8). while the target domain uses Warget = 0.65 unless otherwise specified. In each domain, GSAC is trained for K outer iterations (depending estimation, we collect T, 20 trajectories per domain. To evaluate adaptation performance in the target domain, we compare GSAC against three baselines:
convergence), with inner loop horizon T 10. For domain factor
GSAC (ours): Learn (e(a|s, w) by optimizing over source domains. At test time, estimate w' and directly deploy Ta (alss, in the target domain without further training.
SAC-MTL (multi-task): Learn To(a)s, 8,22, z). where z denotes the pre-specified one-hot encoding of each source domain, and jointly optimize 0 across domains (37;; deploy TT (a]ss, z') in the target domain.
Train a single policy (a(a)s) across source domains without domain-factor conditioning and fine-tune 0 in the target domain.
SAC-FT (fine-tune):
SAC-LFS (learning from scratch): Train (e(a|s) the target domain without prior meta-training.
To evaluate both scalability and generalizability, we vary the grid size of the wireless communication network (grid size {3, 4, 5)), which affects both the number of users and the connectivity structure (see Figure 55. As shown in Figure 2 and Figure [6] GSAC consistently maintains high training and adaptation performance across grid sizes, demonstrating its scalability and generalizability. In particular, GSAC achieves the best few-shot performance (Episodes 1-30), reflecting rapid adaptation from minimal target data. contrast, SAC-MTL exhibits moderate performance during the early adaptation phase but improves steadily over time. This is because multi-task learning leverages shared structure across source domains, yet lacks explicit domain-factor conditioning, leading to slower adaptation compared GSAC. SAC-FT performs worse in the early phase due to its need to fine-tune policy parameters directly in the target domain, resulting higher sample complexity. SAC-LFS suffers the slowest convergence and lowest overall return, underscoring the importance of meta-training and domain factor conditioning for efficient cross -oosinn adaptation.
Conclusions and future work
We presented GSAC, causality-aware MARI. framework that integrates ACR construction with meta actor-critic lcarning, achieving provable scalability, fast adaptation, and strong gencralization across domains. We established quasi-linear sample complexity and finite-sample convergence guarantees, and showed empirically that GSAC consistently outperforms competitive baselines on challenging networked MARL benchmarks.
The main limitation of our current work is that GSAC has only been evaluated on tabular and fully observed benchmarks. Nonetheless, it establishes principled foundation for practical control of large-scale networked systems. Promising future directions include: (i) extending GSAC to continuous state and action spaces with function approximation based on ACRs [I0]; (ii) broadening empirical
evaluation to more diverse and large-scale networked systems [19 and (iii) incorporating partial
observability into the framework.
10

---

Acknowledgments
We would like to thank all the anonymous reviewers for their careful proofreading and constructive feedback, which have greatly improved the quality of this work. In particular, we thank one reviewer for the insightful suggestions on adding the roadmap linking theorems and algorithms in Section 4 as well as for recommending additional discussions reviewers for suggesting the inclusion of two additional baseline methods and an extra benchmark, along with further discussions on the assumptions underlying Theorem [ and Proposition + We are grateful to Chengdong Ma for valuable initial discussions. This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/Y003187/1 and UKRI849].
Assumptions [ We also thank the other
References
[1] Suzhi Bi, Yong Zeng, and Rui Zhang. Wireless powered communication networks: An overview. IEEE Wireless Communications, 23(2):10-18, 2016.
[2] Ioana Bica, Daniel Jarrett, and Mihaela van der Schaar. Invariant causal imitation learning for generalizable policies. arXiv preprint / 1.01489, 2023.
[3] Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, 13(4):2935-2955) 2022.
[4] Jiayu Cheng, Hao Liang, Xiaoying Tang, and Shuguang Cui. An economic and low-carbon dispatch algorithm for microgrids with electric vehicles. 2024 3rd International Conference on Power Systems and Electrical Technology (PSET), pages 556-561. IEEE, 2024.
[5] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: cvar optimization approach. Advances in neural information processing systems, 28, 2015.
[6] Ruohan Ding, Zheng Zhou, Anurag Gupta, Chelsea Finn, and Igor Mordatch. Generalizable goal- conditioned reinforcement learning via causal world models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[7] Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non- stationary reinforcement learning. Advances in Neural Information Processing Systems, 2022.
[8] Hairi, Z. Zhang, and Liu. Local td-update is more sample-efficient than batching for marl policy evaluation with average reward. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2024.
[9] Yulong Hu, Yali Du, and Sen Li. Real-time coordination of human couriers and drones for on- demand food-delivery platforms: A multi-stage risk-aware multi-agent reinforcement learning framework. Transportation Research Part C: Emerging Technologies, 181:105381, 2025.
[10] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
[11] Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from laterngenneuus/nonstaaionny data. Journal of Machine Learning Research, 21(8)): 1-3,, 2020.
[12] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[13] Junhyuk Jiang and Zongqing Lu. Graph neural networks for multi-agent reinforcement learning. arXiv preprint /810.09222 2018.
[14] Hao Liang and Zhi-Quan Luo. Bridging distributional and risk-sensitive reinforcement learning with provable regret bounds. Journal of Machine Learning Research, 25(221):1-56, 2024.
11

---

[15] Justin Lidard, Udari Madhushani, and Naomi E. Leonard. Provably efficient multi-agent reinforcement learning with fully decentralized communication. In Proceedings of the 2022 American Control Conference (ACC), 2022.
[16] Han-Dong Lim and Donghwan Lee. finite-time analysis of distributed q-learning. OpenRe- view, 2024. Submitted to ICLR 2025.
[17] Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in stochastic networked systems. Advances in neural information processing systems, 34:7825-7837, 2021.
[18] Chaochao Lu, José Miguel Hernández- Soriinddzz-obatoo, Lobato, and Bernhard Schölkopf. Invariant causal repre- sentation learning for generalization in imitation and reinforcement learning. In PAIR2Struct Workshop at ICLR, 2022.
[19] Chengdong Ma, Aming Li, Yali Du, Hao Dong. and Yaodong Yang. Efficient and scalable reinforcement learning for large-scale network control. Nature Machine Intelligence, 6(9):1006- 1020, 2024.
[20] Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[21] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CombridgeUnniveesityrresss 19(2);3, 2000.
[22] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems, 2022.
[23] Guanman Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learn- ing for networked systems with average reward. Advances in Neural Information Processing Systems, 33:2074-20883 2020.
[24] Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.
[25] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforrement learning via probabilistic context variables. In International Conference on Machine Learning (ICML), 2019.
[26] Jonas Rothfuss, Jesse Fu, Nikhil Mehrasa, Tianhao Yu, and Sergey Levine. ProMP: Proximal meta-policy search. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[27] Andrzej Ruszezyáski. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125(2):235-261, 2010.
[28] Miloš S. Stanković, Marko Beko, Nemanja Ilić, and Srdan S. Stanković, Multi-agent actor-critic multitask reinforcement learning based on gtd(1) with consensus. In Proceedings of the 6lst IEEE Conference on Decision and Control (CDC), pages 4591-4596, 2022.
[29] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[30] Pravin Varaiya. Max pressure control of network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36:177-195, 2013.
[31] Andrew Wagenmaker, Kevin Huang. Liyiming Ke, Kevin Jamieson, and Abhishek Gupta. Overcoming the sim-to-real gap: Leveraging simulation to learn to explore for real-world rl. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[32] Ziyan Wang, Yali Du, Yudi Zhang, Meng Fang, and Biwei Huang. Macca: Offline multi-agent reinforcement learning with causal credit assignment. Transactions on Machine Learning Research.
12

---

[33] Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for tusk-independent state abstraction. International Conference on Machine Learning, pages 23151-23180, 2022.
[34] Yuming Xiang, Sizhao Li, Rongpeng Li, Zhifeng Zhao, and Honggang Zhang. Decentral- ized adaptive formation via consensus-oriente multi-agent communication. arXiv preprint vrXiv:2307.12287, 2023.
[35] Yaodong Yang, Ying Luo, Ming Li, Minne Zhou, Weinan Zhang, Junliang Wang, Yongxin Zhang, Jun Wang, and Dengyong He. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
[36] Donghao Ying, Yunkai Zhang, Yuhao Ding, Alec Koppel, and Javad Lavaei. Scalable primal- dual actor-critic method for safe multi-agent rl with general utilities. Advances in Neural Information Processing Systems, 36:36524 36:36524-36539, 2023.
[37] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[38] Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference Machine Learning (ICML), 2018.
[39] Shixiang Zhang, Shimon Whiteson, and Yuxi Yu. Learning bisimulation metrics for general- ization in reinforcement learning. In International Conference on Machine Learning (ICML). 2021.
[40] Wanpeng Zhang, Yilin Li, Boyu Yang, and Zongqing Lu. Tackling non-stationarity in reinforce- ment learning via causal-origin representation. arXiv preprint arXiv:2306.0277 0273000002777,7. 2023.
[41] Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforeement learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 7(1):1-51, 2023.
[42] Yudi Zhang, Yali Du, Biwei Huang. Ziyan Wang, Jun Wang, Meng Fang, and Mykola Pech- enizkiy. Interpretable reward redistribution in reinforcement learning: causal approach. Advances in Neural Information Processing Systems, 36:20208-20229 2023.
[43] Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representations (ICLR), 2020.
[44] Alessandro Zocca. Temporal starvation in multi-channel esma networks: analytical frame- work. ACM SIGMETRICS Performance Evaluation Review, 46(3):52-53, 2019.
13

---

A Related works
Networked systems are prevalent across a wide range of real- domains, including power grids A transportation networks 91 and wireless communications [1 44). Our work is closely related to the rapidly growing body of research on multi-agent reinforcement learning in networked systems 23 24]. These studies exploit localized interactions to improve scalability, typically by designing decentralized policy optimization algorithms that learn per-agent policies with theoretical convergence guarantees. Notably, [24] provide the first provably efficient MARL framework for networked systems under the discounted reward setting. Subsequent extensions address average-reward formulations [23] and stochastic, non-local network topologies [17]. However, none of these methods simultaneously address both generalization across domains and scalability in large networked systems. Our work fills this gap by proposing a principled framework- built on causal representation learning and
Networked MARL..
policy optimization that achieves provable
generalization and scalability in networked MARL.
A complementary but distinct line of work focuses on fully decentralized MARL via neighbor-to- neighbor communication. [38] propose a fully decentralized actor-critic algorithm, where agents update local policies using only local observations and consensus-basee messages from neighbors, and prove convergence under linear function approximation. Other strategies such as mean- mean-field MARL K5 approximate interactions by aggregating neighborhood effects, trading some fidelity for tractability. Graph neural networks and caassage-passinng schemes have also gained traction
More recent theoretical efforts in decentralized learning emphasize convergence speed and commu- nication complexity under peer-to-peer protocols. [15) show that message passing improves regret bounds over non-communicative baselines. 228 establish almost-sure convergence for consensus-
based multi-task actor-critic algorithms. M propose a communication protocol for distributed adaptive control with provable bounds consensus error and policy suboptimality. [16] provide the first maite- analysis of distributed tabular Q-learning, linking sample complexity to spectral ceatuunication-fffciiee decentralized "D-Icarning algorithm that matches optimal sample complexity while reducing com- munication overhead.
properties of the network and Markov mixing times. Finally, (8] design
Domain Generalization and Adaptation in RL. RL agents frequently face environmental shifts between training and deployment, motivating research into policy generalization and fast adapta- tion. One line of work leverages causal representation learning to identify invariant features and disentangle spurious correlations. For example, [18] introduce a framework for learning invariant causal representations, with provable generalization across environments. Similarly, (40] trace causal origins of non-stationarity to construct stable representations. [10 propose AdaRL, which learns parsimonious graphical models that pinpoint minimal domain-specifte differences, enabling efficient adaptation from limited target data. Related efforts seek domain- invariant representations through bisimulation metrics [39], which preserve iccision-relevant dynamics while filtering nuisance fac- tors. Causal modeling has also been applied to coal-conditioned transitions [6] and structure-awar meta-learning 10). However, these methods are primarily limited to single-agent settings.
domain-invariant
Complementary to representation-base generalization, growing body of work studies risk-aware or robust RL in the single-agent setting, designed to generalize under uncertainty, distribution shift, tail-risk objectives. Representative approaches include risk-aware RL such as CVaR and entropic risk (27 5 [4, and robust MDP formulations that optimize worst-case or ambiguity-ave performance across environments 122 20 While these methods provide principled guarantees against uncertainty, they primarily focus on single-agent decision making and do not address scalability or structural generalization in multi-agent, networked systems.
Additional approaches not directly aligned with ours include meta-RL and sim-to-real transfer. Meta- RL methods such as PEARL (25). VariBAD 43), and ProMP (26 remain foundational for few-shot adaptation by optimizing meta-objectives or latent context embeddings. In sim-to sim-to-real transfer, [II demonstrate provable improvements in exploration efficiency by leveraging simulated environments during training. imitation learning, [2] propose ICIL, which learns invariant causal features to replicate expert behavior across different domains. While effective in their respective domains, these works do not address the unique structural and scalability challenges posed by large-scale networked MARL.
14

---

B
Table of notation
Symbol
Definition
Networked MARL Setting
Set of agents, indexed by A Agent interaction graph 1-hop and k-hop neighborhood of agent Agents not in .N i.e.,....... ./' \,
4 8)
N₁, N.
No
S,S,
Global and agent state space Global and agent i's action space Global state, local state of agent i, and neighbor states Global action, agent i's action, and neighborhood actions
A,A;
8, Ss; SNG
m
01
Joint policy, I "; Agent i's local policy, parameterized by 0, Joint and local policy parameters Local reward function, Global reward: E, Ti((s,, a.) Initial state distribution Discount factor
0, 0
(S a.), r(s,a)
Pa ?
Causal Modeling and Domain Generalization
Wi, w
Latent domain factor for agent i, and full domain vector
Ω. 8
Domain factor space and local domain space
D
Distribution over domain factors w
Binary causal masks (c.g., ca->x)
(.)())
Transition function for j-th state dimension of agent
(t)
Random noise transition dynamics
$i. di
j-th component of agent i's state
Dimensionality of state and action space of agent
Value Functions and ACR
(" a). Q. a Q7
Global action-value function, Local value contribution of agent Truncated Q-function, ACR-based approximate Q-function
ACR s-hop state, ACR of agent i's local neighborhood state ACR k-hop domain factor, ACR of local domain factor
ACR-based approximately compact local policy Fixed placeholders for unobserved variables in truncation
a,
Algorithm and Learning
Inner-loop episode horizon
K
Number of outer-loop meta-training iterations
Te
Number of trajectories used for estimating w during meta-training
Ta
Number of trajectories used to estimate in target domain
0g, 7e
Critic and actor learning rates
Q
Estimated critic for agent i at time
(k)
Estimated policy gradient at outer iteration A
h, to
Critic learning rate parameters: 0₂ itfe
Theoretical Constants and Bounds Upper bound on local rewards
F
Minimum signal strength for causal discovery
Da
Diameter of domain factor space
Lr. Lq; Ls
Lipschitz constants for dynamics, Q-function, gradient Gradient and smoothness bounds of policy T Minimum visitation probability and mixing time constants Constants in critic approximation bounds Confidence level and domain factor estimation eтror Exponential decay rate Q-function approximation Truncation radius (number of hops) Number of source domains Number of agents Maximum size of local state/action space
L, L'
T,T
Ca. Ch
8, 8
p
M
S.A
15

---

C Background in networked MARL
In this section, we provide background on networked MARL, beginning with the definition of the exponential decay property introduced in [23].
Definition 3 (Exponential decay property). The (c, p)-exnennttial decay property holds if, for any localized policy 0, for any E SNN E SNz, sNn,, ,8=, E SNa, aNg E Awp, ave, avv. E A Q7 satisfies,
o (NN aNr, aws, QT (SNy, Swn, aNy, alw..) S cptt1
Furthermore, [24] proves that the exponential decay property holds generally with p = for discounted MDPs.
Lemma 1, Assume Vi, r S F. Then the (1= -xxpoeenialll decay property holds.
The exponential decay property implies that the dependence Q on other agents shrinks quickly as the distance grows, which motivates the truncated Q-functions [24).
QT (NN;,aN))
Σ
Wi (SN*, aw", aN;- OT (SNY SNM, avy ave.)
8N1, ESM5, ,AAs EAN",
where w/ (SNE,, AN, aN?") any non-negative weights satisfying
Σ
(SNE, avF) 1, v (SNs, ax;) E SNe ANE-
ES AWS, EAvZ
With the definition of the truncated Q-function, when the exponential decay property holds, the truncated Q-function approximates the full Q-function with high accuracy and can be used to approximate the policy gradient.
Lemma 2. Under the (c, -exponnnnntl decay property, the following holds: (a) Any truncated Q-function satisfies,
[2 (sNp. ax:) Q2 Q7 s cp"+1
sup (
(b) Given i, define the following truncated policy gradient,
1 E Fanpp ( h,(0) 1 -
Vo, log o (a, ss)) Σ Q (SN3. (Nr))
n
JEN
Then, if Ve, log n. (a. |s))|| S Lis Va St, we have [ix((0) vo.J
S p"
Lemma 2 shows that the truncated Q-function has much smaller dimension than the true funccioon, and is thus scalable to compute and store. However, despite the reduction dimension, the error resulting from the approximation is small.
D Algorithm design
D.I Roadmap and pipeline
Figure 3 visually illustrates the GSAC pipeline, complementing the roadmap. complements
Algorithm by showing how causal recovery and ACR construction (Phases 1-2) compress the state-action space, enabling efficient meta-policy training (Phase 3), and how this meta-policy rapidly adapts to new domains (Phase 4), guaranteed by Theorems
16

---

Inputs: multi-domain trajectories
Meta-policy Te shared across domains
Thm. 1; Prop. 4-5
Prop. 1-3
Phase 2: ACR Construction
Value ACR s/'
Phase 1: Causal & Domain Recover local causal masks C....
Policy ACR SW;
Estimate latent domain factors w
Domain ACR ware
Exp--decay bounded errors
Identifuble consistent @
Low dim ACRs with bunded apprex. ernor
Thm.
Thm. 2-3
Phase 3: Meta Actor-Critic
Phase 4: Fast Adaptation
Train on M source domains with ACR
Few-shot estimate Wor+1
inputs
Deploy Te (-1 (ir+1)
Critic: TD on Q(s", a," w²)
Actor: localized PG using k-hop returns
Smulll nation gop
Convergent critic and actor updates
Output: domain condain-oonditionee policy
Figure 3: GSAC pipeline (cf. Algorithm 1.
Each phase is annotated with its supporting results.
D.2
Policy ACR
We present the complete pseudocode of the algorithm used in defining the policy ACR in Definition 2 We begin with Algorithm [] which constructs the policy ACR using variable number of steps. shown in Proposition O @ggorithm[ppreciseee
identifies the relevant components of for all agents while preserving the values of the Q-function. However, the number steps required by Algorithm 2 is problem-dependent and may vary. To address this, we propose Algorithm] 3 which constructs the policy ACR with a fixed step budget k. high level, Algorithm identifies all components of s; that influence future rewards within time steps, resulting in approximation error that decays with increasing k. This approximation guarantee formally established in Proposition
Algorithm 2 Policy ACR
Input: causal mask CiE.V'
1: // Initialization:
2: s: + Ø for all i EV
3: for agent i = 1,2, do
4: for component
1, 2, 2.... di do
5: if there is direct link from si, r. ie., o3"== = then s
U {
6: end
7: end for
8: end for
9: // Recursively identify with local communication:
10: while &i not converged for some do
for agent 1,...... do
12: agent i receives (sf)), from its neighbors and si + UVEN, {
13:
for component j = 1,2, ... ds do there an edge from si,j to (component of) Sy, then se + sf (8is) end if
15:
16: end for
17: end for
18:
end while
Output: policy ACR for each agent s
17

---

Algorithm 3 Policy ACR with finite steps
Input: causal mask (Ci)iE.N
Initializatio::
2: (1) Ø for all E N
3: for agent .,2...., do
4: for component j 1, 2. ................................ df do
5:
there a direct link from s4s5 to ie., of" = then si( + s;( U {5
end if 7: end for
6:
8: end for
9: // Recursive identification with local communication in finite steps:
10: for step 1, ...,, do
11: for agent = 1, 1,2..... do
12:
agent i receives
from its neighbors and sx, (1) UVEN, {s)
13:
st + s1)
14:
for component j = 1.2..... ................................ dī do
15:
there an edge from si,j to si, (1) then s/(++ + +;( {sa)) end if
16:
17: end for
18: end for
19: end for
20: for agent 1,2..... do
21: agent i receives (s) (x)) from its neighbors and Si; + UveNs (x)
22: end for
Output: policy ACR sy; for each agent
D.3 GSAC
We provide the complete pseudo-code of GSAC in Algorithm 4
D.4
ACR-free GSAC
We provide the complete pseudo-code of ACR-free
GSAC in Algorithm
5
E
Missing proofs in Section
E.1
Value ACR
Definition recursively identifies the key state components within the k-hop neighborhood that influence an agent's reward, yielding compact subset siz- Sy*" with an exponentially small approximation error in the Q-function. This definition assumes a fixed number of time steps k. In contrast, Definition 4 exactly identifies all state components in Sy² that influence the agent's 's reward but requires a variable number of time steps. Notably, Definition iss stricter and implies Definition [ Specifically, the former identifies only the components that influence the reward within steps, while the latter includes all components in syv". that have direct or indirect influence on the reward, possibly through longer dependencies.
Definition 4 (Value ACR). Given the graph & encoded by the binary masks c, for each agent i and its k-hop neighborhood N?K we recursively define the k-hop ACR s (t) as the set of state components SNY, satisfying at least one of the following:
8i,j has a direct link to agent i's reward r., i.e., c5 = 1, or
(() has an edge to another state component 81 (1 + 1) such that st.N (t) E S
We provide the proof of Proposition here.
18

---

Algorithm 4 GENERALIZABLE AND SCALABLE ACTOR-CRITIC
1: Input: 8,(0); parameter K; T, length of each episode; stepsize parameters h, to
2: If Phuse 1: local causal model learning
3: for source domain index = 2, ... M do
4: Sample w
5: Collect trajectories T (s (t), a (t), rm (t). (t+ 1))
6: Split 7 into T:" (s%) (t), a," (t), (t(,), (. 1)) To
t=1 for each agent
7: Each agent estimate the causal mask C, model parameter v/, and domain factor "o""
8: end for
9: Phase approximately compact representation
10: for each agent i do
11: Identify syx= -- ACR,(c, (c, i, 1)) and SN, + ACR, (c, 12,0 12: Identify WXr +- ACR i, and Ni < ACR,, (c, i) for each 1,2, ,M
m.
end for
14: // Phase 3: meta-learning
for k 0, 1,2,...... K do
16:
Sample domain index m(k) uniformly from (1.....Kt K}
17:
Set ④ = (A))
b Current domain factors
Sample initial state s(0) Po
19:
for each agent do
Take action a;(0) (0) πi; (k(k) (0),
20:
21:
Receive reward (0) = (0),
Initialize critic Q E Sw x xAy xΩN? zero end for
22:
23:
24:
for t = 1 to T do
25:
for each agent do
26:
Get state s.), take action a:(1) m;
0.(1) (())))))
27:
Get reward (t) atss
28:
at-1 = 2-1+6
Update TD target:
(t-1), aw; aw; (t- (t-1), (x) (1- (t-1), (t)
+ + (ri(t-1) 2 (S (f), aN; (v:) (1),
For all other (si; aN:) # ($ (t 1), a^p; (t-1)): 1)), set:
(s aN; (x:)) = - avr: (5
(s
31:
end for
32:
end for
33:
for each agent do
34:
Estimate policy gradient:
T
(k(k)
7
Σ JENN
O ($ (t), aNy (t), Wāx Vo, log प. 8:0k) (ar(t) (s)(t) [ 6))
==
35:
Update policy: 0,(k + 1)= = 0kk)) + (k):()), where mh
7
V
36:
end for
end for
38: // Phase 4: adaptation to new domain
39: Collect few trajectories M+1
1) in the new domain
for each agent do Infer domain factor W ( M+1
42: Deploy policy x, _(()) (-|s;, a M+1,0
end for
19

---

Algorithm 5 GENERALIZABLE AND SCALABLE ACTOR- CRITIC WITHOUT ACR
1: Input: 0c()); parameter n; T, length of each episode; stepsize parameters h, to. 7
2: I/ Phase domain estimation
3: for source domain index m = 2, M do
4: Sample wn D
5: Collect trajectories T (s (I), a (1), 8t), ())))))))))))))) 1))
T. 6: Split 7 into 7," (5% (t), a. (t), ra(t),ss) + 1)) for each agent
7: Each agent estimate the domain factor "
8: end for
9: // Phase meta-learning
10: for k = 0.1...... K do
Sample domain index mkk) uniformly from .L.....K)
Set ma))
Current domain factors
13: Sample initial state s(0) ~
for each agent do
Take action a0)) ; (K(k) ()).
15:
Receive reward (0) Ti (s))
Initialize critic Q E S.p xAye xQN? to zero end for
17:
18:
19:
for t= 1 to 7 do
20:
for each agent do
Get state (t). take action at( ~ =; (.(k)( (t)s.())()))
21:
22:
Get reward (t) (st,t,, (t))
23:
0r-1
F-1+6o Update TD target:
h
24:
(sNs (1-1), (1-1),
(I- (1-1,, WNy)
+ (i (t-1)) + " (f), aN: (1), wxy)))
25:
For all other (SN;. axp)) + (sN; (t (t-1), 1), aA7- (t 1---1 1)), set:
(s aw; WNy) Q - (sN av. wxxe))
26:
end for end for
27:
28:
for each agent do
29:
Estimate policy gradient:
lo - Σ (sNg (()))((),)w aN; (1), (x;; Vo, log "; (a.0)()(()) (s()) 61) 0.(k)
t=
JENT
30:
Update policy: 0.(k + 1) = (k) + megi (k), where v
31: end for
32: end for
33: // Phase 3: adaptation
domain
34: Collect few trajectories M+1 ((s()) + 1)) in the new domain
35: for each agent i do
36: Infer domain factor w M+1
37: Deploy policy , O(K) (+1s;, (.111
38: end for
20

---

s/5 ). (av;, aw_ ;s' (s s//s;;; and 5
Proof of Proposition
Denote
a' a'=(an; aww.) -a,,,,awe,)
Let Pt,x be the distribution of (s:(1), a. (1)) conditioned on (s(0), a0)) (s, a) under policy n, and let PL.s be the distribution of (s) a, (1)) conditioned (s(0), a()) (ss, under policy и.
Due to the local dependence structure, and the localized policy structure, Pt, only depends (SN7. aN;) which is the same (s'vz 8/xz) when S Furthermore, by the definition of ACR, we must have Pr,: = Pt,, for all t S x. per the way the initial state a), (s', a') are chosen. With these definitions, we expand the definition of Q,,
10, (8, a)
O (s',a')
E Ir'r (s)(t), (s(0), = (s, a)] E (s.(),,,, (s(0),,(0)) (s',aa)
m (s - r (s. (s. (s ai)
Σ
|T (8i, ai) - / [Es TV (si, ai)
t=x+1
Σ
PTVVV (pes)Pe.) S - -
1=++1
where TV Pb.)) & is the total variation distance between Pt, and Pl.
E.2
Policy ACR
We begin by presenting Proposition 6 and its proof, which establishes the guarantee of Algorithm and then proceed to the proof of Proposition 2
Proposition 6 (Compactness of policy ACR). Let be an arbitrary localized policy and be the corresponding approximately compact localized policy output by Algorithm for each agent i, and any (s, a) E S A, we have
Of a) = (s.
To prove Proposition 6, we start with some technical lemmas. Using the definition of policy ACR (Algorithm 2) we obtain Lemma
Lemma 3. Fix E M. For any j E [de] St.j E s if and only if either of the following holds
(i) 8i,j has an edge to rt. or
(ii) there exists i' E N such that there exists a path from st,5(t) (t) (t++))
st.(t) (+++ s1(+4)))tt+2)) ... + s(o+))(--)((+t) - rv (t+1), where i(t + () 1' and i(t m) NN for any Moreover, ((()+)) (1 m) s ((44)) for all m [1].
and for some 1.
Proof of Lemma We prove this by induction on the iterations of the while loop in Algorithm 2
Base case: After initialization (lines 2-8 of Algorithm P S contains exactly those state components
Sij that have direct edge to r; (i.e., where c 1). This corresponds precisely condition (i).
Inductive hypothesis: Suppose that after le iterations of the while loop. for any agent i and component 8i.j E &e if and only if either:
Si has direct edge to r;,
there exists path from (t) to some re (1 + m) where S k such that all intermediate state components are in their respective s" sets, and each consecutive agent in the path is neighbor of the previous agent.
21

---

Inductive step: Consider the (k 1)-th iteration. In this iteration, according to Algorithm (lines 10- 17), we add s, to i if there is an edge from šijj some component of Sy,, where Si,, S contains all minimal components of i's neighbors from the previous iterations.
By the inductive hypothesis, for any i' N₁ and any component j', sv. sq. if and only if either:
sj has a direct edge to Tr. or
there exists path from sy (t) to some ry (t + m) where m S k, with all intermediate components in their respective s sets.
Forward direction: Suppose 8i,j added to ii the (k + 1)-th iteration. Then there must be an edge from 8t, to some component f2' E S, for some i' N, By the inductive hypothesis, either:
Case 1: If s has direct edge ry', then we have a path:
(t) st 1) Fv (t 1),
which is path of length 2 (1 1) from (1 re (1 (1+ 1), satisfying condition (ii).
Case If there is path from s 1) T. + 1+ m) for some m k. then we have a longer path:
(t) Sy 1) + r.. + m), which is path of length + 2 m +1 k 1) from (t) to r.. 1 m), satisfying condition (ii).
In both cases, i' E Mg (the neighborhood constraint is satisfied), and all intermediate state components are their respective s" sets by the inductive hypothesis.
Backward direction: Conversely, suppose there exists path from (l) to r (1 + 1) for some 1 S + 1, where intermediate components are their respective sets and each agent is neighbor of the previous agent.
If ( 0, then s has direct edge to r; (condition (i)), so siz s after initialization.
If / 2 1, consider the path:
(1) + 50(+4)))(++(( + ..... re r2
The first transition in this path is from st, to (((+1))((++++ + 1), where i(t + 1) E N₁ (by the neighborhood constraint). By the path condition, (((0-)))(+=((( (1+ 1) ((++)))
Furthermore, there exists path from $044150++1 (t + 1) to r + 1) of length I - < By the inductive hypothesis, (((++))((++1) ((+t1)
Since i(t 1) Nr. have s1(++)),5(+1) sy, Therefore, there is an edge from si,j component in s which means 8új will be added to se in the (k + 1)-th iteration (or earlier).
By induction, 8i.j se if and only either condition (i) or condition (ii) holds, completing the proof.
Lemma 4. Fix i E W. For any j E [d]], 8i, & s² if and only if each of the following holds
(i) $i,j does not have an edge to Ti
(ii) for any e ., there exists no path from (t) to rv (t + I) for some
Proof of Lemma relationship.
This lemma is the logical contrapositive of Lemma S Let's formalize this
Lemma 3 states that s. ss if and only if either:
ši.j has an edge to ri. or
There exists i' V and there exists path from s1)(1) to rv (t + 1) satisfying certain constraints.
22

---

The logical form of Lemma 3 can be written as: 8i,j ES; - V The contrapositive of this statement is: $ij ES; = -((A) V (B)).
Using De Morgan's 's laws, -((A) V (B)) = (A) (B), which gives us:
8LJ $ s
- -B), where:
. (A): ši,j does not have an edge to
-BB): For all i' E V and for all / 2 1, there exists no path from si (t) to (t (t++0 that satisfies the constraints on intermediate nodes.
These are precisely the conditions (i) and (ii) Lemma Therefore, by the logical equivalence of a statement and its contrapositive, Lemma iis proved.
Proof of Proposition 6 We will prove that the Q-functions are identical by showing that the distribu- tion of rewards collected under both policies is identical. This follows if we establish that the relevant state-action trajectories have the same distribution under both policies.
Let us define:
of (s((), (0)) a(0) a): The probability of being in state-action pair (s((), a()) at time 1, starting from (s(0) = s, a (0) = a) and following policy æ.
(s; a;): The reward for agent given its state-action pair.
The Q-function for agent under policy can be expressed as:
Q (s,
Ex
Σ ())((s()),, s(0) s, a(0)
Σ
of (s(t) = s. a0) = a) ((ss(t),a((t))
t=
a(1).(())
Similarly, can define Q (s, a) for the approximately compact policy म. To prove Q (s. a) = Q (s, a). it suffices to show that for all 2 0, the distribution of rewards rs(ss(t), a (t)) is identical under both policies.
From Lemma 3 we know that the reward r: (s; a.) depends only on the components in s and ag This is because any state component not in si has no path to (directly or indirectly), meaning it cannot influence the reward function. Therefore, we need prove that the joint distribution of (t) a, (t)) is the same under both and न.
We prove this by induction l.
Base case 0): The initial state s(0) = is given, so sf (0) the same for both policies. For the initial action, we have:
Under x: a (0) Te SM, (0))
Under म: (0) ~ (-1 (0))
By the definition of the approximately compact localized policy:
# (-1s), n, (,,, ss;
for any fixed SA; and for any configuration of the non-minimal components sx,, $x,/$ Therefore, (0( has the same distribution under both policies when conditioned on the same initial state.
Inductive hypothesis: Assume that for some 2 0, the joint distribution of (s)()). aq(t)) is the same under both и and मे for all agents i.
Inductive step: We need show that the joint distribution of (s) + 1), az 1)) also the same under both policies.
23

---

For sf - 1). we analyze how it depends on the previous state and action. By the Markov assumption, s(t + 1) depends only on s(t) and att). The key insight is show that sy (t + 1) depends only on &2. (t) and (4), not on any component outside Sx; (1).
Suppose, for contradiction, that there exists a component st. É Six, (t) that influences some component $1,y (t + 1) E (t + 1). This means there is causal link from 5k. (t) to 8 (t + 1).
By Lemma since si.y (t + 1) E s + 1). either:
Bi,J (t (t++1 1) has a direct edge to Ti, or
There exists a path from $/ (1 + 1) to some reward ry-(t + + 1) for some / 2 0.
Either way, this creates a path from $k.j (t) to reward (either directly to r, through $1.r (t+ 1). or to [t + 1++1 1) via the path from st.5 + 1)).
This path satisfies the constraints in Lemma conditioo (ii), which would imply sk, E ((). Since & E NG (there is a causal link from agent i: to agent i), we would have 8k (1) E 8/; (!), contradicting our assumption.
Therefore, sy (t 1) depends only on Si, (t) and a.; (t). By the inductive hypothesis, the joint distribution of (s&; (1), aNG (1)) is the same under both policies, so the distribution of ss (1. 1) is also the same.
For (t + 1), we have:
Under x: a, 1) (-Iss + 1))
Under म: at + ~ (. (t +
By the definition of approximately compact policy and the fact that the distribution si; (t + 1) is the same under both policies, at + 1) also has the same distribution under both policies. By induction, for all 2 0, the joint distribution of (s (t), (t)) is the same under both æ and म for all agents i.
Since the reward r, (t). (t)) depends only on s (t) and a, (t), and these have the same distribution under both policies, the expected discounted sum of rewards is also the same.
Therefore, Q (s. QT (s, a) for all agents i and all state-action pairs (s, a).
Corollary Let æ be an arbitrary localized policy and स be the corresponding approximately compact localized policy output by Algorithm 2 for each agent i, and any (s, E S x A, the
approximation error between Q+ (s) av?) and QT (s, a) can be bounded as
sup @ (5 aNF- Q7 (s, a) I -- x+1 (s.A)ESxA
1-7"
Proof.
Using Proposition 6 have
( (5 (NT - O7 (s, a)
S a R 0 (s, + +f (s, a) Q (s,a)
x+l
Proposition 7 (Approximation error of policy ACR), For any policy æ, let be the corresponding approximately compact policy constructed using Algorithm 3 for any i, we have
Q (s, a) Q, (s, a) S
sup (x.a)eSxAA
Proof of Proposition 7 Let us first establish precise definitions:

---

Let i denote the minimal state representation for agent i obtained from Algorithm (with indefinite recursion until convergence).
s; denote the minimal state representation for agent obtained from Algorithm 3 (with
exactly x steps).
Let π be the original localized (1-hop) policy.
Let # be the approximately compact policy derived from Algorithm 2
Let #* be the approximately compact policy derived from Algorithm with steps.
Relationship between minimal representations: By construction of the algorithms, we have for any finite k. This because Algorithm performs exactly g iterations, while Algorithm continues until convergence, which may require more than k. iterations.
S.
Lemma 5. For any agent i, s."" captures all state components that can influence rewards within K time steps.
Proof. By the structure of Algorithm in step 1, (1) includes all state components that directly affect the reward r;. each subsequent step (for I= 1, 1....... K - 1), any state component that can affect component si (I) within one time step added s (I 1). By induction, after steps. si (R) contains all state components that can affect the reward within time steps.
Policy equivalence and divergence: From Proposition 6 we know that for the indefinite recursion case:
of (s. Q2" as.
Therefore, to prove Proposition we need to show:
sup (x
1Q (s, Q, (s.
1 g
F
Trajectory analysis: For any initial state-action pair (s, a), we'll analyze the difference in trajectories under policies 4* and x00
Lemma 6. For any initial state s(0) and action a0)) a, the distribution of states and actions under policies # and 000 are identical for the first time steps, i.e., for = 0, 1.... K.
Proof. We prove this by induction.
Base case (t 0): By definition, s(0) and a()) for both policies.
Inductive step: Assume the distributions are identical for time steps 0, 1,..,, where t < K. At time f. given identical state distributions, both policies make decisions based on the minimal state representations:
# uses S (t)
uses NN, (1)
For any component (1) that affects rewards within x time steps, by Lemma 5, 81, E S (1). Since < both policies have identical information about all state components that can affect rewards within the remaining time steps. Therefore, the action distributions time are identical, which leads to identical state distributions at time 1.
25

---

Q-function decomposition: For any initial state-action pair (s,a), a), we can decompose the Q- functions follows:
OF (s. a) = EEg
(()) = s, a0) a at),
1=
+ Ege
a.) s(0) = s, a(0) a
Ixl+1
2 (8, a) = E
a1) 8()) = s, a(0) = a
+ Ex
(t). s(0) = s, a =
L(=+11
By Lemma 2, the first terms are identical. Therefore:
O (s, a) Of (s,
a') s()) s.a(0)
Eges
(t), ((0) = s, a(0) = a
|=++1
Error bound: Let pf and P be the distributions of (t), (t)) under policies H* and g respectively, conditioned on (s(0), (0)) = (s, a).
From Lemma 6, we know that pt" Pt for 1 S k. For > K, we can bound the total variation distance between these distributions:
TV , 1.
Therefore:
10 (s,a) -
(s) =
Σ 7/ (1 [ri(8;, a:)] E (e (r:(s), ai))
1=
8
N'. TV(p: #* .P.
|=++1
00
7 = 8n x+1 1
1-
t=++1
This gives us the desired bound:
10 (s, -f," (s, a)l
1 1- --=
sup (s.a)ESxA
Since " (s, Q7 (s, a) from Proposition have:
12 (s, a) - Q7 (s) S - 1tt
sup
(S.
-
This completes the proof of Proposition
Proof of Proposition
Using Proposition
we have
@ (/ aN?
Q. (s. a)
Sxp aNF
- Q (s. + +10 (s. -QT (s.a)
3F
7 x+1
26

---

E.3 Cross domain
Denote by WNT (Wi)EnN; and WNo, == w/WNr.
Lemma (Exponential decay property of domain-specifi value functions). For any localized policy T, for any E N,88 S8N E S , &N,, ; ONp,WNe, w'ye E Qy the following holds
Sv=, aN: ANy, aw,, alv, E ,
F
Q7 (SN7, aNy aND, WNy, wNs.) Q Sws, aWp alw, SNG.
Proof. The proof follows from that Lemma | by augmenting the state space of each agent by S Q (st, w)), where the augmented state variable w; degenerates to deterministic state (w: = Wi, Vt).
E.3.1 Value functions
Given the generative model (Equation 6). define w-ACR for Q-functions follows.
Definition 5 (Value w-ACR). Given the graphical representation of the networked system model G that is encoded in the binary masks c c the value ACR s²x- (Definition I), for each agent and its k-hop neighborhood Nix we recursively define domain-speciffe ACR Wiye the latent change factors w that either
w w 1, or Wij has a direct link to agent i's reward ri, i.e., wij wt and og
wyy has an edge a state component 83,1 si i.e., of and sv, E S
Definition 5 also defines agent-wise ACR: U =
Definition 6 (Approximately compact w w-conditioned Q- Q-function). Fix E N and an arbitrary SNT and W.t. Define the approximately compact w-conditioned Q-function as
Q ( aNy wXr+ Q (syz SNy /sx; aNT Sxy (aN:/W8g
Proposition 8 (Approximation eπror of value ACR). For each agent i, the approximation error between Q ( aNp, wNr²) and O (s, a, w) can be bounded as
2F S 1-1 x+1
Q (5 aNy (s.a)ExxA
O7 (s, a))
sup
Proof. The proof follows from that of Proposition | by augmenting the state space of each agent by S Ω4 e (s,, w:), where the augmented state variable w, degenerates to deterministic state (t)(()) = Wi, Vt).
E.3.2 Policies
Definition 7 (w-ACR for localized policies). Given the graphical representation of the networked system model 3 that is encoded in the binary masks the policy ACR si; obtained from Algorithm or Algorithm 3 for each agent and its 1-hop neighborhood N,, we recursively define domain- specific ACR w the latent factors ww; that either
Wij has a direct link to agent i's reward r;, i.e., wij w. and c, o 1, or
wey has edge a state component 83, S i.e., ct and 8 E S
Definition 8 (Approximately compact w-conditioned policy). A domain-specific localized policy (TI...., .T)) is called an approximately compact localized policy if the following holds for all i N, S'y, E SN., WN,
Ti S SNC SN// wix, ww)/ m (-1 SW, S/x, w /wx))
27

---

In other words, there exists ã; :
Aa. such that Ti (: Sy. . /sv, 3 / #1 (. sNc w N.) For convenience, fix arbitrary Sv, /w/, wN// /, and denote by n; (. Si; (x,) == n. (: SN, 1 wN. (. Proposition 9 (Compactness and स be the corresponding approximately compact localized policy (Definition 8 for each agent i, and any as,a,w) w) E S x A x Q we have
SN. x R
cross-domain policy ACR). Let æ be an arbitrary localized policy
Qf (s. a) - Q7 a.
Proof. The proof follows from that of Proposition 6 by augmenting the state space of each agent by S; 12 3 (8;; w:), where the augmented state variable w; degenerates to a deterministic state (w;(t) wi, Vt).
L
Corollary 2. Let æ be an arbitrary localized policy and स be the corresponding approximately compact localized policy output by Algorithm for each agent i, and any (s, a) E S x A f2
the approximation enror between Q (xp avy, ( and Q (s, a) can be bounded as
2F x+1
sup Of aN wN:) (S.a)ESxA
O (s. a. s
Proof. The proof follows from that of Corollary [ by augmenting the state space of each agent by S; f4 3 (s,, wi), where the augmented state variable w, degenerates to deterministic state (w,(t) = wi, w;, Vf).
Proposition 10 (Approximation error). Let æ be arbitrary localized policy and से be the corre- sponding approximately compact localized policy output by Algorithm 3 for each agent i, and any w) E S x A S2 have
sup | (s, a, w) QJ k) S T-7? s+1 (s.a)
Let π be an arbitrary localized policy and के be the corresponding approximately compact localized policy output by Algorithm for each agent i, and any (s, w) E S x A x Q
Corollary
the approximation error between Q (N aN? W and O (s, a) can be bounded as
Of (sir, (S.
Q. (s. aw) s 1-
37
sup
Proof. The proof follows from that of Corollary 2 by augmenting the state space of each agent by S; x Ω4 e (s., wi), where the augmented state variable w, degenerates to a deterministic state (w:(t) Wi, Vt).
F Missing proofs in Section 5
F. Discussion on assumptions in Proposition 5
Our assumptions in Proposition 5 are standard in causal discovery and theoretical MARL..
Faithfulness and sub-uuusueseeee Gaussian noise: these classical and standard requirements ensure identifiability [0] and finite-sample recovery.
Bounded degree: bounded neighborhood degree is natural in networked systems where agents have limited communication range. For example, this holds when each intersection connects to only a few roads, each wireless node has limited interference range.
Full observability: while we currently assume full local observability, the ACR construction can be extended to partially observable settings using learned belief states latent encoders. This part of our future work.
Lipschitz continuity: this mild smoothness assumption satisfied by most smooth policies.
Importantly, our algorithm can still be applied when assumptions are violated, only the theoretical guarantees may not hold exactly.
28

---

F.2 Proof of Theorem 1
Proof. We will prove this theorem by analyzing the causal structure from the perspective of each individual agent while accounting for inter-agent influences through the network structure.
For each agent i, we define the variable set V; that includes:
1)) States of neighboring agents time SNTiet - 1) (sy(t- 1): (i)
States of agent at time t 1: s, 1) ($t,(t
States of agent at time f: st) = (s (t).,,.)...............)
St,d, (t))
Actions of agent at time t 1: a,(t
We denote by m E [I,...,MM) M the domain index, which serves as a surrogate variable for the unobserved change factors wi" Let G. be the causal graph over variables in V, U {k). Our goal is to identify the binary structural matrices ct cf and c;"" which encode the edges in Gt. Under the Markov condition and faithfulness assumption, we can apply the following fundamental result from causal discovery:
Lemma 8. For any two variables V,, V. Vi, they are not adjacent in G3 if and only if they independent conditional on some subset of (Vi + r. L + 8) U {m}.
This lemma follows from the properties of d-separation in causal Bayesian networks, as established therefore identify the skeleton of G, by testing conditional
in [ and (21]. For each agent i, independence relationships among the variables in V; U (m]. The networked MARL system forms a Dynamic Bayesian Network (DBN), where the temporal structure imposes constraints on the edge directions:
1. Edges can only go from variables at time t to variables at time t.
2. There are no instantaneous causal effects between variables at the same time point.
Therefore, for any edge identified in the skeleton of G, between a variable at time t - and a variable at time f, the direction is determined by the temporal order. Specifically, for each state component (t, of agent i, any edge from a state component st,m (t-1) 1) (where could be i or any neighbor of i) Sa,y must be directed from st.m (t 1) to st,5() (t). Similarly, any edge from the action a, 1) to (t,)()) must be directed from at - 1) to si. (t).
Given the directed edges in Gū we can now identify the structural matrices:
1. Identification of ci- For each state component (t) of agent i, the j-th row of cf denoted by c is binary vector where entry is if and only if there a directed edge from S...m (t 1) to (t) G,.
2. Identification of c For each state component $t (t) of agent i, the entry of cf is if and only if there is directed edge from a,(t - 1) to st.s (t) in G.-
To identify of b2 we need to determine which state components have distribution changes across domains.
Lemma 9. For variable V, E V., its distribution module changes across domains if and only if V, and m are not independent given any subset of other variables in V,.
This follows from the definition of the domain-specine changes: if a variable's 's distribution remains invariant across domains, then it is independent of the domain index m conditional on its parents. For each state component $1 (t), we test whether st, (t) JLL m| Z for any subset Z C V, (t), If no such conditional independence holds, then the distribution of s (t) varies across domains, and therefore ci, ++ has at least non-zero entry.
29

---

To identify the specific entries of c,"" "."" we exploit the fact that changes in distributions are localized according to the causal structure.
To complete the proof, we need to show that the structural matrices are uniquely identified from the observed data:
Uniqueness of cf c and c;= These matrices encode the causal connections in the DBN structure. Under the Markov and faithfulness assumptions, the skeleton of the DBN is uniquely identified. Combined with the temporal constraints on edge directions, the matrices ct and cf are uniquely determined.
2. Uniqueness of o=">s: This matrix encodes which state components have distribution changes across domains. Under the faithfulness assumption, state component's 's distribution changes across domains if and only it is not conditionally independent of the domain index given any subset of other variables. This provides unique characterization of the non-zero entries in c.""
A critical aspect of our proof is that we treat each agent's local model separately while accounting for the influence of neighboring agents through their observed states. This is justified by the factorization of the joint transition model:
P(s(t ())s((),
II Ps(s ())x(())))s))))) {=1
This factorization ensures that, for each agent i, we only need to consider the states of its neighbors NG to fully capture the relevant causal influences.
Therefore, under the Markov condition and faithfulness assumption, the structural matrices cf- ct and c cw""+s are identifiable for each agent in the networked MARL system.
F3 Proof of Proposition 4
Proposition 11 (Formal statement of proposition 4. Suppose the following conditions hold: (i) the Markov condition and faithfulness assumption are satisfied for the causal graph 9; of each agent i, (ii) For each true causal edge (X - Y) in Gs. the minimal conditional mutual information is bounde: [(X;Y] Z) 2 À > 0 for any conditioning set Z that d-separates X and in 9i \ X Y). (iii) The maximal in-degree of any node in Gj is bounded by dmaxx (iv) The noise terms ci, (t) are sub-Gaussian with parameter o², (v) Each element of sN; (t) and (t) has bounded magnitude by B. (vi) The transition functions fs.j are Ly-Lisschitz continuous in all arguments.
Then, with probability least 6, the estimated causal masks C+- co-te, and Čye-t will match the true causal masks for
d: dmax log(d; n/o)
T. =2
X
where d; = dim(sx;) is the total dimension of the state space for agent and its neighbors.
The proof consists of four main parts: (1) formulating the statistical estimation problem, (2) analyzing the error in conditional independence tests, (3) bounding the error probability in structure learning, and (4) deriving the sample complexity.
Proof.
Step 1: Formulation of the statistical estimation problem. For each agent i, need estimate the causal graph G; from observed trajectories {(sx (t), (t), + T-
Define the variable set V; for agent that includes:
States of agent at time f 1: s,(t - 1)
States of neighboring agents at time / - 1: (NGr6(1-1)
States of agent at time f: (t)
Actions of agent at time t 1: at 1)
30

---

Domain index k, which serves as surrogate for the unobserved change factors w
We employ constraint-based causal discovery approach, which identifies the causal structure by performing series conditional independence tests based on the key insight from Lemma 13:
Lemma 10. For any two variables V,, V. Vi, they are adjacent in Gj if and only if they are independent conditional on some subset of (Vi : t + r,t o U {m}.
Step 2: Analysis of conditional independence tests. For analyzing conditional independence, use the mutual information measure. For variables X and Y conditional on set Z, the conditional mutual information is:
P(X,YIZ)
I(;; - log P(X(2)P(YIZ)
In practice, estimate this from data using the empirical distributions:
p(),
f(x; (IX
Σ P(x, 2) log P(x, P(y,))
z,y.=
where P denotes empirical probabilities based the observed data.
The estimation error be bounded using concentration inequalities. For any 0:
T.e²
P()((x:\I)) I
S 2xp
C. ||
where |. [ and | are the cardinalities of the respective variable domains, and C₁ is universal constant.
For continuous variables, we can kernel-based estimators or discretization. With appropriate binning of continuous variables, the error bound becomes:
P(1i(x:Y1Z) - [(x; > c) S 2xxp (- C2bxbyb2 z log log2 (Tc)
(-
TEE²
where bx, by, and bz are the number of bins used for each variable, and C2 is constant. The log (Te) term arises from the adaptive discretization continuous variables.
Step 3: bounding the error probability in structure learning. Let Gi be the true causal graph for
agent i, and G. be the estimated graph. We need to bound P(G: + G.).
The structure learning error can occur in ways:
1. Missing a true edge (Type error in a conditional independence test)
2. Adding a spurious edge (Type I error conditional independence test)
For Type errors, set the significance level to 237 where M is the total number of conditional independence tests performed. For Type II errors, we use the fact that when variables are dependent with conditional mutual information at least y, the probability of wrongly concluding independence is:
To?²
P(Tyee II error) < exp
where C33 is constant that depends on the distribution properties.
The total number of conditional independence tests is bounded by M s df EYEO (4:72) s
d2
de (d < d3 (eds/dmass )
danxx.
By the union bound, the probability of any error in structure learning is:
P(g, + (.) S M max a, exp (-
Te??
("ydmmxx
31

---

where we' ve used that the maximum size of any conditioning set is dmax. and the maximum cardinality of any variable domain is bounded by function of dmax given our discretization scheme.
Step deriving the sample complexity. To ensure P() Gi) S 6, need:
a
2M
S
2d (ed. (dmax) Jemax
Ton Cadmaxx
2. exp
2M
2d (ed, Auma Jonax Jonax
From the second condition, we derive:
To7²
2dh (ed,/dmax) ) Amax
2 log
Czdmax
Therefore:
Cadimax
2df. (ed:/ dmaxx) domax
Te
log
7/2
Now, we need to analyze this expression more carefully:
2dk
(ed./dmax)) dmna
2dl
edi dmax
log
log
+ dmax log
8
Under the sparsity assumption (dmax < di), we have:
edi S dinax log(ed;) = O(dmax log(d;))
dmax log
dmuxx
Therefore:
Te Cadhmax
2d + 0 log (d))
log
72
8
This simplifies to:
= (ffiee
Lmax log(d;) + dmax log(1/5)
722
For most practical scenarios where dmax Ollog(d:) (very sparse graphs), this further simplifies to:
T, dmax log(d,) log(d./6)
(d
722
However, for more general bound without assuming extreme sparsity, we have:
T. = drunx log(d)) + dmax log(1/0)
72
Now, accounting for the fact that we're learning causal masks for all n agents, and applying a union bound across agents, replace with å/n, yielding:
= O (fh (eaxx log(d,) + dmax log(n/8)
22
For typical networked systems where scales with the number of neighbors, this gives us our final sample complexity:
T. = (4.
dd. dmax log(d; .n/0)
72
32

---

F.4 Proof of Proposition 5
Proposition 12 (Formal statement of Proposition Suppose the following conditions hold: (i) The true causal masks are estimated accurately, (ii) the transition dynamics are identifiable with respect to the domain factor, i.e., for any two distinct domain factors w; & w/, there exists a of states and actions with non-zero measure such that the conditional distributions P(st, (t. 1)(sx (!), (4),,, wi) and P( + 1)]sy; (1), a, (t), w)) are distinguishable. (iii) The noise terms (t) are independent, sub-Gaus Gaussian with parameter o². (iv) The domain factor w E Q2 belongs to compact set with diameter Dz, (v) For any two distinct domain factors w, + w/ with I|l; w/ll2
(t + 1) ss; (t), (t)), - P(s,(t + 1) (t), (t),), W 2 ae
for some constant > 0. Then, for any domain with true domain factor w", given trajectory of length Te, the estimated domain factor w satisfies:
Da log(n(Te//) Te
ف - w01l2 S Cwp
with probability at least 1 - 6, where Da is the dimension of the domain factor space, and C. is constant that depends on y, σ, a, and the conditioning the estimation problem.
Proof. The proof consists of three main parts: (I) formulating the domain factor estimation problem, (2) analyzing the estimation error using statistical learning theory, and (3) deriving the final sample complexity bound.
Step 1: formulation of the domain factor estimation problem. Given the known causal structure encoded by masks C,- cj and C the domain factor estimation problem can be formulated finding the domain factor that best explains the observed transitions.
For agent we have observed trajectories {(sN, (t), a, (t), s;(t + 1)) generated from the true domain factor w We can define a negative log-likelihood function for each agent i:
(,(w)
log P(s,(t + 1))s,, at),
To
t=
The maximum likelihood estimator (MLE) for the domain factor is:
w
arg min
Σ C.(w).
WEST
Step 2: analysis of the estimation error. To analyze the estimation error, the theory of M-estimators and the properties of the negative log-likelihood function. First, we establish some key properties of the negative mikeeiiidoood
Lemma 11 (Lipschitz continuity of negative log-likelihood). Under the assumption that the transition function fej is LipLsschtitz continuous with respect to all arguments, the negative log- likelihooo function (ww) is L--Liscchizz continuous with respect w, where Le /
Proof. For Gaussian noise with variance o², the conditional probability density is:
P(s.(1 + 1)|s>, (1), ((),
d:
(sng(t (i,i (N, (1) ai(t), c 2 D0 0))
exp
2
20
1=1
Taking the negative logarithm:
log P(s,(t + 1)lsw (t), a,()), at)
d.
(sts(t + 1) 1) - fi,( @ sss ct), (a,)),, (t),))"" 3 ww, 0))
Σ
2 log(2ro7)+
20
33

---

The gradient with respect to w is:
(stj(1 (1++1) Sii) Awlis-
V P(s:(t 1) 1t)ss, = Σ
σ²
Since Ja is T.y-Iiischitz and St S 1= we have:
d
1
|V log 1) (t), a (t), w)]ll
O
r²
Therefore, (,(w) is L Lipischhi continuous with يe /
Lemma 12 (Strong convexity of expected negative log-likelihood). Under the identifiability as- sumption and the total variation distance condition, expected negative log-likelihood [E()(()) is j-strongly convex in neighborhood of the true domain factor w", where depends a and the minimum eigenvalue of the Fisher information matrix.
Proof. The expected negative log-likelihood is:
[[C,(w)] Bss, (,s. P(s 1)) (t), (t),
The Hessian matrix of this function is the Fisher information matrix:
H (w) Esn; (t [w log (t, 1)]sv, a,, V log P(s, at,
w
By the identifiability assumption and the total variation distance condition, for any unit vector have:
H(w")u a² 0.
Therefore, in neighborhood of w*, the expected negative log- log-likelihood is p-strrngly convex with μ 42a a²
Now, we can analyze the estimation error using concentration inequalities for empirical processes. Let:
((w) 1 "
Σ C.ww). n i=1
Proposition 13 (Uniform convergence). With probability at least 1 - 6, for all w E Ω:
Do (Tg(/81)8)
[C(w) - B[[[w]] & C
Te
where C' is a constant depending on Le and Da-
Proof. Since (,(w) is Le-Lischhitz continuous, we can use covering number arguments. The e-covering number of the domain factor space with diameter Da N NO2 E) S (1+ + Du)
By the union bound over an e-net and Hoeffding's 's inequality, we have:
2T
P sup [(w) [[[(w)]] >() < 2N exp
WEB
2LC
B
where Bg is the bound on the negative log-likelihood function. Setting the right-hand side to å and solving for e, we get:
(V
Dr (og(Te/))
==0 0
34

---

Step 3: Deriving the sample complexity bound. By combining the strong convexity of [[](w)) and the uniform convergence bound, can derive the estimation error. For the true domain factor wn and the estimated domain factor ش we have:
[( S L(w*).
This implies:
B[[(W)] S [(() + S (((W)) +e [E([[W)) + 2e.
By the strong convexity of [B), we have:
-[([)]-B[((()) 1 - f-w-w113 w'
Combining these inequalities:
Dr Log(Te/O) Te
|| --w-w11l
w011 2E =0
2
2
Therefore:
||ا - - 0 (1
Da (og(Te/)) PTe
Now, accounting for the fact that we have n agents, we apply union bound across all agents by replacing å with 8/n. This gives us:
Da log(nTT//))
ها w0lla S Cwp
Te
with probability at least - 6, where C. = O(1/v)) ((1/a) is a constant depending on the problem parameters. This completes the proof of the theorem, providing rigorous guarantee on the accuracy of domain factor estimation based on the trajectory length Te-
G Missing proofs in Section
G.1 Discussions Assumption
6
While Assumption I 6 are standard for establishing theoretical guarantees, we discuss whether they hold exactly in practice follows.
Finite MDP (Assumption : some systems, the state or action spaces might not finite or continuous. For such cases, our results can still extend by applying state aggregation clipping methods to ensure boundedness while incurring small approximation error.
Mixing properties (Assumption 2): real-world networked systems may exhibit slow mixing or non- Lipschitz dynamics. Our theoretical results would degrade gracefully, leading larger constants 7 in the sample complexity bounds.
the assumption requires the policy gradients Va, log π. to be Lipschitz and uniformly bounded. This holds for all commonly used policy parameterizations (e.g., softmax policies) with bounded inputs and finiteddimensional parameter vectors. It ensures stable policy updates and is common the convergence analysis of policy gradient and actor-critic algorithms. GSAC uses softmax parameterizations for policies, where log Te, is smooth and has bounded gradients when action probabilities are not close to zero (a condition satisfied by entropy regularization or initialization).
Smoothness (Assumption 3):
Compact parameter space (Assumption 4 policy parameters 8; are always constrained within a finite range due neural network weight regularization, weight clipping, bounded initialization. This is standard in actor-critic methods, where optimization performed with bounded learning rates and regularizer that implicitly keep the parameters within compact set.
Smoothness w.r.t. domain factor (Assumption 5): this reasonable because domain factors, e.g., channel gains, traffic loads, represent physical parameters that vary smoothly. The local dynamics
35

---

P. are typically analytic or polynomial in w, so small perturbations in w; induce proportionally small changes state transitions and rewards. Consequently, the Q-function Q,' (s, a, w:) and policy gradient Vo [0, w) inherit this smoothness. Moreover, standard policy parameterizations and neural network approximators are Lipschitz when combined with bounded weights or gradient clipping. Thus, Assumption 5 aligns well with both the structure of real-world systems and common RL practices. Moreover, even the true dynamics were not strictly Lipschitz, domain factor estimation (Proposition 5) uses rmall-perturbattoon assumption around w for adaptation. In practice, state or reward clipping and function approximation, which are inherently Lipschitz, effectively enforce this assumption.
Compact domain factor space (Assumption 6): the latent domain factors wi represent bounded variations in environment dynamics (e.g.. channel gains, traffic intensities, or reward scaling factors), which are inherently limited by the physical system. In real-world networked systems, these factors are naturally confined to finite ranges determined by system design (e.g., transmission powers, queue lengths). Hence, assuming S2 is compact with bounded diameter Da both reasonable and practical.
For simplicity, we also write Q == Q(:..ww) w) and z == (s, a).
G.2 Proof of Theorem 2
1 suppose the critic stepsize oz t+1o Theorem 5 (Critic error bound). Under Assumptions
satisfies h 2 1 m2x 1=x) 2 max(2h, 4oh, r), and the domain factors are estimated with
Te trajectories. Then, inside outer loop iteration l with domain factor mm(k), for each E E,, with probability least 1
C.
C. + (1 (1- + +Cw
2cp
log(nIe/0)
(i,(s, (SNg, aNY aNN (NT+) S
+to
T
Te
where Ca = I-v7
6F zh [log( (-/²2 + [(K) SA), C² = I-v- max
16thr
2F I- (r + to)), with
=:= 4 F = + 2F, and Cy == LQCwVDD Do.
Proof sketch of Theorem The proof follows similar structure the Theorem in 24 with modifications to account for domain factors. First, we decompose
(z w) Q (ZN;, Qi Qz + Q((, Q7 (ZNz, (Nr))
The first difference term accounts for the error due to domain factor estimation. Using the domain factor identifiability in Proposition and the Lipschitz continuity of Q with respect to w (Assumption
Lipschitz.continutty
5 (i)),, we have
Qi(z, w) - Qi(z, Lo M|! 2ا1l S (To).
For à = (m),)) the critic update follows:
(s 1), ax;
= (1 ()
ta-1(ri 1) + ,Q) (sx; (t), aN; (t), ww;
The steps from the original proof apply with the augmentation of wxn to the state-action represen- tation. The key difference is that need account for the error in domain factor estimation. Let & := sup(x)esxA (Q;(z, ()) Q? w.
Following the decomposition in the original proof (Lemma 7 in 24 ) we have:
& < B- + + sup
Σ (2. )Ex REm
2cp
Σ og B
+
Σ a& BR
1-7
ANT
Following the same induction argument as in the original proof and combining all terms, we get the stated bound.
D
36

---

G.3 Proof of Theorem 3
Theorem 6 (Convergence). Under Assumptions 66 for any ò E (0, 2 3, suppose the critic
stepsize 04 t+ and M/6 = VE+I with vj
S 6L For sufficiently large the inner loop length such
200"t1 and (8/2/21..2) T+t S (1-9)2 Then, with probability +T+
C.
that T +1>
1og7
at least 1 - 6, when domain factors are estimated with T. trajectories:
2
8 1,2
log K log ++
2402
1) log K
Ek=0 1K (((K)))
(1-7)²
(1-7)*
ΣK-1 1/
VK +
12L
2w
log(4nM/0) Te
2Cb(FL Lo )d
log( (M/)))
(1 11-7)50"H
(1 7)2
(1 7)-
M
Proof sketch of Theorem We adapt the proof of Theorem 4 in (24] account for domain factors. The key difference is that the policy gradient now depends on the domain factors, and we need to account for the error in domain factor estimation.
Note that for any 0 E θ, we have
Ve, log T (als, = Vo. Σ log xJ (a))s)))))) Vo, log x. (a. (s, wy).
JEN
Let us define the approximation of the policy gradient and the true policy gradient
0.(1) = Σ
(sN (1), aNy (1), N3 Vo, log 0.00) (a.()))s;()) (a())ss))))))))
JEN
(.(k)= =
2 0( (s(t). a(t), a())) Vo. log 0.(k) (a,t)ssst),
t=0
JEN
M
h,(k) Σ
M
1
E
o(k) stupt
am) (11x,)))))))
a
n
0(k) (0l)) aw. log m; o.(k) 2.1k) (2,ss,.077
JENS
M
J(OK)) DM)
Σ
a. Vo, log ((k) (als, 6mm]
o(k)
M
sript
m=1
1=0
M
Ve, ((O(K),
M
m
M
Vo, Vo,J()(k), M
(100 as Vo, log π 0(k) (a|s, w
M
grupe
o(k)
and 000 (1ew))
m=1
1=0
M
e. (((K))))))))
J((K() = EOnD
Σ
E
xnuph
266
(2 (O000 Vo, log ( (als, (kkss
1=0
= EwwD Ve. (((K)) = Vel = (o.)()
Lemma 13. The following holds almost surely,
FL
(a) [II(
(1 7)
37

---

We decompose the enror between ((k) and the true gradient as follows
@(k) g(k) g(k) + g(k) - -(k) + h(k))
+ VoJ(O(K)) VoJ(O(k),
VoJ DM
(M) (1)
(1))
(OOk) DM) ()
(12)
We will bound each term separately and combine these bounds to prove Theorem
Bounds on e (k). The following lemmas follows similar steps as in the proof 24
Lemma 14 (Lemma 14, 244). When T is large enough s.t.
Ca (2 T)
C₁
I 2cpri11
where
VT
T+
(1 -1-7)1 7)2
(6) = 1
62
Th
log
2TT
+ fn)) log SA
C² =
max
162hT
2F
(r + to)
8
1
σ
1
with & = - + 2F, then we have with probability at least 1
5
|ee (k)]l - 4cT.p==1 0=kcK-1
sup
7)3
Bounds on e² (k).. Let Gj be the o-algebra generated by the trajectories in the first k outer-loop iterations. In particular, let Go be the o-algebra generated the trajectories in the first phase of Algorithm [ Note that w I:M and a I:M are meameusuab. In addition, 0(k) is G Ga-r-measurable, and so (k). Further, by the way that the trajectory {(s(t,, a(t)) generated, we have [[g(k))G h(kk.. As such, me (V)((())) e² (k)) is martingale difference sequence w.r.t. Gt. and we have the following bound which is direct consequence of Azuma- haem--Hoeiiding bound.
Lemma 15 (Adapted Lemma 15
24 With probability at least 1 8/4, we have
K
2F $ (1-7)4 | /
K-1
8 8
My ((((
2
log n
k=0
Bounds on (k)..
c(1 7)
log
+ (w+ + log
f
Lemma 16 (Adapted Lemma 16, (4)). When T +12
we have
log
almost surely,
Le
[ke(( < [1 7) pntl
Bounds on (A).. Since the enror between 4h and " is bounded as ||@" w < (T:)
Cw V Jogin/8) with probability at least 8. Using Assumption 5 (iii), we have with probability To
least 1 - 8/4
M Σ
Ak, ((k) (OJ(O(K),DM) VoJ(((()) L,
A " ||2
(13)
log(4nM/6)
<LSWW
Te
38

---

Bounds on (k()). We first state the uniform concentration bound for vector-valued functions.
...,...,........................ 2MM be i..d. samples from a distribution D on X. Let C R and consider f X x ④ Rd with the following properties: (i) For all j 1, 1,..., d, [ (2. 0) I B for all 0); (ii) For all x E X and j I,...,d, d. (2, 0) - (a, 0') I L 10 0 21 Then, with probability at least
Lemma 17. Let (I1, (2,, 22,,..
8/4,
M
=0 - -0(B+D)) dp log M log (4/6)
1 M
sup
- Blf,Blf(x 0)]
f(,
AEe
M
where the hidden constant is universal.
Observe that for VoJ DM) (O). Using Lemma 17
we have with probability least
1 8/4,
sup |<>())||
[o, DM) VoJ(
sup N
r1
log(4M/5)
(1 7)2 + Lo
de
M
(14)
(FL + Lo)
(og(4M/8)
0
(1-
M
where the second inequality is due Lemma 13
Putting all bounds together. With the above bounds on all el()), we are now ready to prove the Theorem
Since (()) is T/-Lipschitz continuous, have
L' -1) 0(k)]1
J[ + 1)) 2 J(O(K)) + (V 0(k + -(k))
2 - 110kk
(15)
L'n (10(1)
=](((k)) + 7 (VJ(()))
Using the decomposition of g(k) in Equation 12
we get,
[| <
(1) + (00
We bound (V.J(((+)),g((t)) as follows
Σ e' -+002//0000000000==0000000000011
[[T][(([)](<(](((() +|ee [k) || (A))))
Plugging the above bounds on [ and (() into Equation 15 we have
J(B(K + 1)) 2 (((())) + (m 31>"))11 mcCk.u 7cCk.1 npck22
where 6k.0 == (V-J((A))), (k)), C,11 == [[]{(( (A+) |I + [lc3 + (|e((() + (
6R. = 31 Σ (x)] It follows from telescoping sum that
K-1
K-1
J(((K)) J(B(O))
(n 3L' 3L' [J(()))
Σ T&CK.0
Σ
k=
k=U
R-1
K-1
K-1
The [[\
TIEEO
Σ
M
K=D
where we 7 3L' n m1 3L'm)) 2 1 given that 17
After rearranging, get
617
Σ
K-1
K-1
2 J(B(K)) -(((0))
Σ mkCko +
McCk.1 + Σ n
(16)
A=U
k=
k=0
39

---

We now apply our bounds on each e² (k). By Lemma
15
we have with probability 1
6
2
K 1
2F
K --1
2
m log
(17)
(1 - 7)
By Lemma 14
Lemma 16
Equation
13 and Equation 14
we have with probability 1
2*
FL
sup
E111
S
(1 7)2
sup A
|ee( + sup
|ke) +
sup ASK-1
|e*(k)]
+ sup kSK-1
|Te
kSK-1
KSK-1
FL
4cLpptt
Le
log(4nM/0)
(FL + Lo)ed
log(4M//)
+2 (1 -1-
patt + L.
+ Cs
(1 7)²
(1 (1-
Te
(1 7)2
M
cF
FLL log(4n11/)) Te
Ca + Lo)d (1 7)
log( (4488))
(1 7)5
(1 7)2
M
(18)
By Lemma 13 we have almost surely,
et max (1le (k)lll) S 21
15/53
and hence,
||el (m))l 2 60. Σ
sup 6,, =3T,) KSK-1
(1 71-
,=1
Using a union bound, we have with probability 6, all three events hold, thus
K-1 k=
nhl (((()))
Σ 1/8
2(J(O(K)) I(O(O))) + 2
k== TACKDD
K-1
+ 2sup Ck2
K mh
+ 2 Sup
Σx Tlk
47 L2
i TH log 4 3
120
(1 - 7)2 V
+
(1 -1-77 -)) Σk= Th
Σk=0 1/8:
2F
log(4nM/8)
2C5(FL+ + Lo)c 71-
log (og(4M/0)
(1 7)3
6+1
+
(1 7)2
Te
M
17
Since 77k
we have
Vk +
K
K-1
mh
+I and Σ
n
(1+Log K) < 2n2 log K (using K 2 3).
D=
k=U
Further we use the bound J(O(K)) < results, get with probability 1 5,
and J(O(O)) 2 0 almost surely. Combining these
1
ΣK-1 Th ((((K))
2F
+
[1-7)²
8F
log log + 210F2 L'12 (1-7)* loo K
ΣK-1 Thk
VK +
12L
2FLL C.
log(4nM/0)
2Cs (FL + Lg )dd
log (M
r+
+
+
(1 - 7/5
(1 7)2
T.
(1 (1-
M
40

---

G.3.1 Empirical convergence
Corollary 4 (Empirical convergence). Under the same setup of Theorem 1- 6, when domain factors are estimated with T. trajectories:
with probability at least
EK-1 The DM()
2F + ((1->)
8F (1-7)a
240 log log
(1-7)²² n) log K
Ek 7
VK
(1 7)5 pa+1+ 2F
12
log(4n Te
+
(1 71-
ΣK-1 Th ((O(k), DM) )I
((-1) + BF2
2F
(1-7)4
log K log
2'
(1-7)² n n K
Σx=0 7k
VK
12 (1 7)
G.4 Proof of Theorem
Proof. We use the Lipschitz continuity of J0, w'; w) with respect to w'
J(O(K), M+1 : (O((K), w M+
+1 w W - LL ||0 w+1l
log(n/0)
2 - I Cw/
To
It follows that
log(n/8)
E [J([(K), & WMT ( 2 E [J(((K), o - ( (
Iw Cw
Ta
log(n/8) To
J(O(K)) Iw Cwp
H Comprehensive Experiments
We begin clarifying the distinction between the meta-RL and multi-task RL paradigms to highlight the key differences between our algorithmic design and baseline methods.
Clarification: Meta-RL vs. Multi-task RL The goal of multi-task RL 37 learn single mask-conditioned policy that maximizes the average expected return across all source domains drawn from a domain distribution. However, multi-task RL does provide any guarantee of rapid few-shot adaptation to new, unseen domains. In contrast, meta-RL explicitly aims to learn to adapt: it leverages the set of source domains train policy that can be efficiently adapted a small number of trajectories at test time.
domain using only
Therefore, GSAC falls under the meta-RL framework. Our objective is to achieve fast adaptation in a new domain with limited data. Specifically, we train a shared, policy across source domains and, during adaptation, estimate the domain factors @ from only few trajectories to enable immediate policy deployment without additional fine-tuning.
factor-conditione
H.1 Wireless communication
Environment setup. The network consists of n users positioned on 2D grid. Each user maintains a queue of packets with a fixed deadline d, = every timestep, a new packet arrives at user with probability p. Unif(0, 1]. The user either transmits the carliest packet to randomly selected
41

---

access point (AP) (t(t) E Y, or remains idle. A transmission to AP yr (t) succeeds with probability di ~ Onif(O, 1] no other user simultaneously transmits the same AP; otherwise, a collision occurs and no transmission succeeds. Each successful transmission yields a reward of 1.
The local state of each user (t) (b).,(),....> de (t); 21,1 (t), (t)) b₂(t) {0, 1] encodes
queue status by deadline bins (e.g.. (t) = (1, 1, 0) means the first two deadline bins are occupied while the third is empty). 2,, (t) and (t) are irrelevant components
decision quality. 24, is the channel quality indicator, which does not influence the packet success probability as qi depends solely on collisions. (t) the grid coordinates, which adds no useful information because the connectivity graph already encodes all relevant interactions.
The action space is A, = U (mull), where null denotes the idle action. The interaction graph is defined by overlapping access point sets: two users are neighbors if they share at least one AP. This induces local dependencies, which define each agent's k-hop neighhorhood for state, action, and domain-factor inputs.
visualizes the communication graph induced by different grid sizes: 3 3, 4 x 4, and x 5. Each user is placed at grid cell and is connected APs located at the corners of grid blocks. Users that share AP form communication links, resulting in structured interaction graph. This setup highlights the challenge of scalability in networked MARL, as larger grids yield significantly higher local complexity and increased neighborhood sizes. Figure iillustaate the local neighborhood (i.e.. K-hop graph connectivity) for a representative agent in each grid size. These visualizations demonstrate how local dependencies expand with increasing grid sizes. The number of neighbors within R = 1 grows with the grid size, resulting in higher- aigher-dimensional input spaces for both state and Nomain-factor features.
Figure 4
Benefits of ACR. The ACR procedure automatically prunes such irrelevant or redundant variables, retaining only those that causally influence the local reward dynamics. This yields non-trivial reduction in local dimensionality. For example, consider a 4 x 4 grid with de = 2 and neighborhood
size [ 5:
Raw (nuncatioon-oll): [ x (d: 2) = x 4 20 features.
With ACR: NN| x d; 5 x 2 = 10 features.
This 50% shrinkage directly translates into:
Smaller critics and lower compute: reduced local itte-acctoon support and table size.
Faster learning: lower variance in actor updates and tighter constants in our bounds.
Accuracy preserved: approximation error remains controlled (see Proposition T2.
Improved sample efficiency: by effectively increasing the minimum visitation o and reducing the mixing constant (Assumption 2) ACRs tighten the constants in the convergence guarantee (Theorem
Evaluation protocol. We generate M = 3 source domains by sampling domain factors p; from (0.2, 0.5, 0.8). while the target domain uses Purget 0.65 unless otherwise specified. In each domain, GSAC is trained for K outer iterations (variable depending on convergence), with time horizon of == 10. We use a constant stepsize of a = 0.1 for the critic and r) = 0.01 for the actor. The softmax temperature is set to 7 0.5, and the discount factor is 7 0.95. For domain factor estimation, we collect T3 = 20 trajectories per domain. To evaluate adaptation the target domain, we compare our algorithm against three baselines: SAC-MTL, SAC-FT, SAC-LFS.
Domain shift sensitivity. To assess generalization under varying domain factors, we vary the target packet arrival rate with Purget {0.6, 0.65, 0.7]. As shown in Figure 7 GSAC consistently adapts to new domain settings with minimal performance degradation, even when the target domain differs significantly from the training domains. Notably, it outperforms all three baselines across all configurations. Moreover, Figures 2 and demonstrate that GSAC can rapidly adapt to new environments using only few trajectories.
Across all settings, GSAC demonstrates: (i) consistent improvement over training iterations; (ii) fast adaptation from few trajectories; and (iii) robust performance under domain shifts. These findings
42

---

(a) grid size 3
(b) grid size
(c) grid size
Figure Grid layout.
neighbors
neighbors
neighterrs
0
(a) grid size 3
(b) grid size
(c) grid size
Figure Neighborhood illustration.
support our theoretical claims on scalability and generalizability in large-scale networked MARL systems.
H.2 Traffic control
Environment setup. The environment consists of a network of interconnected road links forming
a directed graph. Each node i represents road link, and the edges define feasible turning movements between links. The local state of link is denoted by s; (()))()),,, where ziy(!) E [S] (0, 1....... S} represents the number of vehicles on link intending turn neighboring link j. Accordingly, the local state space S; [S]
The local action at( = (w)s()))EN))) is binary traffic-signal tuple, where (i)(1) E {0, 1} controls whether the turn movement (i + j) is allowed at time t. Hence, the local action space is A; {0, 1J When Vig (t) 1, random number of queued vehicles (ty(1) will depart link i and flow into link j. subject to capacity and queue constraints. Meanwhile, link receives incoming flows from other connected links & e Ag where random fraction Ray(t) (t) of the inflow is assigned to each downstream queue Zty(t), The resulting dynamics are
(1 + 1) = (1) - min
+ Σ REAR
(19)
where [x]] max(min(r,, S), The random variables (t)(t) and Ray(1) are drawn i.i.d. from fixed distributions, defining stochastic yet locally dependent transitions. This structure ensures that (t + 1) depends only on the states and actions within link i's neighborhood, consistent with the local interaction model in Section 2.1.
43

---

35
200
186
200
320
(a) grid size 3
(b) grid size
(c) grid size
Figure 6: GSAC Training for different grid sizes.
(a) Purget 0.6
(b) Parget 0.65
(c) Purget 0.7
Figure 7: Adaptation comparison for different target domains in wireless communication benchmark.
The local reward is defined as the negative queue length, quantifying congestion each link:
re ri(S: 04) =
Σ
Tij-
(00)
JEA
Intuitively, the objective to minimize network-wide congestion by controlling local signals. Policies such as the max-pressure controller typically rely on known traffic-flow statisties, while our learning framework operates in a model-free fashion, adapting from observed data without prior knowledge of Cij or Riy distributions.
Evaluation protocol. We evaluate performance under M = 3 source domains, each defined by different vehicle flow intensities sampled from (low, medium, high} levels of Cty The target domain adopts an intermediate traffic level unless otherwise stated. Each experiment runs for T = 50 timesteps, with discount factor 7 0.95 and learning rates a 0.1 for the critic and 17 = 0.01 for the actor. For domain factor estimation, T. = 20 trajectories are collected per domain.
To assess adaptation, compare GSAC against SAC-LFS baseline trained directly in the target domain. We also vary the network topology by adjusting the number of intersections n E {9, 16, 25). corresponding
x 4, and x 5 grid layouts. As is shown in Figure g GSAC achieves faster adaptation, lower queue lengths, and improved stability across all configurations.
Domain shift sensitivity. To analyze robustness, we test target domains with altered inflow distribu- tions, varying Ctp by -20% relative to training domains. As is shown in Figure 9 GSAC consistently adapts these changes with minimal degradation in performance, whereas LFS requires substantially more data to achieve comparable results. These findings confirm the scalability and generalizability of our model-free learning framework in stochastic, networked traffic environments.
44

---

Grd Size
Grld Sice
Grid Sice
DMC
Aploode (innecc Phane)
babadde Jomem diitaran Phatoo)
08:0000 Passod
(a) grid size 3
(b) grid size 4
(c) grid size
Figure 8: Adaptation comparison for different grid sizes in traffic control benchmark.
target 0.4
target 0.6
thrget 0.7
Eplocd (Saner outer (hare)
Gaiede Conneaakaaain Phasoe)
12:054 (innerakiaa Fraton)
(a) Purget 0.4
(b) Purget 0.6
(c) Purget 0.7
Figure 9: Adaptation comparison for different target domains in traffic control benchmark..
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the contributions of GSAC-cause discovery, meta actor-critic learning, scalability, gencralizability, and theoretical guaran- tees tees-whiie which are consistently supported by the theoretical and empirical results (see Sections 1, 3, 4, and 6).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations such as dependence on causal identifiability, structural assumptions (e.g., bounded degree), and evaluation on a single benchmark are discussed in the main text (Section 5, 6. and can be inferred from the stated assumptions and settings.
Guidelines:
45

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texs system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions for theoretical results are explicitly stated in Section 6.1, 1, and detailed proofs are provided in the appendices (Appendix D)-F), with theorems referenced and numbered properly.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper describes the simulation environment, domain settings, train- ing/testing procedures, and evaluation metrics (Section and Appendix G), allowing others reproduce the key experiments.
Justification:
Guidelines:
46

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data are not currently provided, but we intend to release them with detailed instructions in the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guut http:///cc/giiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
47

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 7 and Appendix G provide detailed descriptions of the experimental settings, including training iterations, learning rates, adaptation horizons, and domain generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While performance plots are provided (Figure 1), statistical error bars or variance across multiple runs not reported due to limited computational budget.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Compute details such as hardware type and execution time are not explicitly mentioned in the current draft and will be added in the supplemental material.
48

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The work complies with the NeurIPS Code of Ethics. It does not involve sensitive data, human subjects, or applications with foreseeable risks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 8 briefly discusses applications to infrastructure and autonomous systems. While primarily theoretical, the methods could improve safety and efficiency, though care should be taken when deployed in critical systems.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploymennt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
49

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release models or datasets with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The wireless benchmark from Qu et al. (2022) is used and cited appropriately (Section References).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsswieeesseeeeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new datasets models released at submission time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
50

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve human subjects or require IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
The methodology and contributions do not involve large language models (LLMs).
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy for what should or should not be described.
51


Reviews:
[{"review_id": "rNJbs8SboD", "reviewer": "Reviewer_7kCG", "summary": "The paper proposes a framework which leverages causal representation learning within the meta actor-critic learning. The authors introduce approximately-compact representations, which truncate value functions to $\\kappa$-hop neighbourhood in order to improve learning efficiency. \nThe authors address the following theoretical and empirical aspects:\n- Structural identifiability in networked MARL: theoretical proofs for the sample complexity\n- Introducing approximately-compact representations (ACRs) and proposing algorithms\n- Meta-actor-critic (MAC) algorithm which is tasked with learning policies across source domains\n- Theoretical guarantees for the algorithm are provided", "questions": "The citation of the empirical task includes where the authors have got the dataset from (paper [2]), but not the original work [1]\n\n[1]Alessandro Zocca. Temporal starvation in multi-channel csma networks: an analytical framework. Queueing Systems, 91(3-4):241–263, 2019.\n[2] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing Systems, 33:2074–2086, 2020.", "limitations": "The authors mention that 'The major limitation is that GSAC is currently evaluated on a single fully-observed benchmark'. The authors are also suggested to tell about the limitation of the assumptions of the theoretical guarantees. \n\nIn other words, in which ways could the assumptions 1-6, line 256, not to be met in practice, and what would be the implications for the proposed method?", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\n- Soundness: while the paper leans towards the theoretical analysis of the proposed algorithm, the approach is sound and backed up with both theoretical and empirical analysis\n- Correctness: I have checked the derivations, and they appear correct\n- Clarity: the writing is mostly clear, with some suggestions in the disadvantages section\n- Originality: scalabiltity of MARL to large-scale network systems is an important open problem, which is of scope and interest to the NeurIPS community, using the causal structures for adaptation to the domain shifts is a promising and original idea\n\nWeaknesses:\n- Clarity: While it is good in general, I would think the paper would be improved by adding some sort of roadmap in the beginning of the methods section, showing how all these pieces, both theory and the algorithm, fit together into a single story. In Section 4.1, the authors state the parts of the Algorithm, but it would be also good if they linked these statements with the theorems.", "was_revised": true, "final_justification": "After reading the reviews, I believe that the authors have done a really good job addressing the reviews. I think the paper has merits which underpin my vote for acceptance: \n- Soundness: while the paper leans towards the theoretical analysis of the proposed algorithm, the approach is sound and backed up with both theoretical and empirical analysis\n- Correctness: I have checked the derivations, and they appear correct\n- Originality: scalability of MARL to large-scale network systems is an important open problem, which is of scope and interest to the NeurIPS community, using the causal structures for adaptation to the domain shifts is a promising and original idea\n\nDuring the rebuttal, the authors have addressed a number of concerns, including:\n-  clarity: the authors address how they will include the roadmaps, improved description of computational overhead, clarification on theoretical assumptions\n- deficiency of empirical analysis: addition of new experiments for the traffic control network task, which confirm the presented theoretical insights, a number of baselines are added\n\nTherefore, I suggest acceptance and that the authors update the paper according to the rebuttal discussion, which in my opinion would further strenghen the paper.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate your recognition of **soundness**, **originality** and **significance** of our work. We would like to address your concerns/comments as follows.\n### W1:  \"add some sort of roadmap in the beginning of the methods section, showing how all these pieces, both theory and the algorithm, fit together into a single story\" \"In Section 4.1, the authors state the parts of the Algorithm, but it would be also good if they linked these statements with the theorems\"\nWe appreciate the suggestion of adding a roadmap theorem-algorithm linkage in Section 4. In the revised version, \n- we introduce a **high-level roadmap** at the start of Section 4, outlining how causal discovery, ACR construction, and meta actor-critic optimization fit together into a unified story (Phases 1–4, Algorithm 1). \n- we explicitly link the algorithmic components with the corresponding theoretical results (Theorems 1–4 and Propositions 1–5).\n- we also add a **brief summary figure** (Figure 1) to visually illustrate the GSAC pipeline, complementing the roadmap. \n\nWe add **Roadmap and Theoretical Connections** at the start of Section 4 (right before Section 4.1, Algorithm overview). It combines the roadmap with explicit theorem-algorithm linkages:\n> Our GSAC framework (Algorithm 1) integrates causal discovery, representation learning, and meta actor-critic optimization into a unified pipeline, with each phase supported by theoretical results. **Phase 1** (causal discovery and domain factor estimation) is underpinned by **Theorem 1** and **Propositions 4–5**, which establish structural identifiability and sample complexity guarantees for recovering causal masks and latent domain factors. **Phase 2** (construction of ACRs) leverages the causal structure to build compact representations of value functions and policies, with bounded approximation errors rigorously characterized by **Propositions 1–3**. **Phase 3** (meta actor-critic learning) performs scalable policy optimization across multiple source domains, with convergence of the critic and actor updates guaranteed by **Theorem 2** (critic error bound) and **Theorem 3** (policy gradient convergence). Finally, **Phase 4** (fast adaptation to new domains) exploits the learned meta-policy and compact domain factors to achieve rapid adaptation, where the adaptation performance gap is formally controlled by **Theorem 4**. Together, these results demonstrate that each algorithmic component is theoretically justified and collectively leads to provable scalability and generalization in networked MARL.\n    \nWe  also add a one-paragraph explanation referring to Figure 1:\n> Figure 1 complements Algorithm 1 by showing how causal recovery and ACR construction (Phases 1–2) compress the state-action space, enabling efficient meta-policy training (Phase 3), and how this meta-policy rapidly adapts to new domains (Phase 4), as guaranteed by Theorems 2–4. \n### Q1: citation of the empirical task\nThank you for pointing it out. We have updated our references to cite the original work **[1]** on the multi-channel CSMA network model in Section 7 Numerical experiments, alongside the benchmark paper **[2]**. We also added references to cite this paper in Introduction on wireless communication systems.\n### Limitations: \"in which ways could the assumptions 1-6 not to be met in practice, and what would be the implications for the proposed method\"\nYou raised an important point regarding Assumptions 1–6 (line 256). **A1-A4** are standard assumptions for proving convergence of networked MARL algorithms, without consideration of domain generalization/adaptation  **[2-4]**. For example, **[4]** provides the first provably efficient MARL framework for networked systems under the discounted reward setting, and including the first convergence analysis of networked MARL algorithm under **A1-A4**. \n\nOur work further fills the gap between scalability and *generalizability across domains* in networked systems. To this aim, we introduce additional asumptions **A5-A6** regarding the latent domain factor. **A5** is similar with **A3**, which imposes the smoothness w.r.t. the domain factor $\\omega$, while **A3** imposes the smoothness w.r.t. the actor parameter $\\theta$. **A6** is a regularity assmption to ensure the compactness of domain factor space.\n\nWhile **A1-A6** are standard for establishing theoretical guarantees, we discuss whether they hold exactly in practice as follows:\n- **Finite MDP  (A1)**: in some systems, the state or action spaces might not finite or continuous. For such cases, our results can still extend by applying state aggregation or clipping methods to ensure boundedness while incurring small approximation error.\n- **Mixing properties (A2)**: real-world networked systems may exhibit slow mixing or non-Lipschitz dynamics. Our theoretical results would degrade gracefully, leading to larger constants $\\tau$ in the sample complexity bounds.\n- **Smoothness (A3)**: the assumption requires the policy gradients $\\nabla_{\\theta_i} \\log \\pi_i$ to be Lipschitz and uniformly bounded. This holds for all commonly used policy parameterizations (e.g., softmax policies) with bounded inputs and finite-dimensional parameter vectors. It ensures stable policy updates and is common in the convergence analysis of policy gradient and actor-critic algorithms.  $\\texttt{GSAC}$ uses softmax parameterizations for policies, where $\\log \\pi_{\\theta_i}$ is smooth and has bounded gradients when action probabilities are not close to zero (a condition satisfied by entropy regularization or initialization).\n- **Compact parameter space (A4)**: policy parameters $\\theta_i$ are always constrained within a finite range due to neural network weight regularization, weight clipping, or bounded initialization. This is standard in actor-critic methods, where optimization is performed with bounded learning rates and regularizers that implicitly keep the parameters within a compact set. \n- **Smoothness w.r.t. domain factor (A5)**: this is reasonable because domain factors, e.g., channel gains, traffic loads, represent physical parameters that vary smoothly. The local dynamics $P_i$ are typically analytic or polynomial in $\\omega_i$, so small perturbations in $\\omega_i$ induce proportionally small changes in state transitions and rewards. Consequently, the Q-function $Q^\\pi_i(s,a,\\omega)$ and policy gradient $\\nabla_{\\theta_i} J(\\theta,\\omega)$ inherit this smoothness. Moreover, standard policy parameterizations  and neural network approximators are Lipschitz when combined with bounded weights or gradient clipping. Thus, **A5** aligns well with both the structure of real-world systems and common RL practices. Moreover, even if the true dynamics were not strictly Lipschitz, domain factor estimation (Proposition 5) uses a small-perturbation assumption around $\\omega$ for adaptation. In practice, state or reward clipping and function approximation, which are inherently Lipschitz, effectively enforce this assumption.\n- **Compact domain factor space (A6)**: the latent domain factors $\\omega_i$ represent bounded variations in environment dynamics (e.g., channel gains, traffic intensities, or reward scaling factors), which are inherently limited by the physical system. In real-world networked systems, these factors are naturally confined to finite ranges determined by system design (e.g., transmission powers, queue lengths). Hence, assuming $\\Omega_i$ is compact with bounded diameter $D_\\Omega$ is both reasonable and practical.\n\nIn the revised version, we have added discusssions about these potential limitations and their implications in Section 8 and Appendix G, making clear how they might affect the performance of $\\texttt{GSAC}$ in less structured or highly stochastic environments.\n### Additional minor comments\nWe have corrected the typo at Line 220 (“weights them” → “weighs them”). \n***\n**References**\n\n[1] Alessandro Zocca. Temporal starvation in multi-channel csma networks: an analytical framework. Queueing Systems, 91(3-4):241–263, 2019. \n\n[2] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing systems, 33:2074–2086, 2020.\n\n[3] Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in stochastic networked systems. Advances in neural information processing systems,34:7825–7837, 2021.\n\n[4] Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022."}, {"review_id": "igdi5yGqb2", "reviewer": "Reviewer_Db8N", "summary": "This paper introduces GSAC (Generalizable and Scalable Actor-Critic), a multi-agent reinforcement learning framework tailored to large-scale networked systems (e.g., traffic, power, and wireless grids). GSAC leverages causal representation learning to identify sparse local causal masks and construct Approximately Compact Representations (ACRs) of both state and latent domain factors.  A meta actor-critic algorithm then trains a shared, domain-conditioned policy across multiple source environments; at test time, only a few trajectories in a new domain are needed to estimate its factors and adapt the policy.", "questions": "- For experiments, how many layers/parameters is the policy?\n- Why not try this on robotic simulation, such as Metaworld?\n- Why meta learning as opposed to multitask learning? Are there ablations regarding this?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Theoretical Rigor: Establishes structural identifiability of local causal masks (seen in Theorem 1), finite-sample bounds for mask recovery (Prop. 4), and convergence and adaptation guarantees for the meta actor-critic (Theorem 2–4).\n- Clear and structured Algorithm - Presents a four-phase pipeline (causal recovery, ACR construction, meta actor-critic training, rapid adaptation) with detailed pseudocode (Alg. 1).\n\nWeaknesses:\n- Computational Overhead: The causal discovery and ACR construction phases (Phase 1 and Phase 2 in Section 4.1) may incur significant upfront cost, and the required neighborhood masking and meta-training could be prohibitive at very large scales.\n- Minimal Experiment - This paper seems mainly theoretical and only has one experiment (and no baselines)\n- Strong Assumptions: Relies on faithfulness, sub-Gaussian noise, bounded degree, full observability, and Lipschitz continuity (Propositoin 4)—conditions that may not hold in many real systems.", "was_revised": true, "final_justification": "Better experiments and good explanation on ACR's low cost.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate your recognition of **theoretical rigor** and **clear and structured** algorithm. We would like to address your concerns/comments as follows.\n### W1: computational overhead of causal discovery and ACR construction\nWe acknowledge that Phase 1 (causal discovery) and Phase 2 (ACR construction) introduce upfront costs. However, these phases only incur **one-time, local, and amortized cost**. \n- They are one-time preprocessing steps for each source domain and do not need to be repeated during meta-training or adaptation. \n- Both steps are local (per agent and over small neighborhoods),  parallel across agents, and the results are re-used for the entire meta-training horizon and for adaptation. We make this explicit in Algorithm 1 (Phases 1–2 precede meta-training and adaptation).\n    \nIn addition, **ACRs reduce overall cost**. As shown in Section 3 and Theorem 2–4, ACRs reduce the effective input dimensionality from $O(|s_{N^k_i}|)$ to $O(|s_{N^{\\kappa}_i}|)$, resulting in quasi-linear scaling in the number of agents. This leads to smaller mixing times $\\tau$ and higher minimum visitation probabilities $\\sigma$, improving both convergence speed and sample efficiency (see Section 6.3).\nOverall, while causal discovery and ACR construction are non-trivial, they are (i) local/parallel, (ii) one-time per source domain, and (iii) reduce the total training/adaptation cost via smaller representations and better constants in the convergence bounds.\n### W2: seems mainly theoretical and only has one experiment (and no baselines)\nOur primary contribution was to establish the theoretical foundations of GSAC, including the structural identifiability, convergence bounds, and adaptation guarantees. However, we admit that broader empirical coverage will strengthen the paper. Here are clarifications:\n- **What is already included**: Appendix H details multiple grid sizes (3×3, 4×4, 5×5), target-domain sweeps for the target packet-arrival rate, and a Learning-From-Scratch ($\\texttt{LFS}$) baseline trained directly in the target domain. Across settings, GSAC adapts in far fewer steps and outperforms $\\texttt{LFS}$ (see Fig. 1b, 4–6)\n- **Add benchmark**: we extend experiments to **traffic control network [1,2]** on a road-link graph, where  state per link is queue counts, action controlling the on-oﬀ of turn movement, and reward is negative total queue length. The preliminary results already show consistent advatanges (much faster adaptation compared to $\\texttt{LFS}$).\n\n| Adaptation Episode | $\\texttt{GSAC}$ Return | $\\texttt{LFS}$ Return | $\\texttt{GSAC}$ Queue | $\\texttt{LFS}$ Queue |  Return Difference |  Queue Difference |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| 1 | -54.9942 | -57.9916 | 4.7782 | 5.1239 | 2.9973 | 0.3457 |\n| 50 | -54.5165 | -57.4359 | 4.7710 | 5.0569 | 2.9194 | 0.2859 |\n| 150 | -54.2193 | -56.0777 | 4.7234 | 4.9296 |1.8584 | 0.2061 |\n| 300 | -53.2257 | -55.0790 | 4.6568 | 4.7744 | 1.8534 | 0.1176 |\n| 400 | -53.1087 | -53.8259 | 4.6359 | 4.6433 | 0.7172 | 0.0073 |\n\n- **Add baselines**: we implement **two additional baselines**, (i) a multi-task variant of GSAC, (ii) a domain-factor-free baseline. We put the experimental results and discussions to the response to **Q3**.\n### W3: \"Relies on faithfulness, sub-Gaussian noise, bounded degree, full observability, and Lipschitz continuity (Proposition 4)—conditions that may not hold in many real systems\"\nOur assumptions in Proposition 4 are standard in causal discovery and theoretical MARL. \n- **Faithfulness and sub-Gaussian noise**: these are classical and standard requirements to ensure identifiability **[3,4]** and finite-sample recovery. \n- **Bounded degree**: bounded neighborhood degree is natural in networked systems where agents have limited communication range. For example, this holds when each intersection connects to only a few roads, or each wireless node has limited interference range.\n- **Full observability**: while we currently assume full local observability, the ACR construction can be extended to partially observable settings using learned belief states or latent encoders. This is part of our future work.\n- **Lipschitz continuity**: this mild smoothness assumption is satisfied by most smooth policies.\n\nImportantly, our algorithm can still be applied when assumptions are violated, only the theoretical guarantees may not hold exactly. We have added a “Limitations and assumptions” paragraph to discuss where these conditions may fail and how GSAC could be extended.\n### Q1: layers/parameters of the policy\nIn the wireless benchmark the local action space is discrete, and we **do not use a neural network** for the policy. We use a softmax (Boltzmann) policy over local actions conditioned on ACR inputs:\n $$  \\pi(a|s,\\omega) = \\text{softmax}( \\theta[a]^\\top (s,\\omega)/\\tau ),  $$\nwhere $\\theta\\in\\mathbb{R}^{|A|\\times d}$ is the parameter matrix, $(s,\\omega)\\in\\mathbb{R}^{d}$ are the ACR features, and $\\tau=0.5$ is the softmax temperature. \n### Q2: Why not try this on robotic simulation, such as Metaworld?\nOur focus is on is large-scale networked systems (traffic, power, wireless), where agents interact over a graph and where $\\kappa$-hop truncation and local causal masks are principled. Metaworld **[5]** tasks lack the explicit networked graph structure that GSAC is designed to exploit. It is primarily *single-agent*, and running multiple independent MetaWorld tasks in parallel does not induce the structured inter-agent dynamics our theory leverages. Our theorems and ACR constructions are explicitly graph- and neighborhood-based. That said, applying GSAC to robotic settings with explicit interaction graphs (e.g., multi-robot navigation with communication/collision constraints or multi-arm coordination with shared resources) is a natural and valuable next step. We have added discussion of this direction in the revised paper.\n### Q3: Why meta learning as opposed to multitask learning? Are there ablations regarding this?\n**Clarification of meta-RL vs. multi-task RL paradym**: the goal of multi-task RL **[5]** is to learn a **single, task conditioned** policy that maximize the average expected return across all source domains from a domain distribution. However, it offers no guarantee of rapid few-shot adaptation from new, unseen domain. Meta-RL, in contrast, explicitly learns to adapt: it leverage the set of source domains to learn a policy that, at test time,  can quickly adapt to a new domain with few trajectories.\n\n**Why meta-learning**: our goal is is **fast adaptation in a new domain from few trajectories**. We train a shared, **domain-factor-conditioned** policy across sources and, at test time, estimate domain factors $\\hat\\omega$ from a few trajectories to adapt immediately. Our theory shows:\n- Meta-gradient error terms shrink with the **number of source domains $M$** and the **samples used to estimate domain factors $T_e$**.\n- The **adaptation gap decays as $O(1/\\sqrt{T_a})$** with the number of adaptation trajectories.\n\n**New baselines**: \n- $\\texttt{SAC-MTL}$ (multi-task): learns  $\\pi_{\\theta}(a|s,z)$, where $z$ indicates the **onehot encoding** of the task; optimized jointly over source domains **[5]**. \n- $\\texttt{SAC-FT}$ (fine-tune, suggested by **Reviewer y5CC**):  trains a single policy $\\pi_{\\theta}(a|s)$ across sources **without domain-factor conditioning** and **fine-tunes $\\theta$**.\n- $\\texttt{GSAC}$ (ours): learns $\\pi_{\\theta}(a|s,\\omega)$, and adapts by adjusting $\\omega$ (estimated from a few trajectories), not $\\theta$\n\nReturn evaluation on a 4×4 network (source domain $p_i \\in \\\\{0.2, 0.5, 0.8\\\\}$, $p_{target}=0.65$, higher is better). “Ep” denotes adaptation episodes collected in the target domain.\n| Ep | $\\texttt{GSAC}$ | $\\texttt{SAC-MTL}$ | $\\texttt{SAC-FT}$ | $\\texttt{LFS}$ |\n|:-:|-----:|-----:|-----:|-----:|\n|1|**3.5093**|2.6802|3.3715|2.4125|\n|10|3.5314|2.6870|3.3283|2.4249|\n|30|3.5537|2.7723|3.3681|2.6146|\n|50 |3.5573|2.8693|3.3337|2.7913|\n|100 |3.5514|3.2814|3.3677|3.0481|\n|**130** |3.5350|**3.5466**|3.4910|3.1683|\n|150 |3.5463|3.6869|3.4841|3.1947|\n|200 |3.5552|3.7961|3.5201|3.3476|\n|**250** |3.5517|3.7882|**3.5602**|3.4625|\n|**300**|3.5465|3.8011|3.6294|**3.5623**|\n\nInitially, performance ranks $\\texttt{GSAC}$>$\\texttt{SAC-FT}$>$\\texttt{SAC-MTL}$>$\\texttt{LFS}$, gaps that persist through Ep 50, highlighting sizable initial gains over $\\texttt{LFS}$ (+45% for $\\texttt{GSAC}$, +40% for $\\texttt{SAC-FT}$, +11% for $\\texttt{SAC-MTL}$). $\\texttt{GSAC}$ achieves the best few-shot performance (Ep 1–50), reflecting rapid adaptation from minimal target data. $\\texttt{SAC-MTL}$ overtakes after ~130 episodes and remains best thereafter. $\\texttt{SAC-FT}$ outperforms $\\texttt{SAC-MTL}$ initially but lags it later, and surpasses $\\texttt{GSAC}$ after ~250 episodes. $\\texttt{LFS}$ improves the slowest and only exceeds $\\texttt{GSAC}$ after ~300 episodes. This matches our design goal: optimize early-episode adaptation by adjusting $\\omega$ rather than re-training $\\theta$.\n\n***\n**References**\n\n[1] Varaiya, P. Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177-195, 2013.\n\n[2] Guannan Qu, et al. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.\n\n[3] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.\n\n[4] Huang, Biwei, et al. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729 (2021).\n\n[5] Tianhe Yu, et al. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Proceedings of the Conference on Robot Learning, PMLR 100:1094-1100, 2020."}, {"review_id": "NjaVRB3kkA", "reviewer": "Reviewer_y5CC", "summary": "The paper studies the problem of designing a provably generalizable and scalable MARL algorithm for networked systems. Specifically, building on the exponential decay property proposed in previous works, they construct approximately compact representations of states and domain factors, which can further reduce the dimension of the truncated local Q-function. They also propose a meta actor-critic algorithm which enables learning scalable localized policies across multiple source domains. Theoretically, they provide guarantees on finite sample convergence of their algorithm as well as the adaptation gap. They also perform a numerical experiment on a wireless communication problem, where they demonstrated that their algorithm performed better than one than trained from scratch.", "questions": "1) Could the authors discuss an example of a realistic problem where the ACR reduces dimensionality significantly? For instance, is this the case in the wireless communication simulations? This can help strengthen this component of the contribution.\n\n2) Could the authors discuss how the sample complexity of adapting to a new domain depends on the similarity of the new domain to earlier domains seen in training? The current relevant result (Thereom 4) only shows the sample complexity in expectation.\n\n3) Could the authors comment on how their approach can be modified to handle the case with unknown $f_{i,j}$?\n\n4) How might the algorithm compare against an alternative benchmark, where the controller is trained with a variety of domains, and then allowed to fine-tune on the new domain data, without explicitly conditioning the policy on the domain factor? This seems to me a more reasonable benchmark than training from scratch. Performing well versus this baseline will add more value to the contribution of the paper.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe idea of conditioning the policy and value function on domain factors is interesting, and as the empirical results suggest, this seems to facilitate quicker adaption when a new domain is seen at test-time. The authors also provide rigorous theoretical analysis for the finite-time convergence of their algorithm.\n\n**Weaknesses*\n\n1) I am not sure how much value the approximate compact representation (ACR) can bring. From what I read, the authors did not give any concrete examples of realistic problems where this reduces dimensionality.\n\n2) I believe a crucial assumption required for estimation of the domain factor is that the dynamics function $f_{i,j}$ is known, so that $\\nabla_{\\omega} f_{i,j}$ can be computed. This limits the approach to only be valid for cases where the dynamics function is known, which I believe the authors did not explicitly mention in the paper.\n\n3) The algorithm only applies to finite state and action spaces, while some recent MARL network algorithms can in fact achieve provable and efficient performance for continuous state and action space network MARL problems, which arises often in practice. Since an important aspect of the algorithm is its potential for empirical improvement (the theoretical aspects of the algorithm are not particularly novel compared to previous network MARL algorithms), this is an important drawback.  \n\n4) The empirical validation is rather limited, since the authors only consider one benchmark, and the latent domain factor in this problem is quite simple (since it is one-dimensional). \n\n5) There are some typos which the authors should be aware of. For instance, in Line 126, I believe the summation should be replaced by a product (the input dimensionality is a product of the local input dimensions).", "was_revised": true, "final_justification": "The authors have mostly addressed my concerns, and I will thus increase my score.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate your recognition of **theoretical rigor**. We would like to address your concerns/comments as follows.\n### W1/Q1: \"how much value the ACR can bring\" \"example of a realistic problem where this reduces dimensionality\"\nIn our wireless benchmark, the local state of each user $i$ is represented as $s_i(t) = (b_{i,1}(t), \\ldots, b_{i,d_i}(t), z_{i,1}(t), z_{i,2}(t))$. $b_i(t) \\in \\\\{0,1\\\\}^{d_i}$ encodes queue status by deadline bins (e.g., $b_i(t)=(1,1,0)$ means the first two deadline bins are occupied while the third is empty). $z_{i,1}(t)$ and $z_{i,2}(t)$ are **irrelevant components** to decision quality:\n\n- $z_{i,1}(t)$: channel quality indicator, which does not influence the packet success probability $q_i$, as $q_i$ depends solely on collisions.\n- $z_{i,2}(t)$: grid coordinates, which adds no useful information because the connectivity graph already encodes all relevant interactions.\n    \nThe ACR procedure **automatically prunes** such irrelevant or redundant variables, retaining only those that causally influence the local reward or dynamics. This yields a **non-trivial reduction** in local dimensionality. For example, on a $4\\times4$ grid with $d_i=2$ and a neighborhood size $|N_i|=5$:\n\n- **Raw (truncation-only):** $|N_i|\\times(d_i+2)=5\\times4=20$ features. \n- **With ACR:** $|N_i|\\times d_i=5\\times2=10$ features.\n\nThis **50% shrinkage** directly translates into:\n\n- **Smaller critics and lower compute:** reduced local state–action support and table size. \n- **Faster learning:** lower variance in actor updates and tighter constants in our bounds. \n- **Accuracy preserved:** approximation error remains controlled (see Prop. 1–2). \n- **Improved sample efficiency:** by effectively increasing the minimum visitation $\\sigma$ and reducing the mixing constant $\\tau$ (Assumption 2), ACRs **tighten the constants** in the convergence guarantee (Thm 2).\n\nMore generally, ACR is valuable when:\n- State representations include auxiliary information, e.g., sensor readings not directly affecting rewards\n- Network structure creates redundancies, e.g., traffic flow where only upstream neighbors matter\n- Domain factors have sparse influence, e.g., weather affecting only outdoor equipment in power grids\n\nWe have included this concrete wireless example, and highlighted which variables are pruned and by how much, along with mask sparsity and critic-size statistics, in the revised paper.\n### W2/Q3: \"crucial assumption required for domain factor estimation\" \"how to modifify to handle the case with unknown $f_{i,j}$\"\nWe would like to clarify that our approach does **not** assume explicit knowledge of the closed-form dynamics $f_{i,j}$. We estimate domain factors using observed trajectories and causal masks (see Section 5, Lines 226–233). The structural matrices $c_{\\cdot \\to \\cdot}$ are identifiable under standard causal discovery assumptions (Thm 1). As shown in Appendix F (Prop. 4 and 5),  the domain factor $\\omega$ are estimated  from observed transitions using a maximum likelihood estimation. This likelihood can be evaluated using a learned parametric approximation $\\hat{f}$ rather than the true $f$.  Thus, our analysis and results remain valid for **unknown dynamics**, provided that $\\hat{f}$ is a consistent estimator of the transition function. Furthermore, Prop. 5 in Appendix F ensures that as long as $\\hat{f}$ approximates $f$ within $\\epsilon$, the error in domain factor estimation increases by at most $O(\\epsilon)$ due to the Lipschitz continuity of the transition likelihood. To avoid confusion, we have clarified these details in the revised version.\n\n### W3: only applies to finite state and action spaces\n**Scope**. Thank you for raising this point. Our contribution targets the gap between **provably scalable, localized MARL on network graphs** (where most analyses are tabular/finite) and cross-domain generalization/adaptation. To the best of our knowledge, we provide the *first unified algorithm and theoretical analysis* that couples causal locality (ACRs) with domain-factor conditioning and delivers finite-sample convergence and adaptation guarantees for networked systems. We instantiate the theory in finite spaces to keep assumptions transparent and to isolate the impact of ACRs and meta-conditioning without function-approximation confounders.\n\n**Extensions**. The ACR and GSAC framework extend naturally to **continuous** spaces by (i) replacing tabular modules with function approximators, and (ii) using entropy-regularized localized actor–critic updates. Because ACRs prune inputs rather than restrict the function class, the variance/conditioning benefits remain, and the exponential-decay/causal-mask assumptions carry over under mild Lipschitz/mixing conditions. In this paper we focus on the finite-space instantiation and position the continuous extension as **complementary future work**. In the revision we have added a concise discussion of assumptions, connections to continuous MARL, and implementation pointers, and we explicitly cite representative continuous MARL works to position our approach as complementary.\n\n### W4: only one benchmark, the latent domain factor is quite simple\nThe wireless environment is widely used as a stress test for decentralized MARL algorithms **[1]**. Despite its simple domain factor (packet arrival probability $p_i$), the interaction graph and collision dynamics induce highly nontrivial coordination challenges, especially as grid size grows (Fig 2). That said, we agree that broader empirical validation would strengthen the work.\n- **Add benchmark**: we extend experiments to **traffic control network [1,2]** on a road-link graph, where  state per link is queue counts, action controlling the on-oﬀ of turn movement, and reward is negative total queue length. The preliminary results already show consistent advatanges (much faster adaptation compared to $\\texttt{LFS}$).\n\n| Adap. Ep.| GSAC Return | LFS Return | GSAC Queue | LFS Queue |  Return Diff. |  Queue Diff. |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n|1 |-54.9942 |-57.9916 |4.7782 |5.1239 |2.9973 |0.3457|\n|50 |-54.5165 |-57.4359 |4.7710 |5.0569 |2.9194 |0.2859|\n|150 |-54.2193 |-56.0777 |4.7234 |4.9296 |1.8584 |0.2061|\n|300 |-53.2257 |-55.0790 |4.6568 |4.7744 |1.8534 |0.1176|\n|400 |-53.1087 |-53.8259 |4.6359 |4.6433 |0.7172 |0.0073|\n\n- **Add baselines**: we implement **two additional baselines**, (i) a multi-task variant of GSAC, (ii) a domain-factor-free baseline. We put the detailed experimental results and discussions to the response to **Q4**.\n### W5: typos\nThank you for pointing this out. We have corrected this typo and carefully proofread the paper.\n### Q2: how the sample complexity of adapting to a new domain depends on the similarity of the new domain to earlier domains seen in training? \nTheorem 4 bounds the adaptation gap as $O(1/\\sqrt{T_a})$. The key idea of proving Theorem 4 is the inequality\n$$J(\\theta(K),\\hat{\\omega}^{M+1};\\omega^{M+1})-J(\\theta(K),\\omega^{M+1};\\omega^{M+1}) \\ge -L_{\\omega'} || \\hat{\\omega}^{M+1}-\\omega^{M+1} ||.$$\nThus, when target domains are close to training domains, the total variation distance $|| \\hat{\\omega}^{M+1}-\\omega^{M+1} ||$ is smaller. This leads to smaller adaptation gap, resulting in faster adaptation. We have added this discussion in the revised version.\n### Q4: compare against the controller trained with a variety of domains, and then allowed to fine-tune on the new domain data, without explicitly conditioning the policy on the domain factor\nThank you for this excellent suggestion. Following your and **ReviewerDb8N**' suggestions, we add two new baseliness:\n- $\\texttt{SAC-MTL}$ (multi-task): learns  $\\pi_{\\theta}(a|s,z)$, where $z$ indicates the **onehot encoding** of the task; optimized jointly over source domains **[5]**. \n- $\\texttt{SAC-FT}$ (fine-tune, suggested by **Reviewer y5CC**):  trains a single policy $\\pi_{\\theta}(a|s)$ across sources **without domain-factor conditioning** and **fine-tunes $\\theta$**.\n- $\\texttt{GSAC}$ (ours): learns $\\pi_{\\theta}(a|s,\\omega)$, and adapts by adjusting $\\omega$ (estimated from a few trajectories), not $\\theta$\n\nReturn evaluation on a 4×4 network (source domain $p_i \\in \\\\{0.2, 0.5, 0.8\\\\}$, $p_{target}=0.65$, higher is better). “Ep” denotes adaptation episodes collected in the target domain.\n| Ep | $\\texttt{GSAC}$ | $\\texttt{SAC-MTL}$ | $\\texttt{SAC-FT}$ | $\\texttt{LFS}$ |\n|:-:|-----:|-----:|-----:|-----:|\n|1|**3.5093**|2.6802|3.3715|2.4125|\n|10|3.5314|2.6870|3.3283|2.4249|\n|30|3.5537|2.7723|3.3681|2.6146|\n|50 |3.5573|2.8693|3.3337|2.7913|\n|100 |3.5514|3.2814|3.3677|3.0481|\n|**130** |3.5350|**3.5466**|3.4910|3.1683|\n|150 |3.5463|3.6869|3.4841|3.1947|\n|200 |3.5552|3.7961|3.5201|3.3476|\n|**250** |3.5517|3.7882|**3.5602**|3.4625|\n|**300**|3.5465|3.8011|3.6294|**3.5623**|\n\nInitially, performance ranks $\\texttt{GSAC}$>$\\texttt{SAC-FT}$>$\\texttt{SAC-MTL}$>$\\texttt{LFS}$, gaps that persist through Ep 50, highlighting sizable initial gains over $\\texttt{LFS}$ (+45% for $\\texttt{GSAC}$, +40% for $\\texttt{SAC-FT}$, +11% for $\\texttt{SAC-MTL}$). $\\texttt{GSAC}$ achieves the best few-shot performance (Ep 1–50), reflecting rapid adaptation from minimal target data. $\\texttt{SAC-MTL}$ overtakes after ~130 episodes and remains best thereafter. $\\texttt{SAC-FT}$ outperforms $\\texttt{SAC-MTL}$ initially but lags it later, and surpasses $\\texttt{GSAC}$ after ~250 episodes. $\\texttt{LFS}$ improves the slowest and only exceeds $\\texttt{GSAC}$ after ~300 episodes. This matches our design goal: optimize early-episode adaptation by adjusting $\\omega$ rather than re-training $\\theta$.\n\n***\n**References**\n\n[1] Guannan Qu, et al. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.\n\n[2] Varaiya, P. Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177-195, 2013."}]

Author rebuttal:

Meta-review: The paper proposes a policy learning approach which leverages causal representation learning within the meta actor-critic learning. The reviewers agree this work contributes to this line of research, introducing GSAC (Generalizable and Scalable Actor-Critic), a multi-agent reinforcement learning framework tailored to large-scale networked systems (e.g., traffic, power, and wireless grids), which leverages causal representation learning to identify sparse local causal masks and construct Approximately Compact Representations (ACRs) of both state and latent domain factors; addressing theoretical and empirical challenges. 

Other favourable comments included that the approach is sound and backed up with both theoretical and empirical analysis, the idea of conditioning the policy and value function on domain factors is interesting and seems to facilitate quicker adaption when a new domain is seen at test-time as the empirical results suggest, and scalabiltity of MARL to large-scale network systems is an important open problem, using the causal structures for adaptation to the domain shifts is a promising and original idea which is of scope and interest to the NeurIPS community.

There was clear support for the recommendation to accept this work. Importantly, reviewers diligently suggested some potential improvements during the discussion, to clear questions on the experiments and the assumptions. I strongly encourage the authors to integrate them into the revised version of their paper conscientiously.
Decision: Accept (spotlight)