Title: Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Networked Systems

Domain Generalization

Domain Adaptation

Causal Structure Learning
Abstract: Large‑scale networked systems, such as traffic, power, and wireless grids, challenge reinforcement‑learning agents with both scale and environment shifts. To address these challenges, we propose \texttt{GSAC} (\textbf{G}eneralizable and \textbf{S}calable \textbf{A}ctor‑\textbf{C}ritic), a framework that couples  causal representation learning with meta actor‑critic learning to achieve both scalability and domain generalization. Each agent first learns a sparse local causal mask that provably identifies the minimal neighborhood variables influencing its dynamics, yielding exponentially tight approximately compact representations (ACRs) of state and domain factors. These ACRs bound the error of truncating value functions to $\kappa$-hop neighborhoods, enabling efficient learning on graphs. A meta actor‑critic then trains a shared policy across multiple source domains while conditioning on the compact domain factors; at test time, a few trajectories suffice to estimate the new domain factor and deploy the adapted policy. We establish finite‑sample guarantees on causal recovery, actor-critic convergence, and adaptation gap, and show that \texttt{GSAC} adapts rapidly and significantly outperforms learning-from-scratch and conventional adaptation baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of designing a provably generalizable and scalable MARL algorithm for networked systems. Specifically, building on the exponential decay property proposed in previous works, they construct approximately compact representations of states and domain factors, which can further reduce the dimension of the truncated local Q-function. They also propose a meta actor-critic algorithm which enables learning scalable localized policies across multiple source domains. Theoretically, they provide guarantees on finite sample convergence of their algorithm as well as the adaptation gap. They also perform a numerical experiment on a wireless communication problem, where they demonstrated that their algorithm performed better than one than trained from scratch.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe idea of conditioning the policy and value function on domain factors is interesting, and as the empirical results suggest, this seems to facilitate quicker adaption when a new domain is seen at test-time. The authors also provide rigorous theoretical analysis for the finite-time convergence of their algorithm.\n\n**Weaknesses*\n\n1) I am not sure how much value the approximate compact representation (ACR) can bring. From what I read, the authors did not give any concrete examples of realistic problems where this reduces dimensionality.\n\n2) I believe a crucial assumption required for estimation of the domain factor is that the dynamics function $f_{i,j}$ is known, so that $\\nabla_{\\omega} f_{i,j}$ can be computed. This limits the approach to only be valid for cases where the dynamics function is known, which I believe the authors did not explicitly mention in the paper.\n\n3) The algorithm only applies to finite state and action spaces, while some recent MARL network algorithms can in fact achieve provable and efficient performance for continuous state and action space network MARL problems, which arises often in practice. Since an important aspect of the algorithm is its potential for empirical improvement (the theoretical aspects of the algorithm are not particularly novel compared to previous network MARL algorithms), this is an important drawback.  \n\n4) The empirical validation is rather limited, since the authors only consider one benchmark, and the latent domain factor in this problem is quite simple (since it is one-dimensional). \n\n5) There are some typos which the authors should be aware of. For instance, in Line 126, I believe the summation should be replaced by a product (the input dimensionality is a product of the local input dimensions).\n\nQUESTIONS:\n1) Could the authors discuss an example of a realistic problem where the ACR reduces dimensionality significantly? For instance, is this the case in the wireless communication simulations? This can help strengthen this component of the contribution.\n\n2) Could the authors discuss how the sample complexity of adapting to a new domain depends on the similarity of the new domain to earlier domains seen in training? The current relevant result (Thereom 4) only shows the sample complexity in expectation.\n\n3) Could the authors comment on how their approach can be modified to handle the case with unknown $f_{i,j}$?\n\n4) How might the algorithm compare against an alternative benchmark, where the controller is trained with a variety of domains, and then allowed to fine-tune on the new domain data, without explicitly conditioning the policy on the domain factor? This seems to me a more reasonable benchmark than training from scratch. Performing well versus this baseline will add more value to the contribution of the paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GSAC (Generalizable and Scalable Actor-Critic), a multi-agent reinforcement learning framework tailored to large-scale networked systems (e.g., traffic, power, and wireless grids). GSAC leverages causal representation learning to identify sparse local causal masks and construct Approximately Compact Representations (ACRs) of both state and latent domain factors.  A meta actor-critic algorithm then trains a shared, domain-conditioned policy across multiple source environments; at test time, only a few trajectories in a new domain are needed to estimate its factors and adapt the policy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Theoretical Rigor: Establishes structural identifiability of local causal masks (seen in Theorem 1), finite-sample bounds for mask recovery (Prop. 4), and convergence and adaptation guarantees for the meta actor-critic (Theorem 2–4).\n- Clear and structured Algorithm - Presents a four-phase pipeline (causal recovery, ACR construction, meta actor-critic training, rapid adaptation) with detailed pseudocode (Alg. 1).\n\nWeaknesses:\n- Computational Overhead: The causal discovery and ACR construction phases (Phase 1 and Phase 2 in Section 4.1) may incur significant upfront cost, and the required neighborhood masking and meta-training could be prohibitive at very large scales.\n- Minimal Experiment - This paper seems mainly theoretical and only has one experiment (and no baselines)\n- Strong Assumptions: Relies on faithfulness, sub-Gaussian noise, bounded degree, full observability, and Lipschitz continuity (Propositoin 4)—conditions that may not hold in many real systems.\n\nQUESTIONS:\n- For experiments, how many layers/parameters is the policy?\n- Why not try this on robotic simulation, such as Metaworld?\n- Why meta learning as opposed to multitask learning? Are there ablations regarding this?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework which leverages causal representation learning within the meta actor-critic learning. The authors introduce approximately-compact representations, which truncate value functions to $\\kappa$-hop neighbourhood in order to improve learning efficiency. \nThe authors address the following theoretical and empirical aspects:\n- Structural identifiability in networked MARL: theoretical proofs for the sample complexity\n- Introducing approximately-compact representations (ACRs) and proposing algorithms\n- Meta-actor-critic (MAC) algorithm which is tasked with learning policies across source domains\n- Theoretical guarantees for the algorithm are provided\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Soundness: while the paper leans towards the theoretical analysis of the proposed algorithm, the approach is sound and backed up with both theoretical and empirical analysis\n- Correctness: I have checked the derivations, and they appear correct\n- Clarity: the writing is mostly clear, with some suggestions in the disadvantages section\n- Originality: scalabiltity of MARL to large-scale network systems is an important open problem, which is of scope and interest to the NeurIPS community, using the causal structures for adaptation to the domain shifts is a promising and original idea\n\nWeaknesses:\n- Clarity: While it is good in general, I would think the paper would be improved by adding some sort of roadmap in the beginning of the methods section, showing how all these pieces, both theory and the algorithm, fit together into a single story. In Section 4.1, the authors state the parts of the Algorithm, but it would be also good if they linked these statements with the theorems.\n\nQUESTIONS:\nThe citation of the empirical task includes where the authors have got the dataset from (paper [2]), but not the original work [1]\n\n[1]Alessandro Zocca. Temporal starvation in multi-channel csma networks: an analytical framework. Queueing Systems, 91(3-4):241–263, 2019.\n[2] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing Systems, 33:2074–2086, 2020.\n\nLIMITATIONS:\nThe authors mention that 'The major limitation is that GSAC is currently evaluated on a single fully-observed benchmark'. The authors are also suggested to tell about the limitation of the assumptions of the theoretical guarantees. \n\nIn other words, in which ways could the assumptions 1-6, line 256, not to be met in practice, and what would be the implications for the proposed method?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-09"}