Title: Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
Venue: NeurIPS 2025
Accepted: yes
Keywords: Nash equilibrium

zero-shot generalization

pursuit-evasion game

reinforcement learning

dynamic programming

policy decomposition

shortest path distance

graph neural network
Abstract: Equilibrium learning in adversarial games is an important topic widely examined in the fields of game theory and reinforcement learning (RL). Pursuit-evasion game (PEG), as an important class of real-world games from the fields of robotics and security, requires exponential time to be accurately solved. When the underlying graph structure varies, even the state-of-the-art RL methods require recomputation or at least fine-tuning, which can be time-consuming and impair real-time applicability. This paper proposes an Equilibrium Policy Generalization (EPG) framework to effectively learn a generalized policy with robust cross-graph zero-shot performance. In the context of PEGs, our framework is generally applicable to both pursuer and evader sides in both no-exit and multi-exit scenarios. These two generalizability properties, to our knowledge, are the first to appear in this domain. The core idea of the EPG framework is to train an RL policy across different graph structures against the equilibrium policy for each single graph. To construct an equilibrium oracle for single-graph policies, we present a dynamic programming (DP) algorithm that provably generates pure-strategy Nash equilibrium with near-optimal time complexity. To guarantee scalability with respect to pursuer number, we further extend DP and RL by designing a grouping mechanism and a sequence model for joint policy decomposition, respectively. Experimental results show that, using equilibrium guidance and a distance feature proposed for cross-graph PEG training, the EPG framework guarantees desirable zero-shot performance in various unseen real-world graphs. Besides, when trained under an equilibrium heuristic proposed for the graphs with exits, our generalized pursuer policy can even match the performance of the fine-tuned policies from the state-of-the-art PEG methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using reinforcement learning (RL) to learn pursuit-evasion game (PEG) policies that generalize across different graph environments. The authors simulate the adversary's policy using a dynamic programming (DP) algorithm for no-exit Markov PEGs and a heuristic for multi-exit cases. They then train a soft-actor-critic agent against this equilibrium adversary. Experiments on real-world graphs demonstrate zero-shot generalization of the learned policy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1) The paper shows that training RL agents against an equilibrium adversary on a graph corpus enables generalization to unseen graphs.\n2) Experiments on diverse real-world maps demonstrate the method's strong zero-shot performance.\n3) A sequential decision-making framework is proposed to address the large joint action space, enabling decentralized execution.\n\n**Weaknesses**:\n1) The claim that the RL method is computationally more efficient than traditional DP is not supported. The DP baseline reaches equilibrium in just 10 seconds (as mentioned in Line 569), raising questions about the RL method’s efficiency given its additional training cost.\n2) The second claim—that the RL approach avoids overfitting to specific behaviors—would benefit from further evidence. As both training and testing rely on Nash equilibrium policies, it remains unclear how the method performs against other types of adversaries, such as random, SPS, or alternative RL agents in testing environments.\n3) The training and test graphs are insufficiently differentiated, making the claimed generalization unclear.\n4) The method is tested on a limited number of agents; its scalability to larger settings (e.g., >10 pursuers) is untested.\n\nQUESTIONS:\n1) How does the RL algorithm perform when tested against non-equilibrium policies (e.g., random or heuristic agents)? Also, what is the performance of $\\text{RL}_p$–$\\text{RL}_e$ in Table 1?\n2) Please provide more details on the graphs used in Table 1 (e.g., number, average degree, diameter). Also, the authors could explain why $\\text{RL}_p$–DP underperforms on graphs like Hollywood Walk of Fame, Sagrada Familia, and The Bund, to help the readers have a full understand of the proposed method.\n3) There is an inconsistency in Table 3: Line 572 mentions 1,000 trials, while the caption states 100 episodes. Please clarify the correct number. Additionally, since computational efficiency is a key contribution, it would be more informative to report and compare the average computation time per episode for EPG and DP in the main text rather than in the appendix.\n\nLIMITATIONS:\nThe proposed method’s scalability is constrained by the shortest-path-based state representation, which incurs O($n^3$) complexity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work describes a framework to generally solve pursuit-evasion graph games. These are games where pursuer and evader agents exist on a graph and the pursuers must catch the evaders in a zero-sum setting. The authors propose an equilibrium policy generation method to learn generalized policies than can work zero-shot on unseen graph structures. They train RL agents against oracle policies which are discovered using dynamic programming. \n\nMy current rating is borderline reject. I would be willing to increase this score is better statistical data is provided, and if the paper is updated with a proper comparison of EPG in the no-exit scenarios against appropriate baselines, or if at least a good justification is provided for not comparing against them.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The use of RL in this problem is well-motivated, as existing methods for PEGs are computationally intensive to fine-tune, and are not guaranteed to generalize across different graph structures.\n2. The construction of oracle policies for training using DP is well-explained: the authors walk through a logical progression of the near-optimality of their DP approach with no-exit graphs, then an extension to exits, and then an extension to many pursuers. I have not rigorously checked the proof of Theorem 1.\n3. The use of an attention mechanism to encode the observations of changing graph structures is well explained.\n4. The experiments evince good results for EPG with the proposed training scheme and policy loss and distance features.\n\nWeaknesses:\n1.  Definition of the NE on pg 3 should include 'at all states'. Also, a Markov game must be finite to guarantee the existence of a NE. Suitable citations should also be provided for the Nash value and the 2 last claims in the NE paragraph.\n2. The experiments are unfortunately a bit lacking as the authors only present ablations of their own method and comparisons to the DP solution. While the use of RL was well-motivated as other PEG solving algorithms would be difficult to generalize, it still would be useful to see if a naive implementation of PSRO in no-exit PEGs. \n3. As the authors mention in the conclusion, a test with partial observability would also greatly improve the viability of this method, since this is a typical assumption in MARL. Since the observation encodings use an attention mechanism, it should not be too difficult to implement this.\n\nQUESTIONS:\n1. Why are there no deviation data provided for the data in Table 1? additionally, what do the error bars in Figure 3 represent? There is a huge difference in the significance of standard error, standard deviation, and confidence intervals.\n2. Why are the testing generalization plots in Figure 4 plotted over time in seconds? Why not flops, or timesteps? This would make for a computationally fair comparison.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Equilibrium Policy Generalization (EPG) framework, a reinforcement learning method aimed at achieving robust zero-shot generalization for pursuit-evasion games (PEGs) across varying graph structures. The authors present a novel approach wherein reinforcement learning is guided by equilibrium policies obtained through dynamic programming (DP) or heuristic approximation. Specifically, an efficient DP algorithm is designed to find pure-strategy Nash equilibria for no-exit PEGs, and a heuristic based on bipartite matching is proposed for multi-exit scenarios. Experimental results demonstrate strong zero-shot performance and detailed ablation studies are provided to validate various components of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n \n- The paper is well-organized, and the figures clearly illustrate the proposed ideas and framework.\n- The core idea is intuitive and the motivation for equilibrium-guided reinforcement learning is well explained.\n- The experiments and ablation studies are thorough and insightful, providing solid empirical evidence supporting the method’s effectiveness.\n\nWeaknesses\n \n\n- The proposed framework critically depends on the availability of an equilibrium opponent policy, which could be a strong assumption. Although the paper discusses two approaches (DP and heuristic) for constructing such equilibrium oracles, these solutions might be difficult to transfer to larger-scale or more general game scenarios, such as Atari environments or complex board/card games (e.g., chess).\n\n- In essence, the method leverages a model-based or expert prior (guidance opponent), and thus the innovation from a purely RL-algorithm perspective is somewhat limited. Other RL methods such as PSRO, SAC, or even simple Q-learning could similarly benefit from having such a guidance opponent. Therefore, the comparisons presented in Figure 4 may not be entirely fair or balanced.\n\n- The claimed zero-shot generalization capability is demonstrated primarily in scenarios with relatively modest variations. Even conventional RL methods, provided with a sufficiently large graph corpus (e.g., > 100 graphs), may achieve reasonable generalization. It would have been more convincing if the proposed approach could generalize beyond graph-based games.\n\nQUESTIONS:\n- Have you tried using Graph Neural Networks (GNNs) to model the scenarios? If yes, how do they compare to your proposed sequence model with shortest path features? If no, can you clarify why such an approach was not considered?\n\n- Why is the proposed approach limited specifically to pursuit-evasion games (PEGs)? It appears that the basic idea could also be applicable to more general two-player adversarial games (for example, two-player Texas Hold'em poker or simplified board games). Could you clarify the reasons behind focusing exclusively on PEGs?\n\n- Could you further discuss the scalability of your equilibrium oracle methods (DP and heuristic)? Specifically, how realistic would it be to apply these methods to larger-scale games, and what practical considerations or limitations would arise?\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Equilibrium Policy Generalization (EPG) framework for pursuit-evasion games (PEGs), aiming to enable zero-shot generalization across graph structures. The core idea is to train reinforcement learning (RL) policies against equilibrium strategies of individual graphs using dynamic programming (DP) and a grouping mechanism for scalability. The authors claim that EPG outperforms state-of-the-art methods in zero-shot performance, supported by experiments on real-world graphs. While the framework shows promise, several critical limitations and theoretical gaps warrant discussion.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \nThe work addresses a critical gap in real-world PEG applications: existing methods struggle with dynamic graph changes and require extensive fine-tuning, limiting real-time applicability. EPG’s focus on zero-shot generalization via equilibrium guidance is innovative and practical.  \n\n **Weaknesses**  \n- The method requires pre-solving Nash equilibria for all training graphs, which is computationally intractable for large-scale scenarios. This limits applicability to real-world problems with dynamic or large graphs.  \n\n- The use of KL divergence between the current policy and precomputed equilibria as a loss raises a fundamental question: If the goal is to mimic equilibrium strategies, why not use direct supervised learning instead of RL? The paper lacks a clear justification for combining RL with equilibrium guidance over pure supervised training, which would be more efficient if equilibria are known.  \n\n- The training paradigm assumes that optimizing against Nash equilibria yields robust policies, but this is theoretically unsound. In zero-sum games, Nash equilibria are optimal against any strategy, but training an RL agent against an equilibrium does not guarantee the agent will converge to a NE. For instance, in rock-paper-scissors (a simple 3x3 game), RL training against this NE without constraints could yield arbitrary policies (e.g., a pure strategy).  \n\n - The evaluation focuses on performance against precomputed NE, but the standard security metric for policies is *exploitability* (i.e., the maximum gain an opponent can achieve by best-responding to the policy). Failing to report exploitability results leaves uncertainty about whether EPG policies are truly robust or merely performant against a fixed NE.  \n - The framework is evaluated in simplified scenarios , lacking scalability tests on large-scale graphs or complex real-world environments.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}