ViPER: Visibility-based Pursuit-Evasion via Reinforcement Learning

Authors: yizhuo  wang
yuhong  cao
jimmy  chiun
subhadeep  koley
mandy  pham
guillaume  sartoretti
iiest  shibpur
u c berkeley
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:16:44

Abstract: In visibility-based pursuit-evasion tasks, a team of mobile pursuer robots with limited sensing capabilities is tasked with detecting all evaders in a multiply-connected planar environment, whose map may or may not be known to pursuers beforehand. This requires tight coordination among multiple agents to ensure that the omniscient and potentially arbitrarily fast evaders are guaranteed to be detected by the pursuers. Whereas existing methods typically rely on a relatively large team of agents to clear the environment, we propose ViPER, a neural solution that leverages a graph attention network to learn a coordinated yet distributed policy via multi-agent reinforcement learning (MARL). We experimentally demonstrate that ViPER significantly outperforms other state-of-the-art non-learning planners, showcasing its emergent coordinated behaviors and adaptability to more challenging scenarios and various team sizes, and finally deploy its learned policies on hardware in an aerial search task.

Main paper: 
 introduction: 
 Pursuit-evasion broadly encompasses general problems that consider detecting/capturing mobile or even adversarial targets (i.e., evaders) by a team of pursuer agents. It has broad applicability in robotic deployments, including human localization after disasters, area patrols to prevent unauthorized activities, and environmental monitoring of wildlife habitats [1, 2, 3, 4] . In this work, unlike many pursuit-evasion problems that assume full observability of the evader's position and motion dynamics [5, 6, 7] -allowing the focus to be merely on chasing the evader(s) without the need for searching-we address a more general worst-case scenario: visibility-based pursuit-evasion, where agents have no information about the evaders. In this scenarios, agents with limited sensing range must coordinate to eventually detect all omniscient evaders, regardless of their movements.In visibility-based pursuit-evasion, based on the agents' sequential observations, the environment is classified into contaminated areas, where the presence of evaders is uncertain, and cleared areas, where it is guaranteed that no evaders can reside/enter without being detected by any agent [2, 8, 9] . Agents are tasked with expanding frontiers (i.e., the boundaries between cleared and contaminated areas) to methodically explore the entire environment, turning all contaminated areas into cleared areas. During the task, agents need to coordinate their actions to control recontamination, where evaders might re-enter cleared areas undetected, turning them back into contaminated areas (see Fig. 1b ). Therefore, this problem is also referred to as adversarial search [5] , or the clearing problem [8, 10] . Due to the complexity of this problem, [8] deploys rule/constraint-based planners to reduce the complexity by prohibiting agents' actions that lead to recontamination, thereby significantly reducing the action space of agents. However, as shown in Fig. 1c , recontamination is unavoidable to complete the task in some challenging scenarios. Following works [10, 11] utilize centralized (meta-heuristic) sampling-based methods to seek efficient search directly in unconstrained action spaces. However, these methods are still limited to relatively small-scale scenarios or require extensive optimization.Motivated by the advanced performance recently achieved by deep reinforcement learning (DRL) over sampling-based methods in various single-robot informative path planning tasks [12, 13, 14] , we propose ViPER, a MARL-based solution for pursuit-evasion, allowing agents to learn to actively collaborate in a decentralized manner, to achieve significantly improved performance over conventional methods in relatively large-scale pursuit-evasion tasks. To develop such a desired learning-based planner, we make efforts to address the following key challenges: (1) Informatively yet concisely encode the observed/communicated information through the agents' neural network to enable efficient policy learning. Complex observations (including cleared/contaminated areas, obstacles, other agents, etc.) need to be represented in an efficient format so that the neural network can learn smoothly and efficiently. In some scenarios, the optimal actions of the team might depend on sensible decisions of specific agents, making efficient credit assignment critical for cooperation learning. ViPER addresses the challenges above by:• incorporating the cleared area dynamics and the other agents' information into a graph attention network to allow context-aware policy learning; • leveraging privileged ground-truth information during training to address the inaccuracy and instability inherent to model-free RL's state value estimation; • employing an attentive critic for credit assignment that integrates agent action features to enable agents learning more efficient, decentralized cooperation.To the best of our knowledge, ViPER is the first neural solution for the visibility-based pursuitevasion problem that considers worst-case adversaries. Our approach improves team coordination, significantly outperforming state-of-the-art non-learning planners in unknown environments. We further empirically show that ViPER, once trained, can adapt to varying team sizes and (significantly more challenging) map structures. We also highlight its robustness in the face of agent failure, and finally experimentally validate learned policies on hardware in an aerial search task. 
prior works: 
 The initial formulation of the guaranteed worst-case pursuit-evasion problem was introduced by Parsons [15] , followed by Suzuki and Yamashita [16] who proposed a visibility-based variant, transitioning the setting from discrete graphs to continuous polygonal environments. In such scenarios, agents aim to find an evader who can move at unpredictable and potentially unbounded speeds.There is a rich literature on visibility-based pursuit-evasion problems, with the majority of works considering fully known environments in which agents precalculate their routes offline before execution. Many of these studies focus on completeness and optimality of single-agent pursuit-evasion algorithms, considering an omnidirectional sensor with infinite range [17, 18] . Furthermore, Gerkey et al. [2] provided a complete algorithm for agents with a limited angle of view and proved that computing the minimum number of such agents required to search an area is NP-hard. Since there is no known closed-form solution for the multi-agent pursuitevasion game, Quattrini Li et al. [19] decomposed the environment into convex hulls and introduced a heuristic search approach, whereas Olsen et al. [11] proposed solving it with graph sampling-based method, aiming to mitigate the inherent complexity of the complete algorithms [20] .Unknown environments: Sachs and Rajko [21] presented the first single-agent algorithm that does not require a complete map of the environment, while Kolling and Carpin [9] utilized coordinated sweep lines by the team to simultaneously clear and explore unknown environments. Based on this, Durham et al. [8] developed an algorithm that guarantees capture if the number of pursuers is sufficient. They introduced a method to update the global frontiers between cleared and contaminated areas using solely local information. All these methods assume a relatively simple polygonal topology of the unknown environments and require more than sufficient agents to carry out the tasks [10] . 
problem formulation: 
 Environments: We consider a bounded, multiply-connected environment represented by an occupancy map M ⊂ R 2 that is initially known (M = M k ) or unknown (M = M u ) to the agents. In the case of an unknown environment, the map is gradually explored and revealed by the agents such that M k ∪ M u = M. Based on their omnidirectional sensor measurements, agents will classify the known areas into either traversable free areas M f or occupied areas M o within their line-of-sight visibility range (i.e., M f ∪ M o = M k ). The free space is further categorized into either cleared areas M c or contaminated areas M f \ M c . Note that we assume perfect communication among agents, allowing them to exchange information and share the same map throughout the task.Agents and evaders: We task n agents to coordinate their trajectories Ψ = {ψ 1 , ..., ψ n }, with the objective of capturing evaders capable of moving at potentially unbounded speeds v e . Here, ψ i : {0, 1, 2, ...} → M f represents the trajectory of agent i over time, with ψ i (t) indicating the agent's location at timestep t. Agents scan their surrounding environment within their sensor footprint S(ψ i (t)) ⊆ M which has a range of r fov (with visibility also limited by obstacles), communicate with other agents to exchange their measurements, and consequently update their map M.We hypothesize a worst-case scenario where the evaders are positioned right outside the frontiers between the cleared and contaminated areas, denoted by F = ∂M c \ M o , poised to intrude once there are any frontiers not within the visibility of any agent. Such frontiers are termed the uncovered frontiers F u , where evaders in contaminated areas can penetrate and cause recontamination, confined to their speed v e . Since the locations of an agent in two consecutive timesteps are within each other's visibility (explained in Section 4), the intersection of two consecutive sensor footprints will also be visible during the agent's movement. This applies to any agent in the team and we denote this intersected area asEQUATIONRecontamination will spread through the free areas M f at speed v e , encroaching upon the cleared areas M c . In particular, when evaders are arbitrarily fast (i.e., v e = ∞), the contaminated areas will expand through each connected M f until reaching the boundary of S(Ψ t:t+1 ).Objective: The objective of the visibility-based pursuit-evasion problem is to determine the optimal agent trajectories Ψ * such that the cleared areas can expand to encompass the entire free space while minimizing the length of the trajectory for all agents:EQUATIONwith M * f the ground-truth free areas. Note that we optimize the sum of the maximum length per step, assuming synchronous decision-making and execution, whereby all agents must reach their viewpoints to proceed to the next step. 
multi-agent coordination method: 
 This section casts visibility-based pursuit-evasion as an RL problem, outlines our proposed policy and critic networks based on graph attention, and details its training procedure. We provide insights into the design of our observation, reward, and network architecture to foster advanced coordination.Sequential decision-making: While agents can pre-plan their trajectories offline in a known environment, navigating an unknown environment relies on an adaptive online planner to continually update the map. This casts the task as a sequential decision-making problem, specifically a Markov decision process (MDP). We first uniformly distribute candidate viewpoints (also referred to as nodes hereafter) V t = {v 0 , v 1 , ...}, ∀v j = (x j , y j ) ∈ M f across the free areas, and append additional nodes into the newly identified free areas (if any) with each decision timestep. We then connect each node with its collision-free neighboring nodes within a specified range (usually ≤ r fov ), forming the graph G t = (V t , E t ) with its corresponding traversable edge set E t . Upon all agents reaching their previously selected viewpoints, each agent will simultaneously choose the next viewpoint to visit, sequentially constructing the agent's trajectoryψ i (t) ∈ V t .Observation space: Inspired by recent single-agent DRL-based informative path planning planners [12, 13] , the observationo t = (G ′ t , Ψ t ) includes the augmented graph G ′ t = (V ′ t , E t )and the current locations of the agent team Ψ t . The augmented graph G ′ t shares the same edge topology E t as G t , but also incorporates map information to assist agents in making informed and coordinated decisions. For each v ′ j ∈ V ′ , the property of the node is v ′ j = (∆x i,j , ∆y i,j , p j , u j , u u,j , s j , s c,j , g j ):(1) Relative position (∆x i,j , ∆y i,j ): the node's position relative to agent j, i.e., (∆x ij , ∆y ij ) = v j -ψ i (t). For simplicity, we omit the agent enumerator i in graph and observation. (2) Occupancy of other agents p j : a binary value indicating whether any other agent is positioned at v j . (3) Utility u j : represents the number of visible frontiers at node v j [22] , i.e.,u j = |{f ∈ F : ∥f -v j ∥ < r fov , v j f ∩(M o ∪M u ) = ∅}|,where v j f ⊂ M is a straight line connecting the node to the frontier. The node utility in contaminated areas is set to -1. (4) Uncovered utility u u,j : similar to utility, but counts only the uncovered frontiers not within any agents' visibility, i.e., replace F above with F u . (5) Cleared signal s j : a binary value indicating whether v j is within cleared areas. (6) Counterfactual cleared signal s c,j : hallucinates a scenario without the presence of agents, considering how cleared areas would change, to help improve understanding of the evader's speed. Specifically, when v e = ∞, s c,j ≡ 0. (7) Guidepost g j : a binary value indicating whether v j is on the agent's A * trajectory to its nearest frontier, which significantly aids in agent navigation to frontiers, especially when they are distant. The node coordinates and utility values in the observation are normalized before being fed into the neural network.Action space: Each time all agents reach their designated viewpoints Ψ t , the graph attentionbased networks, parameterized by θ and shared among all agents, output stochastic policiesπ θ (a i,t | o i,t ) , i = 1, ..., n (s.t. ψ i (t + 1) = v j , (v j , ψ i (t)) ∈ E t ).Agents select the next neighboring node based on these policies, then navigate to their selected viewpoints along straight paths, updating their maps with the information collected en route. If multiple agents select the same viewpoint, the others will alternatively choose the closest node to that viewpoint.Reward structure: Our reward is designed to incentivize agents to optimize the task objective as outlined in Eq. (2) . That is, we assign rewards/penalties for expanding/shrinking cleared areas, impose penalties for the largest distance traveled between viewpoints among all agents, and assign a substantial reward r f upon the successful task completion (0 otherwise):EQUATIONwhere c 1 , c 2 are two constants for scaling. This reward r t is equally shared among all agents, as it is challenging to design a reward structure that aligns with the actual objective while accurately assigning individual rewards to each agent. Inspired by recent advances in graph attention networks [23, 24] , we design a network to learn policy π θ that orchestrates coordinated behaviors among multiple agents. Our graph attention-based policy network consists of an encoder and a decoder: the encoder extracts information from nodes across the entire known map, while the decoder yields which neighboring node to visit next, based on the aggregated global information.We train a critic network to approximate the state-action values. These values represent the expected cumulative future rewards (i.e., return) and guide our policy network by evaluating the quality of the chosen actions. The environment is fully observable if it is known a priori: s t = o t . However, in unknown environments, the policy network can only access partial information about the environment as agents explore. In our work, unlike the policy network, the critic network is provided with complete ground-truth information (e.g., entire map/graph, see Fig. 2 ) during training phase. This privileged learning approach, leveraging the critic's access to the full state, contributes to a more stable training process [13] . To harness the full potential of the well-established centralized training with decentralized execution (CTDE) paradigm [25] , we further provide the critic with the actions of other agents within the team a t . That is, during the training phase, the integration of the states t = (G ′ *t , Ψ t ) along with a t enables the critic to estimate state-action values with greater accuracy, thus in turn implicitly assisting agents in predicting the structure of the environment as well as the behaviors of other agents.Encoder: Our encoder is essentially a sequence of masked self-attention layers (introduced in Appendix A) that enhances the features of each node by incorporating information from other nodes within the graph structure. We first apply a feed-forward layer to embed the node properties in V ′ t into d-dimensional node features h n . These node features are then processed through N stacked self-attention layers (i.e., h q = h k,v = h n , and with N = 6 in our practice), where each layer uses the output of the previous one as its input. We calculate the mask M t of the graph using the adjacency matrix derived from E t . This mask is applied to ensure each node has access only to features of its neighboring nodes (i.e., attention weight w i,j = 0, ∀(v i , v j ) / ∈ E t ). Although attention is constrained to neighboring nodes at each layer, nodes can still acquire non-neighboring node features by repeatedly aggregating features through this stacked self-attention structure. We find that, in contrast to unmasked self-attention, our method of masked self-attention, which propagates neighboring features layer by layer, leads to a more effective learning of pathfinding solution [26] . The encoder thus enhances the node features h n to hn , as each updated set of node features contains dependencies on other nodes in the graph, depending on M t and N .The decoder is used to determine the final policy based on the enhanced node features hn . From these node features, we select the one at the robot's current position as query:h q = h c = hn ψi(t). The current node features h c , along with its neighboring node features (presented as key-value pairs) from hn , are fed into an attention layer, thereby enhancing the output hc with information from its neighboring nodes. Subsequently, we adopt a single-head attention mechanism with hc and its neighboring node features, and directly use its attention scores as the final output policy π θ = (a i,t | o i,t ) = w i,j , from which the agent selects the next viewpoint to move to. This design relaxes the need for a predefined policy size, instead allowing the policy's dimensions to adapt dynamically to the number of neighboring nodes, thus naturally accommodating graphs with arbitrary connectivity.The critic network Q ϕ , parameterized by ϕ, shares the same encoding structure with the policy network π θ , but employs a different decoder that incorporates other agent's actions. We mirror the idea in MAAC [27] that introduced an attentive critic to select relevant information for each agent. From the enhanced node features hn output by the encoder, we filter the node features of other agents hn ψ-i(t) and their corresponding selected features hn ψ-i(t+1) at the next timestep based on their current actions a t . These features are then concatenated as key-value pairs to enhance the current node feature h c . Subsequently, informed by the actions of other agents, the current node features are further enhanced by incorporating neighboring features, identical to the process in the policy decoder. The enhanced current node features hc are then concatenated with neighboring features and are finally mapped to state-action values Q ϕ (s t , a t ) by a feed-forward layer.We train our RL agents using the soft actor-critic (SAC) algorithm on 4, 000 randomly created maps for our training dataset, and an additional set of 100 unseen maps as the test dataset. The training details (algorithm, dataset, graph and sensing configurations) can be found in Appendix B. ViPER's code and trained model are available at https://github.com/marmotlab/ViPER. 
experiments: 
 In this section, we compare our method with state-of-the-art (SOTA) baseline planners designed for unknown environments, showcasing its superiority in terms of performance, generalization, and robustness. We also discuss the emergent coordinated behavior of our trained policy and present the results of a real-life aerial search task involving three drones and one ground robot.We compare ViPER with two planners from Durham et al. [8] and Gregorin et al. [10] . Durham et al. proposed a distributed planner that makes decisions based solely on local frontiers and communication, thus it does not require prior mapping of the environment. Suggested by the settings in their original paper, we tuned their planner to output high-quality solutions for a fair comparison. While many studies evaluate their algorithms in simply structured environments, our map dataset is more structurally complex, requiring agents to implicitly predict the map and actively respond to updates. Our evaluation map dataset comprises 100 maps that were unseen during model training. We also use multiply-connected and conjugated maps from Gregorin et al. and compare our performance against the results reported in their paper, using the same r fov . Following [10] , we test each map 30 times with random start positions. Note that throughout all evaluations, we use our model trained with 3 agents and a bounded evader speed (v e ≤ 20m/timestep, with map size 100 × 100m 2 ) to facilitate faster environment simulation during training.We report the success rate and the maximum trajectory length T t=0 max n i=1 ∥ψ i (t + 1) -ψ i (t)∥ in Table 1 , averaged across 100 maps with unbounded evader speed, i.e., v e = ∞. We mark failure of the task if the trajectory length of any agent exceeds 1, 000m. Our results show that ViPER outperforms the other two baselines in both metrics. Upon closer inspection, the performance gap is more pronounced when the team size is smaller. We believe that this likely comes from the restriction in Durham's algorithm of not allowing recontamination, potentially requiring a larger team size than Table 1 : Comparison against SOTA planners in different map sets and team sizes n. We report the success rate and averaged trajectory length (in parentheses).Multiply-connected maps [10] Conjugated maps [ necessary. This is evident in Fig. 4 , which plots the evolution of the cleared rate |M c |/|M * f | compared to trajectory length. When n = 4, the performance disparity is significant, while at n = 8, both approaches achieve a 100% success rate, resulting in averagely monotonic increases in the cleared rate. Gregorin's approach, on the other hand, is centralized and allows a certain level of recontamination. Their method combines random walking with an evolutionary approach to address rarely explored states. However, in conjugated maps, its success rate still significantly drops as the environment size and the number of agents increase, limiting its scalability to larger team sizes.Known environments: We evaluate our model in known environments and present our results in Table 2 . Despite being trained in unknown environments, our model is capable of adapting to known environments (and vice versa) without much performance change/degradation, and it scales effectively to various team sizes. This is largely due to the fact that the agent observations only include the number of observable frontiers between cleared and contaminated areas (i.e., utility) by design, while excluding information on the exploration frontiers between known and unknown areas. That is, we regard the exploration of the environment as an auxiliary task of the pursuit-evasion.We adopt an additional map dataset of 100 maps with significantly more complex topologies than those used during training and 4 times larger the area to evaluate the algorithm's generalizability with 10 agents. There, the task is deemed unsuccessful once any agent's trajectory length exceeds 3, 000m. The results are presented in the last row of Table 2 and example solutions can be seen in Fig. 3 . In unknown environments, the agents may initially misjudge the layout, leading to inaccurate agents distribution. Consequently, certain agents need to wait at junctions for a long period, awaiting the arrival of other agents before continuing to collaboratively explore the environment and minimize recontamination, thereby increasing the difficulty of the task. We believe these results suggest that ViPER can generalize to vastly different/challenging environments with minimal impact on map size or the number of agents involved, without the need for retraining.Agent failure: In Appendix C, we exhibit ViPER's robustness to mid-task agent failures, showing its ability to adapt and prevent further recontamination from failed agent(s).Our agents exhibit interesting emergent coordinated behaviors during training, thanks to our graphattention-based network and centralized privileged critic (an ablation study of these modules can be found in Appendix D). One of our map set is shown in Fig. 5 , where we display the solution snapshots at various timesteps and trajectory lengths (in parentheses). We observe that, once the green agent clears the top-right room and begins moving towards the blue agent for assistance, the blue agent starts to move downward in advance upon observing the green agent's approach, allowing for some areas of controllable recontamination to reduce trajectory length. Then, after clearing the room, the blue agent joins forces with the green agent to clear the upper room, which has an obstacle situated in the middle requiring at least two agents to clear. Our additional examples in Appendix E further confirm that, although agents make synchronous decisions in MARL framework, they can intelligently allocate themselves on-demand and dynamically redistribute as the task progresses. We carried out experiments to validate ViPER in an aerial search task using three Crazyfile 2.1 drones as searching agents and one TurtleBot3 robot as an imaginary evader (see Fig. 6 ). The experiment was conducted in a 4 × 4m 2 mockup arena with a waypoint node resolution of 0.2m. We opted to use the optical flow sensors on the drones for position tracking, with the origins set at their take-off positions, and transformed the updated positions into a global frame. Waypoints were iteratively published to drones by a remote computer, while the TurtleBot3, controlled by a human as an omniscient evader, only navigated in contaminated areas. This experiment confirms that ViPER can be deployed on robots with relatively low computational costs, and exhibit high performance. For each agent, forward inference takes less than 0.01s, and the combined environment update and planning take less than 1s of wall clock time. 
conclusion: 
 Summary: We introduce the first neural framework based on multi-agent reinforcement learning for visibility-based pursuit-evasion tasks that addresses worst-case scenarios. Our approach relies on a graph attention network to allow agents to individually process their shared information and achieve the necessary level of tight coordination that can ensure the detection of omniscient and potentially arbitrarily fast evaders. In our evaluation, our learned policies exhibit emergent coordinated teamwork behaviors that outperform state-of-the-art planners by a large margin. We also demonstrate ViPER can adapt to different team sizes, handle diverse scenarios, and remain effective in the face of agent failures.Limitations and future work: Although ViPER can efficiently handle exploration and recontamination compared to other methods, the initial positions of the agents still greatly affects their performance and can sometimes result in task failure. When all agents are in a position that necessitates additional support from other agents, they are unlikely to sacrifice their own, usually extensive, cleared areas to assist others. This reluctance can lead to deadlocks in actions, despite the theoretical existence of a solution. Future work will explore multi-level graph representations and hierarchical decision-making for improved graph structure reasoning and state exploration. 
 
 Appendix: Our network relies on attention layers in the Transformer [28] , where each layer transforms a query vector h q and a collection of key-value pair vectors h k,v into an output vector h ′ i by computing a weighted sum of values. The attention scores are derived by measuring the similarities between queries and keys within each attention head:EQUATIONwhere q, k, v are derived by three separate learnable matrices in R d×d asq i = W Q h q i , k i = W K h k,v i , v i = W V h k,v i .Following [29] , we adopt Identity Map Reordering (IMR) between attention layers, placing layer normalization before attention layer to improve network stability in RL tasks.Algorithm: We train our DRL agents with the soft actor-critic (SAC) algorithm. SAC learns a soft value function by modifying the policy gradient to include an entropy term, H(π(• | o t )), which encourages the exploration of more diverse actions and helps prevent premature convergence [30] :EQUATIONwhere T is the number of timesteps and γ ∈ [0, 1] the discount rate, D the replay buffer, and α serves as a temperature parameter that scales the importance of the entropy term against the reward. The temperature parameter is automatically adjusted to match the target entropy H.We use a dungeon map generator [31] to randomly create a set of 4 × 1, 000 multiplyconnected maps for our training dataset, with each map oriented four times for map set augmentation, and an extra set of 100 different maps that are unseen during training for performance evaluation. Each map have a size of 100m × 100m.Sensing: The onboard omnidirectional sensor with a radius of r fov = 20m will monitor/identify potential evaders (and map the environment). Here, assuming a circular field of view (FoV) aligns with most related works [9, 10] and helps incorporate ViPER with both downward-facing cameras and LiDAR. Each agent scans its surrounding environment and communicates with other agents to update a shared belief.Graph: The candidate viewpoints V t are uniformly distributed throughout the entire known free areas M f . The spacing between two adjacent nodes (i.e., resolution) is set at 4m. Note that these parameters are tunable to meet specific practical requirements. Each node is connected with its k = 25 nearest neighboring nodes (including itself) to establish the edge set E t , provided the paths between them are collision-free. We consider a successful task completion to be the closure of the cleared areas M c (i.e., no frontiers between cleared and contaminated areas).Training: During training, the episode is terminated once its length exceeds 128 timesteps, with a replay buffer size of |D| = 10, 000, a discount rate of γ = 1, and the target entropy set to 0.01•log k.Our model is trained on a workstation with an AMD 7950X3D CPU where 12 environments run in parallel, and an NVIDIA GeForce RTX 3090 GPU to train the network once an episode terminates, after accumulating 5, 000 steps of data in the replay buffer. The model converges after 40 hours of training, and we adopt the model at 40, 000 episodes for experiments in Section 5.We evaluate the robustness of ViPER to agent failures occurring in the middle of the task. Specifically, we simulate agent failure by deactivating a random agent, including its movement and sensor, once agents have explored half of the environment. The result is plotted in Fig. 7 , averaged over 100 maps. Our results indicate that the performance remains similar to (or slightly superior to) that of the fully functional agent group (comparing n → n -1 to n -1). This demonstrates that the remaining functional agents are able to reactively adapt and prevent further recontamination from the failed agent, thereby avoiding the need to almost entirely restart the task should the recontamination spread uncontrolled. We perform a comprehensive ablation study to evaluate the effectiveness of our graph attention architecture, attentive critic, and privileged learning module.Graph attention encoding: To evaluate the effectiveness of our graph attention architecture, we additionally train two ablation baselines. In comparison to ViPER with a 6-layer graph encoder, one baseline was trained with a 3-layer graph encoder, while the other was trained without any encoder. As shown in Fig. 8 , the performance significantly degrades with a shallower encoder during training, as agents' limited graph perception results in more myopic decisionmaking. Without a graph encoder, the agents struggle to learn any effective long-term policies necessary to successfully complete the task, leading to frequent task failures.Critic module: We train three additional ablation baselines: ViPER without the MAAC (attentive critic module), ViPER without GT (ground truth information during training), and ViPER without both MAAC and GT. We present the reward and length curves during training in Fig. 8 . It is evident that ViPER consistently outperforms its baselines throughout the training process. We further observe a significant performance decline without privileged learning, indicating the importance of access to ground truth information for the critic to achieve a more accurate value estimate during training. Our ablation study demonstrates that both the attentive critic and privileged learning (individually and combined) enhance ViPER's ability to learn a better policy for multi-agent coordination in visibility-based pursuit-evasion tasks.We handcraft several specific scenarios to showcase team coordination and provide insights into ViPER's policy. As illustrated in Fig. 9a , when there is an isolated obstacle/chamber in the center of a closed environment, it is impossible for a single agent to clear the entire areas, as the evaders can continuously hide behind the obstacle to elude detection. In this case, two agents must simultaneously extend their frontiers from both sides of the obstacle to ensure the evaders are detected. Similarly, as shown in Fig. 9b , each passage must be monitored by an agent to prevent recontamination. In Fig. 9c , four agents are initially paired off to direct toward rightward and downward. However, upon discovering more frontiers below, the green agent reactively reverses and adjusts its trajectory to head downward instead. Fig. 9d depicts scenarios where, if an agent inspects each room along a corridor, an evader could potentially slip past and conceal itself in an already cleared room. One solution is to use two agents, with one consistently monitoring the corridor. Our two agents alternatively choose to take turns clearing rooms and monitoring the corridor, resulting in an even shorter trajectory length. 


Decision: Strengths:

• Novel approach using graph attention networks and multi-agent reinforcement learning for pursuit-evasion tasks
• Outperforms state-of-the-art non-learning planners by a significant margin
• Demonstrates adaptability to unknown environments and various team sizes 
• Successfully deployed on real aerial robots, validating practical applicability

Weaknesses:

• Insufficient graphical explanations of system architecture and integration
• Limited real-world experiments (only 3 drones and 1 evader tested)
• Comparison only to older baselines (2012, 2017), missing more recent learning-based approaches

Reviews: review: title: title: Simulation based research, limited real world robotics evaluation
paper_summary: paper_summary: This paper introduces ViPER, a new approach using a graph attention network and multi-agent reinforcement learning to improve coordination in visibility-based pursuit-evasion tasks where robots detect evaders in complex environments. ViPER outperforms traditional methods by adapting to various scenarios and team sizes, and has been successfully deployed on aerial search hardware.
main_review: main_review: The paper proposes a neural solution that leverages a graph attention network to learn a coordinated
yet distributed policy via multi-agent reinforcement learning, verifying the idea with the pursuit-evasion task. 
However, as far as this reviewer comes from a robotics control background, the motivation for such a methodology is not grounded in robotics, and the application of pursuit-evasion appears to be a toy problem. 
The presentation of the originality and the contributions can be enhanced. Especially, considering what kind of variables are inputted to the model, and what kind of variables are outputted to control the robotic systems.
The authors claim that the problem is complex, however, the description is abstract and it is hard for readers from a robotics background to acknowledge the difficulty. 
Strength:
* The theoretical background is well presented with mathematical correctness.

Weakness:
* Less graphical explanations, such as system diagram, how aerial robotic systems integrate with the model, what the variables controlled, and how the perception is performed.
questions: questions: The paper appears well-written and could potentially be enhanced by addressing the following points:

* The motivation for targeting the pursuit-evasion problem and the proposed approach should be clarified to improve readability, particularly for those primarily familiar with robotics.
* In the experimental evaluation, it should be specified whether the observed camera images are from the aerial robot or if they capture the entire field.
* The method for tracking the position of the drones should be detailed. Specifically, if coordinate transformations are used, what is the origin of these coordinates?
* The actions outputted by the model should be defined. Are they the desired velocity of the drone or the path of the drone?
* The definition of visibility during the experiments should be clarified. Is the drone equipped with a camera that scans the environment?

These clarifications will enhance the understanding and applicability of the research findings.
limitations: limitations: 2
review_summary: review_summary: The research is presented mainly from a theoretical perspective, and the real-world experiments appear to be limited. Therefore, from the viewpoint of robotics, including applications, the research could still be improved.
title: title: Overall interesting paper solving a limited sensor range visibility pursuit evasion problem with multiple robots
paper_summary: paper_summary: The paper proposes a method for finding a plan for multiple robots with limited sensor range to clear an environment so that all potential evaders are found. The problem is solved formulating it as a multi agent reinforcement learning problem, with the method that is based on a graph attention network to learn the distributed policy. Experiments in simulation and one demonstration with real robots are performed.
main_review: main_review: The paper addresses an interesting problem in the pursuit-evasion space and approaches it with a technically sound learning-based approach. It is interesting to see it formulated in this way to have a practical way to solve the problem, rather than having only theoretical analysis and having an algorithm that addresses the worst case. The paper appears to provide convincing results with experiments performed in unseen environments, as well as with real robots. 

It would be useful to perform an ablation study to actually see the impact of the graph-attention based network as claimed in the results.

Specific clarifications are needed:
- it would be useful to discuss also the failure cases for the proposed method. Is it happening for the fact that there are not enough agents, as discussed for the other methods? Or is there something else happening?
- it seems that the problem is to detect the evaders based on the objective, however, the paper also mentions about the capture of them in the problem formulation. Please clarify it.
- with the average results it is useful also to report the standard deviation.
- one limitation worth including is also the guarantee that the proposed method can provide.

Presentation/typos comments:
- Following works [10, 11] utilize -> Some works [10, 11] utilize
questions: questions: 1. It would be useful to include an ablation study on the proposed architecture to see the impact of each component.
limitations: limitations: 3
review_summary: review_summary: While there can be an additional ablation study, overall the paper proposed an interesting learning-based method to solve a pursuit evasion problem, with convincing results, including experiments in unseen environments.
title: title: Review
paper_summary: paper_summary: This paper introduces a method that utilizes graph attention networks to learn centralized planning decentralized control policies for the pursuit-evasion task.
main_review: main_review: **Strengths**
- The masked graph attention network that extracts information across known nodes in order to coordinate future actions across the team appears is an intuitive approach to solving the pursuit-evader problem involving multiple agents.  
- The method outperforms the reported baselines by a significant margin while accounting for recontamination
- The authors validated their approach with an albeit small team of aerial vehicles and a single evader. 
- The method is capable of adapting to unknown environments thanks to the privileged critic learning / partial observability of any individual agent - training paradigm 

**Weaknesses** 
- I understand that real robot experiments are difficult and time consuming, but it would have been nice to see results involving more than three drones and a single evader, especially since the evaluation results included larger teams and maps. 
- The authors claim to compare to SOTA planners, but the two baselines are papers from 2012 and 2017. While I do not directly work in this field, I would be incredibly surprised if there weren't other more relevant baselines. Further, the reported baselines are not data driven approaches. It would only be fair to see how this method stacks up against other similar learning-based methods.
questions: questions: 1. How does your method scale / perform with multiple evaders? 

2. Since the graph attention network incorporates other environment information alongside the graph, is it possible for this method to work with heterogenous robot teams? If not, what changes would be needed to enable this?
limitations: limitations: 3
review_summary: review_summary: The authors put forward a strong learning-based method that is trained on a dataset of thousands of maps, leading to generalization and adaptability to unseen environments. The method was validated with real robot experiments. However, the robot experiments were relatively small in scale and the baseline comparisons seem weak.