Differentiable Approximations for Multi-resource Spatial Coverage Problems

1 INTRODUCTION . Allocation of multiple resources for efficient spatial coverage is an important component of many practical single-agent and multi-agent systems , for e.g. , robotic surveillance , mobile sensor networks and security game modeling . Surveillance tasks generally involve a single agent assigning resources e.g . drones or sensors , each of which can monitor physical areas , to various points in a target domain such that a loss function associated with coverage of the domain is minimized ( Renzaglia et al. , 2012 ) . Alternatively , security domains follow a leader-follower game setup between two agents , where a defender defends a set of targets ( or a continuous target density in a geographical area ) with limited resources to be placed , while an attacker plans an attack after observing the defender ’ s placement strategy using its own resources ( Tambe , 2011 ) . Traditional methods used to solve single-agent multi-resource surveillance problems often rely on potential fields ( Howard et al. , 2002 ) , discretization based approaches ( Kong et al. , 2006 ) , voronoi tessellations ( Dirafzoon et al. , 2011 ) and particle swarm optimization ( Nazif et al. , 2010 ; Saska et al. , 2014 ) . Similarly , many exact and approximate approaches have been proposed to maximize the defender ’ s expected utility in two-agent multi-resource security domains against a best responding attacker ( Kiekintveld et al. , 2009 ; Amin et al. , 2016 ; Yang et al. , 2014 ; Haskell et al. , 2014 ; Johnson et al. , 2012 ; Huang et al. , 2020 ) . Notably , most existing traditional approaches focus on exploiting some specific spatio-temporal or symmetry structure of the domain being examined . Related Work : Since spatial coverage problems feature continuous action spaces , a common technique used across most previous works is to discretize the area to be covered into grid cells and restrict the agents ’ actions to discrete sets ( Kong et al. , 2006 ; Yang et al. , 2014 ; Haskell et al. , 2014 ; Gan et al. , 2017 ) to find the equilibrium mixed strategies or optimal pure strategies using integer linear programming . However , discretization quickly becomes intractable when the number of each agent ’ s resources grows large . While some games can be characterized by succinct agent strategies and can be solved efficiently via mathematical programming after discretizing the agents ’ actions spaces ( Behnezhad et al. , 2018 ) , this is not true for most multi-resource games . Recent works in spatial coverage domains have focused on incorporating advances from deep learning to solve the coverage problems with more general algorithms . For instance , Pham et al . ( 2018 ) focus on the multi-UAV coverage of a field of interest using a model-free multi-agent RL method while StackGrad ( Amin et al. , 2016 ) , OptGradFP ( Kamra et al. , 2018 ) , PSRO ( Lanctot et al. , 2017 ) are model-free fictitious play based algorithms which can be used to solve games in continuous action spaces . However model-free approaches are sample inefficient and require many interactions with the domain ( or with a simulator ) to infer expected utilities of agents ’ actions . Secondly , they often rely on the policy gradients to compute the derivative of the agents ’ expected utilities w.r.t . their mixed strategies , which induces a high variance in the estimate . To alleviate these issues , more recent works take an actor-critic based approach ( Lowe et al. , 2017 ) , which additionally learns a differentiable approximation to the agents ’ utilities ( Kamra et al. , 2019a ; Wang et al. , 2019 ) and calculate gradients of strategies w.r.t . the utilities . But this requires learning accurate reward/value functions which becomes combinatorially hard for multi-resource coverage . Contributions : To address the above challenge , we present a framework to tractably approximate a general class of spatial coverage objectives and their gradients via spatial discretization without having to learn neural network based reward models . We only discretize the target domain to represent integrals and all set operations over it , but not the action spaces of the agents . Hence we mitigate the intractability caused by discretizing high dimensional action spaces of agents with large number of resources , while also keeping agents ’ actions amenable to gradient-based optimization . By combining our framework with existing solution methods , we successfully solve both single-agent and adversarial two-agent multi-resource spatial coverage problems . 2 MULTI-RESOURCE SPATIAL COVERAGE PROBLEMS . In this section , we formally introduce notation and definitions for multi-resource allocation problems along with two example applications , which will be used for evaluation . Multi-agent multi-resource spatial coverage : Spatial coverage problems comprise of a target space Q ⊂ Rd ( generally d ∈ { 2 , 3 } ) and a set of agents ( or players ) P with each agent p ∈ P having mp resources . We will use the notation −p to denote all agents except p i.e . P\ { p } . Actions : An action up ∈ Rmp×dp for agent p is the placement of all its resources in an appropriate coordinate system of dimension dp . Let Up denote the compact , continuous and convex action set of agent p. Mixed strategies : We represent a mixed strategy i.e . the probability density of agent p over its action set Up as σp ( up ) ≥ 0 s.t . ∫ Up σp ( up ) dup = 1 . We denote agent p sampling an action up ∈ Up from his mixed strategy density as up ∼ σp . Joints : Joint actions , action sets and densities for all agents together are represented as u = { up } p∈P , U = ×p∈P { Up } and σ = { σp } p∈P respectively . Coverage : When placed , each resource covers ( often probabilistically ) some part of the target space Q . Let cvgp : q × u→ R be a function denoting the utility for agent p coming from a target point q ∈ Q due to a joint action u for all agents . We do not assume a specific form for the coverage utility cvgp and leave it to be defined flexibly , to allow many different coverage applications to be amenable to our framework . Rewards : Due to the joint action u , each player achieves a coverage reward rp : u → R of the form rp ( u ) = ∫ Q cvgp ( q , u ) impp ( q ) dq , where impp ( q ) denotes the importance of the target point q for agent p. With a joint mixed strategy σ , player p achieves expected utility : Eu∼σ [ rp ] = ∫ U rp ( u ) σ ( u ) du . Objectives : In single-agent settings , the agent would directly optimize his expected utility w.r.t . action up . But in multi-agent settings , the expected utilities of agents depend on other agents ’ actions and hence can not be maximized with a deterministic resource allocation due to potential exploitation by other agents . Instead agents aim to achieve Nash equilibrium mixed strategies σ = { σp } p∈P over their action spaces . Nash equilibria : A joint mixed strategy σ∗ = { σ∗p } p∈P is said to be a Nash equilibrium if no agent can increase its expected utility by changing its strategy while the other agents stick to their current strategy . Two-player settings : While our proposed framework is not restricted to the number of agents or utility structure of the game , we will focus on single-player settings and zero-sum two-player games in subsequent examples . An additional concept required by fictitious play in two-player settings is that of a best response . A best response of agent p against strategy σ−p is an action which maximizes his expected utility against σ−p : brp ( σ−p ) ∈ arg max up { Eu−p∼σ−p [ rp ( up , u−p ) ] } . The expected utility of any best response of agent p is called the exploitability of agent −p : −p ( σ−p ) : = max up { Eu−p∼σ−p [ rp ( up , u−p ) ] } . Notably , a Nash equilibrium mixed strategy for each player is also their least exploitable strategy . Example 1 ( Single-agent Areal Surveillance ) . A single agent , namely the defender ( D ) , allocates m areal drones with the ith drone Di having three-dimensional coordinates uD , i = ( pD , i , hD , i ) ∈ [ −1 , 1 ] 2 × [ 0 , 1 ] to surveil a two-dimensional forestQ ⊂ [ −1 , 1 ] 2 of arbitrary shape and with a known but arbitrary tree density ρ ( q ) . Consequently , uD ∈ Rm×3 . Each drone has a downward looking camera with a circular lens and with a half-angle θ such that at position ( pD , i , hD , i ) , the drone Di sees the set of points SD , i = { q | ||q − pD , i||2 ≤ hD , i tan θ } . A visualization of this problem with m = 2 drones is shown for a sample forest in Figure 1a . We assume a probabilistic model of coverage with a point q being covered by drone Di with probability PH ( hD , i ) = eK ( hopt−hD , i ) ( hD , i hopt ) Khopt if q ∈ SD , i and 0 otherwise . With multiple drones , the probability of a point q being covered can then be written as : cvg ( q , uD ) = 1− ∏ i|q∈SD , i P̄H ( hD , i ) where P̄H stands for 1− PH . Hence , the reward function to be maximized is : rD,1p ( uD ) = ∫ Q ( 1− ∏ i|q∈SD , i P̄H ( hD , i ) ) ρ ( q ) dq with the tree density ρ ( q ) being the importance of target point q ( subscript 1p denotes one agent ) . Example 2 ( Two-agent Adversarial Coverage ) . Two agents , namely the defender D and the attacker A , compete in a zero-sum game . The defender allocatesm areal drones with the same coverage model as in example 1 . The attacker controls n lumberjacks each with ground coordinates uA , j ∈ [ −1 , 1 ] 2 to chop trees in the forest Q. Consequently , uA ∈ Rn×2 . Each lumberjack chops a constant fraction κ of trees in a radius RL around its coordinates uA , j . We denote the area covered by the j-th lumberjack as SA , j = { q | ‖q − pA , j‖2 ≤ RL } . A visualization of this problem with m = n = 2 is shown for a sample forest in Figure 1b . A drone can potentially catch a lumberjack if its field of view overlaps with the chopping area . For a given resource allocation u = ( uD , uA ) , we define Ij = { i | ‖pA , j−pD , i‖2 ≤ RL+hD , i tan θ } as the set of all drones which overlap with the j-th lumberjack . The areal overlap αij = ∫ SD , i∩SA , j dq controls the probability of the j-th lumberjack being caught by the i-th drone : PC ( hD , i , αij ) = PH ( hD , i ) PA ( αij ) where PH is the same as that in example 1 and captures the effect of drone ’ s height on quality of coverage , while PA ( αij ) = 1−exp ( −Kaαij πR2L ) captures the effect of areal overlap on probability of being caught . Hence , the reward achieved by the j-th lumberjack can be computed as : rA , j ( uD , uA , j ) = κ ∫ SA , j∩Q ρ ( q ) dq with probability∏ i∈Ij P̄ ( hD , i , αij ) , and −κ ∫ SA , j∩Q ρ ( q ) dq otherwise i.e . the number of trees chopped if the j-th lumberjack is not caught by any drone or an equivalent negative penalty if it is caught . Hence , the total agent rewards are : rA,2p ( uD , uA ) = −rD,2p ( uD , uA ) = ∑ j rA , j ( uD , uA , j ) ( subscript 2p denotes two-agent ) . Note that in the above examples drones provide best probabilistic coverage at a height hopt . By increasing their height , a larger area can be covered at the cost of deterioration in coverage probability . Further , the defender can increase coverage probability for regions with high tree density by placing multiple drones to oversee them ; in which case , the drones can potentially stay at higher altitudes too . Example 2 further adds additional interactions due to overlaps between defender and attacker ’ s resources1 . Hence , these examples form a challenging set of evaluation domains with multiple trade-offs and complex possibilities of coverage involving combinatorial interactions between the players ’ resources . For both examples , we use the following constants : θ = π6 , hopt = 0.2 , K = 4.0 , RL = 0.1 , Ka = 3.0 , κ = 0.1 . However , note that these values only serve as practical representative values . The techniques that we introduce in this paper are not specific to the above probabilistic capture models or specific values of game constants , but rather apply to a broad class of coverage problems where the agents act by placing resources with finite coverage fields and agents ’ rewards are of the form : rp ( u ) = ∫ Q fp ( u , q ) dq . Dealing with zero gradients : In the two-agent game , the attacker ’ s reward depends on the locations of its resources , but the defender ’ s reward solely depends on overlaps with the attacker ’ s resources . In absence of such overlap , the gradient of rD,2p w.r.t . uD , i becomes 0 . Hence , we propose to use the reward from the one-agent game as an intrinsic reward for the defender similar to how RL algorithms employ intrinsic rewards when extrinsic rewards are sparse ( Pathak et al. , 2017 ) . Then the reward function for the defender becomes : r̃D,2p ( uD , uA ) = rD,2p ( uD , uA ) + µrD,1p ( uD ) . We use a small µ = 0.001 to not cause significant deviation from the zero-sum structure of the game and yet provide a non-zero gradient to guide the defender ’ s resources in the absence of gradients from rD,2p . 1In reality , lumberjacks might act independent of each other and lack knowledge of each others ’ plans . By allowing them to be placed via a single attacker and letting them collude , we tackle a more challenging problem and ensure that not all of them get caught by independently going to strongly covered forest regions .

Summary: This paper studies the coverage game where agents allocate their resources to target spaces to maximize their coverage, and the goal of this paper is to (approximately) compute the Nash Equilibrium. The proposed method simulates the game by iteratively updating the best response, and the main contribution is an algorithm to approximate the gradient of the utility function with respect to the resource allocation (over the space). In particular, the paper proposes to decompose the gradient into two parts and estimate each part by discretization.