Title: DECEPTIVE OPPONENT MODELING WITH PROACTIVE NETWORK INTERDICTION FOR STOCHASTIC GOAL RECOGNITION CONTROL

Abstract: Goal recognition based on the observations of the behaviors collected online has been used to model some potential applications. Newly formulated problem of goal recognition design aims at facilitating the online goal recognition process by performing offline redesign of the underlying environment with hard action removal. In this paper, we propose the stochastic goal recognition control (S-GRC) problem with two main stages: (1) deceptive opponent modeling based on maximum entropy regularized Markov decision processes (MDPs) and ( 2) goal recognition control under proactively static interdiction. For the purpose of evaluation, we propose to use the worst case distinctiveness (wcd) as a measure of the nondistinctive path without revealing the true goals, the task of S-GRC is to interdict a set of actions that improve or reduce the wcd. We empirically demonstrate that our proposed approach control the goal recognition process based on opponent's deceptive behavior.

1. INTRODUCTION:

Goal recognition (GR), also called intention recognition, is the task of inferring the goals of an agent according to the observed actions or states collected online (Sadri, 2011) , which enables humans or agents to make proactive response plans. Goal recognition design (GRD) is to deliberately redesign the environment for improved online goal recognition ability (Keren et al., 2014) , which includes two key models, the first measures how efficiently and effectively the online goal recognition system performs in a given setting, the second optimizes the goal recognition setting via redesign (Keren et al., 2019) . Goal recognition control (GRC) aims at soft action interdiction under bounded resource by adding cost compared to the hard action removal of GRD (Luo et al., 2019) , in which the interdiction can be proactively static inhibition or online dynamic block by allocating security resources to protect goals against the attacker.

In fact, under complex environment (both adversarial and cooperative), the information of an opponent's goals are asymmetric and can not be obtained through communication (Le Guillarme, 2016) . Most recent advancements in GR utilize planning recognition as planning (PRAP) (Ramírez & Geffner, 2009) , generative game-theoretic frameworks (Ang et al., 2017) , and goal recognition as planning (GRAP) (Pereira et al., 2019) . There are three key assumptions widely used: (1) agents performs optimal plans to real the goals; (2) the environment is fully observable, that is the states and actions of the agents are observable; and (3) the agent's actions are deterministic. However, these existing frameworks seldom address the deceptive behaviors of actively misleading the goal recognition process. We will relax the first and third assumptions to handle non-optimal agents and stochastic actions. Such as one game-theoretic approach provided with a unified treatment of both threat assessment and response planning in (Guillarme et al., 2017) , after evaluating the threat, the defender will allocate road barrier or patrolling force to protect critical infrastructure, as illustrated in Figure 1 .

In order to identify the goals and improve security, we employ opponent modeling to deal with the non-stationary strategies stemming from deception, in which maximum entropy regularized Markov decision process (MDP) is utilized to shape the multi-modal adversarial strategies (Shen & How, 2019). After finding the maximal non-distinctive agent path, we are seeking some optimal modifications (e.g., interdiction to block or inhibit action) to accelerate the online goal recognition.

In this work, we propose the stochastic goal recognition design (S-GRC) problem with deceptive opponent modeling and proactive network interdiction: the opponent's deceptive policy is modeled as one soft multi-criteria decision policy with one tunable parameter to balance goal achievement and deception preference; and the primary objective is offline redesigning the environment by soft action interdiction to control online GR.

To validate the model, we evaluate our approach in two different environment representations: random generated connected graph and real road network. Our experiments demonstrate that (1) deceptive opponent modeling with entropy regularization make it robust to multi-modal stochastic behaviors; (2) soft decision policy based stochastic goal recognition design model for action removal bridge the gap between observation and decision making.

2. RELATED WORK:



2.1. GOAL RECOGNITION AND DESIGN:

Goal Recognition (GR): The ability of inferring the goals of others can assist us to reason about what they are doing, why they are doing it, and what they will do next (Sukthankar et al., 2014) . The recognition setting can be generally be divided into keyhole, where the agent is unaware of being observed and recognized as if the agent is looking through a keyhole; adversarial, where the agent is actively hostile to the observations of the actions and the inference of the goals; and intended, where the agent wants to convey the goal to be understood and helps the recognizer to detect the objective (Keren et al., 2019) . In this work, we focus on the adversarial setting, where the agent would hide the goal and attempt to thwart the recognition process by deception. GR is strongly related to the topic of privacy-preserving and explainable planning (Keren et al., 2016; Chakraborti et al., 2019; Kulkarni et al.) .

Goal Recognition Design (GRD)::

The GRD problems contains two models: one goal recognition setting analyzed environments, acting agent (attacker), and recognition system (recognizer, observer); one design model of possible ways to change the environment (Keren et al., 2019) . The environment will induce a set of possible behaviors for the attacker, GRD can be divided into deterministic GRD (Keren et al., 2019) , stochastic GRD (S-GRD) (Wayllace et al., 2018; Wayllace, 2019) , GRD for Agents with partial knowledge (GRD-APK) (Sarah et al., 2019) , and gametheoretic GRD (Ang et al., 2017; Masters & Sardina, 2017) . Also, many design measures have been employed, such as action removal (AR), action sensor refinement (SAR), and action conditioning (AC) (Keren et al., 2018; Wayllace et al., 2018) .

2.2. OPPONENT MODELING AND NETWORK INTERDICTION:

Deceptive Opponent Modeling: Opponent modeling has been employed to reason the nonstationary strategies of autonomous agents (Albrecht & Stone, 2018; Papoudakis et al., 2019) . Deception involves combination of hiding the truth and showing the false (Masters & Sardina, 2017) , which articulated three distinct strategies in each case: dissimulation (masking, repackaging, and dazzling), simulation (mimicking, inventing, and decoying) (Ettinger & Jehiel, 2010) . As for deceptive path planning, game-theoretic method (Root, 2005) and model based (Masters & Sardina, 2017) method have been investigated.

Network Interdiction: In recent researches, interdiction has been used to counter the attacker's behavior, MDP interdiction (Panda & Vorobeychik, 2017) , plan interdiction (Vorobeychik & Pritchard, 2018) , and network interdiction (Xu et al.) show one promising research thread of Observe-Orient-Decide-Act (OODA).

2.3. REGULARIZED MDPS:

Most sequential decision-making problems can be described as MDP based reinforcement learning problems or stochastic game problems. Regularization for MDPs can be divided into temporal regularization problems in temporal space (Thodoroff et al., 2018) , spatial regularization in feature or state space (Farahmand, 2011; Harrigan) , and entropy or information regularization in policy space (Neu et al., 2017; Geist et al., 2019; Belousov & Peters, 2019) . Regularized reinforcement learning focuses on the smoothness and sparsity of the value function (Farahmand, 2011) , acquiring of diverse robot skills (Haarnoja, 2018) , and sparsity and multi-modality of the optimal policy (Lee et al., 2019) . Regularized stochastic game employees various entropy constraints to balance or control behavior (Grau-Moya et al., 2018; Savas et al., 2019; Ahmadi et al., 2018; Tian et al., 2019) .

3. STOCHASTIC GOAL RECOGNITION CONTROL:

The goal recognition control (GRC) problems try to balance the intended and adversarial goal recognition process under network interdiction (Luo et al., 2019) , which extends to assume that the agent can choose to reveal or obfuscate (Kulkarni et al., 2019b; a) , share or hide (Strouse et al., 2018) the goals. In this paper, we focus on stochastic GRC (S-GRC) under adversarial, cooperative, and non-stationary environment. The framework of our proposed approach is illustrated in Figure 3 .

We embed deceptive opponent modeling and proactive network interdiction into the OODA cycle as two modules, which could bridge the gap between observation and decision-making. The deceptive opponent modeling module will shape the multi-modal opponent, the proactive network interdiction module will provide bounded resource allocation for defense. So, GRC will identify the goals with topological soft q learning and provide proactive interdiction resource allocation.

3.1. SSP-MDP FOR S-GRC:

Different from the GRD problem, we employ more applicable and soft measure of interdiction with bounded resource to control the goal recognition process. With this motivation in mind, we use the Stochastic Shortest-Path MDP (SSP-MDP) (Kolobov & Kolobov, 2012 ) model, which is widely used for uncertainty given an initial state, a set of goal states, actions with probabilistic outcomes and an action cost function. A SSP-MDP is represented as a tuple S, s 0 , A, T, C, G , which consists a set of states S, a initial state s 0 , a set of actions A, a transition function T : S × A × S → [0, 1] of state transition from s to s after executing the action a, and a cost function C : S × A × S → R that gives the cost of executing action a, and a set of terminal goals G ⊆ S. With a cost function to replace the reward function, the objective is to find a optimal policy π so as to minimize the smallest expected cost or travel time of the policy, which forms the map from states to actions. So, the optimal policy of the SSP-MDP can be find by using the Value Iteration (VI) algorithm, which use a value function V to represent expected costs. The expected cost of an optimal policy π * for the starting state s 0 ∈ S is the expected cost V (s 0 ), and the expected cost for all states s ∈ S can be calculated by the Bellman equation (Kolobov & Kolobov, 2012) : 

EQUATION

A previous work on stochastic shortest path network interdiction modeled the problem as one sequential bi-level attacker-defender game, where the defender moves first by deploying sensors to the arcs in the network maximizing the expected shortest traveling time of the adversary (Zhang et al., 2018) . Here we formulate the stochastic goal recognition control (S-GRC) problem as an extension of the GRD problem to allows non-optimal agents and stochastic actions, which means the agent's action are not deterministic and the possible successor states are with probability, the measures we can take are interdictions with cost. The bounded interdiction resources allocations can better model the real world applications. A S-GRC problem is represented as a tuple P = D, G , the domain information D = S, s 0 , A, T, C, G can be represented as SSP-MDPs. Similar to the S-GRD (Wayllace et al., 2018) , we employ a metric to assess the largest number of actions an agent can take or the largest cost an agent will incur before revealing the goal. Given a problem P = D, G , the worst case distinctiveness (wcd) of the problem can be defined as:

wcd(P ) = max π∈II V π (s 0 ) (2)

So the objective of S-GRC under network interdiction can be reformulated as one optimization problem of finding a subset of actions ∆A * ⊂ A:

EQUATION

where P = D, G is the problem with domain D = S, s 0 , Â, T, C, G , Â is the redesigned action set with ∆A * interdicted, I is the corresponding cost to the interdiction resource, the total interdiction resource allocation is bounded by k.

3.2. AUGMENTED SSP-MDP WITH SCCS:

With the metric wcd, we can measure the maximal number of actions an agent can take before its goal is revealed. However, the formal wcd related metrics for S-GRDs are inconsistent with the intuitive definition, the wcd computation is not Markovian owing to the set of possible goals depend on the observed trajectory to a state. To deal with the uncertain inconsistence, wcd ag and ECD together with topological value iteration are employed to model the problem, in which the augmented MDPs with strongly connected components do not have loops (Wayllace et al.) .

The state space of an MDP can be represented as a directed connectivity graph. One example MDP is shown in Figure 3 (a), where the states are denoted by nodes, the actions are denoted by edges, the transitions are related to the arrow. The wcd may be infinite when the state space contains some cycle loop, here we can use the Tarjan algorithm (Tarjan, 1972; Hou et al., 2014) to generate strongly connected components (SCCs), which form a directed acyclic graph (DAG) as shown in Figure 3 Augmented SSP-MDP for S-GRC: Given a SSP-MDP, an augmented SSP-MDP S, s0 , Ã, T, C, G augments each component of the tuple:

• Each state ŝ ∈ Ŝ is represented by s, G .

• The augmented initial state is s0 = s 0 , G .

• Each augmented action â ∈ Â is a tuple a, G , where aA and G is the set of all goals for which that action is an optimal action.

• • The cost function C : S × Ã × S → R + gives the cost C (s, ã, s ) of executing action ã in augmented state s and arriving in s . This cost equals the cost C (s, ã, s ) = C(s, a, s ) under the same condition as above.

• The augmented goal states G ⊆ S are those augmented states s, G for which any execution of an augmented action will transition to an augmented state s , G with one goal or no goals.

3.3. DECEPTIVE OPPONENT MODELING:

As the opponent may change the goals midway to mislead the goal recognition process. We use a binary variable λ ∈ {0, 1} to characterize whether the opponent is a optimal or deceptive with decoy goals.

Depending on λ, the opponent is expected to exhibit different behaviors, which is fully described by an opponent policy π o (a o t |s t ). This model is restrictive since the action probability only depend on the current state. Nonetheless, we use this model only for policy learning, and use a general history dependent opponent policy for the evaluation of the learned autonomous agent policy. Another implicit assumption of this model is that the opponent has full observability over the states. This assumption could be released by modeling the opponent as a POMDP agent.

Neutral Opponent: If the opponent is a civilian, i.e. λ = 0, we assume a simple reactive policy π cil (a o t |s t ) is available to model the opponent: Soft Q Learning: We use soft-Q learning (Haarnoja, 2018) to learn a stochastic belief space policy. The soft Q learning objective is to maximize the expected reward regularized by the entropy of the policy, t E bt,st,at∼ρπ γ t [r(b t , s t , a t ) + σH(π(•|b t , s t ))].

EQUATION

(7)

The parameter σ controls the 'softness' of the policy. The nice interpretation of this objective function is maximizing accumulative reward while behaving as uncertain as possible, which is a desired property against an adversary. This maximum entropy problem is solved using soft Q iteration. For discrete action space, the fixed point iteration: 

Q soft (b t ,

converges to the optimal soft value functions Q * soft and V * soft (Haarnoja, 2018) , and the optimal policy can be obtained from: 

π * MaxEnt (

5. CONCLUSION AND FUTURE WORK:

Risk-Sensitive MDPs (Hou et al., 2014) illustrate one applicable scenario where the cost function is constrained with one threshold or deadline.

See Table 1 .

:



Figures and Tables:

Figure 1: Illustration of critical infrastructure protection (CIP) (Guillarme et al., 2017): the defender can perform proactive road network interdiction to accelerate the opponent's goal recognition even with possible deceptive behaviors.

Figure 2: The OODA cycle embedded with stochastic goal recognition control (S-GRC): one deceptive opponent modeling module and one goal recognition control module.

(b).

Figure 3: Augmented MDP with strong connected components (SCCs).

The new transition function T : S × Ã × S → [0, 1] gives the probability T (s, ã, s ), where s = s, G , ã = a, G , and s = s , G ∩ G , T (s, ã, s ) = T (s, a, s ), if |G ∩ G | > 1 and equals 0 otherwise.

Sample table titleThe temperature parameter α in (6) represents the level of rationality of the adversary. The other parameter β indicates the level of deception. Z(s t ) is the partition function that normalizes π MDP

Table content: <table><tr><td/><td colspan="2">PART</td><td>DESCRIPTION</td></tr><tr><td/><td colspan="3">Dendrite Input terminal</td></tr><tr><td/><td>Axon</td><td/><td>Output terminal</td></tr><tr><td/><td>Soma</td><td/><td>Cell body (contains cell nucleus)</td></tr><tr><td colspan="4">Deceptive Opponent: We use the following equation to model an adversarial agent's policy π o :</td></tr><tr><td colspan="4">π o (a o t |s t , λ = 1; α, β) =argmin π∈∆ {KL(π|π MDP α</td><td>) + βKL(π|π o (•|s t , λ = 0))},</td><td>(5)</td></tr><tr><td/><td>π MDP α</td><td colspan="2">(a o t |s t , λ = 1) = e αQ(st,a o t ) /Z(s t ),</td><td>(6)</td></tr><tr><td colspan="4">where KL(•|•) denotes the Kullback-Leibler divergence between two distributions, The goal-</td></tr><tr><td>achieving policy π MDP α</td><td colspan="3">is associated with the optimal Q function Q(s t , a o t ), of a goal-achieving</td></tr><tr><td colspan="2">adversary MDP defined later.</td><td/></tr></table>

s t , a t ) ← r t + γE bt+1,st+1∼ps [V soft (b t+1 , s t+1 )], (8) V soft (b t , s t ) ← σ log

Table content: <table><tr><td>a∈A</td><td>exp(</td><td>1 σ</td><td>Q</td></tr></table>

a t |b t , s t ) = exp( 1 σ (Q * soft (b t , s t , a t ) -V * soft (b t , s t ))).

Table content: <table><tr><td>(10)</td></tr><tr><td>3.4 PROACTIVE NETWORK INTERDICTION</td></tr><tr><td>Proactive network interdiction</td></tr><tr><td>4 EXPERIMENTS</td></tr><tr><td>4.1 GRID WORLD</td></tr><tr><td>4.2 NETWORK GRAPH</td></tr><tr><td>4.3 RESULTS</td></tr></table>

Peer review (iclr2020):
This paper proposes to use reinforcement learning to model an agent that is reaching goal that it is intended to reach. The authors consider the case where the agent is (1) indifferent to being observed, (2) trying to help an observer reach its goal, and (3) trying to fool the observer into not reach its goal. The paper propose to use a value function to quantify how easy it is to predict where the agent is going (*worst case distinctiveness*). The authors propose to then train an agent to modify the action space to make it difficult for an agent to fool the observer. While this is an interesting idea, the paper is clearly incomplete. The experiment section is missing, and much of the method section is partly written.