# Learning Deceptive Strategies in Adversarial Settings:
## 3 Game-Theoretic Reward Shaping
### Police Officer Reward Shaping  
The police officer's per-step reward, \(r_{\mathsf{P}}(t)\), must encourage accurate inference of the robber's hidden objective while maintaining coverage of alternative exits. We begin by measuring the change in (Manhattan) distance from the robber's current position \(\mathbf{x}_{\mathsf{R}}(t)\) to each exit \(\boldsymbol{g}_{i}\):  
\[d_{i}(t)=\left\|\mathbf{x}_{\mathsf{R}}(t)-\boldsymbol{g}_{i}\right\|_{1}, \quad\Delta d_{i}(t)=d_{i}(t)-d_{i}(t-1).\]  
Notice that, due to the grid-world nature of the environment and the up/down/left/right movement, \(\Delta d_{i}(t)\) can only take one of the two values: \(-1\) and \(1\). Therefore, \(\Delta d_{i}(t)\) is a direct indicator of whether the robber moved closer to the goal \(\boldsymbol{g}_{i}\) (i.e., when \(\Delta d_{i}(t)=-1\)) or moved further away. In general, one movement can lead to \(\Delta d_{i}(t)\) being \(-1\) for multiple goals. Therefore, to separate the goal(s) for which the distance was minimized by the robber, we define the set  
\[G_{\min}=\{\,\boldsymbol{g}_{i}\in G\ :\ \Delta d_{i}(t)=-1\}.\]  
[PERSON]'s allocation vector \(a_{t}=[a_{1}(t),\ldots,a_{n}(t)]\) is then rewarded according to the following:  
_Unique Minimum:_ If \(|G_{\min}|=1\) with \(\boldsymbol{g}_{j}\) being the only goal in the set \(G_{\min}\), we define [PERSON]'s reward as  
\[r_{\mathsf{p}}(t)=\beta\,a_{j}(t)-\beta\sum_{ \begin{subarray}{c}i=1\\ i\
eq j\end{subarray} }^{n}a_{i}(t), \tag{5}\]  
where \(\beta>0\) scales the incentive to concentrate force on the most likely exit. Notice that the allocation vector \(a(t)\) satisfies the condition \(\sum_{t=1}^{n}a_{i}(t)=1\) as mentioned in (1). Consequently, \(r_{\mathsf{p}}\) in (5) can equivalently be written as  
\[r_{\mathsf{p}}(t)=\beta(2a_{j}(t)-1).\]  
This shows that at least half of the allocation must be assigned to the unique goal (i.e., \(a_{j}(t)>0.5\)) for which the robber decreased its distance to receive a non-zero reward. If \(a_{j}(t)<0.5\), the reward is negative, implying that the police officer is being penalized. It should be noted that in our earlier formulation, \(a_{i}(t)\in\{0,1\}\) for each goal \(i\) (see (1)), which forces [PERSON] to fully commit to the goal to receive a positive reward, as partial allocations are not permitted. To introduce exploration into [PERSON]'s strategy, one could relax this constraint and allow fractional allocations, i.e., \(a_{i}(t)\in[0,1]\). While this adds flexibility to [PERSON]'s strategy, it also increases the complexity of the problem, as [PERSON]'s action space becomes continuous. Beyond this added complexity, such a formulation may be less practical, as in real-world settings, [PERSON] may not be able to allocate arbitrary fractions of a unit to different goals. To strike a balance between practical applicability and strategic flexibility, we retain a discrete action space but expand the set of possible allocations beyond what was allowed in (1). A detailed description of \(A_{\mathsf{P}}\) will be provided later in the simulation section. _Tied Minima:_ If \(|G_{\min}|>1\), we refine the decision by introducing a new quantity  
\[c_{i}(t)=d_{i}(t)+p_{i}(t)\]  
which denotes the sum of the current allocation at goal \(\boldsymbol{g}_{i}\) and the current distance to that goal. Essentially, \(c_{i}(t)\) represents the maximum possible amount of reinforcement that could be allocated to goal \(\boldsymbol{g}_{i}\) if it were the true goal, assuming that [PERSON] begins moving directly toward it and [PERSON] fully commits its resources to that goal for the remainder of the time. In our framework, \(d_{i}(t)\) is _precisely_ the number of lattice steps needed to reach \(\boldsymbol{g}_{i}\), and the police officer receives exactly one unit of reinforcement per time step. Consequently, boththe distance and allocation terms are already expressed in identical, naturally normalized units.