# Learning Deceptive Strategies in Adversarial Settings:
## 3 Game-Theoretic Reward Shaping
### Police Officer Reward Shaping
The unweighted sum \(d_{i}(t)+p_{i}(t)\) therefore captures the total number of remaining allocation steps required to _saturate_ goal \(i\) if the police officer were to devote all future resources there from time \(t\) onward. In other scenarios, one may need to use a weighted sum \(w_{d}d_{i}(t)+w_{p}p_{i}(t)\). Let us define \(c_{\min}=\min\{c_{i}(t):\;\;g_{i}\in G_{\min}\}\) and the set for which \(c_{i}\)'s are the lowest as follows  
\[C_{\min}=\{g_{i}\in G_{\min}\;:\;\;c_{i}(t)=c_{\min}\}\]  
Now, using \(C_{\min}\), we define P's reward as  
\[r_{p}(t)=\beta\Big{(}\sum_{i\in C_{\min}}\frac{a_{t,i}}{|C_{\min}|}-\sum_{i\
ot \in C_{\min}}a_{t,i}\Big{)},\]  
encouraging the police officer to allocate proportionally among the most effective candidate goals. Finally, to penalize overcommitment to decoy exits, we grant a conservation bonus \(r_{\text{bonus}}>0\) whenever the robber has previously visited exit \(i\) and the police officer allocates zero mass there. Collectively, these shaping terms drive the police officer to rapidly infer the robber's intended exit, discourage wasteful dispersion of resources, and maintain a calibrated balance between exploration and exploitation in the allocation strategy. The entire reward computation process is presented in Algorithm 1. ```
0:robber_pos\(\mathbf{x}_{\text{R}}(t)\), goal_alloc\(\{\mathbf{x}_{\text{P}}(s)\}_{s=0}^{t-H}\), robber_action\(\mathbf{a}_{\text{R}}\), police_action\(\mathbf{a}_{\text{P}}=[a_{1},\dots,a_{n}]\)
1:\(r_{\text{R}}\gets 0\;\;\;r_{\text{P}}\gets 0\)
2:\(\mathbf{x}_{\text{R}}(t+1)\leftarrow\text{next\_position}(\mathbf{x}_{\text{R}}(t), \mathbf{a}_{\text{R}})\).\(\triangleright\) Robber's next position
3:if\(\mathbf{x}_{\text{R}}(t+1)=g_{\text{T}}\)then
4:\(r_{\text{R}}\gets r_{\text{R}}+r_{\text{goal}}\)
5:elseif\(|\text{Boundary}(\mathbf{x}_{\text{R}}(t+1))\)then
6:\(r_{\text{R}}\gets r_{\text{R}}-r_{\text{P}}\)\(\triangleright\) Out of grid boundary penalty
7:elseif\(|\text{Obstacles}(\mathbf{x}_{\text{R}}(t+1))\)then
8:\(r_{\text{R}}\gets r_{\text{R}}-r_{\text{inv}}\)\(\triangleright\) Obstacle collision penalty
9:elseif\(\mathbf{x}_{\text{R}}(t+1)=\mathbf{x}_{\text{R}}(t)\)then
10:\(r_{\text{R}}\gets r_{\text{R}}-r_{\text{stay}}\)\(\triangleright\) Non-movement penalty
11:endif
12:compute \(r_{\text{resource}}(t)=\sum_{k=1}^{H}\alpha(k)\;p_{\text{T}}(t-k)\) and \(r_{\text{time}}(t)=\lambda\;t\)\(\triangleright\) Equation (4)
13:\(r_{\text{R}}\gets r_{\text{R}}-\big{(}r_{\text{resource}}+r_{\text{ time}}\big{)}\)\(\triangleright\) robber reward
14:\(d_{i}(t)\leftarrow\|\mathbf{x}_{\text{R}}(t)-g_{i}\|\),\(d_{i}(t-1)\leftarrow\|\mathbf{x}_{\text{R}}(t-1)-g_{i}\|\) for all \(g_{i}\in G\). 15: Compute \(\Delta d_{i}=d_{i}(t)-d_{i}(t-1)\) for all goals \(g_{i}\)
16:\(G_{\min}\leftarrow\{\;i:\Delta d_{i}=-1\}\)
17:if\(|G_{\min}|=1\)then
18: let \(g_{j}\) be the sole element of \(G_{\min}\)
19:\(r_{\text{P}}\gets r_{\text{P}}+\beta\,a_{j}-\beta\sum_{i\
eq j}a_{i}\)
20:else
21: for each \(i\in G_{\min}\) compute \(c_{i}(t)=d_{i}(t)+p_{i}(t)\)
22:\(c_{\min}=\min\{c_{i}(t):\;\;g_{i}\in G_{\min}\}\) and \(C_{\min}=\{g_{i}\in G_{\min}\;:\;c_{i}(t)=c_{\min}\}\)
23:for\(i=1\) to \(n\)do
24:if\(i\in C_{\min}\)then
25:\(r_{\text{P}}\gets r_{\text{P}}+\beta\,\frac{a_{i}}{|C_{\min}|}\)
26:else
27:\(r_{\text{P}}\leftarrow r_{\text{P}}-\beta\,a_{i}\)
28:endif
```  
**Algorithm 1** One-Step Reward Computation```
1: replay capacity \(C\), batch size \(B\), discount \(\gamma\), soft-update \(\tau\), grad-clip bound \(G\)
2:trained Q-networks \(Q_{\theta_{R}},Q_{\theta_{P}}\)
3:Initialize online networks \(Q_{\theta_{R}},Q_{\theta_{P}}\) and targets \(Q_{\theta_{R}^{-}}\gets Q_{\theta_{R}}\) and \(Q_{\theta_{P}^{-}}\gets Q_{\theta_{P}}\).