# Learning Deceptive Strategies in Adversarial Settings:
## 3 Game-Theoretic Reward Shaping
### Police Officer Reward Shaping
4:Initialize replay buffers \(\mathcal{D}_{\mathsf{R}},\mathcal{D}_{\mathsf{P}}\) with capacity \(C\)
5:for\(t=1,2,\ldots\)do
6: Observe game-states \(\mathbf{s}(t)\) \(\triangleright\) Defined in Equation (2)
7: Select \(\mathbf{a}_{\mathsf{R}}(t),\mathbf{a}_{\mathsf{p}}(t)\) via \(\varepsilon_{t}\)-greedy policies
8: Execute \(\mathbf{a}_{\mathsf{R}}(t),\mathbf{a}_{\mathsf{p}}(t)\); observe \((r_{\mathsf{R}}(t),r_{\mathsf{p}}(t),\mathbf{s}(t+1),\mathsf{done})\)\(\triangleright\) done=1 iff \(\mathbf{x}_{\mathsf{R}}(t+1)=g_{{}_{T}}\)
9: Store transitions \((\mathbf{s}(t),\mathbf{a}_{\mathsf{R}}(t),\mathbf{a}_{\mathsf{p}}(t),r_{\mathsf{ R}}(t),\mathbf{r}_{\mathsf{p}}(t),\mathbf{s}(t+1),\mathsf{done})\) in \(\mathcal{D}_{\mathsf{R}},\mathcal{D}_{\mathsf{P}}\)
10:if\(|\mathcal{D}_{\mathsf{R}}|\geq B\)then
11:\(L_{\mathsf{R}}\gets 0\), \(\triangleright\) Loss function for R
12: Sample minibatch of size \(B\) from \(\mathcal{D}_{\mathsf{R}}\),
13:for each experience \((s,\mathbf{a}_{\mathsf{R}},\mathbf{a}_{\mathsf{p}},r_{\mathsf{R}},\mathbf{s}^{ \prime},\mathbf{done})\) in \(B\)do
14: Compute targets \(y_{\mathsf{R}}=r_{\mathsf{R}}+(1-\mathsf{done})\,\gamma\,\max_{\mathbf{a}\in A _{\mathsf{R}}}Q_{\theta_{R}^{-}}(\mathbf{s},\mathbf{a})\)
15:\(L_{\mathsf{R}}\gets L_{\mathsf{R}}+\texttt{HuberLoss}(y_{\mathsf{R}}, \mathcal{Q}_{\theta_{R}}(s,\mathbf{a}_{\mathsf{R}}))\)
16:endfor
17: Update \(\theta_{\mathsf{R}}\) by minimizing \(L_{\mathsf{R}}\); clip gradients with bound G
```

**Algorithm 1**_Cont._

## 4 Deep Q-Network Training

To jointly induce deceptive and counter-deceptive behaviors, we optimize the robber's and police officer's policies via deep Q-learning with _soft_ target-network updates. Each agent maintains an _online_ deep Q-network (DQN) \(Q_{\theta}\) and a lagging _target_ DQN \(Q_{\theta^{-}}\), both parametrized by multi-layer perceptrons with weights \(\theta\) and \(\theta^{-}\), respectively. There are total of four networks \((\theta_{\mathsf{R}},\theta_{\mathsf{R}}^{-},\theta_{\mathsf{P}},\theta_{ \mathsf{P}}^{-})\), two for each player. _Experience_ (in RL, _experience_ is a tuple that contains the current state, current action, reward, next state, and occasionally a binary variable denoting whether the episode ended in the next state or not) is stored in a replay buffer of symbolic capacity \(C\), from which minibatches of size \(B\) are sampled to decorrelate updates. We employ a discount factor \(\gamma\in(0,1)\), gradient-norm clipping bound \(G\), and a soft-update coefficient \(\tau\ll 1\) so that after each gradient step the target parameters blend smoothly:

\[\theta^{-}\ \longleftarrow\ \tau\,\theta\ +\ (1-\tau)\,\theta^{-}.\]

Exploration is governed by time-dependent rates \(\{\varepsilon_{t}\}\): at each decision point an agent selects the action that maximizes its current Q-estimate with probability \(1-\varepsilon_{t}\), and otherwise samples uniformly from its action set. The schedule for decaying \(\varepsilon_{t}\) is chosen to ensure ample early exploration followed by gradual consolidation on high-value strategies. Updates proceed whenever the replay buffer contains at least \(B\) transitions. A minibatch is drawn, and for each experience a bootstrapped target is computed against the target network using the _Huber loss_ to mitigate sensitivity to outliers. Gradients are clipped to norm \(G\) before each parameter update, reinforcing stability in the non-stationary, adversarial training setting (Algorithm 2). ```
1: replay capacity \(C\), batch size \(B\), discount \(\gamma\), soft-update \(\tau\), grad-clip bound \(G\)
2:trained Q-networks \(Q_{\theta_{R}},Q_{\theta_{P}}\)
3:Initialize online networks \(Q_{\theta_{R}},Q_{\theta_{P}}\) and targets \(Q_{\theta_{R}^{-}}\gets Q_{\theta_{R}}\) and \(Q_{\theta_{P}^{-}}\gets Q_{\theta_{P}}\).