## 5 Reinforcement Learning Algorithms Compared  
Reinforcement learning (RL) provides a robust framework for sequential decision-making under uncertainty, allowing autonomous agents--such as patrol vehicles or drones--to learn optimal navigation strategies from trial-and-error interaction with dynamic environments [1]. In this study, we evaluate and compare three RL algorithms widely used in both foundational and applied domains: Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Proximal Policy Optimization (PPO). Each approach is assessed for its effectiveness in adapting to crime-aware patrolling tasks, especially within non-stationary, sensor-rich, urban environments. Deep Q-Network (DQN) uses a single neural network to approximate the action-value function \(Q(s,a)\), estimating expected cumulative rewards for taking action \(a\) in state \(s\) and following an optimal policy thereafter. Originally developed for playing Atarigames [1], DQN has since been adopted in urban mobility applications such as traffic light optimization. Ref. [19] and autonomous vehicle routing [7]. However, in our implementation with 2024 DC crime data, DQN displayed tendencies to overfit high-frequency crime zones and failed to generalize effectively across diurnal cycles. This overestimation led to repetitive zone visitation and suboptimal spatial coverage. To address this limitation, we implemented Double Deep Q-Network (DDQN), which decouples action selection and evaluation by employing two separate neural networks. The primary policy network chooses actions, while the target network evaluates them, thereby reducing overestimation bias. This architecture, widely used in recent patrol and robotics planning applications [8], improved stability and convergence in our patrol simulations. In particular, DDQN policies adapted smoothly during high-risk periods (e.g., 10 p.m.-2 a.m. in clusters 20, 22, and 24 of DC), yielding higher average rewards over 100 episodes. The DDQN update rule is expressed as  
\[Q(s_{t},a_{t})\gets r_{t}+\gamma Q^{\prime}(s_{t+1},\arg\max Q(s_{t+1},a; \theta)) \tag{5}\]  
where \(r_{t}\) is the immediate reward, \(\gamma\) the discount factor, and \(Q^{\prime}\) the target network. Proximal Policy Optimization (PPO), a policy-gradient method developed for continuous action spaces, directly learns a stochastic policy \(\pi(a|s)\) instead of estimating Q-values. PPO constrains policy updates through a clipped surrogate objective to ensure stable learning [14]. Its ability to generalize in high-variance settings has led to applications in swarm-based drone patrols [10] and dynamic evacuation planning [9]. In our implementation, PPO was tested with synthetic AV telemetry--including vehicle speed, fuel level, and camera-based threat detection--forming a continuous, sensor-rich state space. The PPO agent employed Generalized Advantage Estimation (GAE) and a shared actor-critic network structure to balance exploration and exploitation. While PPO exhibited slower convergence than DDQN, it demonstrated superior resilience to noisy or incomplete inputs. In simulated scenarios involving real-time crime surges and crowding anomalies, PPO adjusted its patrol patterns more responsively than value-based methods. These findings align with recent evidence from urban surveillance systems where PPO facilitated adaptive policy learning in emergency evacuations and multi-agent congestion zones [9]. Figure 4 presents the reward trajectory of DDQN and PPO over 100 episodes. DDQN converged faster and attained higher peak rewards, while PPO's lower variance suggests greater robustness in unpredictable environments. Overall, DDQN is well-suited for moderately dynamic environments requiring high reward precision, whereas PPO is advantageous in high-variance, real-time applications that demand frequent adaptation.