## 3 Materials and Methods
### Hybrid DRLCC Framework
#### 3.2.3 Module 3: Decision-Making Layer (PPO Agent)
* total episodes \(E\), total jobs \(J\), seeds \(S=\{0,1,2,42,100,999\}\)
* final trained policy \(\pi^{*}_{\theta}\); per-seed CSV logs
* **procedure**TRAIN_DRLCC(\(E,J,S\))
* **for all \(s\in S\)do**
* SET_GLOBAL_SEED(\(s\)) \((\pi_{\theta},V_{\phi})\leftarrow\textsc{InitPPO}\big{(}\gamma=0.95,\;\lambda=0.95,\;\varepsilon=0.20,\) \(\textsc{lr}=2\times 10^{-4},\;K=6,\;\textsc{mini\_batch}=64\big{)}\)
* \(best\_reward\leftarrow-\infty\)
* \(metrics\_buf\leftarrow\emptyset\)
* **for**\(episode\gets 1\)to \(E\)do**
* \(env\leftarrow\textsc{Environment}()\)
* \(AMR\leftarrow\textsc{InitAMRS}(capacities=[150,200,250,150,200])\)
* **for all**\(amr\in AMR\)do
* \(env.\textsc{process}(\textsc{BATTERY\_CHECK}(amr))\)
* \(Q\leftarrow\emptyset\)
* \(metr\leftarrow\textsc{InitMetrics}()\)
* \(env.\textsc{process}(\textsc{JOB\_ARRIVAL}(env,AMR,Q,\pi_{\theta},metr,\textsc{ total\_jobs}=J,\lambda=0.4))\)
* \(env.\textsc{run}()\)
* **if**\(|\textsc{MEMORY}|\geq\textsc{mini\_batch}\)then
* \((\pi_{\theta},V_{\phi})\leftarrow\textsc{PPO\_UPDATE}(\pi_{\theta},V_{\phi})\)
* \(R\leftarrow\sum_{r\in\textsc{MEMORY}}r\)
* **if**\(R>best\_reward\)**then**
* \(best\_reward\gets R\)
* \(\textsc{SaveCheckpoint}(\pi_{\theta},V_{\phi},seed=s,episode)\)
* \(\textsc{Append}(metrics\_buf,metr)\)
* \(\textsc{LogEpisodesToCSV}(metrics\_buf,seed=s)\)
* **return**\(\pi^{*}_{\theta}\)shows the value-loss, policy-loss, and training-reward curves for our PPO-based DRLCC agent over six independent seeds. Key convergence diagnostics were as follows:  
1. Reward plateau Across both small (4 AMRs, 80 jobs) and larger scenarios (up to 8 AMRs, 200 jobs), the mean episodic reward rose sharply in early episodes. It plateaued by \(\sim\)350 episodes, with inter-seed variance dropping below 5%. 2. Loss stabilization Policy and value losses showed oscillations during the first \(\sim\)250 update steps but settled into a steady band by \(\sim\)300 steps. We applied gradient clipping (max norm 0.5) and a StepLR schedule (\(\gamma\) = 0.9 every 100 steps) to dampen these oscillations. 3. Hyperparameter robustness A \(\pm\)20% sweep on the learning rate, discount factor (\(\gamma\)), and clip ratio (\(\epsilon\)) changed final makespans by \(<\)3% in all tested configurations, confirming that performance is not acutely sensitive to precise hyperparameter values. Figure 7: Training curves for the PPO-based DRLCC agent over 6 independent seeds for scenario 4 AMR with 80 jobs. Mean value-loss (bold blue) with individual seed traces (faded lines) plotted against update step, demonstrating a steady decline and plateau by \(\sim\)4000 steps. Mean policy-loss (bold red) with individual seed traces (faded lines), oscillating within a narrow band around zero after initial stabilization. Mean episodic reward (orange) and its exponential moving average (green dashed) over update steps, showing convergence to \(\sim\)650 reward by \(\sim\)3500 steps. Shaded regions indicate \(\pm\)1 standard deviation across seeds. Transient dips  
Less than 1% of training runs showed short-lived reward drops--typically due to random latency spikes--yet the clipping mechanism and entropy bonus enabled rapid recovery, with no lasting harm to the learned policy. By combining smoothed reward curves, loss traces, and a modest sensitivity sweep, we ensured that our DRLCC models reliably converge under a variety of AMR fleet sizes and job arrival patterns before proceeding to comparative evaluation.