## 3 Materials and Methods
### Hybrid DRLCC Framework
#### 3.2.3 Module 3: Decision-Making Layer (PPO Agent)
This rich, multi-dimensional state vector enables the DRL agent to reason over complex scheduling and resource allocation decisions within a dynamic environment. Action Space: The action space consists of assigning a specific task to an available AMR. The agent selects an action based on the observed state, aiming to maximize system performance while respecting priority and resource constraints. The action space defines all the agent's possible decisions at a given state. In our case, each action corresponds to the assignment of a specific AMR to a specific waiting job. Formally,  
* If there are \(N\) AMRs and \(J\) waiting jobs, the agent's action space consists of \(N\ \times\ J\) potential pairings. * At each step, the agent selects a (job, AMR) pairing, subject to validation by the PCP before execution. Invalid actions (e.g., assigning a task to a charging AMR or violating the PCP ceiling) are filtered out by the PCP module, ensuring policy compliance with system constraints. * State \(s_{t}\ \in\ \mathbb{R}^{\wedge}\ (4N\ +\ 8)\) as detailed in Module 1. * Action \(a_{t}\) chooses one of the \(N\) AMRs for the current job. * We mask out infeasible AMRs (capacity, battery, or ceiling violations) so the policy is only assigned from valid AMRs. Reward Mechanism: The agent receives a composite reward that reflects the three operational objectives of the system: efficient task completion, responsiveness to high-priority jobs, and avoidance of priority inversions (see Table 2). After each epoch, the updated actor weights are copied into the frozen \"old\" policy network so that probability ratios in the next epoch are computed against freshly learned parameters. A StepLR scheduler (factor 0.9 every 100 updates) is applied to the Adam optimizer to improve long-horizon convergence. The complete training loop, including data collection, multi-epoch PPO updates, and checkpointing, is provided in Figure 6.  
Table 3 provides a concise description of the helper routines and update steps invoked within Figure 6. All hyperparameter settings were kept constant across the six random seeds listed above. A one-off sensitivity check (\(\pm\)20% on \(\gamma\), \(\varepsilon\), and LR) showed \(<\) 3% variation in makespan, confirming that performance is not acutely sensitive to these values. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Trigger** & **Value** \\ \hline \(\mathbf{R}_{\text{Comp}}^{(2)}\) & Base reward per completed high-priority task & +5 \\ \hline \(\mathbf{R}_{\text{comp}}^{(0/1)}\) & Base reward per completed medium/low task & +2 \\ \hline \(\mathbf{R}_{\text{wait}}\) & \(-\)0.1 \(\times\) task-wait-time (s) & -0.1 s\({}^{-1}\) \\ \hline \(\mathbf{P}_{\text{block}}\) & \(-\)0.2 \(\times\) job-block-count & -0.2 block\({}^{-1}\) \\ \hline \(\mathbf{P}_{\text{inv}}\) & Priority inversion penalty & -10 \\ \hline \(\mathbf{R}_{\text{quick}}\) & Bonus if a high-priority job finishes \(<\) 20 s after arrival & +20 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Reward values are provided during the training of the model.