## 3 Materials and Methods
### Hybrid DRLCC Framework
#### 3.2.3 Module 3: Decision-Making Layer (PPO Agent)  
While the Priority Ceiling Protocol (PCP) guarantees mutual exclusion and prevents unbounded priority inversion, it cannot, by itself, optimize resource utilization in highly dynamic, unpredictable manufacturing scenarios. By integrating DRL, we enable the system to learn from experience and continually improve scheduling decisions, balancing throughput, delay reduction, and adherence to safety constraints. RL Agent Overview: The PPO agent sits atop the simulated environment, closing the loop. Receive a normalized state vector from Module 1. Sample a candidate AMR assignment from the policy network, masked to only valid robots. Module 2's PCP logic validates or rejects the action, yielding either execution or a penalty. The environment returns a scalar signal reflecting task urgency, wait penalties, block counts, and inversion events. At the end of an episode, PPO ingests all \((s,a,r,log\pi)\) tuples and updates its policy and value networks via a clipped surrogate objective with Generalized Advantage Estimation (GAE). Proximal Policy Optimization (PPO) was selected as the DRL backbone for our framework due to its strong balance between stability, simplicity, and sample efficiency, qualities essential for dynamic, distributed environments like Federated Digital Twin-based manufacturing systems. PPO stands out due to its balance of stability, sample efficiency, and ease of implementation. These qualities are especially critical in our context, where frequent policy updates, limited training time, and high variability in system states are key factors. The objective function is formally expressed as follows:  
\[L_{CLIP}(\theta)=\ E_{t}[min(r_{t}(\theta)A_{t},\ clip(r_{t}(\theta),1-\epsilon, \ 1+\epsilon)A_{t})] \tag{6}\]  
where  
* \(\theta\) is the policy parameters;
* \(E_{t}\) is the expectation over timesteps;
* \(r_{t}(\theta)\) is the probability ratio between new and old policies;
* \(A_{t}\) is the advantage function estimating the relative benefit of an action;
* \(\epsilon\) is the clipping threshold. Our PPO agent is implemented using PyTorch 2.5.0+cu118, leveraging its deep learning and auto-differentiation capabilities for efficient policy gradient computation. A three-layer fully connected backbone with two 256-unit ReLU hidden layers feeds parallel policy and value heads; complete architectural and training settings are summarized in Table 1.  
State representation: The DRL agent receives a state vector at each decision step, which captures the current status of the manufacturing system. This state representation is carefully designed to provide the agent with a comprehensive view of all relevant parameters needed for effective decision-making. The state includes the following features:  
* Job Queue Status: The number of pending jobs, their priority levels (0: low, 1: medium, 2: high), and their waiting times;
* AMR Status: Availability (free or occupied), current location, battery level, and charging status of each AMR;
* Resource Occupancy: Information about which AMRs are currently assigned and to which job; \begin{table}
\begin{tabular}{l l l} \hline \hline
**Category** & **Item** & **Value** \\ \hline Architecture & Input dimension & 4 N + 8 \\ \hline  & Hidden layers & 2 \(\times\) 256 (ReLU) \\ \hline  & Policy head & Fully connected, M outputs, Softmax \\ \hline  & Value head & Fully connected, 1 output, linear \\ \hline Training & Optimizer & Adam \\ \hline  & Learning rate & 1 \(\times\) 10\({}^{-4}\) \\ \hline  & Clip ratio \(\epsilon\) & 0.10 \\ \hline  & Discount \(\gamma\) & 0.95 \\ \hline  & GAE \(\lambda\) & 0.95 \\ \hline  & Mini batch/Epochs & 64/6 \\ \hline  & Entropy bonus & 0.01 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: PPO agent neural network architecture and training hyperparameters. Task-Specific Data: The number of subtasks remaining and the estimated processing times.