# Deep Reinforcement Learning-Based Joint Scheduling of 5G and TSN in Industrial Networks
## 6 Results
### Results Analysis and Discussion  
The convergence speed of the proposed model and the final stable reward value are very important for the reinforcement learning model. In order to accelerate the model convergence process and increase the reward value, several DDPG model related parameters are fine-tuned, including learning rate, model structure and other hyperparameters. \begin{table}
\begin{tabular}{c c} \hline \hline The probability of packets arriving within each TTI & 0.5 \\ \hline The number of packets arriving within each TTI & The number is valued from 1–3, followed by a uniform distribution \\ \hline Time-triggered application packet size & 200 Bytes \\ \hline Video flow packet size & 800 Bytes \\ \hline gNB transmission power & 20 dBm \\ \hline Channel noise power & \(-90\) dBm \\ \hline \(\kappa_{t}\) & 100 \\ \hline Subcarrier interval & 15 kHz \\ \hline \(\Delta\) & 0.1 \\ \hline EMLR of time-triggered application (\(d_{i}^{\text{obs}}\)) & 6 ms \\ \hline Deadline of video flow & 20 ms \\ \hline \(\lambda_{\max}\) & 0.2 \\ \hline \(\beta\) & 0.01 \\ \hline \(\alpha\) & 0.01 \\ \hline \(R_{j}^{\min},j\in\mathbf{F}_{vi}\) & 80,000 Bytes per second \\ \hline Discount factor \(\mu\) & 0.9 \\ \hline min trainset num & 800 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. As shown in Figure 5, we optimize model training process by adjusting the learning rate, using layer normalization and reshaping the reward function with Equation (15). As shown in Figure 5, the DDPG model without optimization has serious oscillation during training, because the learning rate and the step size are too large. When the learning rate is reduced from 0.1 to 0.01, it alleviates the problem of model oscillation. In addition, we can see that the reward value appears to decrease during the model training process when there is no layer normalization, because the distribution of input data may change continuously due to parameters updating, which leads to less optimal model training. Therefore, layer normalization is introduced for model training, which makes the reward value of the DDPT model increase markedly. Finally, the potential function is adopted for reward function, which makes the convergence process faster and the final obtained stable reward value larger. In order to compare 5G system performance under various scheduling algorithms, Proportional Fairness (PF) and Earliest Deadline First (EDF) are used for resource allocation in 5G-TSN cooperation architecture. PF and EDF are commonly used scheduling algorithms in 4G and 5G. PF ensures that all applications have the opportunity to be scheduled. To some extent, PF can provide a throughput guarantee for the required flows. EDF is a classical scheduling for time-sensitive applications. For the algorithm \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layers** & **Actor Network** & **Critic Network** \\ \hline \multirow{3}{*}{Input layer} & Fully connected layer (40 \(\times\) 400) \\ \cline{2-3}  & Rectified Linear Unit (Relu) nonlinear activation function \\ \cline{2-3}  & Normalization layer \\ \hline \multirow{3}{*}{Hidden layer} & Fully connected layer (400 \(\times\) 400) \\ \cline{2-3}  & Relu nonlinear activation function \\ \cline{2-3}  & Normalization layer \\ \hline \multirow{3}{*}{Output layer} & Fully connected layer (400,10) & Fully connected layer (400,1) \\ \cline{2-3}  & Tanh nonlinear activation function & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Network parameters. Figure 5: Average reward value vs. training process of DDPG. proposed in this article, the objective is to provide latency guarantee for time-triggered flows as well as to provide throughput guarantee for video flows, so we take these two schemes as a benchmark to compare the latency and throughput performance with the proposed algorithm. As shown in Figures 6 and 7, DDPG with a potential function reward is compared with DDPG without a potential function reward, EDF and PF.