# Deep Reinforcement Learning-Based Joint Scheduling of 5G and TSN in Industrial Networks
## 5 DDPG-Based Resource Scheduling Algorithm  
Based on the above elaboration, we have designed an MDP model with the appropriate state space, action space and reward function according to the optimization problem described in Equation (17) with constraints in Equations (1), (2) and (6). Utilizing the reinforcement learning model for resource scheduling, the best policy achieved from the training process can maximize the expected reward value, which is defined as follows. \[\mathbf{\mathcal{A}}_{t}^{*}=\operatorname*{argmax}_{\mathbf{\mathcal{A}}_{t}}\sum_{k =t+1}^{L}\sum_{i=1}^{N}\psi_{k}(\gamma_{i}) \tag{17}\]  
In DQN, the general output consists of multiple dimensions of Q-values for each action. For example, when scheduling users, it outputs Q-values for both scheduling and not scheduling each user. However, DDPG combines the advantages of both policy-based and value-based methods. The actor network in DDPG learns a deterministic policy that directly maps states to actions, allowing for fine-grained control and explicit action outputs. In the case of scheduling users, it can directly output the actions for each individual user. Therefore, in this section, a DDPG-based resource scheduling algorithm is proposed to achieve the optimal action from the above formula, which is more suitable for multi-user tasks. DDPG includes two neural networks: the actor network is denoted as \(\pi(\mathbf{S};\mathbf{\Theta})\), which takes the state \(\mathbf{S}\) as input and outputs actions \(\mathbf{A}\). The critic network is denoted as \(Q(\mathbf{S},\mathbf{\mathcal{A}};\mathbf{W})\), and it takes the state \(\mathbf{S}\) and actions \(\mathbf{\mathcal{A}}\) as inputs and outputs the evaluation Q-value of actions \(\mathbf{\mathcal{A}}\). Besides this, \(\mathbf{\Theta}\) and \(\mathbf{W}\) indicate the parameters of the actor and critic networks, respectively. The training process of DDPG is to make the actor achieve an action maximizing the approximate Q-value given by the critic network, as well as to make the critic evaluate an approximate Q-value for the actions learnt from the actor closer to the true Q-value calculated by the Bellman Equation in Equation (18). \[Q_{t}^{true}=\psi_{t}(\gamma_{i})+\mu Q_{t+1} \tag{18}\]  
where \(\mu\) is the discount factor. Once the number of transition sets \((\mathbf{S}_{t},\mathbf{\mathcal{A}}_{t},\psi_{t}(\gamma_{i}),\mathbf{S}_{t+1})\) meets the training requirements, the parameters of the actor and critic networks are updated according to Equations (19)-(23). Figure 4: Relationship between \(\lambda\) and queueing delay in gNB. The critic calculates \(Q_{t}\) of action \(\mathbf{\mathcal{A}}_{t}\) in state \(\mathbf{S}_{t}\):  
\[Q_{t}=Q(\mathbf{S}_{t},\mathbf{\mathcal{A}}_{t};\mathbf{W}) \tag{19}\]  
The critic ranks the \(Q_{t+1}\) of the state \(\mathbf{S}_{t+1}\) and action \(\mathbf{\mathcal{A}}_{t+1}\):  
\[Q_{t+1}=Q(\mathbf{S}_{t+1},\mathbf{\mathcal{A}}_{t+1};\mathbf{W}),\textit{where }\mathbf{\mathcal{A}}_{t+1}=\pi(\mathbf{S}_{t+1};\mathbf{\Theta}) \tag{20}\]  
Calculation loss is defined as the difference between true Q-value and predicted Q-value by the critic with Equation (20):  
\[\delta_{t}=Q_{t}-Q_{t}^{true} \tag{21}\]  
Then the critic parameters are updated using gradient descent:  
\[\mathbf{W}=\mathbf{W}-\mathbf{\alpha}*\delta_{t}*\frac{\partial Q(\mathbf{S}_{t},\mathbf{\mathcal{ A}}_{t};\mathbf{W})}{\partial\mathbf{W}} \tag{22}\]  
The goal of the actor network is to achieve an action-maximizing approximate \(Q_{t}\) given by the critic network.