# Deep Reinforcement Learning-Based Joint Scheduling of 5G and TSN in Industrial Networks
## 3 System Model and Problem Formulation
### Problem Formulation  
For a time-triggered application, its QoS requirement is to satisfy end-to-end latency; meanwhile, the QoS requirement of video is to guarantee transmission rate. Therefore, the agent, acting as a scheduler, aims to achieve a QoS tradeoff between time-triggered and video applications by jointly optimizing radio resource allocation. In this article, the purpose of scheduling is to provide a solid latency guarantee for time-triggered applications as well as to improve the throughput for video. In order to achieve this goal, the utility function of flow \(i\) at the \(t\)th TTI is defined as follows.  
\[U_{i}(t)=\frac{p_{i}(t)}{d_{i}^{pos}-d_{i}^{sub}+I_{ds-tt}(t)+\Delta} \tag{3}\]  
where \(p_{i}(t)\) is the resource allocation indicator of flow \(i\) at the \(t\)th TTI. \(p_{i}(t)\in\{0,1\}\), \(p_{i}(t)=1\) denotes the flow \(i\) has been scheduled at this TTI; otherwise, \(p_{i}(t)\)\(=0\). \(d_{i}^{obs}\) and \(d_{i}^{sub}\) are the EMLR and queueing and processing delay in gNB of flow \(i\), respectively. Moreover, \(I_{ds-tt}(t)\) is a latency factor related to the GCL state in DS-TT. \(\text{GCL}(t)=1\) means that the GCL is _OPEN_ for time-triggered queues in DS-TT at the \(t\)th TTI; otherwise, \(\text{GCL}(t)=0\). As shown in Equation (4), this indicates that \(I_{ds-tt}(t)\) only affects the utility value of time-triggered applications. Finally, in order to keep the utility function stable, a small positive constant \(\Delta\) is introduced.  
\[I_{ds-tt}(t)=\left\{\begin{array}{l}0,\forall i\in\mathbf{F}_{vi}\text{ or }\forall i\in\mathbf{F}_{tt}\text{ and }\text{GCL}(t)=1\\ 1,\forall i\in\mathbf{F}_{tt}\text{ and }\text{GCL}(t)=0\end{array}\right. \tag{4}\]  
As discussed in Equation (2), the video flow is concerned more with the average throughput, and the average throughput of any video flow \(j\) is defined as follows.  
\[\overline{R}_{j}=\frac{\sum\limits_{k=1}^{t}n_{j}^{k}\cdot m_{j}^{k}}{t\cdot T _{TTI}} \tag{5}\]  
This assumes that the scheduling is starting from the first TTI, and the period of each TTI is \(T_{TTI}\). \(n_{j}^{k}\) represents the allocated Resource Blocks (RBs) for video flow \(j\) at the \(k\)th TTI, and \(m_{j}^{k}\) is the bit volume per RB, which depends on the radio channel quality of flow \(j\) at the \(k\)th TTI.  
In each TTI, the available resources are finite, so the resources allocated to all active flows should not be larger than the available resources in gNB, which are described as  
\[\sum\limits_{i=1}^{N}p_{i}(t)\bullet n_{i}^{t}\leq\kappa_{t},i\in\mathbf{F} \tag{6}\]  
where \(\kappa_{t}\) represents the available RBs at the \(t\)th TTI.  
As discussed above, the object of 5G-TSN joint scheduling is to ensure end-to-end latency requirements for time-triggered flows as well as to improve the overall throughput of video applications in each TTI. The optimization problem is formulated as follows:  
\[\text{max}\sum\limits_{i=1}^{N}U_{i}(t) \tag{7}\]  
subject to Equations (1), (2) and (6).  
Based on Equation (3), it can be found that the problem described in Equation (7) is a non-convex function with integer and continuous variables, which is computationally complex. Therefore, a reinforcement learning model is proposed to achieve the optimal resource allocation policy following Equation (7).