# Deep Reinforcement Learning-Based Joint Scheduling of 5G and TSN in Industrial Networks
## 3 System Model and Problem Formulation  
### System Model  
Usually, there are several applications transmitted simultaneously in industrial networks, such as video monitoring, data collections from Programmable Logical Controller (PLC) and control information among PLCs or between PLCs to field devices. In our system model, the industrial network is composed of 5G and TSN, and applications are separated into two categories: time-triggered applications and video applications, which are denoted as \(\mathbf{F}_{tt}\) and \(\mathbf{F}_{vi}\), respectively. The active applications in the 5G system are denoted as \(i=\{1,2,3,\ldots,N\}\) (\(i\in\mathbf{F}\), \(\mathbf{F}\equiv\mathbf{F}_{tt}\cup\mathbf{F}_{vi}\)).  
Furthermore, 5G-TSN industrial networks should satisfy various QoS requirements for different applications, such as end-to-end maximum latency requirements (EMLR) for both time-triggered and video, and minimum data rate requirements for video. The QoS requirements for various applications are described as follows:  
\[d_{i}<d_{i}^{qos},i\in\mathbf{F} \tag{1}\]  
\[\overline{R}_{j}\geq R_{j}^{\min},j\in\mathbf{F}_{vi} \tag{2}\]  
where \(d_{i}\) is the queueing delay and transmission delay in a 5G system of the \(i\)th active flow, and \(d_{i}^{qos}\) is the latency requirement of the corresponding flow. \(\overline{R}_{j}\) denotes the average data rate achieved by the \(j\)th video flow, and \(R_{j}^{\min}\) denotes the minimal throughput requirement of the corresponding video flow.  
Figure 3 illustrates the procedures of 5G and TSN joint scheduling for downlink data transmission based on an artificial intelligence agent, where the agent is a logical entity in gNB to make dynamic resource scheduling decisions in each Time Transmission  
Interval (TTI); the agent can learn from the environment and make the appropriate \"action\". Detailed procedures are described as follows.  
Step 1: Environment information collection. At the beginning of each TTI, environmental information is collected by an agent. The environment information includes the radio channel quality of each User Equipment (UE), the GCL state of the time-triggered queue in DS-TT, and the queue state of each activate flow in gNB, such as queueing packet number and queueing delay of each flow. The agent is physically located in the gNB, and is responsible for environment information collection, so the queue state in gNB is delivered to the agent with an internal interface, but the channel quality and GCL state should be reported from the UE to the agent using radio signaling.  
Step 2: Decision-making and configuration. Packets of all applications are buffered for downlink scheduling in 5G gNB. Based on information collected from UEs, gNB and DS-TT, the agent makes decisions depending on the number of packets waiting for transmission, service priority, obtained average data rates, queueing time and deadline of the flow, and selects which flow can be scheduled and how many resources are allocated for packet delivery. After resource decision-making, the corresponding allocated resources are configured for packet delivery of each eligible flow. To avoid impacting the scheduling in the next TTI, resource decision-making, configuration and packet delivery with allocated resources should be implemented in a TTI.  
After the agent allocates the radio resources for the corresponding flow, the environment, including GCL state in DS-TT, queue state in gNB and channel state of the user terminal, will be changed, and the agent will learn all these new states and make the appropriate action in the next TTI, which means that step 1 and step 2 will be executed cyclically in each TTI.  
Furthermore, the GCL pre-defined in DS-TT is demonstrated in Figure 3; the packets of time-triggered flows are mapped into queue 7 in DS-TT, and the promised transmission duration for the time-triggered queue is \(1\) TTI, which means that the ON/OFF state is changed alternately in each TTI.  
Figure 3: System model of joint scheduling in 5G-TSN architecture.