## 3 DRL-Based Joint User Selection and Blocklength Allocation Design
### MDP Formulation  
UE selection problem (16a) can be considered as a sequential decision-making process. At any given frame, we can only access the present and historical states, and make decisions under the current state which will influence future states and decisions. Therefore, obtaining the optimal global solution for this problem is impossible. However, it can be modeled as a MDP and solved by reinforcement learning methods. Reinforcement Learning tasks are usually modeled by MDP. Each state of the Reinforcement Learning system must have the Markov property, meaning once the current system state is known, future states are solely dependent on it and not on past states. An MDP in Reinforcement Learning can be described with the tuple \((S,A,P,R,\gamma)\). \(S\) represents the state space, which is the set of all possible states of the environment. \(A\) is the action space representing the set of all possible actions the agent can take. \(P\) is the set of state transition probabilities when the agent performs an action in a given state. \(R\) is the reward function. \(\gamma\) is the discount factor that dictates the degree to which future rewards brought by actions are discounted when calculating cumulative rewards. The classical reinforcement learning process includes the following steps:  
1. At every time step \(t\), the agent observes the current environment state \(s_{t}\in S\). 2. The agent selects an action \(a_{t}\) according to the policy \(\pi(s_{t})\) based on the current environmental state. 3. The environment transitions to state \(s_{t+1}\) due to action \(a_{t}\) and gives a reward \(r_{t}\) to the agent. The learning process of reinforcement learning is an interactive process between the agent and the environment. It allows the agent to explore in an unfamiliar environment where different actions lead to different rewards. As the agent interacts continuously with the environment, it constantly improves its strategy based on changes in rewards and environmental states. After many trials and errors, it obtains the strategy with the most cumulative rewards. The agent continuously updates and iterates its strategy according to the rewards obtained from the actions taken in different environmental states in the ongoing interactive process, aiming to obtain as much cumulative reward as possible. In our low-latency MEC scenario, the definitions of state space, action space, and reward function are as follows. * State Space: The state space is composed of three components, which are the age and size of the oldest data packet in each UE's queue (the oldest data packet will be at the head of the queue), and the real-time SNR of the channels. Therefore, the state in frame \(i\) can be represented as: \[s_{t}=\{\Delta_{1,t},\Delta_{2,t},\cdots,\Delta_{K,t},m_{1,t},m_{2,t},\cdots,m_{K, t},\gamma_{1,t},\gamma_{2,t},\cdots,\gamma_{K,t}\}\] (17) where \([\Delta_{1,t},\Delta_{2,t},\cdots,\Delta_{K,t}]\) is the vector of length \(K\) containing the age of the oldest data packet in each UE, \([m_{1,t},m_{2,t},\cdots,m_{K,t}]\) is the vector of length \(K\) containing the size of the oldest data packet in each UE, and \([\gamma_{1,t},\gamma_{2,t},\cdots,\gamma_{K,t}]\) is the vector of length \(K\) containing the real-time SNR of the channels. * Action Space: From (15), we use a vector of binary decision variables to determine whether the K-th UE transmits in frame \(t\): \[a_{t}=\{x_{1,t},x_{2,t},\cdots,x_{K,t}\},x_{K,t}\in\{0,1\}\] (18) where \(x_{K,t}=1\) means UE K transmits a data packet to an edge server in frame \(t\), and \(x_{K,t}=0\) means UE K does not transmit a data packet in frame \(t\). Since each \(x\) can be either 0 or 1, there are a total of \(2^{n}\) possible actions for \(a_{t}\).