## 3 DRL-Based Joint User Selection and Blocklength Allocation Design
### EMPPO Algorithm  
EMPPO comes from the standard PPO algorithm and its adapted version maskable PPO. The PPO algorithm was proposed by [PERSON] in 2017 [35] and has been widely adopted due to its simplicity, stability and efficiency. It is a policy gradient method, which provides an improvement to trust region policy optimization [36]. Then, considering scenarios where actions cannot always be selected, the maskable PPO has been proposed in 2022 [37]. By masking nonviable actions, it reduces ineffective exploration by agents, thereby enhancing the efficiency and performance of the algorithm. The framework of EMPPO is shown in Figure 4. Within the EMPPO framework, action masking is critical in addressing the reliability constraint of low-latency MEC networks. Two main enhancements reflect EMPPO's key advancements over the standard PPO. Firstly, EMPPO integrates a dynamic mask layer before the policy network's output to evaluate and filter potential actions based on current conditions and ensure compliance with error probability constraints arising from variable CSI and packet sizes. Secondly, during forward propagation, EMPPO incorporates an additional step where the mask is actively applied to refine the action probability distribution, ensuring that only feasible actions are considered in decision-making processes. In practice, EMPPO converts binary-encoded action vectors into one-hot encoded vectors, representing a unique mapping for all potential UE selections. Under the FBL regime, the mask function comes into play by selectively filtering actions according to real-time CSI. EMPPO uses a clipped surrogate objective function to achieve a balance between exploitation and exploration, which can be expressed as:  
\[L^{CLIP}(\theta)=\hat{\mathbb{E}}_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t },\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]} \tag{20}\]  
where \(\theta\) is the policy network parameter, \(r_{t}(\theta)\) is the probability radio of action, \(\hat{A}_{t}\) is the advantage estimate at timestep \(t\), and \(\epsilon\) is a hyperparameter. The probability radio \(r_{t}(\theta)\) can be denoted as:  
\[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{old}}(a_{t}\mid s _{t})} \tag{21}\]  
where \(\pi_{\theta}(a_{t}\mid s_{t})\) is the current policy, \(\pi_{\theta_{old}}(a_{t}\mid s_{t})\) is the old policy. If \(r_{t}(\theta)>1\), the agent is more likely to take action based on the current policy. If \(0<r_{t}(\theta)<1\), the agent is more likely to take action based on the old policy. The value network predicts the expected return from a given state, and the policy network outputs a probability distribution over actions given the current state. Generalized Advantage Estimation (GAE) is a method often used in PPO to reduce the variance of advantage estimates without greatly increasing bias, improving the stability of policy updates. The advantage function \(\hat{A}_{t}\) can be calculated as:  
\[\hat{A}_{t}=\delta_{t}+(\gamma\lambda)\delta_{t+1}+\cdots+(\gamma\lambda)^{T -t+1}\delta_{T-1} \tag{22}\]  
where \(r_{t}\) is reward, \(\gamma\) is the discount factor for future rewards and \(\lambda\) is GAE parameter. The temporal difference errors \(\delta_{t}\) can be calculated as:  
\[\delta_{t}=r_{t}+\gamma V_{\phi}(s_{t+1})-V_{\phi}(s_{t}) \tag{23}\]  
where \(V_{\phi}(s_{t+1})\) is the value of the subsequent state, \(V_{\phi}(s_{t})\) is the value of the current state. We employ an experience buffer for optimization. The agent's experience is stored as tuples \((s,a,r,s^{\prime})\) in an experience buffer and is then processed in epochs using mini-batches to improve policy parameters iteratively. Based on the discussion, we can deploy the EMPPO algorithm in low-latency MEC environments. The specific steps are shown in Algorithm 1. Figure 4: EMPPO framework.