## 4 Optimal Frame Generation Time Threshold Calculation
### Overall Structure  
In the real-time frame generation (RTFG) module, under the condition that the system buffer capacity is fixed, if the frame generation waiting time threshold \(T_{th}\) is set too small, the average packet delay will decrease. However, this will require the addition of a large number of idle packets in the MPDU, leading to a reduction in the frame multiplexing efficiency. On the other hand, if \(T_{th}\) is set too large, the frame multiplexing efficiency will improve, but the average packet delay will increase, which will negatively impact real-time data with stringent delay requirements. Therefore, it is necessary to calculate the optimal frame generation time threshold \(T_{th}\).  
In this paper, the problem is modeled as a Markov decision process \(M=(S,A,P,R)\) and solved with the Proximal Policy Optimization algorithm. The overall structure of the optimal frame generation time threshold calculation model is shown in Figure 3. The model involves three main components: state space, action space, and reward.  
As illustrated in Figure 3, the proposed framework consists of three main interacting modules: the agent, the environment, and the reward computation module.  
* Agent: The agent module contains the actor network, critic network, and the weight parameter vector of the policy function. The agent implements a Proximal Policy Optimization algorithm with an actor-critic structure. The actor network generates actions \(a_{t}\), representing the selected frame generation threshold based on the current state \(s_{t}\), while the critic network evaluates the expected return of each action. These  
Figure 2: Research methodology flowchart.  
networks are trained using a clipped surrogate loss function, value function regression, and policy entropy regularization.
* Environment: The environment models the delay-sensitive queuing system and real-time frame generation process in the AOS scenario. It includes idle packet filling and transmission scheduling based on traffic load and queue conditions. At each time step, the environment produces a new state \(s_{t}\), capturing features such as packet arrival rate, queue length, frame generation time, and frame length.
* Reward: The reward module quantifies the QoS outcome of each action by combining parameters such as frame multiplexing efficiency and average delay into a composite reward signal \(r_{t}\). This reward is used to reinforce beneficial actions and suppress suboptimal ones, guiding the agent to converge toward optimal framing policies.  
Moreover, the buffer stores critical transition data collected during agent-environment interactions. Specifically, it logs tuples of \((s_{k},a_{k},\pi_{old}(a_{k}\mid s_{k}),r_{k})\), including the current state, selected action, action probability under the old policy, and corresponding reward. These stored experiences are used for batch policy updates, enabling efficient and stable training.  
These components operate in a closed feedback loop where the agent continuously observes the environment, takes action, receives rewards, stores transitions in a buffer, and updates its policy through backpropagation. The integrated design ensures adaptive, real-time optimization of framing behavior in delay-sensitive service transmission.  
Figure 3: Overall structure of the optimal frame generation time threshold calculation model.