## 1 Introduction
These algorithms result in a low application efficiency in space systems with various types of services and time-varying network states. Reinforcement learning (RL) is an intelligent method used to achieve the iterative optimization of a policy through a continuous interaction with the environment. This method ensures that the agent selects the best action that is suitable for task requirements under different network conditions. RL does not need to predefine a large number of rules, and it has strong adaptability. Typical RL algorithms include value-based Q-learning and SARSA [12; 13; 14], policy-based REINFORCE and Actor-Critic methods, and Proximal Policy Optimization (PPO), which is popular recently [15; 16; 17]. PPO introduces restrictive optimization when updating strategies to ensure stability and efficiency. It is an advanced algorithm in deep reinforcement learning, which is widely used in robot control, autonomous driving, intelligent recommendation systems and other fields [18; 19; 20; 21]. To this end, in this paper, we introduce the PPO algorithm in reinforcement learning into the framing process. We propose the advanced intelligent frame generation (AIFG) algorithm in Advanced Orbiting Systems for differentiated QoS requirements. The main contributions are as follows:* For massive heterogeneous services in AOS, we establish a service classification framework for delay-sensitive services and non-delay-sensitive services. Among them, delay-sensitive services include audio data, telemetry data, telecommand data, etc., while non-delay-sensitive services include video data, remote control data, observation data, experimental data, etc. Based on this, we design real-time framing algorithms for delay-sensitive services and efficient framing algorithms for non-delay-sensitive services, which are discussed in detail in Sections 2 and 5. * In order to effectively reduce the queuing delay of delay-sensitive services, we model the problem of calculating the optimal time threshold for framing such services as a Markov decision process. This model captures the sequential decision-making nature of the problem. Then, we employ the PPO algorithm to obtain the optimal solution based on performance indicators such as frame multiplexing efficiency, average framing time, and average packet delay. This modeling and optimization process is elaborated in Section 4. * We develop a joint simulation platform based on NS-3 and PyTorch, which enables a comprehensive performance evaluation of the proposed AIFG algorithm under realistic network conditions. The results show that compared with the traditional frame generation algorithm, the average queuing delay of the service under the AIFG algorithm is reduced by an average of 31.69%, the average multiplexing efficiency is increased by an average of 60.88%, and the average throughput is increased by an average of 31.92%. The AIFG algorithm accommodates different transmission needs of real-time and non-real-time services, enhancing the network QoS of the AOS. The detailed simulation setup and performance analysis are presented in Section 6. This article is organized as follows. In Section 2, the system model is established, including the data arrival model, service classification, and queuing structures for different service types. Section 3 describes the materials and methods of this paper. Then, Section 4 formulates the optimal frame generation time threshold calculation as a Markov decision process and presents a solution based on the Proximal Policy Optimization algorithm. Section 5 introduces the design of the advanced intelligent frame generation (AIFG) algorithm to address differentiated QoS requirements. Section 6 details the simulation setup, parameter settings, and performance evaluation of the proposed algorithm compared with baseline methods. The results of the simulation and theoretical analysis are given in Section 6. Finally, Section 7 concludes this article and highlights potential future research directions.