## 4 COIN-LEO Algorithm
### Deep Reinforcement Learning Using the PPO Algorithm  
Reinforcement learning (RL) is an important branch of machine learning, whose goal is to enable an agent to learn how to make decisions in a specific environment through interaction with the environment in order to maximize cumulative rewards. Reinforcement learning is a method of learning the optimal strategy through trial and error, similar to how humans try different actions during the learning process to achieve the best results. In reinforcement learning, the agent interacts with the environment, observes the current state and takes actions based on the current state. After taking an action, the agent receives an immediate reward or punishment to evaluate the goodness of that action. The goal of the agent is to learn the policy that maximizes the cumulative reward, which means selecting the optimal action in each state to achieve the best decision-making outcome.  
Deep reinforcement learning (DRL) is a specific subset of reinforcement learning (RL). It involves using deep neural networks as function approximators to learn complex policies or value functions from high-dimensional state spaces. The use of deep neural networks allows DRL to handle large and complex environments, making it suitable for tasks that involve raw sensory input.  
The key components of deep reinforcement learning include:  
1. Environment: The environment is the context in which the agent operates. It can be a simulated environment, a real-world scenario, or even a virtual game environment;
2. State: The state represents the current condition of the environment, which the agent observes to make decisions;
3. Action: The action is the decision made by the agent based on the observed state. It determines the agent's interaction with the environment;
4. Reward: The reward is a scalar value that provides feedback to the agent after taking an action. It serves as a measure of how well the agent is performing and guides its learning process;
5. Policy: The policy is the strategy or behavior that the agent uses to select actions based on the observed state;
6. Value Function: The value function estimates the expected long-term return or reward from a particular state. For the modeling of this problem, we provide the following symbol interpretations: S: The set of states in the reinforcement learning model. S : \(\{\overset{\rightarrow}{c},\overset{\rightarrow}{m},\overset{\rightarrow}{l},\overset{\rightarrow}{b},\text{p},\text{i}\}\); \(\overset{\rightarrow}{c}\): The remaining CPU resources of each satellite in the cluster; \(\overset{\rightarrow}{m}\): The remaining memory resources of each satellite in the cluster; \(\overset{\rightarrow}{l}\): The signal propagation delay of each inter-satellite link in the cluster; \(\overset{\rightarrow}{b}\): The remaining link rate of each inter-satellite link; p: The layer number before the last splitting point (the last action) in the neural network model; _i_: The step number (also the satellite number to place the model);A: The set of actions in the reinforcement learning model. A: \(\{\text{a}\in[\text{p}+1,\text{L}]\}\); a: Splitting the model after layer a; L: The total number of layers in the neural network model; \(dp^{i}_{p+1,a}\): Putting the layer from p + 1 to a on the i th satellite, the processing latency of the section; \(dt_{i,a}\): Latency of transmitting the intermediate results of layer a from the i th satellite to the next satellite; \(d^{stream}\): Time for the entire inference process; \(d^{stream}_{max}\): Configurable upper time limit for the reasoning process.