## 4 Problem Equation
### A3C Algorithm  
In this paper, we need to consider the coverage of the LEO satellite, transmission status of the GEO data relay satellite, communication link status, cache status, and computing power of the MEC server. Moreover, the SIN is a dynamic network system which is constantly changing. Therefore, this paper adopts the A3C algorithm in the deep reinforcement learning algorithm. The A3C algorithm is a deep reinforcement learning algorithm which combines a use value function and a strategy gradient. The actor part can dynamically change the strategy according to the learned value function. The critic part estimates the current state (action) value function and evaluates the actor's strategy [38]. The basic framework of the A3C algorithm based on the SIN is shown in Figure 5.  
In the A3C algorithm, first of all, we define the learning strategy as \(\iota\). The value function \(V^{t}(s)\) and action value function \(Q^{t}(s,a)\) are used to judge the learning strategy. The value function \(V^{t}(s)\) of the current initial state \(s\) is defined as  
\[V^{t}(s)=E^{t}[\sum_{k=0}^{\infty}{{}^{k}R_{u}(t+k+1)_{i}}|S_{t}=s ], \tag{27}\]  
where \(E^{t}[*]\) represents mathematical expectations under certain state transition probabilities and learning strategies, \(R_{u}(t)\) represents the reward function, and is the discount factor, \(\in[0,1]\). is used to measure the role of reward function in value function. The farther it is away from the current state, the smaller the value of will be. Each strategy represents a mapping from state to action space, i.e., \(a=\iota(s)\). The action value function \(Q^{t}(s,a)\) is defined as  
\[Q^{t}(s,a)=E^{t}[\sum_{k=0}^{\infty}{{}^{k}R_{u}(t+k+1)}|S_{t}=s,a_{u}(t)=a]. \tag{28}\]  
Actor networks can be divided into three parts. Assuming that the network parameter of the Actor part is \(\Theta\), the following results are obtained:  
(1) Revenue function: \(J(\Theta)=V^{\iota_{\Theta}}(s)=E_{\iota_{\Theta}}[V]\);  
(2) Derivation of strategy function: \(\
abla_{\Theta}\iota_{\Theta}(s,a)=\iota_{\Theta}(s,a)\
abla_{\Theta}\log \iota_{\Theta}(s,a)\);  
(3) Renewal of income gradient through gradient: \(\
abla_{\Theta}J(\Theta)=E_{\iota_{\Theta}}(s,a)[\
abla_{\Theta}\log\iota_{ \Theta}(s,a)V^{\iota_{\Theta}}(s)]\). For the Critic part, set the network parameter as \(\Theta_{c}\). When the Actor network and the Critic network are finally determined, \(V_{\Theta_{c}}(s)\approx V^{\iota_{\Theta}}(s)\). The optimal strategy obtained through Actor and Critic networks is the same. Therefore, the gradients of the two should be equal, i.e., \(\
abla_{\Theta_{c}}V_{\Theta_{c}}(s)=\
abla_{\Theta}\log\iota_{\Theta}(s,a)\). Figure 5: Framework of the Asynchronous Advantage Actor-Critic (A3C) algorithm based on the SIN. After the above deduction, we define the loss function as \(\varepsilon=E_{i}[\left(V^{i\omega}(s)-V_{\Theta_{c}}(s)\right)^{2}]\). When the loss function is minimized, its minimum value is obtained when the derivative is 0. It can be concluded that \(\
abla_{\Theta_{c}}\varepsilon=0\). Further derivation shows that  
\[ \begin{array}{l}E_{i}[\left(V^{i\omega}(s)-V_{\Theta_{c}}(s)\right)\
abla_{ \Theta_{c}}V_{\Theta_{c}}(s)]=0\\ E_{i}[\left(V^{i\omega}(s)-V_{\Theta_{c}}(s)\right)\
abla_{\Theta}\log\iota_{ \Theta}(s,a)]=0\\ E_{i}[V^{i\omega}(s)\
abla_{\Theta}\log\iota_{\Theta}(s,a)]=E_{i}[V_{\Theta_{ c}}(s)\
abla_{\Theta}\log\iota_{\Theta}(s,a)]\end{array}  \tag{29}\]  
Therefore, the gradient of the income function \(J(\Theta)\) is  
\[\
abla_{\Theta}J(\Theta)=E_{i\omega}[\
abla_{\Theta}\log\iota_{\Theta}(s,a)V_{ \Theta_{c}}(s)]. \tag{30}\]  
It is known that the network parameter of Actor part is \(\Theta\) and that of the Critic part is \(\Theta_{c}\). Since there are multiple threads in the A3C algorithm, we have two parameters in the thread: \(\Theta^{\prime}\) and \(\Theta_{c}{}^{\prime}\). Set the global counter \(T=0\); thus, each thread has its own counter \(t\). The flow chart of the A3C algorithm is shown below.