## 4 Problem Equation
### A3C Algorithm
```  Initialize thread step counter \(t\gets 1\) repeat  Reset gradients:\(d\Theta\gets 0\) and \(d\Theta\gets 0\)\(d\Theta_{c}\gets 0\)  Synchronize thread-specific parameters \(\Theta^{\prime}=\Theta\) and \(\Theta_{c}{}^{\prime}=\Theta_{c}\) \(t_{start}=t\)  Get state \(S_{t}\) repeat  Perform \(a_{u}(t)\) according to policy \(\iota(a_{u}(t)|S_{t};\Theta^



(4) After the number of iterations is reached, the parameters \(\Theta^{\prime}\) and \(\Theta_{c}{}^{\prime}\) in each thread are used to update the network parameters \(\Theta\) and \(\Theta_{c}\) of the whole Actor and Critic parts. ## 5 Simulation Analysis

### Simulation Parameter Setting

In the experiment, the hardware environment was an Intel Core i7-8750 CPU, with 8 GB of memory and 1 TB of hard disk space. The software environment was Python3.6.1 with Tensorflow1.4.0, MATLAB R2014a [39]. We assumed that there were three GEO data relay satellites, five LEO communication satellites, seven MEC servers, and seven caches. The altitudes of the five LEO satellites were 500 km, 780 km, 1000 km, 1200 km, and 1400 km. The elevation angle between user \(u\) and LEO satellite \(l\) conforms to Markov chain model. Assuming that the elevation angle is excellent, \(w_{u}^{l}=10\), better, \(w_{u}^{l}=8\), medium elevation, \(w_{u}^{l}=6\), lower elevation, \(w_{u}^{l}=4\), and extremely bad, \(w_{u}^{l}=2\). We assume that the elevation state transition probability matrix is

\[\kappa=\left[ \begin{array}{cccc}0.4&0.1&0.2&0.2&0.1\\ 0.1&0.4&0.1&0.2&0.2\\ 0.2&0.1&0.4&0.1&0.2\\ 0.2&0.2&0.1&0.4&0.1\\ 0.1&0.2&0.2&0.1&0.4\end{array} \right]. \tag{31}\]

Similarly, when the communication efficiency between user \(u\) and satellite \(l\) is very excellent, the spectrum utilization ratio is \(v_{u}^{l}(t)=10\), better, \(v_{u}^{l}(t)=8\), medium condition, \(v_{u}^{l}(t)=5\), lower condition, \(v_{u}^{l}(t)=1\), and extremely bad, \(v_{u}^{l}(t)=0.2\). Its state transition probability matrix is

\[l=\left[ \begin{array}{cccc}0.5&0.1&0.05&0.15&0.3\\ 0.3&0.5&0.1&0.05&0.15\\ 0.15&0.3&0.5&0.1&0.05\\ 0.05&0.15&0.3&0.5&0.1\\ 0.1&0.05&0.15&0.3&0.5\end{array} \right]. \tag{32}\]

Assuming that there is a space task, whether it needs a GEO relay satellite transmission conforms to a Markov chain model, and its state transition probability matrix is

\[\diamond=\left[ \begin{array}{cccc}0.3&0.7\\ 0.7&0.3\end{array} \right]. \tag{33}\]

The cache state of the space task conforms to the Markov chain model, and its state transition probability matrix is

\[\Phi=\left[ \begin{array}{cccc}0.6&0.4\\ 0.4&0.6\end{array} \right]. \tag{34}\]

For MEC servers, when the computing state is excellent, the computing rate is \(\Xi_{u}^{m}(t)=50\), better, \(\Xi_{u}^{m}(t)=30\), medium condition, \(\Xi_{u}^{m}(t)=10\), lower condition, \(\Xi_{u}^{m}(t)=3\), and extremely bad, \(\Xi_{u}^{m}(t)=0.5\). Its state transition probability matrix is

\[E=\left[ \begin{array}{cccc}0.5&0.15&0.05&0.25&0.05\\ 0.05&0.5&0.15&0.05&0.25\\ 0.25&0.05&0.5&0.15&0.05\\ 0.05&0.25&0.05&0.15\\ 0.15&0.05&0.25&0.05&0.5\end{array} \right]. \tag{35}\]The remaining parameters in the simulation are shown in Table 2.

In this experiment, we simulated the expected benefits of unit resources in the following six situations as follows:

(1) Unified consideration of LEO satellite elevation state, communication link state, GEO data relay satellite transmission state, caching state, and computing state, expressed as A3C-based all scheme. (2) Unified consideration of GEO data relay satellite transmission state, caching state, and computing state, regardless of LEO satellite elevation state and communication link state, expressed as A3C-based without coverage communication scheme. (3) Unified consideration of LEO satellite elevation state, communication link state, caching status, and computing state, regardless of GEO data relay satellite transmission state, expressed as A3C-based without GEO communication.