# Cooperative Downloading for LEO Satellite Networks: A DRL-Based Approach
## 3 Deep-Reinforcement-Learning-Based Cooperative Downloading Scheme
### Discretized SAC-Based Learning Algorithm  
To train the optimal cooperative downloading policy to maximize the amount of downloaded data, we adopt the SAC-based algorithm [34]. The SAC algorithm includes an entropy term in the training objective function that evaluates the policy and trains it to maximize not only the accumulated reward but also the entropy. We first define an actor network \(\pi_{\phi}(\cdot)\) with parameter \(\phi\), main critic networks \(Q_{\theta_{i}}\) with parameters \(\theta_{i}\), and target critic networks \(Q_{\theta_{i}}\) with parameters \(\hat{\theta}_{i}\) for \(i\in\{1,2\}\). The detailed procedure of the SAC-based algorithm is shown in Algorithm 1. First, the algorithm initializes the experience experience buffer \(D\), the weight \(\alpha\), and the parameters (i.e., \(\phi\), \(\theta_{i}\), and \(\hat{\theta}_{i}\) for \(i\in\{1,2\}\)) (see lines 1-5 in Algorithm 1). For each episode, the algorithm observes the initial state \(s_{t}\) (see line 7 in Algorithm 1). In addition, in each time slot \(t\), the algorithm generates an action \(a_{t}\) by using the actor network, i.e., \(a_{t}=\pi_{\phi}(s_{t})\) (see line 9 in Algorithm 1), and observes the next state \(s_{t+1}\) and the reward \(r_{t}\) after executing the action \(a_{t}\) (see lines 9-11 in Algorithm 1). As a result, the algorithm stores the experience \((s_{t},a_{t},r_{t},s_{t+1})\) at the experience buffer \(D\) (see line 12 in Algorithm 1). ```
1:Initialize the experience buffer \(D\)
2:Initialize the weight \(\alpha\) with 1
3:Initialize the actor network \(\pi_{\phi}\) with random parameter \(\phi\)
4:Initialize main critic networks \(Q_{\theta_{i}}\) with random parameters \(\theta_{i}\) for \(i\in\{1,2\}\)
5:Initialize target critic networks \(Q_{\theta_{i}}\) with parameters \(\hat{\theta}_{i}\) as main critic networks \(Q_{\theta_{i}}\)
6:for each training episode do
7: Observe initial state \(s_{t}\)
8:for each step \(t=1,2,...,T\)do
9: Generate the action \(a_{t}=\pi_{\phi}(s_{t})\)
10: Execute the action \(a_{t}\)
11: Observe the next state \(s_{t+1}\) and the reward \(r_{t}\)
12: Store the experience \((s_{t},a_{t},r_{t},s_{t+1})\) at the experience buffer \(D\)
13: Sample a mini-batch \(D\) of a few experiences from the buffer \(D\)
14: Calculate the target state value \(V_{\theta}(s^{\prime})\) based on Equation (9)
15: Update the main critic network \(Q_{\theta_{i}}\) based on the gradient \(\
abla_{\theta_{i}}J_{Q}(\theta_{i})\) in Equation (10)
16: Update the actor network \(\pi_{\phi}\) based on the gradient \(\
abla_{\phi}J_{\pi}(\phi)\) in Equation (12)
17: Update the weight \(\alpha\) based on the gradient \(\
abla_{\alpha}J(\alpha)\) in Equation (13)
18: For every \(B\) steps, use soft update for the target critic networks based on Equation (14)
19:endfor
20:endfor
```  
**Algorithm 1** Discretized soft-actor-critic algorithm. Now the algorithm enters into the parameter update phase. To this end, the algorithm randomly samples a mini-batch \(\hat{D}\) of a few experiences from the buffer \(D\) (see line 13 in Algorithm 1). Based on these sampled experiences, the algorithm calculates the target state value \(V_{\hat{\theta}}(s^{\prime})\) as  
\[V_{\hat{\theta}}(s^{\prime})=\mathbb{E}_{q^{\prime}\sim\pi_{\hat{ \varphi}}(s^{\prime})}\Big{[}Q_{\hat{\theta}}^{min}(s^{\prime},a^{\prime})- \alpha\log\left(\pi_{\hat{\varphi}}(s^{\prime})\right)\Big{]}, \tag{8}\]  
where \(s^{\prime}\) denotes the next state of a tuple \((s,a,r,s^{\prime})\in\hat{D}\), which is one of the mini-batch samples and \(a^{\prime}\) denotes an action obtained from \(s^{\prime}\) and the actor network \(\pi_{\hat{\varphi}}(\cdot)\). In addition, \(Q_{\hat{\theta}}^{min}(s^{\prime},\hat{a})\) represents the minimum value between \(Q_{\hat{\theta}_{1}}(s^{\prime},\hat{a})\) and \(Q_{\hat{\theta}_{2}}(s^{\prime},\hat{a})\). Note that the generated actions by the cooperative downloading policy should be the indices for GSs or adjacent LEOs receiving the data.