# Cooperative Downloading for LEO Satellite Networks: A DRL-Based Approach
## 3 Deep-Reinforcement-Learning-Based Cooperative Downloading Scheme
### Discretized SAC-Based Learning Algorithm
That is, the policy should be defined over a discrete action space, and, therefore, we adopt an SAC parameter update algorithm considering discrete action spaces [35]. Specifically, the output layers of neural networks are organized with \(|\mathrm{L}|\) FC layers in parallel so that each layer corresponds to each satellite, and the output size of each layer is set as \(|V|\) so that each output corresponds to the discretized action space. These network designs allow the actor and critic networks to map their states into the action probabilities and Q-values, respectively. Consequently, the expectation operation in (8) can be replaced with the direct output of the actor network without deriving any action probability density function. That is, the target state value in the discretized action spaces can be computed as (see line 14 in Algorithm 1)  
\[V_{\hat{\theta}}(s^{\prime})=\pi_{\hat{\varphi}}(s^{\prime})^{ \top}\Big{(}Q_{\hat{\theta}}^{min}(s^{\prime},\cdot)-\alpha\log\left(\pi_{ \hat{\varphi}}(s^{\prime})\right)\Big{)}. \tag{9}\]  
Note that the transformed critic networks \(Q_{\hat{\theta}}^{min}(s^{\prime},\cdot)\) do not require the action \(a^{\prime}\) as an input. This is because the critic networks generates soft \(Q\)-values over selectable actions directly. Based on the target state value, the algorithm updates the main critic networks (see line 15 in Algorithm 1) with a gradient  
\[\
abla_{\theta_{i}}J_{Q}(\theta_{i})=\
abla_{\theta_{i}}\mathbb{E}_{(s,a,r,s^ {\prime})\sim\hat{D}}\bigg{[}\frac{1}{2}\big{(}Q_{\theta_{i}}(s,a)-\big{(}r+ \psi\mathbb{E}_{s^{\prime}}\big{[}V_{\hat{\theta}}(s^{\prime})\big{]}\big{)} \big{)}^{2}\bigg{]}. \tag{10}\]  
Furthermore, the algorithm updates the actor network with a gradient  
\[\
abla_{\phi}\hat{J}_{\pi}(\phi)\approx\
abla_{\phi}\mathbb{E}_{s\sim D}[ \mathbb{E}_{\mathbb{I}\sim\pi_{\hat{\varphi}}(s)}[\alpha\log(\pi_{\hat{\varphi }}(s))-Q_{\hat{\theta}}^{min}(s,\hat{a})]]. \tag{11}\]  
By the same discretization, the inner expectation of (11) can also be calculated directly, and, thus, (11) can be rearranged as  
\[\
abla_{\phi}\hat{J}_{\pi}(\phi)\approx\
abla_{\phi}\mathbb{E}_{s\sim\hat{D}} [\pi_{\hat{\varphi}}(s)^{\top}(\alpha\log(\pi_{\hat{\varphi}}(s))-Q_{\hat{ \theta}}(s,\cdot))]. \tag{12}\]  
Then, the algorithm updates the actor network accordingly (see line 16 in Algorithm 1). Additionally, the weight \(\alpha\) is automatically adjusted as the training progresses (see line 17 in Algorithm 1), with a gradient  
\[\
abla_{\alpha}J(\alpha)=\
abla_{\alpha}\pi_{\hat{\varphi}}(s)^{ \top}\big{(}-\alpha\big{(}\log\left(\pi_{\hat{\varphi}}(s)\right)+\hat{H} \big{)}\big{)}, \tag{13}\]  
where \(\hat{H}\) denotes the target entropy. Finally, the algorithm updates the target critic networks with the main critic networks by using an exponentially moving average (i.e., soft update) for every \(B\) steps (see line 18 in Algorithm 1) as  
\[\hat{\theta}_{i}=\tau\cdot\theta_{i}+(1-\tau)\cdot\hat{\theta}_{i},i\in\{0,1\}, \tag{14}\]  
where \(\tau\) denotes a coefficient for the soft update.