# A Green Computing Business Aggregation Strategy for Low Earth Orbit Satellite Networks
## 4 Problem Solution
### Aggregating Businesses
#### 4.2.2 Double DQN  
The double DQN approach is designed to reduce the overestimation of Q-values by decoupling the action selection from the action evaluation. The implementation details are as follows:  
Utilize Double DQN to learn the optimal business aggregation policy that maximizes cumulative rewards. Double DQN employs two Q-networks to mitigate the overestimation of Q-values:  
* Online network (\(Q_{o}(s_{t},a_{t};\theta)\)): The primary network used for selecting actions.
* Target network (\(Q_{T}(s_{t},a_{t};\theta^{-})\)): A secondary network with parameters \(\theta^{-}\) that are periodically updated to stabilize training.
* Action selection using \(\varepsilon\)-greedy strategy: Based on the \(\varepsilon\)-greedy policy, an action \(a_{v}\) is selected in a given state \(s_{t}\) as per the following: \[a_{v}=\begin{cases}\text{random action}&\text{if random}<\varepsilon\\ \arg\max_{a}Q_{o}(s_{t},a;\theta)&\text{otherwise}\end{cases}\] (23) where \(\varepsilon\) is a probability between 0 and 1 representing the likelihood of choosing a random action for exploration, and \(Q_{o}(s_{t},a;\theta)\) is the action-value function of the online network.  
* Calculating transition probabilities: The probability of transferring business \(\mathbb{B}_{uj}\) to satellite \(S_{v}\) is computed as: \[P(B_{v}+B_{uj}\mid B_{v},a_{v})=\begin{cases}\frac{B_{v}^{\max}-B_{uj}-B_{v}}{B_{ v}^{\max}}&\text{if }B_{uj}+B_{v}<B_{v}^{\max}\\ 0&\text{otherwise}\end{cases}\] (24) where \(B_{v}^{\max}\) is the maximum business volume that satellite \(S_{v}\) can handle, and \(B_{v}\) represents the current business volume of satellite \(S_{v}\).
* Reward calculation: The reward for taking action \(a_{v}\) in state \(B_{v}\) is calculated as: \[R(B_{v},a_{v})=\frac{\tau_{ujv}\cdot P_{\text{C}}^{\text{idle}}-\tau_{ujv} \cdot(P_{T}^{a}+P_{R}^{a})}{\tau_{ujv}+\tau_{ujv}^{c}-\tau_{uj}^{c}}\] (25) where \(\tau_{ujv}\) is the energy difference associated with the business transfer; \(P_{C}^{\text{idle}}\) is the idle power consumption; \(P_{T}^{a}\) and \(P_{R}^{a}\) are the transmission and reception power consumptions, respectively; and \(\tau_{ujv}^{c}\) and \(\tau_{uj}^{c}\) represent delay components.
* Updating the Q-values using the Bellman equation: We update the Q-value of the online network using the Bellman equation as shown: \[Q_{o}(s_{t},a_{v};\theta)\gets Q_{p}(s_{t},a_{v};\theta)+\alpha\Big{[}R(B_ {v},a_{v})+\gamma\max_{a^{\prime}}Q_{T}(s_{t+1},a^{\prime};\theta^{-})-Q_{p}(s _{t},a_{v};\theta)\Big{]}\] (26) where \(Q_{p}(s_{t},a_{v};\theta)\) is the previous Q-value before the update, \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and \(\theta^{-}\) are the parameters of the target network.
* Convergence check: Perform a convergence check to determine if the Q-values have stabilized: \[\Delta Q=\big{|}Q_{o}(S_{t},a_{v};\theta)-Q_{p}(S_{t},a_{v};\theta)\big{|}<\delta\] (27) where \(\delta\) is a small threshold value. If \(\Delta Q\) remains below \(\delta\) for multiple consecutive updates, the Q-values are considered to have converged, indicating that the optimal business aggregation strategy for \(\mathbb{B}_{uj}\) has been found.  
Repeat the above steps to make aggregation decisions for all businesses that need to be transferred, and form the final business-working satellite mapping.