# Joint Optimization of Relay Communication Rates in Clustered Drones under Interference Conditions
## 4 Intelligent Channel Selection Scheme
### Discrete Soft Actor-Critic Algorithm  
The DSAC algorithm requires discussion primarily around the agent and the interfering environment. The agent mainly consists of the following three components: an action selection network, a value evaluation network, and an entropy structure. The Soft Actor-Critic algorithm adds an entropy component to the previous actor-critic network architecture to enhance the algorithm's ability to explore global solutions and prevent convergence to local optima. Entropy is a measure of uncertainty, with higher randomness corresponding to greater entropy values. In the DSAC algorithm, the presence of entropy shifts the focus of the evaluation network from determining the cumulative rewards of actions for each state to exploring actions that could yield the maximum cumulative rewards in a given state. This adjustment addresses the shortcomings of traditional AC architectures in deep reinforcement learning. The working framework of this architecture is illustrated in Figure 3. For ease of expression, this paper denotes the soft predictive value network with \(Q^{soft}\) as \(Q\). From the diagram, it can be observed that the Discrete Soft Actor-Critic algorithm architecture consists of one policy network and four evaluation networks \(Q(\cdot)\). The evaluation networks are divided into two sets, as follows: predictive value networks and their corresponding target value networks. The configuration within each set aims to prevent overestimation and ensure the stability of the learning process, while the arrangement among the sets is designed to enhance learning efficiency and further improve stability. In terms of the evaluation network, unlike algorithms for continuous action spaces, in discrete action spaces, the state is solely input into the evaluation network, which then outputs values for all possible actions corresponding to that state, denoted as \(S\rightarrow\mathbb{R}^{|A|}\). The expression for the loss function is \(L_{Q}(\omega^{u})\). \[L_{Q}(\omega^{u})=\mathop{\mathbb{E}}_{(s,a,r,s^{\prime})\sim D _{I}}\left[\left(Q(s_{t},a_{t};\omega^{u})-\left(\left[\underset{\omega^{u}_{ t}}{min}\big{\{}Q(s^{\prime};\omega^{u}_{t})\big{\}}-\alpha\ln(\pi(s^{\prime}; \theta^{u}))\big{]}\right)\right)^{2}\right] \tag{41}\]  
Figure 3: Schematic diagram of the Discrete Soft Actor-Critic algorithm architecture. In this setup, \(r_{t}(s_{t},a_{t})\) represents the expected reward value when action \(a_{t}\) is executed in state \(s_{t}\). \(\alpha\) denotes the entropy coefficient, which signifies the weight of the entropy, \(H(\pi(s^{\prime};\theta^{u}))=-\alpha\pi(s^{\prime};\theta^{u})^{T}\ln(\pi(s^{ \prime};\theta^{u}))\) represents the entropy itself, and \(D_{J}\) indicates the experience replay buffer used to store interaction data between the agent and the environment. Regarding the policy network, its input remains the state, but its output is the probability distribution of discrete actions, unlike the output of mean and variance in continuous action spaces, denoted as \(S\rightarrow[0,1]^{|A|}\). The expression for the loss function of the policy network is \(L_{\pi}(\theta^{u})\). \[L_{\pi}(\theta^{u})=\mathop{\mathbb{E}}_{s\sim D_{J}}\left[\pi(s;\theta^{u})^ {T}\left[\alpha\ln\pi(s;\theta^{u})-\underset{\omega_{t}^{u}}{min}Q(s;\omega_ {i}^{u})\right]\right] \tag{42}\]  
Regarding entropy, the entropy coefficient, \(\alpha\), determines the importance of the entropy term. To enable the adaptive variation in the entropy coefficient at different stages of learning, its loss function is designed as \(L(\alpha)\), as follows:  
\[L(\alpha)=\mathop{\mathbb{E}}_{s\sim D_{J}}\left[\pi(s;\theta^{u})^{T}\ln\pi( s;\theta^{u})-\widetilde{H}\right] \tag{43}\]  
In this context, \(\widetilde{H}\) represents the target entropy. The application of gradient descent to these loss functions facilitates the updating of network parameters.