# A Fuzzy-Logic-Based Load Balancing Scheme for a Satellite-Terrestrial Integrated Network
## 2 Materials and Methods
### The Fuzzy-Logic- and Reinforcement-Learning-Based Access Algorithm
In order to minimize the resource utilization ratio and balance the resource utilization, the third part of the reward is defined as  
\[r_{3}^{t}=\sum_{n=1}^{N-1}\eta_{n}^{t}+w_{s}\cdot\eta_{N}^{t}+\frac{\sum_{n=1} ^{N}\left(\eta_{n}^{t}-\frac{\sum_{n=1}^{N}\eta_{n}^{t}}{N}\right)^{2}}{N}, \tag{14}\]  
where the weighted value of the sum and the variance of RRUR is included in \(r_{3}^{t}\). Due to the large bandwidth, the NTN resources are enough to load many users. If there is no additional limit on accessing NTN, the agent will tend to let as many users as possible access the NTN to avoid overloading in TN cells. In order to prevent the transmission delay of users from being affected, we use the additional weight \(w_{s}\) to increase the impact of resource utilization of the NTN cell. (4) The training process of the FDDPG algorithm: As in DDPG, FDDPG has two components, actor and critic. The actor network defined as \(\mu\left(s^{t}\right)\) takes \(s^{t}\) as input and returns action \(a^{t}\). The critic network defined as \(Q\left(s^{t},a^{t}\right)\) returns long-term reward based on states and actions. \(Q\left(s^{t},a^{t}\right)\) can be expressed as  
\[Q\left(s^{t},a^{t}\right)=E[r^{t}|s^{t},a^{t}]\approx E[r^{t}+\gamma Q\left(s ^{t+1},\mu\left(s^{t+1}\right)\right)] \tag{15}\]  
according to the Bellman equation, where \(E[\cdot]\) means expectation and \(\gamma\) means the reward discount. DDPG combines actor-critic and DQN, so there are four networks in total. The actor network \(\mu\left(s^{t}\right)\) and actor target network \(\mu^{\prime}\left(s^{t}\right)\) have the same structure, but different parameters \(\theta^{\mu}\) and \(\theta^{\mu^{\prime}}\) and different update frequencies. The critic network \(Q\left(s^{t},a^{t}\right)\) and critic target network \(Q^{\prime}\left(s^{t},a^{t}\right)\) have the same structure, but different parameters \(\theta^{Q}\) and \(\theta^{Q^{\prime}}\) and different update frequencies. For the activation function, the linear rectification function (ReLU) is utilized in hidden layers and the hyperbolic tangent function is utilized in output layers. Figure 3 shows the training structure of the proposed fuzzy reinforcement learning. In each episode during the training process, users' positions and velocities are randomly reset to reset the environment. In each training step, the three fuzzy inputs are obtained from the environment and the state \(s^{t}\) is obtained based on Equation (8). In order to speed up training, \(s^{t}\) is normalized to get  
\[ \begin{split}& s^{t}=\big{(}s^{t}(m,n)-mod\big{(}s^{t}(m,n), \beta\big{)}\big{)}\end{split}  \tag{16}\]  
with the normalization coefficient \(\beta\), where \(m\in[1,M],n\in[1,N]\). To explore new states, the output of the actor network is added by random noise. The action \(a^{t}\) is obtained by  
\[a^{t}=\mu\big{(}s^{t}|\theta^{u}\big{)}+N(0,var), \tag{17}\]  
in which the noise is a Gaussian random number with mean value of 0 and variance of \(var\). After performing \(a^{t}\) on the environment, \(r^{t}\) and the next state \(s^{t+1}\) can be obtained from the output of the environment. To break the association between data, \(\big{(}s^{t},a^{t},r^{t},s^{t+1}\big{)}\) is stored in a replay buffer. \(\theta^{u}\), \(\theta^{u^{\prime}}\), \(\theta^{Q}\) and \(\theta^{Q^{\prime}}\) are updated by sampling a mini-batch with size \(K\) from the replay buffer. The loss function of the critic network is defined as  
\[ \begin{split}& L\Big{(}\theta^{Q}\Big{)}=E_{\
u^{t}}\big{[} \Big{(}y^{t}-Q\Big{(}s^{t},a^{t}|\theta^{Q}\Big{)}\Big{)}^{2}\big{]},\\ & y^{t}=r^{t}+\gamma Q^{\prime}\Big{(}s^{t+1},\mu^{t}\Big{(}s^{t+ 1}\Big{)}\big{|}\theta^{Q^{\prime}}\Big{)},\end{split}  \tag{18}\]  
which is the temporal difference error between the outputs of \(\theta^{Q}\) and \(\theta^{Q^{\prime}}\).