# A Fuzzy-Logic-Based Load Balancing Scheme for a Satellite-Terrestrial Integrated Network
## 2 Materials and Methods
### The Fuzzy-Logic- and Reinforcement-Learning-Based Access Algorithm  
Traditionally, users access the cell with the best reference signal receiving power (RSRP) [30]. According to the link budget, this may not work well in the STIN studied in this paper. In order to reduce the occurrence of overloading, to reduce the frequency of calling offloading algorithms and ensure QoS, the intelligence of reinforcement learning is introduced to make access decisions. In the following, the proposed FLRL-AC is described. Reinforcement learning is a common method for intelligent decision. It obtains learning information and updates model parameters by calculating the rewards for actions in the current state of the environment. Reinforcement learning is divided into two categories: One is value learning, which uses a neural network to approximate the optimal action value function, such as Q-Learning or a deep Q-network. The other is policy learning, which uses a neural network to approximate the policy function, such as the actor-critic method. In this paper, reinforcement learning is used to select the initial access cell for each user so that the action dimensions are \(N^{M}\). Due to exponential expansion and large dimensions of action space, a deep deterministic policy gradient (DDPG) [31] which is compatible with a large dimension state and actions is utilized in the proposed algorithm. FEM is utilized in DDPG in order to further reduce the difficulty of training and enable the decision to have a better impact on the future state, which is called fuzzy deep deterministic policy gradient (FDDPG) in this paper. In the following, we will describe the [PERSON] decision process of the problem and the FDDPG training process. (1) State space: The sate space describes the environment. It reflects the relative positions, relative motions and channel states between cells and users. Thus, the state at time \(t\) is defined as  
\[s^{t}=\big{(}f^{t}(1,1),f^{t}(1,2),\cdots,f^{t}(M,N)\big{)}, \tag{8}\]  
where \(f^{t}(m,n)\) means the FEM between user \(m\) and cell \(n\) at time \(t\). (2) Action space: To make better access decisions, the index of cell is set as the action. In Equation (9), \(AC^{t}(m)\in[0,N]\) means the index of cell which user \(m\) accesses at time \(t\). \[a^{t}=\big{(}AC^{t}(1),AC^{t}(2),\cdots,AC^{t}(M)\big{)}. \tag{9}\]  
(3) Reward function: The proposed access decision method aims at maximizing the total reward of the access selections for all users. The discounted total reward is  
\[R^{t}=\sum_{T=0}^{\infty}\gamma^{T}\cdot r^{t+T}=\gamma^{T}\Big{(}a_{1}\cdot r _{1}^{t+T}+a_{2}\cdot r_{2}^{t+T}+a_{3}\cdot r_{3}^{t+T}\Big{)}, \tag{10}\]where \(\gamma\) is the reward discount and \(r^{t}\) is the per-time reward. In this paper, the per-time reward is composed of three parts, \(r_{1}^{t}\), \(r_{2}^{t}\) and \(r_{3}^{t}\). For the first part,  
\[r_{1}^{t}=\frac{\sum_{m=1}^{M}f^{t}\big{(}m,AC^{t}(m)\big{)}}{M}, \tag{11}\]  
the design goal is to reduce the overloading tendency in the future. Thus, \(r_{1}^{t}\) equals the average value of the FEM of all users. At meanwhile, for the second part,  
\[r_{2}^{t}=\sum_{n=1}^{N}O^{t}(n), \tag{12}\]  
we hope to keep load balanced at the current time by minimizing the number of overloaded cells. \(r_{2}^{t}\) equals the overload penalty for all cells, where  
\[O^{t}(n)= \begin{cases}1,\text{ }if\text{ }\eta_{n}^{t}\geq\Theta\\ 0,\text{ }otherwise\end{cases}  \tag{13}\]  
indicates whether cell \(n\) is overloaded at time \(t\). \(\eta_{n}^{t}=\frac{B_{n}^{t}}{B_{C}}\) is the radio resource utilization ratio (RRUR) of cell \(n\) at time \(t\), where \(B_{n}^{t}\) is the occupied portion of the bandwidth of cell \(n\) and \(B_{C}\) is the total bandwidth resources of the cell. For terrestrial cells, \(B_{C}\) is equal to \(B_{g}\). Additionally, for the satellite, \(B_{C}\) is equal to \(B_{s}\). \(\Theta\) is the overload threshold. For TN cells, the threshold is \(\Theta_{g}\). Additionally, for the NTN cell, the threshold is \(\Theta_{g}\).