# Federated Reinforcement Learning Based AANs with LEO Satellites and UAVs
## 3 System Design
### Environment Configuration
#### 3.4.2 State  
In order for an agent to perform an optimal action suitable for the situation, the state should include appropriate information. We designed the state in the environment to be composed of 4 value sets, and each set has 4 values. Algorithm 1 shows the pseudocode for getting the state, and Table 1 lists the variables used in Algorithm 1. Figure 2: The overall operations of FRL in the proposed system. ```
1:for each UAV \(u\) in U do
2:if\(u\) is not the chosen drone then
3:if u.x \(<\) x then N[0] \(+=1\)
4:if u.x \(>\) x then N[1] \(+=1\)
5:if u.y \(<\) y then N[2] \(+=1\)
6:if u.y \(>\) y then N[3] \(+=1\)
7:endif
8:endfor
9:T\({}_{o}\)[0] = sum(sum(M\({}_{o}\)[0:x+1,0:v\)])
10:T\({}_{o}\)[1] = sum(sum(M\({}_{o}\)[x:h,0:v]))
11:T\({}_{o}\)[2] = sum(sum(M\({}_{o}\)[0:h,0:y+1]))
12:T\({}_{o}\)[3] = sum(M\({}_{o}\)[0:h,y:v]))
13:temp = sum(T\({}_{o}\))
14:for i in range(0,4) do
15:S[i] = T\({}_{o}\)[i] / temp
16:endfor
17:for i in range(4,8) do
18:S[i] = T\({}_{o}\)[i-4] / N[i]
19:endfor
20:temp = sum(S[4:8])
21:for i in range(4,8) do
22:S[i] = S[i] / temp
23:endfor
24:T\({}_{r}\)[0] = sum(sum(M\({}_{r}\)[0:x+1,0:v]))
25:T\({}_{r}\)[1] = sum(sum(M\({}_{r}\)[x:h,0:v]))
26:T\({}_{r}\)[2] = sum(sum(M\({}_{r}\)[0:h,0:y+1]))
27:T\({}_{r}\)[3] = sum(M\({}_{r}\)[0:h,y:v]))
28:temp = sum(T\({}_{r}\))
29:for i in range(8,12) do
30:S[i] = T\({}_{r}\)[i-8] / temp
31:endfor
32:for i in range(12,16) do
33:S[i] = T\({}_{r}\)[i-12] / N[i]
34:endfor
35:temp = sum(S[12:16])
36:for i in range(12,16) do
37:S[i] = S[i] / temp
38:endfor
39:return S
```  
**Algorithm 1** Algorithm for getting the state  
Figure 3: An example of network traffic map. In the scenario, it is reasonable for the UAV to provide a network service located on an area with a lot of traffic. Therefore, we designed the first value set to include traffic information, and the first value set in the state is composed of traffic sum values on the agent's left, right, bottom, or top side. For example, in Figure 3, the traffic sum value on the left side of UAV 2 is the sum of traffic in the regions indicated by the numbers 2 and 3 with an underline. Lines 9 to 16 in Algorithm 1 are relevant to these operations. Lines 13 to 16 are required to normalize the traffic values for better learning. As explained in the above, it is reasonable for UAVs to move to locations with a lot of traffic. However, it is inappropriate for all the UAVs to flock to one location because they cannot provide services to a wide area. Therefore, in addition to the traffic information, relative locations of neighboring UAVs should also be considered. The second set is composed of the values for taking into account the traffic and the positions of surrounding UAVs together. The fifth to eighth values of the state are the traffic sum value divided by the number of neighboring UAVs on the left, right, bottom, or top of the UAV, respectively. For example, the first value of the second set is the value obtained by dividing the sum of traffic in the regions by the number of neighboring UAVs on the left side of the UAV. Lines 1 to 8 and 17 to 23 in Algorithm 1 are relevant to these operations. Until now, we considered traffic on the ground without taking account of the network services provided by UAVs. Unlike Figure 3, Figure 4 shows an example of UAVs providing data communications to the ground in the assumed scenario. As shown in the figure, each UAV provides a network service for traffic on the ground around one's location. The white part in the figure means an area where there is no traffic left because of UAVs providing network services. The yellow lines indicate communications between the UAVs and satellites, and the number on the lines shows the amount of transmission. The most ideal result is that UAVs are well placed to minimize the sum of the demanded traffic remaining in the traffic map. Thus, unlike the sets 1 and 2, we designed the sets 3 and 4 to contain information about the remaining traffic.