## 3 FGRL for Mapless Navigation
### Federated Growing Reinforcement Learning (FGRL)  
As mentioned before, in this work, we borrow the idea of FL to speed up the learning efficiency, and we propose the FGRL approach that will evaluate each agent's performance to calculate the weights of local models. Figure 3 illustrates the model aggregation of the proposed algorithm. Suppose that \(K\) agents can be trained in parallel and that a global model exists among all the agents. In the beginning, the global model parameters are sent to each agent to initialize each agent's local model. Here, the parameters specifically configure the actor and critic networks. Then, each agent makes decisions based on its local model and starts to collect new experiences. At the same time, each agent can also update its local model based on new experiences. We define a global update period (also called an aggregation round) with \(E\) episodes, and we also define two indicators for performance evaluation of \(E\) episodes, i.e., the success rate \(\zeta=\frac{n_{\text{steps}}}{\zeta}\), and the average steps to the desired location during the successful episodes \(\varphi=\frac{n_{\text{steps}}}{n_{\text{success}}}\). Here, \(n_{\text{success}}\) denotes the number of successful episodes in \(E\), and \(n_{\text{steps}}\) indicates the moving steps of each successful episode. Thus, in this work, we adopt \(\zeta\) and \(\varphi\) to evaluate the performance of each agent during its individual learning episodes. We use \(m\) to indicate an aggregation round, where not all agents can perform better than the previous round. Thus, we define a growth value \(g_{m}\) to quantify the performance of each agent in the \(m\)-th aggregation round as follows:  
\[g_{m}=\psi\frac{\Delta\zeta_{m}}{\zeta_{m}}+(\psi-1)\frac{\Delta\varphi_{m}}{ \varphi_{m-1}}, \tag{10}\]  
where \(\Delta\zeta_{m}=\zeta_{m}-\zeta_{m-1}\), \(\zeta_{m}\) and \(\zeta_{m-1}\) represent the success rates of the current \(m\)-th and the previous aggregation round, while \(\Delta\varphi_{m}=\varphi_{m}-\varphi_{m-1}\), \(\varphi_{m}\) and \(\varphi_{m-1}\) indicate the average steps of the current \(m\)-th and the previous aggregation round. \(\psi\in(0,1)\) is a normal constant. Thus, we can use the value of \(g_{m}\) to express how much an agent has grown in this \(m\)-th aggregation round compared to the previous round. In order to reduce the fluctuation of the growth value for model aggregation, we use \(\eta_{m}\) to replace \(g_{m}\) as follows:  
\[\eta_{m}=\frac{1}{1+e^{-g_{m}}}. \tag{11}\]  
We also set a threshold \(\Omega\) for the growth value \(\eta_{m}\) to measure whether the agent grows enough or degrades in an aggregation round. If an agent's performance of the current \(m\)-th aggregation round does not exceed the threshold, its local model will not participate in the model aggregation. To update the global model, we compare two consecutive rounds. For the first update round, we assume that all the agents have the same growth value, \(\eta_{1}=1\), and we can average the local models to update the parameters of the global model. After the first update, we will calculate their respective growth values, using Equations (10) and (11), to weight the contribution of each agent:  
\[w_{m}^{\text{global}}=\sum_{k=1}^{K^{\prime}}\frac{\eta_{m}^{k}}{\sum_{k=1}^{K^{ \prime}}\eta_{m}^{k}}w_{m}^{k}, \tag{12}\]  
where \(w_{m}^{\text{global}}\) denotes the global network parameter, and \(w_{m}^{k}\) represents the local model of \(k\)-th agent in the \(m\)-th aggregation round. \(K^{\prime}\) indicates the number of agents that are filtered by threshold \(\Omega\) and participate in the model aggregation. Once the global model is upgraded, it will be distributed to each agent for the soft update of each local model:  
\[w_{m+1}^{k}=\lambda w_{m}^{k}+(1-\lambda)w_{m}^{\text{global}}, \tag{13}\]  
where \(w_{m+1}^{k}\) represents the local model of \(k\)-th agent in the \((m+1)\)-th aggregation round.