## 5 Training
### Training Parameters and Bootstrapping  
We employ bootstrapping methods to guide and speed up the training convergence. The basic task of the neural network is to obtain a policy that guides the robot to the goal position. However, since there is only one goal in the environment, there is a low probability that a policy based on randomly initialized weights will be able to reach it. This means that the trajectories with successful goal reaching will be under-represented in the replay buffer. To mitigate this issue, we set a distance threshold of how far a goal position can  
Figure 6: Gazebo simulator environment used for training neural network policy. Three robots are randomly placed in the environment and shown with green circles around them. Such obstacles as cylinders and boxes are also placed in the environment as obstacles, and their locations are randomized at the start of each episode.  
be randomly placed from its respective robot. We expand this distance every time a robot reaches its goal. Another method employed is adding Gaussian noise to the obtained action from the policy. This allows us to take slightly different actions than the ones calculated and allows for exploration of the policy. We start the training with a set maximum variance for the Gaussian noise and slightly reduce it at every time step. However, Gaussian noise by definition is centered around the already existing policy and, in the majority of cases, will produce very slight deviations from it, especially later in the training process with a small variance. This is conducive to optimizing an already well-behaving policy. But there are cases in which drastically different actions could produce more positive outcomes in the long run. Therefore, for time steps that satisfy the criteria:  
\[p>R \tag{7}\]  
where \(R\) is a random scalar drawn in the range \([0,1]\), we take a completely random action drawn from a uniform distribution. This allows us to randomly take out-of-distribution actions while still generally executing trajectories following the policy.  
The reinforcement algorithm uses rewards to evaluate performance. In our training setup, we employ the following reward function:  
\[r(s_{t},a_{t})=\begin{cases}r_{g}&\text{if }d_{t}<\eta_{\delta}\\ r_{c}&\text{if }l_{min,t}<\eta_{c}\\ r_{t}&\text{otherwise,}\end{cases} \tag{8}\]  
Here, \(r_{g}\) is the reward for reaching the goal when the distance to the goal from the robot is below a threshold \(\eta_{\delta}\). The collision reward \(r_{c}\) is given when the minimum laser reading \(l_{min,t}\) is below a threshold \(\eta_{c}\). The immediate reward \(r_{t}\) is calculated as:  
\[r_{t}=\beta_{v}\times v_{t}-\beta_{omega}\times|\alpha_{t}|-\beta_{l}\times r_ {l} \tag{9}\]  
where \(\beta\) is the coefficient for the respective reward element and \(r_{l}\) is a negative reward depending on closeness to obstacles, obtained as follows:  
\[r_{l}=\begin{cases}\eta_{l}+\eta_{c}-l_{min}&\text{if }l_{min}<\eta_{l}+\eta_{c} \\ 0&\text{otherwise,}\end{cases} \tag{10}\]  
where \(\eta_{l}\) is a threshold for the distance to obstacles. The full list of training parameters is available in Table 3.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{3}{c}{**Training Parameters**} \\ \hline
**Parameter** & **Value** & **Metric** \\ \hline
**Reward Function**: & & \\ \(r_{g}\) & +100 & - \\ \(r_{c}\) & \(-100\) & - \\ \(\eta_{l}\) & 1 & m \\ \(\eta_{c}\) & 0.35 & m \\ \(\beta_{v}\) & 1 & - \\ \(\beta_{\omega}\) & 0.5 & - \\ \(\beta_{l}\) & 0.5 & - \\
**Training Parameters**: & & \\ Epochs & 100 & - \\ Time steps per epoch & 5000 & - \\ Max episode length & 500 & time steps \\ Seed value & 42 & - \\ \hline \hline \end{tabular}
\end{table}
Table 3: Network parameters and structure of both critic networks.