## 4 Methodology
### Deep Reinforcement Learning Setup
#### 4.1.3 Reward Function  
The reward function is designed to incentivize each robot to choose and move towards a unique goal position while minimizing the time consumed to reach the goal position. It penalizes collisions with obstacles or other robots and multiple robots reaching the same goal position. Equation (5) shows the reward \(r\) received by robot \(i\) at time step \(t\).  
\[r_{i}^{t}=\left(r^{d}\right)_{i}^{t}+\left(r^{col}\right)_{i}^{t}+\left(r^{o} \right)_{i}^{t}+\left(r^{t}\right)_{i}^{t} \tag{5}\]  
where \(\left(r^{d}\right)_{i}^{t}\) is used to incentivize the ego-robot to go towards a goal position. It is calculated in Equation (7) using the matrix \(\left(G\mathbb{R}^{2\times N}\right)\), which contains the positions of all of the goals. The first row of \(G\) describes the \(x\) position of each goal, while the second row describes the \(y\) position. The number of columns is the number of the goal positions \(N\). The robot position vector at time step \(t\) is described as \(\left(P_{i}^{t}\mathbb{\in}\mathbb{R}^{2}\right)\). The ego-robot receives a positive reward value \(r_{terminal}\) if it reaches the goal position; otherwise, it is either rewarded or penalized if it moves towards or away from the nearest goal position \(\left(\min\left(\|G-P_{i}^{t}\mathbb{1}_{N}^{\intercal}\|\right)\right)\). \(q^{t}\) in Equation (6) is an intermediate variable that is used in Equation (7) to simplify it.  
\[q^{t}=\frac{\min\left(\|G-P_{i}^{t}\mathbb{1}_{N}^{\intercal}\|\right)-\min \left(\|G-P_{i}^{t-1}\mathbb{1}_{N}^{\intercal}\|\right)}{dt} \tag{6}\]  
\[\left(r^{d}\right)_{i}^{t}=\begin{cases}r_{terminal}&,if\ \min\left(\|G-P_{i}^{t} \mathbb{1}_{N}^{\intercal}\|\right)<0.2\\ w_{g}q^{t}&,\ otherwise\end{cases} \tag{7}\]  
Equation (10) describes \(\left(r^{col}\right)_{i}^{t}\), which is the reward function that penalizes the ego-robot in case of collisions. The collisions with obstacles are detected using the 2D laser scanner readings on the ego-robot \(\left(L^{t}\mathbb{\in}\mathbb{R}^{360}\right)\). The collisions with other robots are detected by thresholding the distance between the ego-robot and other robots. The distance with other robots is calculated using the matrix \(\left(P_{eth}^{t}\mathbb{\in}\mathbb{R}^{2\times N-1}\right)\), which describes the positions of the other robots at time step \(t\). In case a collision is detected, the ego-robot receives a negative reward value \(-r_{terminal}\). \(cond_{1}\) in Equation (8) and \(cond_{2}\) in Equation (9) are intermediate variables that are used in Equation (10) to simplify it.  
\[cond_{1}=\min\left(L^{t}\right)<1.1R \tag{8}\]  
\[cond_{2}=\min\left(\|P_{oth}^{t}-P_{i}^{t}\mathbb{1}_{N-1}^{\intercal}\|\right) \leq 2R \tag{9}\]  
\[\left(r^{col}\right)_{i}^{t}=\begin{cases}-r_{terminal}&,if\ cond_{1}\ or\ cond_{2}\\ 0&,\ otherwise\end{cases} \tag{10}\]Equation (12) describes \(\left(r^{o}\right)_{i}^{t}\), which rewards or penalizes the ego-robot if it is moving towards a unique goal position or towards a goal position that another robot is moving towards. \(d_{oth}\) in Equation (11) is an intermediate variable that is used in Equation (12) to simplify it.  
\[d_{oth}=\left\|P_{oth}^{t}-G\left[\argmin\left(\|G-P_{i}^{t}\mathbb{1}_{N}^{ \intercal}\|\right)\right]\mathbb{1}_{N-1}^{\intercal}\right\| \tag{11}\]  
\[\left(r^{o}\right)_{i}^{t}=w_{oth1}\log_{10}(\min\left(d_{oth}\right))+w_{oth2} \tag{12}\]  
Finally, Equation (13) describes \(\left(r^{t}\right)_{i}^{t}\), which penalizes the ego-robot depending on the amount of time it takes to reach a goal position. The ego-robot receives a negative reward \(-r_{terminal}\) if the maximum episode duration \(T_{max}\) is exhausted before reaching a goal position.  
\[\left(r^{t}\right)_{i}^{t}=\begin{cases}-r_{terminal}&,\ if\ t\geq T_{max}\\ w_{t}\frac{t}{T_{max}}&,\ otherwise\end{cases} \tag{13}\]  
All the constants \(r_{terminal},\ w_{g},\ w_{oth1},\ w_{oth2},\ T_{max}\), and \(w_{t}\) are set during experimentation, and the constant \(R\) depends on the robot dimensions.