## 3 Integration of the PPO Algorithm for Robot Navigation
### Reward Function  
The reward function has two components, each influencing the agent's actions and learning process. The components include a goal-oriented reward function, which calculates the reward contributed by the location of the goal relative to the agent, and an obstacle-oriented reward function, which calculates the reward contributed by the location of the closest obstacle relative to the agent. The composition of these elements is carefully designed to meet the specific objectives and shape the agent's behavior accordingly. The goal-oriented reward function is formally represented as Equation (5):  
\[R_{t_{Goal}}= \begin{cases}-10&\Delta d_{Goal}>0.5\;\text{m},\Delta d_{Goal} \leq 0\;\text{m}\\ 200\times\Delta d_{Goal}\times\left(1-4\times\left|0.5-\left\{\frac{d}{\pi} \right\}\right|\right)&0\;\text{m}<\Delta d_{Goal}\leq 0.5\;\text{m}\end{cases} , \tag{5}\]  
The goal distance rate is calculated as in Equation (6), which represents the change of agent to goal distance between steps:  
\[\text{Goal distance rate}=\Delta d_{Goal}=\mathit{PastGoalDistance}- \mathit{GoalDistance} \tag{6}\]  
Intuitively if \(\Delta d_{Goal}\leq 0\;\text{m}\), the agent is further away from the goal than the previous time step. \(G_{H}=\arctan\frac{\eta_{Goal}-y_{Pool}}{\eta_{Goal}-\pi_{Label}}-\theta\) represents the goal heading, where \((x_{goal},y_{goal})\) and \((x_{robot},y_{robot})\) are the Cartesian coordinates of the goal and robot, respectively, and \(\theta\) is the robot's orientation. The \(\left\{\cdot\right\}\) function takes the fraction part of any calculated result inside, where \(\mathcal{A}=\left(0.5\times\left(G_{H}+\pi\right)\right)\pmod{2\pi}\), which is the remainder after division by \(2\pi\). We partially follow the implementation in the original work [36] to define \(\mathcal{A}\) with an offset value of \(\pi\) to ensure that the robot receives the maximum rewards when facing the goal directly. Figures 5 and 6 show the hardware representations of the robot, obstacles, and goal positions, where Figure 5 shows an example of how the goal distance and heading change for the agent. Figure 7a shows the plot of the goal-oriented reward function. As can be seen in the plot, the agent receives the most reward if it is facing the goal directly and is close to it. The agent receives the largest penalty when facing away from the goal. Note that the green arrow line represents the path of the agent. The obstacle-oriented reward function is formally represented as Equation (7):  
Figure 4: Overall workflow. The user can specify the parameters needed at the beginning of the current training round. In each episode, the agent receives the new state \(s_{t}\), when taking action \(a_{t}\). When the episode termination condition is met (i.e., the binary episode termination flag \(Flag\) is set to 1), the current goal rate will be calculated and saved to an array, which will be saved to a CSV file for further analysis. \[R_{t_{Obstacle}}= \begin{cases}5&\Delta d_{Obstacle}\leq 0\text{ m}\\ -400\times\Delta d_{Obstacle}\times\left(1-4\times\left|0.5-\left\{\frac{\mathcal{B }}{\pi}\right\}\right|\right)&\Delta d_{Obstacle}>0\text{ m}\end{cases} , \tag{7}\]  
The obstacle distance rate is calculated as in Equation (8), which represents the change of agent to closest obstacle between steps:  
\[\text{Obstacle distance rate}=\Delta d_{Obstacle}=PastObstacleDistance- ObstacleDistance \tag{8}\]  
Intuitively if \(\Delta d_{Obstacle}\leq 0\text{ m}\), the agent is further away from the closest obstacle than the previous time step. \(O_{H}\) represents the obstacle heading, where \(\mathcal{B}=(0.5\times(O_{H}+\pi))\) (mod \(2\pi\)). We partially follow the implementation in the original work [36] to define \(\mathcal{B}\) with an offset value of \(\pi\) to ensure that the robot receives the minimum rewards when directly facing the closest obstacle. Figure 6 demonstrates pictorially one case of how the obstacle distance and heading might change for the agent. Figure 7b shows the plot of the obstacle-oriented reward function.