## 5 Path-Planning Training
### Rewarding Formulation  
The rewards are used to tell the agent how good its taken actions are, which in this case are the planned path to the destination. Rewarding is an essential task in any reinforcement learning model. Different from rule-based models, rewarding is usually based on the results of the agent's actions or the effect of the agent's actions on the states. The rewarding formulation problem is equivalent to the task of instructing the pedestrian agent to aim at certain aspects. Consequently, to conduct a natural navigation behavior, we studied how humans consider the navigation as natural or not. As a result, we adopt the idea of human comfort, introduced by [PERSON] et al. (2018). The idea consists of several factors contributing to a simulated movement, which could make observing humans feel comfortable or human-like. Within the scope of our study, we choose to adopt the following factors for our rewarding mechanism:  
* choosing the shortest path to the destination;
* avoiding frequently changing direction;
* following basic navigation rules and common-sense standards; and
* colliding with obstacles. The first factor, which is also considered a decisive factor in many studies, is to plan the shortest path to the destination. While in real life navigation, human pedestrians may subconsciously aim at the shortest navigation time, they still consider shortest path to be the highest-ranking factor, as in a study conducted by [PERSON] (2011). As each rewarding factor correlates with the aspect that the human pedestrian is aiming at or wants to achieve, planning the shortest path would be formulized. Consequently, we calculate the rewarding for this behavior by placing a negative reward corresponding to the sum of the squared length of each component path. This means if the path is longer, the agent would receive a larger penalty. This rewarding is formulated as follows:  
\[\mathcal{R}_{1}=-\lambda\sum_{i=0}^{11}\|p_{i}\|^{2}, \tag{5}\]  
where \(\lambda\) is the environment's scale, and \(p_{i}\) is the vector of each component path. Regarding the rewarding for changing direction, we only consider the changes in angle which are larger than 30 degrees. Any changes in angle which are smaller than this could be acceptable and are still considered natural. For this reason, we formulate the rewarding for this behavior by placing a penalty each time there is a large change of direction in the planned path as follows:  
\[\mathcal{R}_{2}=-\sum_{i=0}^{10}\theta(angle(p_{i},p_{i+1}))\, \tag{6}\]  
where \(angle(p_{i},p_{j})\) is the angle value between the vectors \(p_{i}\) and \(p_{j};\theta(x)\) is the Heaviside step function, specified by  
\[\theta(x)= \begin{cases}0\,&\text{if }x<0\,\\ 1\,&\text{if }x\geq 0\.\end{cases}  \tag{7}\]  
As for the rewarding based on following basic navigation rules and common-sense standards, the rules may vary between different regions and cultures. From our observation, the following rules are applied in our study:  
1. Following the flow of navigation by walking parallel to the sides. 2. Walking on the left side of the road. While pedestrians are not required to strictly follow this, in real life, people still choose to follow this as a general guideline to avoid accidents. Similarly, in right-side walking countries, pedestrians would choose to walk on the right side of the road. 3. Avoiding getting close to the sides. To define the appropriate rewarding formulations, the planned path of the agent is sampled into \(N\) values \(s_{i}\) with \(i\) ranges from 0 to \(N\). The respective rewarding functions are calculated as follows:  
\[\mathcal{R}_{3}=-\lambda\sum_{i=0}^{N}\theta\big{(}\|x_{pos}(s_{i+1})-x_{pos}(s _{i})\|-H_{1}\big{)}, \tag{8}\]  
\[\mathcal{R}_{4}=-\lambda\sum_{i=0}^{N}\theta\big{(}-x_{pos}(s_{i})\big{)}, \tag{9}\]  
\[\mathcal{R}_{5}=-\sum_{i=0}^{N}\theta\big{(}\|x_{pos}(s_{i})\|-H_{2}\big{)}, \tag{10}\]  
where \(x_{pos}(s_{i})\) function returns the \(x\) coordinate of the point \(s_{i}\).