# Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder
## 4 Results
### Influence Factors of the Obstacle-Avoiding MDP
#### 4.2.1 Reward Functions  
Firstly, to explore the effects of prior experience in reward functions, we compared the concise reward with the comparative value reward. The comparative value reward consists of the concise reward and some supplemental rewards concerning the risk of the current state. Secondly, to reduce action-switching frequencies, we added action-changing punishment into the original reward functions.  
To reduce the randomness of training, we trained 100 times and randomly collected 60,000 samples each time. To evaluate the learned obstacle-avoiding policy, we let the robot move from the start  
Figure 7: **(a,b)** The blue line is the path planned by SSG based on the map without unexpected obstacles. The black line is the trajectory planned by SG–RL without replanning the subgoal sequence. The dark-gray region is unpassable, while the light-gray region is traversable for robots. Red regions deemed close to obstacles denote alert areas. **(a)** Simplified office map with unexpected obstacles, **(b)** real office map with unexpected obstacles; **(c,d)** Two magnified details. **(c)** Magnified detail 1, **(d)** Magnified detail 2.  
position to the finish line on the test map (Figure 8). If the robot collided with obstacles, the policy was considered as a failure, otherwise it was considered a success.  
As seen in Table 6, the ratio of successful OA policies with the concise reward is five times higher than that with the comparative value reward. The other evaluation indicator of concise reward is over five times higher than that of the comparative value reward. Obviously, training with the concise reward can more easily get an obstacle avoidance policy than training with the comparative value reward. The comparative value reward including prior experience makes rewards dense; however, meanwhile, it brings difficulties for training. Therefore, training with the concise reward can perform better, and adding prior experience into reward functions may bring some negative effects on training. The better place to impart to the agent prior experience may be the initialization.  
As seen in Table 6, the learned policies did not perform well in the aspect of action-switching frequencies; thus, we brought the punishment concerning action changes into reward functions to reduce action-switching frequencies while avoiding obstacles.  
The punishment concerning action changes suggests that when current action is different from last one, the agent obtains a value of \(-0.2\). Then, the entire combined reward has \"strong\" requirements (i.e., avoid obstacles) and \"weak\" requirements (i.e., reduce action changes) which means that, after \"strong\" requirements are satisfied, \"weak\" requirements can be satisfied as much as possible.  
Shown in Table 7, the evaluation factor is the ratio of policies with action-switching frequency below 30%. Obviously, original reward functions with action-changing punishment can increase this evaluation factor. Therefore, action-changing punishment does a great favor to both the concise reward and the comparative value reward, reducing the action-switching frequency by increasing the performance of learned policies, thereby also demonstrating that the combined rewards work better than separate ones.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Reward Function** & \begin{tabular}{c} **Ratio of Successful OA** \\ **Policies** \\ \end{tabular} & \
\begin{tabular}{c} **Ratio of Policies with Action-Switching** \\ **Frequency Below 30\%** \\ \end{tabular} \\ \hline \hline
\begin{tabular}{c} concise reward \\ comparative value reward \\ \end{tabular} & 45\% & 38\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of concise reward and comparative value reward.  
Figure 8: This is a successful sample of one robot with learned obstacle-avoiding policy completing the test. The red square is the start position, and the pink dotted line is the finish line. The green square is the position that the robot arrived at the finish line.