# A Novel Hybrid Path Planning Method Based on Q-Learning and Neural Network for Robot Arm
## 3 Simulation and Experiment Results
### Experiment Results
As Figure 9 shows, the running time of Q-learning code is generally longer than that of neural network code because a learning algorithm requires to interact with the environment modeled in programming software to learn optimal actions, but the neural network does not. Therefore, we call them as active and passive phases, respectively. As shown in the below plot, since the resolution in our experiment is eight by four, we have thirty-two cells, so the total running time of this hybrid method for that particular resolution is around 1 s.  
To evaluate the performance of the method in different situations, we test it in various configurations. The start, obstacle, and target cells are placed in different cells arbitrarily. An attempt has been made to cover all different types of configurations. Those configurations are presented in Figure 10.  
The running time of these different configurations are obtained and are presented in Table 3. It is obvious that when the start and target cells are close to each other, the running time is slightly less than when they are far apart. This is because, since they are close to each other, during the exploring for the target cell, the path is found in fewer episodes. Additionally, since the number of actions is small due to the short path, fewer joint angles are calculated by the trained neural network. Therefore, the total running time of the code is reduced. In order to compare the speed of our hybrid method with conventional path planning methods such as RL-based, APF, PRM, and RRT, we run a simple code of each method on the same computer to compute its running time. It is worth mentioning that we downloaded [PERSON]'s APF and PRM (2020), [PERSON]'s RRT (2019), and [PERSON]'s RL-based (2018) from Github [19; 20; 21]. Then, according to our subject, we change and modify their codes and make them similar to our work to become comparable  
Figure 9: Running time vs. number of cells: 1. Total running time in black, 2. Q-learning running time (active phase) in blue, 3. Neural network running time (passive phase) in red. with ours. As can be seen in Figure 11, our hybrid method needs less running time to do the path planning process under the same conditions, which means it is faster than traditional methods due to using separate active and passive approaches. In fact, our hybrid method makes finding the path faster and simpler in two ways. First and foremost, in the passive phase, we save time by spending a fraction of time before path planning and using it during path planning. In other words, we spend time training the neural network before path planning and use the trained weights during path planning. This is because, unlike the training mode of a neural network which is time-consuming, the using mode takes less time. Second, in the active phase, instead of linking simulators with programming software to obtain a sequence of joints angles, we use simplified state-action spaces to simulate the agent-environment interaction. These two factors have led our hybrid method to reduce the running time and increase the speed. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Configurations Shown** & **Q-Learning Time [s]** & **Neural Network Time [s]** & **Total Time [s]** \\
**in Figure 10** & & & \\ \hline (a) & 0.756 & 0.042 & 0.798 \\ (b) & 0.451 & 0.028 & 0.479 \\ (c) & 0.459 & 0.028 & 0.487 \\ (d) & 0.808 & 0.049 & 0.857 \\ (e) & 0.635 & 0.036 & 0.671 \\ (f) & 1.012 & 0.080 & 1.092 \\ (g) & 0.687 & 0.049 & 0.736 \\ (h) & 0.929 & 0.070 & 0.999 \\ (i) & 1.073 & 0.081 & 1.154 \\ (j) & 0.807 & 0.056 & 0.863 \\ (k) & 0.618 & 0.040 & 0.658 \\ (l) & 0.889 & 0.064 & 0.953 \\ (m) & 0.489 & 0.036 & 0.525 \\ (n) & 0.691 & 0.042 & 0.733 \\ (o) & 1.13 & 0.088 & 1.218 \\ (p) & 1.179 & 0.090 & 1.269 \\ Average & 0.788 & 0.055 & 0.843 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Result of running time test for different configurations in the 8 by 4 environment.