# A Novel Hybrid Path Planning Method Based on Q-Learning and Neural Network for Robot Arm
## 3 Simulation and Experiment Results
### Simulation Results  
Robot simulation is an essential tool in every robot researcher's toolbox. A well-designed simulator makes it possible to test algorithms, design robots rapidly, and perform regression testing using realistic scenarios. Many simulators can be used in the simulation. RoboDk, Webots, Gazebo, SimSpark, Roboguide, MotoSim, RobotExpert, RobotStudio, and RobotSim are some of the most important of them. While robot simulation software vary in feature sets and capabilities, they all share a few common things. They are simulation tools built to replicate real-world robotic applications, considering every environmental and physical factor and testing for possible variables. In this section, to ensure the performance of presented hybrid method, we test it in the RoboDK simulator before implementing it in the experimental setup. This helps to make sure that the technique is safe and the robot does not collide with itself, the obstacle, and its environment. RoboDK, developed by [PERSON], is an offline programming and simulation software for industrial robots. This simulation software can be used for many manufacturing projects, including milling, welding, pick and place, packaging and labeling, palletizing, painting, and calibration [16]. As explain in step 1, we take a photo from the top view of the workspace, as shown in Figure 4. This includes a rectangular workspace with dimensions of 1000 mm by 500 mm, a yellow square as the start point, a red square as the obstacle, and a green square as the target with dimensions of 125 mm by 125 mm. In step 2, we use the KNN algorithm to find the coordinate of the center of the colorful areas. In this image, given that the reference coordinate system shown in Figure 4, which is located in the center of the robot arm, the found coordinates of start, obstacle, and target via image processing are (+565,+189), (+801,\(-\)193), (+920,\(-\)434), respectively. In step 3, the indexes of the cells of start, obstacle, and target are determined. The indexes of start, obstacle, target cells whose coordinates mentioned in the previous paragraph are (3,1), (6,3), (8,4), respectively. Then, we use them in the Q-learning algorithm to find a sequence of actions from the start cell to the target cell while avoiding collision with the obstacle cell. Inspired by the windy grid world problem, in each cell of the gridded workspace, the best action is obtained so that the agent gets the maximum accumulate reward. In this article, the values of learning rate (\(\alpha\)), discount factor (\(\gamma\)), and probability of exploration (\(\epsilon\)) are equal to 0.5, 0.9, and 0.1, respectively. The cumulative reward value is monitored while running the Q-learning algorithm for the environment shown in Figure 4. Given Figure 5, we can observe that the cumulative reward converges quickly as the episode increases, which implies the algorithm finds the optimum actions efficiently. It additionally demonstrates that the agent finds actions by the proposed action-finding method such that it not only reaches the given target mostly but also maximizes the return due to the quickly converged reward. Furthermore, since this method uses a simplified state, action, and environment compare with other RL-based path planning methods [3; 8; 17; 18], it reaches the maximum cumulative reward value in fewer episodes. Consequently, it significantly reduces the required time for path planning. Figure 4: Top view image from the 2D workspace: start in yellow, obstacle in red, target in green, which are all placed arbitrarily. Figure 5: Cumulative Reward vs. Episodes. As Figure 5 shows, during training, the cumulative reward generally increases until finding a path that leads to the maximum reward and begins fluctuating. This fluctuation happens due to the marginal probability of a non-optimal action being chosen at each step in epsilon-greedy exploration policy, even though an optimal path has been found.