# NRNH-AR: A Small Robotic Agent Using Tri-Fold Learning for Navigation and Obstacle Avoidance
## 3 Experiments  
### Static Environment  
In the first phase of the experiment, the agent was tasked with locating an object within a static physical environment. To accomplish this, three distinct static environments were utilized, in which the initial position of the robot agent and the position of the target object were varied. Figure 7 shows three static environments, with an initial position of the robotic agent (black box) and a position of the target object (green box). As mentioned before, the positions of the agent and the target varied. In this experiment, the SSL, USL, and DRL phases were trained in environment (a), while environments (b) and (c) involved training and inference of the DRL phase. The learning process started in the SSL phase. If the agent consistently achieved the upper threshold of learning level \(A\) for ten consecutive iterations (T = 10), it indicated the need for a more advanced learning system, the USL phase. The choice of 10 consecutive successful iterations as a threshold for transitioning between learning stages (from SSL to USL, and from USL to DRL) was based on empirical testing [24]. Through experimentation, we observed that, when the agent exhibited consistent navigation behavior, such as completing trajectories with minimal deviation or corrective oscillation, for at least 10 consecutive episodes, it indicated that the current internal representation or learning stage was sufficiently stable to advance. Similarly, if the agent maintained the highest level of learning for ten consecutive iterations during the USL phase, it suggested the application of the DRL phase. This sequential progression from SSL to DRL via USL is necessary when dealing with complex environments. By avoiding the direct implementation of DRL without the preceding SSL and USL phases, the agent minimizes the time and resources required to comprehend a complex environment effectively. Figure 8 showcases the trajectory followed by the previously trained robotic agent through all learning phases using static environments (a)-(c), evaluated in (c) of Figure 7. The agent started from the initial position marked by a black box, which is position (a), and ultimately reached the location where a target object was found. The agent's position in each time step from (a) to (f) corresponds to photograph numbers from (a) to (f) of Figure 8. In the initial position, (a), the agent selected the action \"forward\" from some possible actions {\"forward\", \"turn left\"}, according to ultrasonic signals. This selected action provided the best quality based on the agent's experience as reflected in the trained predicted actor network. Similarly, the agent selected \"forward\" in time step (b) and reached position (c). In this position, the agent found an obstacle to its front, so three pictures were taken. Because none of the three pictures presented the target object, the agent generated \(O_{\omega}\), then trained CNN 2 interpreted that collision case \(C_{c}\) is 3, meaning to [1, 1, 0]. Using its experience, the agent took the \"right turn\" as the best action, and then took the \"forward\" action to reach position (e). In position (e), the agent found an obstacle to its front, so three pictures were taken by the agent. In this case, CNN1 detected the target  
Figure 7: Configuration of three physical static environments (**a–c**) used for evaluation. object in one of these three pictures (left side picture), and then the robotic agent took the \"left turn\" action and accomplished the mission in position (f). The same set of hyperparameters was used across all three learning phases (SSL, USL, and DRL) to maintain consistency. These include a learning rate of 0.001, discount factor \(\gamma\) = 0.9, smoothing factor \(\tau\) = 0.1, policy update delays every two iterations, batch size = 1, and replay buffer size of 10,000. The optimizer used was Adam. These values were empirically tuned for stable training under the computational constraints of the Jetson Nano.