# NRNH-AR: A Small Robotic Agent Using Tri-Fold Learning for Navigation and Obstacle Avoidance
## 2 Materials and Methods
### Self-Supervised Learning Phase  
In this phase, the ultrasonic sensor data and pre-trained CNN 2 are used. If the physical robot agent detects an obstacle to its front receiving the active ultrasonic sensor signal, then the concatenated observation \(O_{co}\) is taken by the agent's camera. \(O_{co}\) is introduced to pre-trained CNN 2, which produces one of the four collision cases \(C_{c}\in\{1,2,3,4\}\) shown in Table 1. According to the ultrasonic signal, the real action is determined randomly from some possible actions to avoid collision. The selected action is called real action \(a_{r}\). Since the learning process proceeds using real actions generated by the agent itself, we call this phase SSL. In the SSL phase, an actor network and two critic networks are used. The two critic networks are composed of a real critic network and a predicted critic network, as shown in Figure 4. The actor network \(\mu_{\varphi}\) produces a predicted action \(a_{p}\) from collision case \(C_{c}\) as input data. The real critic network \(\pi_{\phi}\) outputs the real quality value \(Q_{r}\) from collision case \(C_{c}\) and real action \(a_{r}\), while the predicted critic network \(\pi^{\prime\prime}_{\phi}\) outputs the predicted quality value \(Q_{p}\) using the predicted action \(a_{p}\) and collision case \(C_{c}\). All of these networks comprise two hidden layers: the first hidden layer consists of 250 neurons, while the second hidden layer consists of 200 neurons and one output neuron, which represents predicted action \(a_{p}\) in the actor network, and qualities \(Q_{r}\) and \(Q_{p}\) in the real and predicted critic networks. \begin{table}
\begin{tabular}{c c} \hline \hline
**Collision Case** & **Class Case (\(C_{c}\))** \\ \hline
[0 1 0] & 1 \\
[0 1 1] & 2 \\
[1 1 0] & 3 \\
[1 1 1] & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Four collision classes that CNN 2 classifies \(O_{co}\). \begin{table}
\begin{tabular}{c c} \hline \hline
**Layer Name** & **Parameters** \\ \hline Input & \(150\times 150\times 1\) \\ Conv1 & \(3\times 3\times 32\) \\ Maxpool 1 & \(2\times 2\) \\ Conv2 & \(2\times 2\times 64\) \\ Maxpool 2 & \(2\times 2\) \\ Fc1/ReLU & \(120\) \\ Fc2/SoftMax & \(4\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: CNN 2 architecture. The predicted critic network's weights are updated, and these updated weights are copied to the real critic network. The actor network's weight is updated to maximize the predicted quality value \(Q_{p}\). The loss function used to update the critic networks is given by (1), and the loss function used to update the actor network is given by (2). \[\mathcal{L}_{critic}=\frac{1}{N}\underset{t=1}{\overset{N}{\sum}}\left(\pi_{ \phi}(C_{c},\ a_{r})-\pi^{\prime}_{\phi^{\prime}}\big{(}C_{c},a_{p}\big{)} \right)^{2} \tag{1}\]  
\[\mathcal{L}_{actor}=-\frac{1}{N}\underset{t=1}{\overset{N}{\sum}}\pi^{\prime} _{\phi^{\prime}}\big{(}C_{c},\mu_{\phi}(C_{c})\big{)} \tag{2}\]  
where \(N\) is the batch size. In our case, \(N=1\) to keep continuous movements of the robotic agent. The weight update of the real critic network is performed using the soft-target method, given by (3). \[\phi\leftarrow\tau\phi^{\prime}+(1-\tau)\phi \tag{3}\]  
where \(\phi\) and \(\phi^{\prime}\) are weights of real and predicted critic networks, respectively, and \(\tau\) is the smoothing factor, which is set to 0.1. After a real action is performed, the robotic agent obtains the next collision case \(\overset{\sim}{C}_{c}\), capturing the ultrasonic signal. If any obstacle is not detected, \(\overset{\sim}{C}_{c}\) is equal to 0; in other cases, three images are taken and concatenated to obtain the next \(\overset{\sim}{C}_{c}\) via pre-trained CNN 2. This experience, composed of \"\(C_{c}\)\", \"\(a_{r}\)\", and \"\(\overset{\sim}{C}_{c}\)\", is saved in the experience buffer memory in the agent. If, in any time step, a target object is detected in one of the three images, CNN 1 is applied, the current episode is finished, and the learning phase is iterated using updated networks.