# New Approach in Human-AI Interaction by Reinforcement-Imitation Learning
## 4 Experiments and Results
```
0: Exact requirements of Algorithm 1,  Initialization:  Same as Algorithm 1, while\(Q\) is not converged do  select \([O,A]\in O_{s}\) while\(O\) is not terminated do \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) \(A^{\prime}\leftarrow\phi(O^{\prime})\) \(e(O,A)\gets e(O,A)+1\) \(\sigma\leftarrow\lambda Q(O_{n+1},A_{n+1})-Q(O_{n},A_{n})\) for\([O^{\prime},A^{\prime}]\in O_{s}\)do if\(f\) is exist: then  Consider Equation (5)  Consider Equation (1)  Consider Equation (3)  Consider Equation (4) endif \(Q(O^{\prime},A^{\prime})\gets Q(O^{\prime},A^{\prime})+\alpha*\gamma* \sigma*e(O^{\prime},A^{\prime})\) \(e(O^{\prime},A^{\prime})\leftarrow\gamma*\lambda*e(O^{\prime},A^{\prime})\) \(O^{\prime}\gets O\) \(A^{\prime}\gets A\) endfor endwhile endwhile return\(Q\)
```  
**Algorithm 3** Hybrid State-action reward-state-action imitation learning (SARSA/IL)  
```
0:  requirements of Algorithm 1,  Initialization:  Meta parameters: \(\theta\), \(\omega\) while\(T<T_{max}\)do \(\theta\gets 0\) and \(\omega\gets 0\) while\(O\) is not terminated do \(A^{\prime}\leftarrow\phi(O^{\prime})\) if\(f\) is exist:then  Consider Equation (5)  Consider Equation (1)  Consider Equation (3)  Consider Equation (4) endif \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) for\(i=t-1:t_{start}\)do  update \(R,d\theta\) and \(d\omega\) endfor endwhile return\(Q\)
```  
**Algorithm 4** Hybrid Asynchronous Advantage Actor-Critic Agents Imitation Learning (Hybrid A3C/IL)  
```
0: \(\theta\gets 0\) and \(\omega\gets 0\) while\(O\) is not terminated do \(A^{\prime}\leftarrow\phi(O^{\prime})\) if\(f\) is exist:then  Consider Equation (5)  Consider Equation (1)  Consider Equation (3)  Consider Equation (4) endif \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) for\(i=t-1:t_{start}\)do  update \(R,d\theta\) and \(d\omega\) endfor endwhile endwhile return\(Q\)
```  
**Algorithm 5** Hybrid Asynchronous Advantage Actor-Critic Agents Imitation Learning (Hybrid A3C/IL)  
```
0: \(\theta\gets 0\) and \(\omega\gets 0\) while\(O\) is not terminated do \(A^{\prime}\leftarrow\phi(O^{\prime})\) if\(f\) is exist:then  Consider Equation (5)  Consider Equation (1)  Consider Equation (3)  Consider Equation (4) endif \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) for\(i=t-1:t_{start}\)do  update \(R,d\theta\) and \(d\omega\) endfor endwhile return\(Q\)
```  
**Algorithm 6** Hybrid Asynchronous Advantage Actor-Critic Agents Imitation Learning (Hybrid A3C/IL)  
```
0: \(\theta\gets 0\) and \(\omega\gets 0\) while\(O\) is not terminated do \(A^{\prime}\leftarrow\phi(O^{\prime})\) if\(f\) is exist:then  Consider Equation (5)  Consider Equation (1)  Consider Equation (3)  Consider Equation (4) endif \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) for\(i=t-1:t_{start}\)do  update \(R,d\theta\) and \(d\omega\) endfor endwhile return\(Q\)
```  
**Algorithm 7** Hybrid Asynchronous Advantage Actor-Critic Agents Imitation Learning (Hybrid A3C/IL)
Moreover, the achievements of utilizing Algorithms 1-4 on continuous classic mountain-car in an OpenAI-Gym environment are shown in Figure 4b. In this environment, the agent receives a punishment (negative reward) for each step of an episode. The maximum performance of this environment is \(-100\) cumulative rewards and it shows that the minimum number of steps in an epoch to gain the flag on top of the hill in the continuous environment is \(100\). Like the cart-pole environment, the results for both \"Hybrid A3C/IL\" and \"Hybrid SARSA/IL\" show that the proposed algorithms based on the integration-imitation learning and reinforcement learning outperform the A3C and SARSA, as examples of policy-based and value-based RL. Figure 4b presents Hybrid A3C/IL and Hybrid SARSA/IL converging faster concerning the two RL algorithms. However, Hybrid SARSA/IL shows lots of oscillations before stabilising at episode # 20. The reason for these fluctuations is that the value-iteration-based RL cannot act well in complicated continuous environments regarding exploration and evaluation of the tuple of state and action.