# New Approach in Human-AI Interaction by Reinforcement-Imitation Learning
## 3 Teacher Assistance Structure
### Reinforcement Learning (RL) Policy  
In RL, we consider a predefined environment. In such an environment, an agent performs actions and reactions sequentially to complete a task, using its observations of the environment and the rewards it gets from it. The agent can choose an action from an action space, \(A_{s}=A_{1},A_{2},...,A_{n}\). That action passes to the stochastic environment and return a new observation space, \(O_{s}=O_{1},O_{2},...,O_{n}\), and reward, \(R_{s}=R_{1},R_{2},...,R_{n}\). At each step, the agent observes the current state game state and cannot understand the whole task by observing just the current game state. Moreover, the Marcov Decision Process (MDP) is a fundamental of RL. MDP can be used in a cooperative structure for the decision-making tasks to partly or completely control and balance agents.  
The first RL algorithm in this study is SARSA and its details are presented in Algorithm 1. SARSA is very similar to Q-learning. The key difference between them is an on-policy, whereas Q-learning is a class of off-policy Temporal Difference (TD). This implies that the SARSA learning process deals with the actions taken by the current policy instead of the greedy policy. So the SARSA update (see Equation (6)) does not consider the maximum value and greedy policy (Equation (2)).  
\[Q(O_{n},A_{s})=Q(O_{n},A_{s})+\alpha[R_{n+1}+\gamma Q(O_{n+1},A)-Q(O_{n},A_{s})] \tag{6}\]On the other hand, \(Q\)-learning consists of a multi-layer neural network (NN), where its inputs would be states of an environment and the outputs would be the action value, \(Q(s,\theta)\), where \(\theta\) is the parameters of \(NN\). In fact, \(Q\)-learning updates for the parameter after taking action \(A_{n}\) after observing the state \(O_{n}\), and receive an immediate reward \(R_{n}\) and \(Q_{n+1}\). So the update equation is given by:  
\[Q(O_{n},A_{s})=Q(O_{n},A_{s})+\alpha[R_{n}+\gamma maxQ(O_{n+1},A)-Q(O_{n},A_{s })] \tag{7}\]  
This equation shows that the policy considered to select an action is a greedy policy calculated by Equation (8):  
\[maxQ(O_{n+1},A)-Q(O_{n},A_{s}) \tag{8}\]  
The second RL algorithm in this study is Asynchronous Advantage Actor-Critic Agents (A3C), a policy gradient algorithm with a special focus on parallel training. In A3C, the critic part learns the value function while the actor part is trained in parallel and becomes synchronized with the global parameters sequentially. In A3C, there is a loss function for state value to minimize the Mean Square Error (MSE) (Equation (9)) and this is the baseline in the policy gradient update. Finally, the gradient descent can be applied to find the optimal value. For more details about the A3C see the Algorithm 2.  
\[J_{V}(\omega)=(G_{t}-V_{\omega}(s))^{2} \tag{9}\]  
```
```  Observations: \(O_{s}=O_{1},O_{2},...,O_{n}\),  Actions: \(A_{s}=A_{1},A_{2},...,A_{n}\),  Reward Function: \(R_{s}=R_{1},R_{2},...,R_{n}\),  Transition: \(T_{s}=T_{1},T_{2},...,T_{n}\),  Initialization:  Learning rate: \(\alpha\in[0,1]\), initialized with = 0.1  Discount factor: \(\gamma\in[0,1]\)  Balancing rate: \(\lambda\in[0,1]\); Trade off between Temporal-Difference and Monte-Carlo  Process Q-learning \((O,A,R,T,\alpha,\gamma,\lambda)\) while\(Q\) is not converged do  select \([O,A]\in O_{s}\) while\(O\) is not terminated do \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) \(A^{\prime}\leftarrow\phi(O^{\prime})\) \(e(O,A)\gets e(O,A)+1\) \(\sigma\leftarrow\lambda Q(O_{n+1},A_{n+1})-Q(O_  
```
0: requirements of Algorithm 1, Initialization: Meta parameters: \(\theta\), \(\omega\) while\(T<T_{max}\)do \(\theta\gets 0\) and \(\omega\gets 0\) while\(O\) is not terminated do \(A^{\prime}\leftarrow\phi(O^{\prime})\) \(R\gets R(O,A)\) \(O^{\prime}\gets T(O,A)\) for\(i=t-1:t_{start}\)do  update \(R,d\theta\) and \(d\omega\) endfor endwhile endwhile return\(Q\)
```  
**Algorithm 2** Asynchronous Advantage Actor-Critic Agents (A3C)