# New Approach in Human-AI Interaction by Reinforcement-Imitation Learning
## 3 Teacher Assistance Structure  
The proposed structure exploits teacher feedback as a rectification of the action domain of the learner; as soon as an action is performed by the agent, this teacher feedback improves the online policy. It can also infer the policy of the agent from infrequent teacher feedback. Four main characteristics of human teacher feedback and their related effects are considered and formulated; namely, duality, reaction time delay, contingency and instability.  
While several studies consider a range for teacher feedback, like very bad, bad, good, very good or \(-100\), \(-50\), \(50\), \(100\), giving feedback by humans from a range is very complicated and requires very good knowledge of the task and environment. This study takes advantage of duality feedback when a human teacher is satisfied with the decision made by agent or not. This kind of feedback can be sent by expert or non-expert human due to its simplicity concerning knowledge transferral.  
The next feature that is considered is the reaction time delay of the human to send feedback. Several studies like [23] present the sample efficiency for training neural reinforcement when it is pre-trained by an expert using the supervised learning method. In fact, in different IL algorithms like DAGGER and GAIL, which are based on offline learning, there is no need to consider the reaction time delay of the human in the model. In the mentioned algorithms just an expert should prepare a time-consuming metadata before starting the training process. Moreover, unprofessional feedback from a non-expert teacher can ruin the training process.  
The proposed structure for human-AI interaction presents a methodology to enable AI agents to interact with the human (teacher, expert or not) completely online; we should deal with the delay in the reaction of humans [24; 25]. Using the teacher feedback in online training without recognizing that delay can make the training process impractical. However the reaction time delay is not constant and it would vary depending on teacher personality, teacher knowledge, complexity of the environment and ambiguity of actions.  
In addition to the reaction time delay, the contingency of the human teacher feedback as a feature of reactive manner is dealt with. Due to limited patience, mostly the human teacher stops to send positive feedback while the agent carries out actions correctly. Moreover, the frequency of releasing the feedback can vary based on human preference [26]. So the proposed methodology considers a module named \"feedback predictor\" (See Figure 2) to present the contingency and stochasticity of correct feedback which is sent in a specific timestamp.  
The details of the structure are explained as follows (See Figure 2):Feedback predictor: This section gathers the previous feedback and advice, and predicts the next action to be taken by the human.
* Supervised policy: This module can improve the variables of the RL policy;
* Feedback evaluator: This module can upgrade the variables of the supervised policy module;
* Feedback flag: This module can manage and control the time-lags and postponement of human response;
* RL policy: This considers an on-policy algorithm of Q-learning and a one policy-based RL algorithm.  
As soon as the agent picks an action, the supervisor can observe the outcome of that action on the environment and send his feedback. This feedback \((f)\) is a positive, negative or neutral value to intimate that the last performed action should be modified increasingly or decreasingly. The neutral value is considered when the teacher is not available or he prefers not to expose his idea. The environment state \((S)\), performed action, \(A(S)\), and teacher feedback \((f)\) are sent back to the \"feedback evaluator\" and \"supervised policy\" sections to update the \(\Phi\) and \(\Psi\).