# Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment
## 3 Materials and Methods  
### Methodology and Implementation of the Agents  
In this work, our focus is on assessing how interactive feedback, used as an assistance method, may affect the performance of a DeepRL agent. To this aim, we implement three different approaches for the RL agents:  
1. DeepRL: where the agent interacts autonomously with the environment;
2. agent-IDeepRL: where the DeepRL approach is complemented with a previously trained artificial agent to give advice; and
3. human-IDeepRL: where the DeepRL approach is complemented with a human trainer. The first approach includes a standard deep reinforcement learning agent, referred to here as DeepRL, and is the basis of both of the interactive agents discussed subsequently. The DeepRL agent perceives the environment information through a visual representation [37], which is processed by a convolutional neural network (CNN) that estimates the Q-values. The deep Q-learning algorithm allows the agents to learn through actions previously experienced, using the CNN as a function approximator, allowing them to generalize states and apply Q-learning in continuous state spaces. To save past experiences, the experience replay [38] technique is implemented. This technique saves the most useful information (experience) in memory, which is used afterward to train the RL agent. The neural network is responsible for processing the visual representation and gives the Q-value of each action to the agent, which decides what action to take. In order for the agent to balance exploration and exploitation of actions, the \(\epsilon\)-greedy method is used. This method includes an \(\epsilon\) parameter, which allows the agent to perform either a random exploratory action or the best known action proposed by the policy. The learning process for the autonomous agent, i.e., DeepRL agent, is separated into two stages. The first pretraining stage consists of 1000 random actions that the agent must perform to populate the initial memory. In the second stage, the agent's training is carried out using the \(\epsilon\)-greedy policy and, after each performed action, the agent is trained using 128 tuples considering state, action, reward, and next state, as \(<s_{t},a_{t},r_{t},s_{t+1}>\) extracted from the memory. Both IDeepRL approaches are based on autonomous DeepRL and include the interactive feedback strategy from an external trainer to improve the DeepRL performance. Therefore, the agents have the same base algorithm, adding an extra interactive stage. For the interactive agents, the learning process is separated into three stages. The first pretraining stage corresponds to 900 random actions that the agent must perform in order to populate the initial memory. In the second stage, the external trainer participates giving early advice about the environment dynamics and the intended task during 100 consecutive time-steps. In the third stage, the agent starts training using the \(\epsilon\)-greedy policy, and, following each action selected, the agent is trained with 128 tuples as \(<s_{t},a_{t},r_{t},s_{t+1}>\) saved previously in the batch memory. The learning process for both autonomous and interactive agents is depicted in Figure 2.  
In the second stage of the IDeepRL approach, the learner agent receives advice either from a previously trained artificial agent or from a human trainer. The artificial trainer agent used in agent-IDeepRL is an RL agent that collected previous experience by performing the autonomous DeepRL approach using the same hyperparameters. Therefore, the knowledge is acquired by interacting with the environment in the same manner as the learner agent does. Once the trainer agent has learned the environment, it is used to then provide advice in agent-IDeepRL over 100 consecutive time-steps. Both the trainer agent and the learner agent perceived the environmental information through a visual representation after an action is performed. Figure 2: The learning process for autonomous and interactive agents.