# Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment
## 4 Results  
In this section, we show the experimental results obtained during the training of three different agents implemented with the three proposed methodologies, i.e., an autonomous agent scenario, a human-agent scenario, and an agent-agent scenario, namely, DeepRL, human-IDeepRL, and agent-IDeepRL. The methodologies are tested with the same hyperparameters, which have been experimentally determined concerning our scenario, as follows: initial value of \(\epsilon=1\), \(\epsilon\) decay rate of 0.9995, learning rate \(\alpha=10^{-3}\), and discount factor \(\gamma=0.9\) during 300 episodes. As discussed in Section 3.1, the first methodology is an autonomous agent using DeepRL, who must learn how the environment works and how to complete the task. Given the complexity of learning the task autonomously, the time required for the learning process is rather high. The average collected reward for ten autonomous agents is shown in Figure 5 represented by the black line. Moreover, this complexity also increases the error rate or misclassification of the objects located in the central table. Next, we perform the interactive learning approaches by using agent-IDeepRL and human-IDeepRL. The average obtained results for ten interactive agents are shown in Figure 5 represented by the blue line and the red line, respectively. The agent-IDeepRL approach performs slightly better than the human-IDeepRL approach, mainly because people needed more time to understand the setup and to react during the experiments. However, both interactive approaches obtain very similar results, achieving much faster convergence when comparing to the autonomous DeepRL approach. Furthermore, the learner agents getting advice from external trainers make fewer mistakes, especially at the beginning of the learning process, and are able to learn the task in fewer episodes. On the other hand, the autonomous agent makes more mistakes at the beginning since it is trying to learn how the environment works and the aim that it has to be accomplished. This is not the case for the interactive agents since the advisors help them during this critical part of the learning. For a qualitative analysis of the obtained results, we have computed the total collected reward \(R_{T}\) defined as the sum of all individual rewards \(r_{i}\) received during the learning process, i.e., \(N=300\) for this case (see Equation (4)). The total collected rewards for the three methods are: autonomous DeepRL \(R_{T}=369.8788\), agent-IDeepRL \(R_{T}=606.6995\), and human-IDeepRL \(R_{T}=589.5974\). Therefore, the methods agent-IDeepRL and human-IDeepRL present an improvement, in terms of collected reward, of 64.03% and 59.40%, respectively, in comparison to the autonomous DeepRL used as a baseline:  
\[R_{T}=\sum_{i}^{N}r_{i} \tag{4}\]  
Due to the trainer having a budget of 100 actions to advise, the interactive feedback is consumed within the first six episodes, taking into account that the minimal amount of actions to complete an episode are 18 actions. Even with such a small amount of feedback, the learner agent receives an important knowledge from the advisor that is complemented with the experience replay method. During the human-IDeepRL approach, to give demonstrative advice, 11 people participated in the experiments, four males and seven females, with ages between 16 and 24 (\(M=21.63,SD=2.16\)). The participants were explained how to help the robot to complete the task giving advice using the same script for all of them (see Section 3.1). Figure 6 shows the collected rewards by an autonomous DeepRL agent and three learner agents trained by different people as examples. All human-IDeepRL approaches work much better than the autonomous DeepRL, making fewer mistakes and, therefore, collecting faster rewards. Between the interactive agents, there are some differences, especially at the beginning of the learning process, within the first 50 episodes.