# Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment
## 2 Related Works
### Deep Reinforcement Learning and Interactive Feedback  
Deep reinforcement learning (DeepRL) combines reinforcement Learning (RL) and deep neural networks (DNN). This combination has allowed the enhancement of RL agents when autonomously exploring a given scenario [15]. If an RL agent is learning a task, the environment gives the agent the necessary information on how good or bad the taken actions are. With this information, the agent must differentiate which actions lead to a better accomplishment of the task aims [5]. Aims may be expressed by a reward function that assigns a numerical value to each performed action from a given state. Additionally, after performing an action, the agent reaches a new state. Therefore, the agent associates states with actions to maximize \(r_{0}+\gamma\cdot r_{1}+\gamma^{2}\cdot r_{2}+...\), where \(r_{i}\) is the obtained reward in the \(i\)-th episode and \(\gamma\) is the discount factor parameter that indicates how influential future actions are. The fundamental base of RL tasks is the Markov decision process (MDP), in which future transitions and rewards are affected only by the current state and the selected action [16]. Therefore, if the Markovian property is present for a state, this state contains all the information needed about the dynamics of a task. For instance, chess is a classic example of the Markov property. In this game, it does not matter the history of plays in order to make a decision about the next movement. All the information is already described in the current distribution of pieces over the board. In other words, if the current state is known, the previous transitions that led the agent to that situation become irrelevant in terms of the decision-making problem. Formally, an MDP is defined by a 4-tuple \(<S,A,\delta,r>\) where:  
* \(S\) is a finite set of system states, \(s\in S\);
* \(A\) is a finite set of actions, \(a\in A\), and \(A_{s_{t}}\in A\) is a finite set of actions available in \(s_{t}\in S\) at time \(t\);
* \(\delta\) is the transition function \(\delta:S\times A\to S\);
* \(r\) is the immediate reward (or reinforcement) function \(r:S\times A\to\mathbb{R}\). Each time-step \(t\), the agent perceives the current state \(s_{t}\in S\) and chooses an action \(a_{t}\in A\) to be carried out. The environment gives a reward \(r_{t}=r(s_{t},a_{t})\) and the agent moves to the state \(s_{t+1}=\delta(s_{t},a_{t})\). The functions \(r\) and \(\delta\) depend only on the current state \(s_{t}\) and action \(a_{t}\), therefore, it is a no memory process. Over time, the agent tries to learn a policy \(\pi:S\to A\) which, from a state \(s_{t}\), yields the greatest value or discounted reward [5], as shown in Equation (1):  
\[Q^{\pi}(s_{t},a_{t})=r_{t}+\gamma\cdot r_{t+1}+\gamma^{2}\cdot r_{t+2}+...= \sum_{i=0}^{\infty}\gamma^{i}\cdot r_{t+1} \tag{1}\]  
where \(Q^{\pi}(s_{t},a_{t})\) is the action-value function following the policy \(\pi\) (e.g., choosing action \(a_{t}\)) from a given state \(s_{t}\). The discount factor \(\gamma\) is a constant (\(0\leq\gamma<1\)), which determines the relative importance of immediate and future rewards. For instance, in case \(\gamma=0\), then the agent is short-sighted and maximizes only the immediate rewards, or in case \(\gamma\to 1\) the agent is more foresighted in terms of future reward. The final goal of RL is to find an optimal policy (\(\pi^{*}\)) mapping states to actions in order to maximize the future reward (\(r\)) over a time (\(t\)) with a discount rate (\(\gamma\in[0,1]\)), as shown in Equation (2). In the equation, \(\mathbb{E}_{\pi}[]\) denotes the expected value given that the agent follows policy \(\pi\) and \(Q^{*}(s_{t},a_{t})\) denotes the optimal action-value function [5]. Table 1 summarizes all the elements shown within Equation (2).