# Boosting Deep Reinforcement Learning Agents with Generative Data Augmentation
## 2 Background  
### RL as Markov Decision Process  
A reinforcement learning task where, considering a sequence of discrete timesteps \(t\), an intelligent agent makes decisions based on its interaction with the environment can be described as a Markov decision process (MDP) [15]. Formally, an MDP is represented by a tuple \(\langle S,A,P,R,\gamma\rangle\) where:  
* \(\mathbf{S}\) is the state space, i.e., the set of all possible states;
* \(\mathbf{A}\) is the action space, i.e., the set of all possible actions;
* \(\mathbf{P_{a}(s,s^{\prime})}:S\times A\times S\rightarrow[0,1]\) is the transition function which determines the probability that given a current state \(s\) and an action \(a\), the next state will be \(s^{\prime}\). Thus, \(P_{a}(s,s^{\prime})=Pr(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)\);
* \(\mathbf{R_{a}(s)}:S\times A\rightarrow\mathbb{R}\) is the reward function which determines the immediate reward obtained by executing action \(a\) while being in state \(s\);
* \(\boldsymbol{\gamma}\in[0,1]\) is a discount factor used to calculate the expected return \(\mathbb{E}[G_{t}]\). In this context, the Markov property is satisfied, as the transition from a state \(s\) to a state \(s^{\prime}\), as well as the immediate reward \(r\) after executing an action \(a\), depend solely on this action and the state \(s\) (Equation (1)). In this respect, the state representation at each timestep \(t\) describes completely the properties of the state and it is independent of the path that led to it. \[P(s_{t+1}=s^{\prime},r_{t+1}=r|s_{t},a_{t})=P(s_{t+1}=s^{\prime},r_{t+1}=r|s_{ t},a_{t},r_{t},s_{t-1},\ldots,r_{1},s_{0},a_{0}) \tag{1}\]  
At each discrete timestep \(t\), the agent selects an action \(a_{t}\in A\), under a state representation \(s_{t}\in S\), and transits to a new state \(s_{t+1}\in S\) according to the environment's dynamics, as determined by \(P\). During this transition, the agent obtains an immediate reward \(r_{t}\in R\) based on the properties of the specific environment. The goal of the agent is to determine a policy for selecting actions that maximizes the expected return value (Equation (2)):  
\[Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}[G_{t}|s_{t},a_{t}]=\mathbb{E}_{\pi}[ \sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}|s_{t},a_{t}] \tag{2}\]  
for each state-action pair, that is a policy \(\pi\) such that \(Q_{\pi}(s,a)\geq Q_{\pi^{\prime}}(s,a)\) for all \(s\in S\), \(a\in A\) and any possible policy \(\pi^{\prime}\). The role of \(\gamma\) is to progressively reduce the effect of future rewards, yet take them into account in the expected reward. Normally \(\gamma\) takes values in the range \([0,1]\); in the extreme case of \(\gamma=0\) only the immediate reward is considered, while in the case of \(\gamma=1\) all rewards are treated equally, independently of when they are obtained. The optimal Q-value function \(Q^{*}(s,a)=\max\limits_{\pi}Q_{\pi}(s,a)\) satisfies the Bellman optimality equation (Equation (3)):  
\[Q^{*}(s_{t},a_{t})=\mathbb{E}[r_{t+1}+\gamma\max\limits_{a_{t+1}}Q^{*}(s_{t+1 },a_{t+1})] \tag{3}\]  
which indicates that, given the Q-value is known for all actions in the next state, the best action in the current state is the one maximizing the quantity \(r+\gamma Q^{*}(s^{\prime},a^{\prime})\). In order to determine the Q-value function in a real reinforcement learning task, a neural network is used as a function approximator such that \(Q(s,a;\theta)\approx Q^{*}(s,a)\), where \(s\in S\) is a state representation of the environment, \(a\in A\) is a possible action, and \(\theta\) are the weights of the model. The loss function (which is derived from the Bellman optimality equation) used for the neural network's training in the deep Q-learning algorithm is as follows (Equation 4)):  
\[L(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim U(D)}[(r+\gamma\max\limits_{a^{ \prime}}Q(s^{\prime},a^{\prime};\theta)-Q(s,a;\theta^{-}))^{2}] \tag{4}\]The samples \((s,a,r,s^{\prime})\) are drawn uniformly from a replay buffer \(D\), where they are stored during the agent's interaction with the environment.