# Boosting Deep Reinforcement Learning Agents with Generative Data Augmentation
## 2 Background
### Denoising Diffusion Probabilistic Models  
Denoising diffusion probabilistic models (DDPMs) [16] are a class of generative models used to produce novel samples by learning the distribution of the training data points. The main idea is to diffuse the data by progressively adding noise to the samples and then train the model to correctly predict the probability density function of the noise in order to remove it and reproduce the original data. The training process is split in two phases. In the forward diffusion process, noise sampled from a Gaussian distribution is iteratively diffused into the data until they are completely mapped to another, more simple distribution. Subsequently, a neural network models the reverse process that makes it capable of generating new samples, by starting from random noisy data and progressively mapping them to the distribution of the training data. Formally, let \(q(x)\) be the underlying distribution of the training data, and \(x_{0}\) a sample drawn from it (\(x_{0}\sim q(x)\)). The distribution of the forward diffusion process is determined in Equations (5) and (6). During the forward process, at each step \(t\), a new latent variable \(x_{t}\) is produced by adding Gaussian noise of variance \(\beta_{t}\) to the previous variable \(x_{t-1}\), as described in Equation (7), where \(\epsilon\sim\mathcal{N}(0,\mathbf{I})\). \[q(x_{t}|x_{t-1}) = \mathcal{N}(x_{t};\mu_{t}=x_{t-1}\sqrt{1-\beta_{t}},\Sigma_{t}= \beta_{t}\mathbf{I}) \tag{5}\] \[q(x_{1:T}) = q(x_{1}|x_{0})q(x_{2}|x_{1})...q(x_{T}|x_{T-1})=\prod_{t=1}^{T} q(x_{t}|x_{t-1})\] (6) \[x_{t} = \sqrt{1-\beta_{t}}x_{t-1}+\sqrt{\beta_{t}}e_{t} \tag{7}\]  
By setting \(a_{t}=1-\beta_{t}\), \(\tilde{a_{t}}=\prod_{i=1}^{t}a_{i}\) and using the reparameterization trick [17], \(x_{t}\) can be sampled directly, without calculating all intermediate latent variables. Given the starting data point \(x_{0}\), the distribution \(q\) takes the form of Equation (8) and the latent variable after \(T\) timesteps can be calculated as shown in Equation (9). \[q(x_{t}|x_{0}) = \mathcal{N}(x_{t};\mu_{t}=x_{0}\sqrt{\tilde{a_{t}}},\Sigma_{t}=( 1-\tilde{a_{t}})\mathbf{I}) \tag{8}\] \[x_{t} = \sqrt{\tilde{a_{t}}}x_{0}+\sqrt{1-\tilde{a_{t}}}e_{0} \tag{9}\]  
For the reverse process, a neural network is employed to model the distribution \(p\) (i.e., the parameters of another Gaussian) that progressively removes the noise and maps any random sample to the original distribution. The goal is to minimize the distance (Kullback-Leibler divergence) between \(p_{\theta}\) and the forward posterior distribution \(q(x_{t-1}|x_{t},x_{0})\), which is defined in Equation (10). \[q(x_{t-1}|x_{t},x_{0})=\mathcal{N}(x_{t-1};\tilde{\mu_{t}}(x_{t},x_{0}),\tilde {\beta_{t}}\mathbf{I}) \tag{10}\]  
where \(\tilde{\mu_{t}}(x_{t},x_{0})=\frac{\sqrt{\tilde{a_{t}}-1}\tilde{\mu_{t}}}{1- \tilde{a_{t}}}x_{0}+\frac{\sqrt{\tilde{a_{t}}}(1-\tilde{a_{t-1}})}{1-\tilde{a _{t}}}x_{t}\) and \(\tilde{\beta_{t}}=\frac{1-\tilde{a_{t-1}}}{1-\tilde{a_{t}}}\beta_{t}\). By setting the variance to a fixed constant \(\beta_{t}\) and using the reparameterization trick to express \(x_{0}\) in terms of \(x_{t}\), \(\tilde{\mu_{t}}\) is defined as follows:  
\[\tilde{\mu_{t}}(x_{t})=\frac{1}{\sqrt{\tilde{\kappa}_{t}}}(x_{t}-\frac{\beta_ {t}}{\sqrt{1-\tilde{a_{t}}}})e_{t} \tag{11}\]  
As a result, the neural network only needs to predict the noise \(e_{\theta(x_{t},t)}\) at each timestep in order to minimize the following loss function:  
\[L(\theta)=\mathbb{E}_{t,x_{0},\epsilon}[\|\epsilon-e_{\theta}(x_{t},t)\|^{2}] \tag{12}\]Concerning the model's architecture, while the only restriction is that the input and output of the network are of the same dimensionality, the most common approach is a U-net-based neural network with residual and self-attention blocks. The model takes as input the timestep and the respective sample (image) and predicts the noise at that point.