# Efficient Estimation of Generative Models Using Tukey Depth
## 2 Approximate Bayesian Computing  
Let \(X\) represent a \(p\)-dimensional random vector with probability distribution \(P(x|\theta)\) and with unknown parameters \(\theta\). Furthermore, let \(P(\theta)\) denote the prior distribution for unknown parameters. Given a set of observations (or a dataset) \(\mathbf{x}=x_{1},\ldots,x_{n}\), the most common approach to estimate the unknown parameters is to compute the posterior distribution using [PERSON] theorem \[P(\theta|\mathbf{x})\propto P(\mathbf{x}|\theta)P(\theta), \tag{1}\]  
where \(P(\mathbf{x}|\theta)\) is the likelihood function for the observations. However, if the likelihood function cannot be computed, the posterior distribution cannot be computed. If it, on the other hand, is possible to generate samples from the likelihood function, ABC algorithms can be used to generate samples from an approximation of the posterior, which again can be used to analyse properties of the posterior distribution. The ABC method is based on the principle that two datasets which are close in terms of summary statistics are likely to have been generated by similar parameters. The idea is as follows: We want to learn about the values of the parameters \(\theta\) that could have generated our observed data, \(\mathbf{x}\). So, we generate data \(\widetilde{\mathbf{x}}\) from our model using parameters \(\widetilde{\theta}\) drawn from the prior distribution. If \(\widetilde{\mathbf{x}}\) is \"close enough\" to our observed data, \(\mathbf{x}\), we reason that \(\widetilde{\theta}\) could be a plausible value of the true parameters. This is the core idea behind ABC. The term \"close enough\" in this context refers to how similar \(\widetilde{\mathbf{x}}\) to the observed data. It is, however, usually challenging to compare every single data point in \(\mathbf{x}\) and \(\widetilde{\mathbf{x}}\). We therefore use a set of statistics and compute it for the observations, \(\mathbf{S}(\mathbf{x})=S_{1}(\mathbf{x}),\ldots,S_{k}(\mathbf{x})\), and for the generated samples \(\mathbf{S}(\widetilde{\mathbf{x}})\), and measure the distance using some metric \(\rho\). We control how strict or lenient we are in accepting \(\widetilde{\theta}\) as plausible through the use of some threshold \(\epsilon\). If we set \(\epsilon\) to be very small, we only accept \(\widetilde{\theta}\) if the generated data is extremely close to our observed data. On the contrary, a larger \(\epsilon\) makes us more lenient. However, there is a trade-off. The smaller the \(\epsilon\), the more accurate the approximation of the posterior distribution becomes, but it is also more difficult (i.e., more computationally expensive) to find acceptable \(\widetilde{\theta}\)'s. Conversely, a larger \(\epsilon\) makes the algorithm faster, but the approximation of the posterior is coarser. Hence, the choice of \(\epsilon\) is crucial in ABC methods, and there is ongoing research on automated and adaptive ways to choose this threshold. The use of summary statistics and a metric \(\rho\) in the ABC is a practical way to approximate the Bayesian posterior distribution, providing a way to carry out Bayesian analysis when the likelihood function is not available or is too computationally expensive to use directly. The simplest and most commonly used ABC algorithm is the rejection algorithm. Each iteration of the rejection algorithm consists of three steps. First, a sample \(\widetilde{\theta}\) is generated from the prior distribution. Second, a random sample \(\widetilde{\mathbf{x}}=\widetilde{x}_{1},\ldots,\widetilde{x}_{n}\) is generated from the likelihood function using parameter values \(\widetilde{\theta}\), \(P(\widetilde{\mathbf{x}}|\widetilde{\theta})\). Finally, if the distance between the \(\mathbf{S}(\mathbf{x})\) and \(\mathbf{S}(\widetilde{\mathbf{x}})\) is less than some chosen threshold \(\epsilon\) using some suitable metric \(\rho\), \(\rho(\mathbf{S}(\mathbf{x}),\mathbf{S}(\widetilde{\mathbf{x}}))<\epsilon\), then \(\widetilde{\theta}\) is accepted as an approximate sample from the posterior distribution.