Understanding Generative Adversarial Networks (GANs)
By: Joseph Rocca
Tags: Machine Learning, Neural Networks, Deep Learning, Towards Data Science, Data Science

Generating random variables

In this section, we discuss the process of generating random variables: we remind some existing methods and more especially the inverse transform method that allows to generate complex random variables from simple uniform random variables. Although all this could seems a little bit far from our subject of matter, GANs, we will see in the next section the deep link that exists with generative models.

Uniform random variables can be pseudo-randomly generated

Computers are fundamentally deterministic. So, it is, in theory, impossible to generate numbers that are really random (even if we could say that the question “what really is randomness ?” is a difficult one). However, it is possible to define algorithms that generate sequences of numbers whose properties are very close to the properties of theoretical random numbers sequences. In particular, a computer is able, using a pseudorandom number generator, to generate a sequence of numbers that approximatively follows a uniform random distribution between 0 and 1. The uniform case is a very simple one upon which more complex random variables can be built in different ways.

Random variables expressed as the result of an operation or a process

There exist different techniques that are aimed at generating more complex random variables. Among them we can find, for example, inverse transform method, rejection sampling, Metropolis-Hasting algorithm and others. All these methods rely on different mathematical tricks that mainly consist in representing the random variable we want to generate as the result of an operation (over simpler random variables) or a process.

Rejection sampling expresses the random variable as the result of a process that consist in sampling not from the complex distribution but from a well known simple distribution and to accept or reject the sampled value depending on some condition. Repeating this process until the sampled value is accepted, we can show that with the right condition of acceptance the value that will be effectively sampled will follow the right distribution.

In the Metropolis-Hasting algorithm, the idea is to find a Markov Chain (MC) such that the stationary distribution of this MC corresponds to the distribution from which we would like to sample our random variable. Once this MC found, we can simulate a long enough trajectory over this MC to consider that we have reach a steady state and then the last value we obtain this way can be considered as having been drawn from the distribution of interest.

We won’t go any further into the details of rejection sampling and Metropolis-Hasting because these methods are not the ones that will lead us to the notion behind GANs (nevertheless, the interested reader can refer to the pointed Wikipedia articles and links therein). However, let’s focus a little bit more on the inverse transform method.

The inverse transform method

The idea of the inverse transform method is simply to represent our complex — in this article “complex” should always be understood in the sense of “not simple” and not in the mathematical sense — random variable as the result of a function applied to a uniform random variable we know how to generate.

We consider in what follows a one dimensional example. Let X be a complex random variable we want to sample from and U be a uniform random variable over [0,1] we know how to sample from. We remind that a random variable is fully defined by its Cumulative Distribution Function (CDF). The CDF of a random variable is a function from the domain of definition of the random variable to the interval [0,1] and defined, in one dimension, such that

In the particular case of our uniform random variable U, we have

For simplicity, we will suppose here that the function CDF_X is invertible and its inverse is denoted

(the method can easily be extended to the non-invertible case by using the generalised inverse of the function but it is really not the main point we want to focus on here). Then if we define

we have

As we can see, Y and X have the same CDF and then define the same random variable. So, by defining Y as above (as a function of a uniform random variable) we have managed to define a random variable with the targeted distribution.

To summarise, inverse transform method is a way to generate a random variable that follows a given distribution by making a uniform random variable goes through a well designed “transform function” (inverse CDF). This notion of “inverse transform method” can, in fact, be extended to the notion of “transform method” that consists, more generally, in generating random variables as function of some simpler random variables (not necessarily uniform and then the transform function is no longer the inverse CDF). Conceptually, the purpose of the “transform function” is to deform/reshape the initial probability distribution: the transform function takes from where the initial distribution is too high compared to the targeted distribution and puts it where it is too low.