Beyond Single Point Estimations using Neural Networks | by Elliott Zhu | Towards Data Science

Machine learning algorithms often estimate the first moment (i.e., the mean) of an unknown data generating process. However, applications that go beyond the first moment of distribution fitting can be found in a wide variety of different fields, such as economy [2], engineering [3] and natural sciences [4]. For example, a neural network that is looking at financial markets and attempting to guide investors may calculate the probability of the stock market movement in additional to the direction of the movement. To do so, it could use a Probability Density Function in order to calculate the total probability that the continuous random variable range will occur.
Traditional approaches for density estimation (such as here) are based on the preliminary choice of a statistical model of the function and subsequent fitting on its parameters. For example, a neural network outputs the mean (μ) and the standard deviation (σ) parameters would be sufficient to describe the density function of a normal distribution:
However, what id. the priori knowledge on the data generating process is not available? An alternative approach based on Bayesian statistics was first proposed in this early work by Aristidis Likas [1]. This simple but effective method does not require any assumption on the available data, but extracts the probability density function from the output of a neural network, that is trained with a suitable database including the original data and some ad hoc created data with known distribution. Unfortunately, the original implementation does not fit the modern computation environment, therefore, in this post, I will demonstrate how the method can be used in the context of Python and TensorFlow.
For illustration purpose, we may create some simple data generating process such as the one below, where the outcome is generated using the conditional normal distribution of unit exponential Xs. Therefore, each individual will have their unique probability density function (PDF) conditioned on X.
def hi_sample(N):    fx = lambda x: np.random.normal(loc = np.mean(x[:,0:-1],1), size=N)    X1 = np.random.exponential(1, size=N)    X2 = np.random.exponential(1, size=N)    Y =  fx(np.array([X1,X2]).T)    hi_data = [X1, X2, Y]    return np.array(hi_data).Ttrain_array = hi_sample(5000)
The generated Y will have the following distribution:
Rather than going through all the math jargon in the original work, here, we highlight the essential steps required for implementation in python:
Augment the original dataset with known data generating process (creating a prior distribution), which is from the reference PDF: P(ref). Here, we assume all individuals’ outcome in the augment dataset will be generated from a standard normal distribution:
Augment the original dataset with known data generating process (creating a prior distribution), which is from the reference PDF: P(ref). Here, we assume all individuals’ outcome in the augment dataset will be generated from a standard normal distribution:
train_array_0 = train_array.copy()train_array_0[:,-1] = metropolis_hastings_normal(train_array_0[:,-1]) #regenerate the outcome f(x) using metropolis_hastings simulators with a standard normal distribution. train_data = [train_array,train_array_0]
2. Fit the augmented dataset with a binary discrimination neural network which estimates the probability, y(X), that an observation pair X:=(x,f(x)) is generated from the pr(X) or from the target PDF: p(X). It can be a simple network like this:
def simple_nn(dim_x):    input_x = tfkl.Input(shape=(dim_x+1))    combined_layer = tfkl.Dense(dim_x)(input_x)    for i in range(5):        combined_layer = tfkl.Dense(dim_x)(combined_layer)        combined_layer = tfkl.Dropout(0.1)(combined_layer)    out = tfkl.Dense(1, activation='sigmoid')(combined_layer)    model = tf.keras.Model(inputs = input_x, outputs=out)    model.compile(loss = 'binary_crossentropy',                  optimizer=tf.keras.optimizers.RMSprop(lr=0.001))    return model
3. Compute the target probability density function using Bayes theory:
Thus, training the NN with the purposely augmented data set produces a neural function y(X) from which the unknown PDF p(X) can be inferred, by using an intermediate arbitrary PDF pr(X):
conditional_y = pdf_function_i(x).numpy()reference_dense =  norm.pdf(x[:,-1]).reshape(-1, 1) #norm(loc = np.sum(x[:,0:-1],1)).pdf(x[:,-1]).reshape(-1, 1)conditional_dense = (conditional_y / np.clip(1 - conditional_y, a_min=1e-3, a_max=1.0))  * reference_dense
Now lets plot the estimated PDF (conditional_dense) with the true PDF:
Note here each individual point has a normal distribution conditioned on given X.
Let’s see the estimations of a few other conditional distributions derived from standard density functions:
We are able to get fairly accurate estimation of individual PDFs from different original data generating process using the same standard normal prior. Are we able to further improve the estimation? Yes. In another work by Leonardo Reyneri et al.[5], authors suggested using the estimated PDF, p’(X), from the initial estimation as the prior for the adjustment estimation. That is, we repeat step 1 to 2, with the standard normal prior being replaced by the p’(X), and in step three we estimate the following quantity:
Thus, we can get the one step adjustment of the original estimation of PDF. The intuition behind this is similar as an autoencoder, where the estimated the PDF will be optimal if the neural network cannot distinguish the estimated PDF and the prior PDF.
[1] Likas, A. (2001). Probability density estimation using artificial neural networks. Computer physics communications, 135(2), 167–175.
[2] Combes, C., Dussauchoy, A.: Generalized extreme value distribution for fitting open- ing/closing asset prices and returns in stock-exchange. Springer Operational Re- search 6(1), 3–26 (2006)
[3] Zaharim, A., Razali, A.M., Abidin, R.Z., Sopian, K.: Fitting of Statistical Dis- tributions to Wind Speed Data in Malaysia. European Journal of Scientific Re- search 26(1), 6–12 (2009)
[4] Wendell Cropper Jr., P., Anderson, P.J.: Population dynamics of a tropical palm: use of a genetic algorithm for inverse parameter estimation. Ecological Modelling 177, 119–127 (2004)
[5] Reyneri, L., Colla, V., & Vannucci, M. (2011, June). Estimate of a probability density function through neural networks. In International Work-Conference on Artificial Neural Networks(pp. 57–64). Springer, Berlin, Heidelberg.