Deep Quantile Regression in Tensorflow | by Jacob Zweig | Towards Data Science

A key challenge in deep learning is how to get estimates on the bounds of predictors. Quantile regression, first introduced in the 70’s by Koenker and Bassett [1], allows us to estimate percentiles of the underlying conditional data distribution even in cases where they are asymmetric, giving us insight on the relationship of the variability between predictors and responses.
As described by Koeneker and Hallock [2], the standard approach in OLS regression is to estimate the conditional mean of the data by drawing a line which minimizes distance between the line and all of the points on the graph. In contrast, the goal in quantile regression is to estimate the conditional median, or any other quantile. To do so, we instead weight the distances between points on the graph and our regression line based on the selected quantile. For example, if we selected the 90th quantile to estimate, we’d fit a regression line so that 90% of the data points are below the line and 10% are above.
A great recent post on medium by Sachin Abeywardana demonstrated how to perform deep quantile regression using Keras. However, a limitation of this implementation is that it requires separate fitting of models per quantile. Using Tensorflow, however, we can instead fit an arbitrary number of quantiles simultaneously.
The notebook for this implementation can be found on the Strong Analytics github.
We first instantiate a simple regression model, and add outputs for each of the quantiles we’d like to estimate
# Define the quantiles we’d like to estimatequantiles = [.1, .5, .9]# Add placeholders for inputs to the modelx = tf.placeholder(tf.float32, shape=(None, in_shape))y = tf.placeholder(tf.float32, shape=(None, out_shape))# Create model using tf.layers APIlayer0 = tf.layers.dense(x, units=32, activation=tf.nn.relu)layer1 = tf.layers.dense(layer0, units=32, activation=tf.nn.relu)# Now, create an output for each quantileoutputs = []for i, quantile in enumerate(quantiles):    output = tf.layers.dense(layer1, 1, name=”{}_q{}”.format(i, int(quantile*100)))    outputs.append(output)
Now, we create the losses for each quantile
losses = []for i, quantile in enumerate(quantiles):   error = tf.subtract(y, output[i])   loss = tf.reduce_mean(tf.maximum(quantile*error, (quantile-1)*error), axis=-1)   losses.append(loss)
Finally, we combine the losses and instantiate our optimizer
combined_loss = tf.reduce_mean(tf.add_n(losses))train_step = tf.train.AdamOptimizer().minimize(combined_loss)
And that’s basically it! Now we can fit our model and simultaneously estimate all of the specified quantiles:
for epoch in range(epochs):   # Split data to batches   for idx in range(0, data.shape[0], batch_size):     batch_data = data[idx : min(idx + batch_size, data.shape[0]),:]     batch_labels = labels[idx : min(idx + batch_size, labels.shape[0]),:]    feed_dict = {x: batch_data,                 y: batch_labels}    _, c_loss = sess.run([train_step, combined_loss], feed_dict)
Fitting this data on a dataset of motorcycle crashes (from here) gives us the following output:
1. Regression Quantiles. Roger Koenker; Gilbert Bassett, Jr. Econometrica, Vol. 46, No1. (Jan., 1978), pp. 33–50.
2. Quantile Regression. Roger Koenker; Kevin F. Hallock. Journal of Economic Perspectives · vol. 15, no. 4, Fall 2001. (pp. 143–156).
Want to work on challenging Machine Learning and AI in a variety of industries with a team of top data scientists in Chicago? We’re hiring talented data scientists and engineers!
Learn more at strong.io and apply at careers.strong.io