Simple Regression Example
By: Jake Batsuuri
Tags: Data Science, Deep Learning, Keras, Python, Machine Learning

Simple Regression Example

K-Fold Validation, Normalization, MAE

Overview

Basics:

Goal

Input & Output

Encoding & Decoding

Architecture

Regularization

Validation

Code:

Import

Model Definition

K-Fold Validation

Validation

Final Model

Goal

We will try to predict the median house price given 13 different parameters. The parameters are attributes such as crime rate, property tax rate, square footage etc.

Input & Output

You can learn more about the data set here and here.

The input variables in order are:

CRIM per capita crime rate by town ZN proportion of residential land zoned for lots over 25,000 sq.ft. INDUS proportion of non-retail business acres per town CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) NOX nitric oxides concentration (parts per 10 million) RM average number of rooms per dwelling AGE proportion of owner-occupied units built prior to 1940 DIS weighted distances to five Boston employment centres RAD index of accessibility to radial highways TAX full-value property-tax rate per $10,000 PTRATIO pupil-teacher ratio by town B 1000(Bk — 0.63)² where Bk is the proportion of blacks by town LSTAT % lower status of the population MEDV Median value of owner-occupied homes in $1000's

If number 12 is what I think it is, that’s really fucked up. Anyway. The output should be the median price in the 1000’s. If we used activation functions in classification, we do no such thing for regression since the output should be a real-valued number.

Encoding & Decoding

In this stage, we don’t necessarily encode so much as normalize the data. The reason we do this is because each attribute has different ranges, which makes learning more difficult.

So we normalize each attribute or feature by making its mean 0 and a standard deviation 1.

Architecture

We will use a relatively small network of 2 hidden layers, each with 64 units. The considerations for this are the size of the data set and the normalization step.

Having a small data set makes overfitting more likely, and a small network is one way to deal with overfitting. And normalizing the data set makes it a bit easier to learn.

These two are the necessary and sufficient conditions.

For the loss function, we will use the MSE or the Mean Squared Error.

Regularization

Our primary method of regularization is early stopping again. But this time, we will use a metric of MAE or the Mean Absolute Error, which is the absolute value of the difference between the predictions and the actual prices.

Validation

Generally, with a bigger data set, we partition it into training and validation sets. But what would happen here is that our validation set would end up being too small. And smaller sets often give a high variance, which isn’t great for validation.

So we use a k-fold validation method. Here we partition the data into k groups, where k is often 3 to 5. And we train on the k-1 groups and evaluate with the last group. Then we repeat this process k times. We are changing the evaluation group every time.

During each iteration, we get a validation score. After all the iterations, we average the scores to get a final validation score, like a GPS measurement error reduction.