The Definitive Way to Deal With Continuous Variables in Machine Learning | by Riccardo Di Sipio | Towards Data Science

Data analysis can be seen as a device (or a black box; or a Chinese room) which, given some inputs, provides outputs. The main difference between rule-based and statistics-based (aka machine learning) is that the former accepts training examples and rules (“algorithms”) as inputs and gives predictions as output, while the latter expects training examples and labels as inputs and gives rules as output. That is, in machine learning the rules are inferred from the relationship between input data and labels, while in the classical approach the rules are pre-determined, usually by humans who spent years trying to figure them out. The difference is not as subtle as one may think: for example, in science the rules are often encoded in programs that perform a simulation of the physical system under investigation (e.g. Earth’s climate or interactions between particles). On the other hand, if those rules are not known beforehand as pre-existing theoretical frameworks, the other option is likely to be only one that makes sense. It is worth mentioning that in some notable cases such as protein folding, a machine learning approach called AlphaFold has been proven to be at least as successful as more traditional means, unlocking new exciting possibilities.
Focusing on the machine learning approach, one very common issue is how to deal with heterogeneous data. In most cases, one ends up with a collection of inputs that are either categorical labels (e.g. cat / dog / fish, hired / dismissed) and possibly mutually exclusive, or continuous (e.g. salary, temperature, distance, height, weight...). If you think the two groups have to be treated in different ways because of their intrinsic nature, think twice or keep reading.
To begin with, let’s review briefly how categorical inputs are dealt with. The most straightforward way is to attach a numerical (integer) label to each category, e.g. dog = 1, cat = 2 and fish = 3 . While this might work in principle, in practice it is commonly known that the statistical model (henceforth just model) may spot a fictitious pattern due to the fact that e.g. fish > dog (because 3 > 1 ), which is unlikely to make any sense. To avoid such problem, it is commonly suggested to use a one-hot encoding. In this method, each category is represented by a vector of N elements (where N = number of categories) that are all 0s except for a 1 placed at the index corresponding to the given category. In our example dog = (1,0,0) , cat = (0,1,0) and fish = (0,0,1) . In this case, you need to expand the categorical feature into N binary features (or N-1 if the “Dummy Variable Trap” or “multicollinearity” can be an issue in the problem under consideration). However, this is not the end of the story. These indices are to be used in a lookup table, which elements will be further processed by the model to match the output presented during the training. If you are not familiar with the concept, a lookup table is a NxM matrix, where the number of rows match the number of categories N, and the M columns span a parameter space often referred to as the vector embedding space. The lookup table is called embedding matrix. For example, if the matrix is:
A = [     [0.1, -0.2, 0.3],      [-0.2, 0.3, 0.1],     [0.7, -0.4, -0.2]]
then the vector embedding for cat is (-0.2, 0.3, 0.1) . In short, this process transforms a sparse categorical to a dense representation, which can be further manipulated numerically e.g. by the means of a recurrent or a convolutional neural network depending on the nature of data. As for the embedding matrix itself, it is either learned during training, or is calculated elsewhere by other means (in this case it is said to be pre-trained).
Now, let’s go back to our original problem of continuous variables. If your model is based on decision trees, you are probably good to go without any manipulation. If the model is based on a neural network, things are less clear. In general, it is common sense that neural network do not like unnormalized numerical data, so some kind of standardization is in order. What to do exactly is a matter of discussion, and usually depends on a number of factors.
To begin with, one may simply scale (“min-max scaling” or “normalization”) the input feature to a fixed range which is usually (0,1) and then send it downstream to the rest of the model:
It is a simple approach which main drawback is that it would squash everything to a narrow range in case the distribution is unevenly spread across the range. Quite often, manuals recommend another procedure usually referred to as standardization, which entails subtracting the mean of the distribution and dividing by the standard deviation:
For this procedure to make sense, data have to be distributed roughly as a Gaussian distribution. In the example below, the hypothetical feature a) can be either normalized or standardized successfully, but feature b) is more likely to be problematic.
There is also another aspect that has to be considered. Some features may be non-negative, or limited below a given value, but unbounded otherwise (e.g. salary). Others might be unbounded both ways (e.g. distance to the left or to the right) or even present symmetry around 0 (as many trigonometric functions). Moreover, the training data may not have a sufficient number of examples at the extremes, so that the scaling parameters (mean, min/max, std) may not be representative of the larger population. Any outlier appearing at inference time would not be treated correctly, possibly giving rise to unintended behaviour or puzzling results (for X > Xmax one has X’>1).
I believe there is a robust way to standardize data, it’s not new, but for some reason it’s seldom mentioned when this topic is discussed. These are the steps:
Pool data into N bins to build a histogram of the feature. If the amount of training data allows it, find the optimal boundaries by calculating the quantiles of the distribution (in this case, bins will be spaced unevenly);Replace the input feature with the bin index and treat it as a categorical input (i.e. which bin does this value correspond to?);Use a lookup table to extract dense vector embeddings for each bin;Pass the vector embedding downstream.
Pool data into N bins to build a histogram of the feature. If the amount of training data allows it, find the optimal boundaries by calculating the quantiles of the distribution (in this case, bins will be spaced unevenly);
Replace the input feature with the bin index and treat it as a categorical input (i.e. which bin does this value correspond to?);
Use a lookup table to extract dense vector embeddings for each bin;
Pass the vector embedding downstream.
If you follow these simple rules, it would not matter if input features have different ranges (e.g. salary and distance from the workplace), outliers do not result in invalid outputs and correlations between different features are preserved to a large extent. Besides, there is a theoretical motivation why histograms are more robust than numerical values. In real life, continuous input features are the result of a measurement which in principle comes with an uncertainty associated with it, but this aspect is usually neglected for the sake of simplicity. And even if that’s not the case (think of salary), chances are it makes more sense intuitively to group inputs into separate categories anyway (e.g. low vs high income). It is also worth mentioning an extension of this method in which the distribution is fitted with a functional form (e.g. polynomial or exponential), and then the quantiles of the fitted function are used instead of the histogram. Finally, with this approach, regression tasks are turned into classification tasks: by quantizing the target distribution, the name of the game is to identify the corresponding bin and not the “exact” numerical value. You might have heard that all classification problems are in fact regression problems with discrete target variables, which is certainly true. But so is its opposite.
Good news is: there is a KBinsDiscretizer class in the scikit-learn library that implements most of the things discussed above.
To make things clearer, I prepared a simple implementation of a quantized features embedding layer based on TensorFlow 2.x. For a full test drive, feel free to fork this Google Colab notebook.
To begin with, you need to quantize the input features. I strongly suggest to use the KBinsDiscretizer class from scikit-learn , but the same procedure can be recreated by making use of the numpy.quantile function.
The input features may look like these:
After quantization with 5 bins for each feature, we get something like this:
At this point, we need to create an extension of the Embedding layer from TensorFlow / Keras . In the original implementation, the embedding matrix is a 2D object indexed by the token id. In our case, we need an extra dimension indexing which feature the embedding refers to, so in total this is a 3D tensor. Luckily for us, the gather_nd function lets us select entries from a 3D tensor using a 2D tensorial index. It sounds complicated, but it isn’t: all it does is to select the vector embedding in a 2D lookup table indexed by (feature id, token id):
You can use this layer as you would with a regular embedding layer, for example:
Training a model with 5 bins per feature for 100 epochs yields acceptable results, with some sign of overfitting in the loss, but still at a reasonable level considering that we didn’t do any optimization. The final (sparse) accuracy is around 65%.
At this point, if you want to turn the predicted bin into an actual number (in this case, the house price), you can throw a random number (uniform or gaussian) within the bin’s boundary, or provide just the mean (i.e. the bin centre). We can argue that the more the bins, the more accurate the final prediction will be, but you also needs more training examples in order to fit a larger number of parameters.
To wrap up, feature quantization (or discretization) is a technique that allows one to manipulate continuous and possibly unbounded inputs in a consistent way, and to turn regression problems into classification problems.
A final word concerning generative models: I myself have been involved in a project to develop a GAN to simulate interactions between particles at the Large Hadron Collider [JHEP08(2019)110], where some input features span several order of magnitudes, some are positive-only but unbounded, others are bounded and show an axial symmetry. At the time, I wasn’t aware of this technique. In hindsight, feature quantization could have spared us lots of thinking and troubles. Similarly, I suspect this method can turn out to be useful for autoencoders.