Normalization Techniques in Python Using NumPy
By: Chris Morrow
Tags: Getting Started, Data Science, Normalization, Statistics, Python

Normalization Techniques in Python Using NumPy

Photo by Author via Flickr

Data Science isn’t only about developing models. A lot of the work involves cleaning data and selecting features. Plugging features into a model that have similar distributions but significantly different means, or are on vastly different scales can lead to erroneous predictions. A common solution to these problems is to first “normalize” features to eliminate significant differences in mean and variance.

The term “normalization” can be misleading (and also shouldn’t be confused with database normalization), because it has come to mean many things in statistics. There is however, a common theme among normalization techniques which is to bring separate datasets into alignment for easier comparison. The two techniques we’ll focus on are Residual Extraction, which shifts the datasets’ means, and Re-scaling which stretches and squeezes the values in the datasets to fit on a scale from 0 to 1. Needless to say, both of these techniques will eliminate the units applied to the datasets. Thankfully, the shifting and scaling techniques can both be accomplished easily in Python and calculated efficiently using the NumPy Python package.

Extracting Residuals

Let’s first explore the Residual Extraction technique. A residual is the relative difference between a value in a dataset and the dataset’s mean. This technique is useful when you have datasets with similar distributions but significantly different means, making comparisons between the datasets difficult. For example, let’s say we have an exam that’s taken by two different classes of equal size. The questions are the same, in the same order, and have the same answers. However, the average scores between the two classes are different. Class 1 averaged an 82 on the test and Class 2 averaged a 77. How can we combine the scores from the two classes?

Lets start by setting up our Python environment:

import numpy as np

import scipy.stats as st

from sci_analysis import analyze

%matplotlib inline np.random.seed(12)

The code above imports the NumPy package as np , the SciPy stats module as st — which will be used for creating our datasets, the analyze function from the sci_analysis Python package — for graphing results, and lastly, we set the random number generator seed value so that the results are reproducible. The number 12 passed as an argument to np.random.seed() was arbitrarily selected. Now, lets create two datasets, one for each class:

dist1 = st.norm.rvs(82, 4, size=25).astype(int)

dist2 = st.norm.rvs(77, 7, size=25).astype(int)

print(dist1)

print(dist2)

Output:

[83 79 82 75 85

75 82 81 78 93

79 83 86 77 87

81 86 78 77 86

84 82 84 84 77] [61 65 64 61 72

73 76 78 74 75

77 70 72 77 72

72 76 86 79 74

72 76 92 54 80]

Class 1 is represented by dist1 and Class 2 by dist2 . Both variables are NumPy arrays of twenty-five normally distributed random variables, where dist1 has a mean of 82 and standard deviation of 4, and dist2 has a mean of 77 and standard deviation of 7. Both arrays are converted to integers to complete our exam score example. We can visualize the class scores with the code below:

analyze(

{'dist1': dist1, 'dist2': dist2},

title='Different means',

nqp=False,

)

print(f'dist1 mean: {np.mean(dist1)} std dev: {np.std(dist1)}')

print(f'dist2 mean: {np.mean(dist2)} std dev: {np.std(dist2)}')

Output:

dist1 mean: 81.76 std dev: 4.197904239022134

dist2 mean: 73.12 std dev: 7.7785345663563135

From the graph above as well as the mean and standard deviation of dist1 and dist2 , we can tell that the distributions are not matched.

Let’s now calculate the residuals for dist1 and dist2 :

dist1_norm = dist1 - int(np.mean(dist1))

dist2_norm = dist2 - int(np.mean(dist2))

print(dist1_norm)

print(dist2_norm)

Output:

[ 2 -2 1 -6 4

-6 1 0 -3 12

-2 2 5 -4 6

0 5 -3 -4 5

3 1 3 3 -4] [ -12 -8 -9 -12 -1

0 3 5 1 2

4 -3 -1 4 -1

-1 3 13 6 1

-1 3 19 -19 7]

The residuals are the difference between each value and the mean of the value’s distribution. In other words, each residual is the distance from each distribution’s mean which is now zero. Since each distribution now has a mean of zero, they can now be compared directly to one another.

analyze(

{'dist1': dist1_norm, 'dist2': dist2_norm},

title='Normalized',

nqp=False,

)

print(f'dist1 mean: {np.mean(dist1_norm)} std dev: {np.std(dist1_norm)}')

print(f'dist2 mean: {np.mean(dist2_norm)} std dev: {np.std(dist2_norm)}')

Output:

dist1 mean: 0.76 std dev: 4.197904239022134

dist2 mean: 0.12 std dev: 7.7785345663563135

We can see that the means are now near zero (and would be zero if it wasn’t for rounding error when converting the values to integers), but the variance of each distribution hasn’t changed. This shows how calculating the residuals is effectively shifting the mean of each distribution to align them for direct comparison.

Min-Max Re-scaling

Another normalization technique is to re-scale datasets. This is useful when trying to compare datasets of different factors or that use different units, such as comparing miles to meters. Let’s generate two new distributions that are on different scales by a factor of 100:

dist3 = st.gamma.rvs(1.7, size=25)

dist4 = st.gamma.rvs(120, size=25)

print(dist3)

print(dist4)

Output:

[ 0.49529541 1.42598239 0.38621773 0.96738928 0.53575876

1.72574991 0.3431045 0.80584646 0.77543188 1.84272915

2.049985 0.76373308 3.54020309 0.36979422 4.67967817

0.6311116 2.51371776 1.12812921 0.62183125 2.03923847

1.15269735 0.72795499 1.86093872 0.52560778 0.65314453] [113.88768554 108.94661696 118.40872068 124.94416222 150.91953839

116.86987547 107.05486021 89.61392457 126.28254195 123.32858014

108.31036684 114.52812809 109.43092709 114.17768634 114.54545154

111.21616394 102.08437696 127.42455395 105.82224292 127.28966453

114.03632754 120.02256655 120.77792085 103.43640076 112.24143473]

From the output above, you can see that dist3 is on a 0 to 10 scale and dist4 is a factor of 100 greater than dist3 . By checking the mean and standard deviation, we can see that these distributions cannot be compared to one another.

analyze(

{"dist3": dist3, "dist4": dist4},

title="Different Scales",

nqp=False,

)

print(f'dist3 mean: {np.mean(dist3)} std dev: {np.std(dist3)}')

print(f'dist4 mean: {np.mean(dist4)} std dev: {np.std(dist4)}')

Output:

dist3 mean: 1.3024508282047844 std dev: 1.0372657584518052

dist4 mean: 115.42402875045401 std dev: 11.29662689178773

Luckily, there’s a technique to re-scale the distributions by using the ratio of the distance of each value from the minimum value in each dataset to the range of values in each dataset. This ratio is represented by the equation:

(x - min) / (max - min)

By applying this equation in Python we can get re-scaled versions of dist3 and dist4 :

max = np.max(dist3)

min = np.min(dist3)

dist3_scaled = np.array([(x - min) / (max - min) for x in dist3])

max = np.max(dist4)

min = np.min(dist4)

dist4_scaled = np.array([(x - min) / (max - min) for x in dist4])

print(dist3_scaled)

print(dist4_scaled)

Output:

[0.03509474 0.24970817 0.00994177 0.14395807 0.04442546

0.3188336 0. 0.10670681 0.09969331 0.34580864

0.39360118 0.09699561 0.7372407 0.00615456 1.

0.06641352 0.50053647 0.18102418 0.0642735 0.39112306

0.18668952 0.08874529 0.35000771 0.04208467 0.07149424] [0.39594679 0.31534946 0.46969265 0.57629694 1.

0.44459144 0.28449166 0. 0.59812822 0.54994402

0.30497113 0.40639351 0.32324939 0.4006772 0.40667608

0.35236968 0.20341453 0.6167564 0.26438555 0.61455612

0.3983714 0.49601725 0.50833838 0.22546836 0.36909361]

By re-scaling dist3 and dist4 the maximum value for each dataset is now 1 while the minimum is now 0. This is convenient since each dataset is now on the same scale, and the shape of each distribution is preserved. In effect, each distribution has been squeezed and shifted to fit between 0 and 1. Now, lets check the mean and standard deviation of the re-scaled dist3 and dist4 :

analyze(

{'dist3': dist3_scaled, 'dist4': dist4_scaled},

title='Scaled',

nqp=False,

)

print(f'dist3 mean: {np.mean(dist3_scaled)} std dev: {np.std(dist3_scaled)}')

print(f'dist4 mean: {np.mean(dist4_scaled)} std dev:{np.std(dist4_scaled)}')

Output:

dist3 mean: 0.2212221913870349 std dev: 0.2391901615794912

dist4 mean: 0.42100718959757816 std dev: 0.18426741349056594

We can now see that means for dist3_scaled and dist4_scaled are significantly different with similar standard deviations.

Using NumPy for Normalizing Large Datasets

Both residuals and re-scaling are useful techniques for normalizing datasets for analysis. However, these datasets are often more than just 25 values, so it’s worth thinking about how to compute residuals and perform re-scaling efficiently. Thankfully, NumPy can help perform these calculations quickly and efficiently.

If you aren’t familiar with NumPy, it is a Python package for high-performance scientific computing. NumPy has two features that make calculations faster and more efficient compared to using regular Python lists. First, NumPy uses arrays as its primary data structure which uses less memory compared to Python lists, and requires each value in the array to be of the same type. Arrays can also be operated on by scalars — applying the scalar operation to each value of the array. In fact, we already saw this when calculating the residuals by subtracting the mean of the distribution (a scalar) from the distribution (an array).

Let’s compare the difference in speed between calculating residuals using a Python list comprehension and an array operation. We’ll start by creating an array of random, normally distributed variables with 100,000 values:

dist5 = st.norm.rvs(82, 5, size=100000)

Let’s see how long it takes to calculate the residuals of dist5 using a list comprehension:

avg = np.mean(dist5)

%timeit [val - avg for val in dist5]

Output:

10 loops, best of 3: 24.9 ms per loop

The list comprehension takes around 25 milliseconds to iterate through each value in dist5 which isn’t too bad. However, let’s calculate the residuals of dist5 again, but with a NumPy scalar operation:

avg = np.mean(dist5)

%timeit dist5 - avg

Output:

10000 loops, best of 3: 144 µs per loop

Just subtracting the mean from dist5 (which is a NumPy array) takes 144 microseconds! This is thanks to the efficient design of the NumPy array.

Now, what if we want to operate on an array with a function and not just a simple scalar? This is NumPy’s second performance boost over Python in the form of universal functions. The universal functions are “vectorized” operations that take advantage of CPU optimizations to achieve computational speedups when iterating over each value in an array. NumPy has numerous built-in universal functions, but we can also write our own by using NumPy’s frompyfunc() function.

Let’s look at the difference in speed between re-scaling the values of an array using a list comprehension and a custom universal function:

min = np.min(dist5)

max = np.max(dist5)

%timeit [(val - min) / (max - min) for val in dist5]

Output:

10 loops, best of 3: 47.2 ms per loop

The list comprehension performs admirably by re-scaling dist5 in 47 milliseconds. Let’s compare this to creating a custom universal function called scale .

scale = np.frompyfunc(lambda x, min, max: (x - min) / (max - min), 3, 1)

We use the np.frompyfunc() function which accepts a callable, the number of inputs, and the number of outputs as its arguments. The callable in this case is our re-scaling equation in the form of a lambda function. If you aren’t familiar with lambda functions, you can think of them as one-line, unnamed functions. Our lambda function has three arguments, x , min , and max . These three arguments are the three inputs specified when calling np.frompyfunc() . Also worth noting, min and max are scalars calculated from dist5 while x represents each value in dist5 . Now, let’s calculate min and max and call the scale function:

min = np.min(dist5)

max = np.max(dist5)

%timeit scale(dist5, min, max).astype(float)

Output:

10 loops, best of 3: 20.4 ms per loop

As you can see, the custom universal function takes half the time to complete compared to the list comprehension.

Conclusion

We examined two normalization techniques — Residual Extraction and Min-Max Re-scaling. Residual Extraction can be thought of as shifting a distribution so that it’s mean is 0. Min-Max Re-scaling can be thought of as shifting and squeezing a distribution to fit on a scale between 0 and 1. Residual Extraction is useful for comparing distributions with different means but similar shapes. Min-Max Re-scaling is useful for comparing distributions with different scales or different shapes.

Both of these normalization techniques can be performed efficiently with NumPy when the distributions are represented as NumPy arrays. Scalar operations on NumPy arrays are fast and easy to read. When more complex operations on arrays are needed, a universal function can be used to perform the operation efficiently.