Feature Transformation for Multiple Linear Regression in Python | by Bonnie Ma | Towards Data Science

Data processing and transformation is an iterative process and in a way, it can never be ‘perfect’. Because as we gain more understanding on the dataset, such as the inner relationships between target variable and features, or the business context, we think of new ways to deal with them. Recently I started working on media mix models and some predictive models utilizing multiple linear regression. In this post, I will introduce the thought process and different ways to deal with variables for modeling purpose.
I will use King County house price data set (a modified version for more fun) as an example.
Let’s import libraries and look at the data first!
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsimport scipy.stats as stats% matplotlib inlinedf = pd.read_csv(“kc_house_data.csv”)df.head()
Identify missing values, and obvious incorrect data types.
df.info()<class 'pandas.core.frame.DataFrame'>RangeIndex: 21597 entries, 0 to 21596Data columns (total 21 columns):id               21597 non-null int64date             21597 non-null objectprice            21597 non-null float64bedrooms         21597 non-null int64bathrooms        21597 non-null float64sqft_living      21597 non-null int64sqft_lot         21597 non-null int64floors           21597 non-null float64waterfront       19221 non-null float64view             21534 non-null float64condition        21597 non-null int64grade            21597 non-null int64sqft_above       21597 non-null int64sqft_basement    21597 non-null objectyr_built         21597 non-null int64yr_renovated     17755 non-null float64zipcode          21597 non-null int64lat              21597 non-null float64long             21597 non-null float64sqft_living15    21597 non-null int64sqft_lot15       21597 non-null int64dtypes: float64(8), int64(11), object(2)memory usage: 3.5+ MB
In most statistical models, variables can be grouped into 4 data types:
Continuous: it can have an infinite number of possible values within a selected range. ‘float’ is usually used for continuous data in python. e.g. the footage of the living area
Nominal: nominal variables use a numerical representation to interpret types or attributes of objects. They are categorical values with 2 or more possible values and have no inherent order or ranking sequence. For example, origin country of cars, US can be 1, Japan 2, Germany 3
Ordinal: ordinal variables are actually presenting numerical values. They usually have 2 or more possible values in a limited range and these values have ordered categories. e.g. grade of house. When you plot ordinal values with target, you will often see clear vertical line. Ordinal data might be read as integer or float in your dataset so data visualization is always helpful to detect them
Binary: only have 2 possible values usually 0 and 1. Dummy variables are binary
Below chart shows clearly the relationship.
Log: Log transformation helps reducing skewness when you have skewed data.
Take house price as an example.
df_log[‘price’] = np.log(df[‘price’])sns.distplot(df_set['price'], fit=norm)fig = plt.figure()
Power: if we know by nature the independent variable has exponential or diminishing relationship with the target variable, we can use power transformation. For example, when we try to model TV ad spend against sales volume, we know that at some point, the impact of TV advertisement on sales will decrease. This is so called “diminishing return”. So we usually use the power of 0.3 to 0.7 to transform TV spend so that we can get better model fit.
Standardization: minus the mean and divide by standard deviation. The standardization does not make data more normal, it will just change the mean and the standard error.
scaled_price = (logprice -np.mean(logprice))/np.sqrt(np.var(logprice))
Mean normalization: the distribution will have values between -1 and 1, and a mean of 0.
Min-max scaling: brings values between 0 and 1
Label Encoding: this is when the actual value is text and you want to create numerical values for the variable. For example:
origin = [“USA”, “EU”, “EU”, “ASIA”,”USA”, “EU”, “EU”, “ASIA”, “ASIA”, “USA”]origin_series = pd.Series(origin)cat_origin = origin_series.astype('category')cat_origin.cat.codes0    21    12    13    04    25    16    17    08    09    2dtype: int8#Or use scikit-learn's LabelEncoder:from sklearn.preprocessing import LabelEncoderlb_make = LabelEncoder()origin_encoded = lb_make.fit_transform(cat_origin)origin_encodedarray([2, 1, 1, 0, 2, 1, 1, 0, 0, 2])
Binning: binning is very handy when comes to ordinal values. In stead of reading the ordinal values as integers in the model, we can transform the variable to categorical by creating bins based on the value distribution and then create dummy variables.
In King County house price example, grade is an ordinal variable that has positive correlation with house price.
df[‘grade’].describe()count    21596.000000mean         7.657946std          1.173218min          3.00000025%          7.00000050%          7.00000075%          8.000000max         13.000000Name: grade, dtype: float64
We can create 4 bins based on percentile values.
bins = [3,7,8,10,13]bins_grade = pd.cut(df[‘grade’],bins)bins_grade.value_counts().plot(kind='bar')
bins_grade0          (3, 7]1          (3, 7]2          (3, 7]3          (3, 7]4          (7, 8]5        (10, 13]6          (3, 7]7          (3, 7]8          (3, 7]9          (3, 7]10         (7, 8]11         (3, 7]12         (3, 7]13         (3, 7]14         (3, 7]15        (8, 10]
We then create dummy variables for them because some of the modeling technique requires numerical values.
bins_grade = bins_grade.cat.as_unordered()grade_dummy = pd.get_dummies(bins_grade, prefix=’grade’)
Create dummy variables
Another way to create dummy variables is to use LabelBinarizer from sklearn.preprocessing package
from sklearn.preprocessing import LabelBinarizerlb = LabelBinarizer()grade_dummies = lb.fit_transform(grade)# you need to convert this back to a dataframegrade_dum_df = pd.DataFrame(grade_dummies,columns=lb.classes_)
The advantage of using dummies is that, whatever algorithm you’ll be using, your numerical values cannot be misinterpreted as being continuous. Going forward, it’s important to know that for linear regression (and most other algorithms in scikit-learn), one-hot encoding is required when adding categorical variables in a regression model!
Again, feature transformation involves multiple iterations. There are many ways to get the data right for the model. Just be curious and patient!