How to improve the accuracy of a Regression Model
By: Shwetha Acharya
Tags: Gridsearchcv, Feature Engineering, Accuracy, Feature Transformation, Regression

Observations based on the above plots:

Males and females are almost equal in number and on average median charges of males and females are also the same, but males have a higher range of charges. Insurance charges are relatively higher for smokers. Charges are highest for people with 2–3 children Customers are almost equally distributed across the 4 regions and all of them have almost the same charges. Percentage of female smokers is less than the percentage of male smokers.

Thus, we can conclude that ‘smoker’ has a considerable impact on the insurance charges, while gender has the least impact.

Let’s create a heatmap to understand the strength of the correlation between charges and numeric features — age, BMI, and children.

sns.heatmap(df[['age', 'bmi', 'children', 'charges']].corr(), cmap='Blues', annot=True)

plt.show()

Correlation Map — Image by Author

We see that age and BMI have an average +ve correlation with charges.

We will now go over the steps of model preparation and model development one by one.

Feature Encoding

In this step, we convert categorical variables — smoker, sex, and region — to numeric format(0, 1,2, 3, etc.) as most of the algorithms cannot handle non-numeric data. This process is called encoding and there are many ways to do this :

LabelEncoding — Represent categorical values as numbers (For example, a feature such as Region with values Italy, India, USA, UK can be represented as 1, 2, 3, 4) OrdinalEncoding — Used for representing rank-based categorical data values as numbers. (For example representing high, medium, low as 1,2,3) One-hot Encoding — Represent categorical data as binary values — 0s,1s only. I prefer to use one-hot encoding over label encoding if there aren’t many unique values in the categorical feature. In here, I have used pandas’ one hot encoding function (get_dummies) on Region and split it into 4 columns — location_NE, location_SE, location_NW, and location_SW. One can also use label encoding for this column, however, one hot encoding gave me a better result.

#One hot encoding

region=pd.get_dummies(df.region, prefix='location')

df = pd.concat([df,region],axis=1)

df.drop(columns='region', inplace=True) df.sex.replace(to_replace=['male','female'],value=[1,0], inplace=True)

df.smoker.replace(to_replace=['yes', 'no'], value=[1,0], inplace=True)

2. Feature Selection and Scaling

Next, we will select features that affect ‘charges’ the most. I have selected all the features except gender as its effect on ‘charges’ is very less(concluded from the viz charts above). These features will form our ‘X’ variable while charges will be our ‘y’ variable. If there are many features, I suggest you use scikit-learn’s SelectKBest for feature selection to arrive at the top features.

#Feature Selection

y=df.charges.values

X=df[['age', 'bmi', 'smoker', 'children', 'location_northeast', 'location_northwest', 'location_southeast', 'location_southwest']] #Split data into test and train

X_train, X_test, y_train, y_test=train_test_split(X,y, test_size=0.2, random_state=42)

Once we have selected our features, we need to ‘standardize’ the numeric ones — age, BMI, children. Standardization process converts data to smaller values in the range 0 to 1 so that all of them lie on the same scale and one doesn’t overpower the other. I have used StandardScaler here.

#Scaling numeric features using sklearn StandardScalar

numeric=['age', 'bmi', 'children']

sc=StandardScalar()

X_train[numeric]=sc.fit_transform(X_train[numeric])

X_test[numeric]=sc.transform(X_test[numeric])

Now, we are all set to create our first basic model😀 . We will try Linear Regression and DecisionTrees to predict insurance charges

Model scores — Image by Author

Mean absolute error (MAE) and root-mean-square error (RMSE) are the metrics used to evaluate regression models. You can read more about it here. Our baseline models give a score of more than 76%. Between the 2, DecisionTrees give a better MAE of 2780. Not bad..!

Let’s see how can we make our model better.

3A. Feature Engineering

We can improve our model score by manipulating some of the features in the dataset. After a couple of trials, I found that the following items improve accuracy:

Grouping similar customers into clusters using KMeans. Clubbing northeast and northwest regions into ‘north’ and southeast and southwest into ‘south’ in Region column. Transforming ‘children’ into a categorical feature called ‘more_than_one_child’ which is ‘Yes’ if the number of children is > 1

from sklearn.cluster import KMeans

features=['age', 'bmi', 'smoker', 'children', 'location_northeast', 'location_northwest', 'location_southeast', 'location_southwest']

kmeans = KMeans(n_clusters=2)

kmeans.fit(df[features]) df['cust_type'] = kmeans.predict(df[features]) df['location_north']=df.apply(lambda x: get_north(x['location_northeast'], x['location_northwest']), axis=1) df['location_south']=df.apply(lambda x: get_south(x['location_southwest'], x['location_southeast']), axis=1) df['more_than_1_child']=df.children.apply(lambda x:1 if x>1 else 0)

All features— Image by Author

3B. Feature Transformation

From our EDA, we know that the distribution of ‘charges’ (Y) is highly skewed and hence we will apply scikit-learn’s target transformer — QuantileTransformer to normalize this behavior.

X=df[['age', 'bmi', 'smoker', 'more_than_1_child', 'cust_type', 'location_north', 'location_south']] #Split test and train data

X_train, X_test, y_train, y_test=train_test_split(X,y, test_size=0.2, random_state=42) model = DecisionTreeRegressor()

regr_trans = TransformedTargetRegressor(regressor=model, transformer=QuantileTransformer(output_distribution='normal'))

regr_trans.fit(X_train, y_train)

yhat = regr_trans.predict(X_test)

round(r2_score(y_test, yhat), 3), round(mean_absolute_error(y_test, yhat), 2), round(np.sqrt(mean_squared_error(y_test, yhat)),2) >>0.843, 2189.28, 4931.96

Woahh… a whopping 84%…and MAE has reduced to 2189!

4. Use of Ensemble and Boosting Algorithms

Now we will use these features on ensemble-based RandomForest, GradientBoosting, LightGBM, and XGBoost. If you are a beginner and not aware of boosting and bagging methods, you can read more about them here.

X=df[['age', 'bmi', 'smoker', 'more_than_1_child', 'cust_type', 'location_north', 'location_south']] model = RandomForestRegressor()

#transforming target variable through quantile transformer

ttr = TransformedTargetRegressor(regressor=model, transformer=QuantileTransformer(output_distribution='normal'))

ttr.fit(X_train, y_train)

yhat = ttr.predict(X_test) r2_score(y_test, yhat), mean_absolute_error(y_test, yhat), np.sqrt(mean_squared_error(y_test, yhat)) >>0.8802, 2078, 4312

Yes! Our RandomForest model does perform well — MAE of 2078👍. Now, we will try with some boosting algorithms such as Gradient Boosting, LightGBM, and XGBoost.

Model Score— Image by Author

All of them seem to perform well:)

5. Hyperparameter Tuning

Let’s tweak some of the algorithm parameters such as tree depth, estimators, learning rate, etc, and check for model accuracy. Manually trying out different combinations of parameter values is very time-consuming. Scikit-learn’s GridSearchCV automates this process and calculates optimized values for these parameters. I have applied GridSearch to the above 3 algorithms. Below is the one for XGBoost:

Best values for parameters in GridSearchCV — Image by Author

Once we have optimum values for our parameters, we will run all 3 models again with these values.

Model Score — Image by Author

This looks much better! We have been able to improve our accuracy — XGBoost gives a score of 88.6% with relatively fewer errors 👏👏

1. Distribution plot of predicted and actual values of charges ; 2. Residual plot — Image by Author

Distribution and Residual plots confirm that there is a good overlap between predicted and actual charges. However, there are a handful of predicted values that are way beyond the x-axis and this makes our RMSE is higher. This can be reduced by increasing our data points i.e. collecting more data.

We are now ready to deploy this model into production and test it on unknown data. Well done👍

In a nutshell, the points that improved the accuracy of my model

Creating simple new features Transforming target variable Clustering common data points Use of boosting algorithms Hyperparameter tuning

You can access my notebook here. Not all of them may work for your model every time. Pick and choose the ones that work best for your scenario:)