Chained multi-output regression solution with Scikit-Learn
By: Kaushik Choudhury
Tags: AI, Python, Machine Learning, Scikit Learn, Programming

In a typical regression or a classification scenario, we have a set of the independent variable and one or more dependent variables. Sometimes it is not possible to predict all the dependent variable values together. Due to the correlations among the targets, we need to consider the predicted value of the other dependent variable in the sequence to predict the next target value. Hold your breath, and it will become clear with a detailed explanation and example discussed in this article. All variables are also known as features and the dependent variables as the targets.

To make predictions with machine learning in such scenarios, we can put together a sequential regression model with Regressorchain in the Scikit-Learn

The first model is trained based on the independent features and the first dependent variable in the sequence of target variables. Other dependent variable values are not taken into consideration while training the first model.

Illustration by the author — Training first model based on independent feature training data and first dependent training data actual values

The second model is fit and trained on the independent feature values in the training datasets, first and second dependent variable values. This model is used to predict the second target value later with test datasets.

Illustration by the author — Training the second model based on independent feature training data, first and second dependent training data actual values

Finally, for predicting the third dependent variable, the last model is trained based on the independent feature values, and all the dependent variable values. This last model will be used to predict the value of the last dependent variable (target) in the sequence of the variable.

Illustration by the author — Training the last model based on independent feature training data, and all three dependent training data actual values

All three trained models will be used with different input data to predict the dependent variable values in the sequence.

First trained model with independent feature test values as input are used to predict the first dependent variable from the specified order of the dependent features.

Illustration by the author —Test dataset independent feature values input for the first trained model

The predicted value for the first dependent variable and independent feature test values are used as input by the trained second model to predict the next dependent value.

Illustration by the author — Test dataset independent feature values and first target predicted value is input for the second trained model

First two dependent variable predicted values along with test independent feature values used as inputs in the third trained model for predicting the last dependent variable.

Illustration by the author — Test dataset independent feature values and two target predicted values are input for the third trained model

Step 1: In Scikit-Learn package, RegressorChain is implemented in the multioutput module. We will use make_regression, math and NumPy for creating the test data.

from sklearn.linear_model import LinearRegression

from sklearn.multioutput import RegressorChain

import math

import numpy as np

from sklearn.datasets import make_regression

Step 2: Sample test data of 20,000 records with ten independent features and one dependent is generated.

X, y,coef = make_regression(n_samples=20000, n_features=10, random_state=0, noise=4.0,coef=True, bias=200)

Step 3: Correlated second and third dependent variable y2 and y3 is created, and all three dependent variables are transformed in a 2000 x 3 array in the below code.

y2=(y*math.sin(45))+(math.pi/3)

y3=y*y

y= np.vstack((y,y2,y3)).T

Step 4: Dependent variables in succession are considered as per the “order” parameter in RegressorChain for training different models. In our example, the first model is trained based on independent feature training values and the second dependent variable value. Then the independent feature values, second and third dependent values are used as an input to train the second model. Finally, with the independent feature test values and all three dependent variable test dataset values, the third model is trained.