Time Forecast with TPOT. a Python Automated Machine Learning... | by Susan Li | Towards Data Science

My colleague at work recommended me several wonderful Machine Learning libraries and some of them were new to me. Therefore, I decided to try them out, one by one. Today is TPOT’s turn.
The data set was about predicting engineers of Daimler’s Mercedes-Benz cars speed of testing system, the purpose is to reduce the time that cars spend on testing, with over three hundreds of features. Frankly, I have little or no domain expertise in automobile industry. Regardless, I will try to make the best predictions I can, using TPOT, a Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
This data set contains an anonymized set of variables, each representing a custom feature in a Mercedes car.
The target feature is labeled “y” and represents the time (in seconds) that the car took to pass testing for each variable. The data set can be found here.
train = pd.read_csv('mer_train.csv')print('Train shape: ', train.shape)
We know what problems we have here: too many features (columns) and not enough rows.
In addition, we don’t know what those features are except “y” and “ID”.
Target feature “y” is the time (in seconds) that the car took to pass testing for each variable. Let’s see its distribution.
plt.figure(figsize = (10, 6))n, bins, patches = plt.hist(train['y'], 50, facecolor='blue', alpha=0.75)plt.xlabel('y value in seconds')plt.ylabel('count')plt.title('Histogram of y value')plt.show();
train['y'].describe()
plt.figure(figsize = (10, 6))plt.scatter(range(train.shape[0]), np.sort(train['y'].values))plt.xlabel('index')plt.ylabel('y')plt.title("Time Distribution")plt.show();
There is one outlier which was the maximum time at 265 seconds.
cols = [c for c in train.columns if 'X' in c]print('Number of features except ID and target feature: {}'.format(len(cols)))print('Feature types :')train[cols].dtypes.value_counts()
Out of all features, we are having 8 categorical features and 368 integer features. What about the cardinality of the features? The following ideas and scripts were from Mikel Bober Irizar.
counts = [[], [], []]for c in cols:    typ = train[c].dtypes    uniq = len(train[c].unique())    if uniq == 1:        counts[0].append(c)    elif uniq == 2 and typ == np.int64:        counts[1].append(c)    else:        counts[2].append(c)print('Constant features: {} Binary features: {} Categorical features: {}\n'.format(*[len(c) for c in counts]))print('Constant features: ', counts[0])print()print('Categorical features: ', counts[2])
There were 12 features in which only contain a single value (0), these are useless for supervised algorithms, and we will drop them later.
The rest of our data set is made up of 356 binary features, and 8 categorical features. Let’s explore categorical features first.
for cat in ['X0', 'X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X8']:    print("Number of levels in category '{0}': \b {1:2}".format(cat, train[cat].nunique()))
Feature X0
sort_X0 = train.groupby('X0').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X0', data=train, order = sort_X0)plt.xlabel('X0')plt.ylabel('Occurances')plt.title('Feature X0')sns.despine();
X0 vs. target feature y
sort_y = train.groupby('X0')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (14, 6))sns.boxplot(y='y', x='X0', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X0 vs. y value')plt.show();
Feature X1
sort_X1 = train.groupby('X1').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X1', data=train, order = sort_X1)plt.xlabel('X1')plt.ylabel('Occurances')plt.title('Feature X1')sns.despine();
X1 vs. target feature y
sort_y = train.groupby('X1')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (10, 6))sns.boxplot(y='y', x='X1', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X1 vs. y value')plt.show();
Feature X2
sort_X2 = train.groupby('X2').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X2', data=train, order = sort_X2)plt.xlabel('X2')plt.ylabel('Occurances')plt.title('Feature X2')sns.despine();
X2 vs. target feature y
sort_y = train.groupby('X2')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (12, 6))sns.boxplot(y='y', x='X2', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X2 vs. y value')plt.show();
Feature X3
sort_X3 = train.groupby('X3').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X3', data=train, order = sort_X3)plt.xlabel('X3')plt.ylabel('Occurances')plt.title('Feature X3')sns.despine();
X3 vs. target feature y
sort_y = train.groupby('X3')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (10, 6))sns.boxplot(y='y', x='X3', data=train, order = sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X3 vs. y value')plt.show();
Feature X4
sort_X4 = train.groupby('X4').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X4', data=train, order = sort_X4)plt.xlabel('X4')plt.ylabel('Occurances')plt.title('Feature X4')sns.despine();
X4 vs. target feature y
sort_y = train.groupby('X4')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (10, 6))sns.boxplot(y='y', x='X4', data=train, order = sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X4 vs. y value')plt.show();
Feature X5
sort_X5 = train.groupby('X5').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X5', data=train, order = sort_X5)plt.xlabel('X5')plt.ylabel('Occurances')plt.title('Feature X5')sns.despine();
X5 vs. target feature y
sort_y = train.groupby('X5')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (12, 6))sns.boxplot(y='y', x='X5', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X5 vs. y value')plt.show();
Feature X6
sort_X6 = train.groupby('X6').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X6', data=train, order = sort_X6)plt.xlabel('X6')plt.ylabel('Occurances')plt.title('Feature X6')sns.despine();
X6 vs. target feature y
sort_y = train.groupby('X6')['y']\                     .median()\                     .sort_values(ascending=False)\                     .indexplt.figure(figsize = (12, 6))sns.boxplot(y='y', x='X6', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X6 vs. y value')plt.show();
Feature X8
sort_X8 = train.groupby('X8').size()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize=(12,6))sns.countplot(x='X8', data=train, order = sort_X8)plt.xlabel('X8')plt.ylabel('Occurances')plt.title('Feature X8')sns.despine();
X8 vs. target feature y
sort_y = train.groupby('X8')['y']\                    .median()\                    .sort_values(ascending=False)\                    .indexplt.figure(figsize = (12, 6))sns.boxplot(y='y', x='X8', data=train, order=sort_y)ax = plt.gca()ax.set_xticklabels(ax.get_xticklabels())plt.title('X8 vs. y value')plt.show();
Unfortunately, we did not learn much from the above EDA, this is life. However, we did notice that some categorical features have effects on the “y” and the “X0” seems to have the highest effect.
After exploring, we now going to encode these categorical features’ levels as digits using Scikit-learn’s MultiLabelBinarizer and treat them as new features.
We then drop the constant features and categorical features which have been encoded, as well as our target feature “y”.
train_new = train.drop(['y','X11', 'X93', 'X107', 'X233', 'X235', 'X268', 'X289', 'X290', 'X293', 'X297', 'X330', 'X347', 'X0', 'X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X8'], axis=1)
We then add the encoded features to form the final data set to be used with TPOT.
train_new = np.hstack((train_new.values, X0_trans, X1_trans, X2_trans, X3_trans, X4_trans, X5_trans, X6_trans, X8_trans))
The final data set is in the form of a numpy array, in the shape of (4209, 552).
It’s time to construct and fit TPOT regressor. When it is finished, TPOT will display the “best” model (based on test data MSE in our case) hyperparameters, and will also output the pipelines as an execution-ready Python script file for a later use or our investigation.
Running above code will discover a pipeline as output that achieves 56 mean squared error (MSE) on the test set:
print("TPOT cross-validation MSE")print(tpot.score(X_test, y_test))
You may have noticed that MSE is a negative number, according to this thread, it is neg_mean_squared_error for TPOTRegressor that stands for negated value of mean squared error. Let’s try it again.
from sklearn.metrics import mean_squared_errorprint('MSE:')print(mean_squared_error(y_test, tpot.predict(X_test)))
print('RMSE:')print(np.sqrt(mean_squared_error(y_test, tpot.predict(X_test))))
So, the difference between our predicted time and the real time is about 7.5 seconds. That’s a pretty good result! And the model that produces this result is one that fits a RandomForestRegressor stacked with KNeighborsRegressor algorithm on the data set.
Finally, we are going to export this pipeline:
tpot.export('tpot_Mercedes_testing_time_pipeline.py')
I enjoyed learning and using TPOT, hope you are the same. Jupyter notebook can be found on Github. Have a great weekend!
Reference: TPOT Tutorial