Storytelling of a Data Project (Regression Problem)
By: Shuki Zanyovka
Tags: Regression, Exploratory Data Analysis, Feature Engineering, Catboost, Data Science

Regression models are models used to predict (estimate) a certain continuous function or statistical phenomena by correlated features which can be the outputs of sensors of a system. The value required to be predicted is measurable by a range of real numbers (such as temperatures or the time it takes for an application to complete a task). This is as opposed to classification problems, where the model attempts to classify a certain sample into a finite set of classes.

The project described in this article was the final project of my 8 month online Data Science course at Practicum by Yandex.

The problem at hand is a regression problem of the final temperature of the substrate of a steel processing factory based on the exact process applied to the input raw steel. Predicting the final temperature of the substrate can be done by developing a complex physics model of the entire system and running it on a high-end computer or by trying to capture the data (features) which are highly correlated to the output temperature then trying to develop a machine learning model based on the limited apriori knowledge of the physics of the process.

We are given a set of data-sets that contain the complete data from the measurements,

In each file, the key column contains the batch number. The batch number is a unique identifier of an execution of a batch (sample) of a process. Following some EDA (Exploratory Data Analysis) of the key column, we can see that each key might have a variable number of rows of temperature measurements (i.e. at different times),

Also, the more important thing to emphasize is that the process itself is described by the data_arc dataset by a key and contains multiple rows per process stage (the same as before, a variable number of rows describe a single process result for a certain key / material).

A simple function can help describing the data structure,

In order to analyze the data per key, we simply call the function for a few different keys,

This function shows that there are multiple rows per key (variable number of rows in different keys, we see different numbers of rows) which describe the specific process identified by the key. Each row has a timestamp of the start and end time a specific power level was applied to the material. There’s a challenge to understand how to reduce the M rows per key to a set of features per key (which includes all the features from the other datasets).

Note that the timestamp of each of the process samples described in data_arc makes sense in time from the first entry of the key — this is a continuous process until the process is complete. Following completion, the processed material is removed and new material (tagged as key+1) can be processed in the machine.

Following this result, we now need to build the features and obviously the variable number of rows must be taken care of.

As for the multiple rows of temperature sampling, our target is simply the last temperature (i.e. the temperature with the latest timestamp that belongs to the key).

The first temperature (in time) for the key can be considered as a feature, along with the last temperature’s timestamp.

All timestamps can be used as features if they are updated to be the time difference (in seconds) from the first temperature measurement time (or from the first row of the processing).

As for the variable number of rows describing the processing, the data_arc rows describe the active power applied to the system, the amount of time it was applied and the reactive power that was output from the machine (the power which flows back and forth that means it moves in both the directions in the circuit or reacts upon itself, and is called reactive Power). A similar time reference update can be set to the time difference from the timestamp we have chosen to be the beginning of the process.

At this point we can see that since the temperature measured is a physical process, we can safely assume it’s a continuous function which is also highly correlated with the mean of all data_arc rows, (the mean active and reactive power and the actual duty cycle — i.e. the time the power was applied) so we summarize all rows per key.

The result of the described feature extraction process (and some EDA to remove outliers from each feature, which I don’t describe here) can be tested by plotting the histogram of the time difference of the last measurement.

data.last_temperature_measurement_delta_t.hist(bins=100)

The distribution of the last time measurement seems to be a normal distribution with an outlier at 0–100 seconds. Still, we can choose to either leave these outlier samples or remove them. If the model we develop will be good enough, then it might also fit nicely with this outlier as it seems these are valid samples of the datas. We keep note of this outlier and continue to look at the correlation of the data columns. A nice way to see the correlation between features and target is to use the pair plot function for certain interesting columns,

The pairplot() function can show the distribution of the target (last temperature) vs. each of the features. So we look at the last row of the pairplot,

Sometimes, looking at the distribution of each feature vs. the target can help understand how it correlates to the target (for example, linear correlation or so).

It appears that there’s no pattern visible for the eye in this model, so the next thing to do would be to simply develop a model and test it — I’ve chosen the CatBoost regression model.

The final set of features in my model were a mix of features from all datasets including the features we engineered from the multiple rows + the start temperature (the last temperature is our target) + the delta_time in seconds from the beginning of the measurement to multiple events in the process.

The CatBoost model is an iterative model that trains in each step on the error of the previous step to correct it. The objective of the project was to reach a mean absolute error smaller than 6.

Below is some code to demonstrate how to use the CatBoost model,

The result on the validation and test set were,

On the validation set, MAE (Mean Absolute Error) = 5.306839916817323

On the test set, MAE = 5.137533647546855

This confirms our first assumption. The mean of the data_arc rows per key are sufficient to reach the MAE (Maximum Absolute Error) requirements of the model.

Looking at the result, we can safely assume that our model predicts the last temperature at high accuracy (this can be further tested by trying to run the pipeline of the training and prediction using different random instantiations of the training / validation and test data).

To summarize, the model we developed can be used by the factory data analysts team to optimize the machine’s processing in terms of power consumption and the time of processing without doing any additional experiments at high accuracy. A better result could be achieved by developing a physics model of the entire machine but it would require much more time and effort and it would be hard to update and monitor daily if the process were to deteriorate for some reason.