Grocery Item’s Nil Pick prediction using a Machine Learning Model
By: Abhinandan Sonvane
Tags: Engineering, Data Science

Data Processing

In this section, we intend to build the input pipeline for training and evaluating the Nil Pick machine learning model.

The input pipeline represents how the data will be passed into the model for each step of training or evaluation. Since training the model requires thousands of steps, the input pipeline must be as efficient as possible.

The store dataset we created was stored in a pandas DataFrame store_dataset. Since the DataFrame is not the most efficient data storage for the input pipeline, we’ll need to perform additional processing to create a more efficient solution.

Splitting the Store Dataset

There are two main components in creating a machine learning model: training and evaluation. Training is the foundation of machine learning, but the evaluation is just as important. Model evaluation gives us a concrete idea of just how good the model is after training, and it allows us to compare the performances for different configurations of the model.

Figure 2. Splitting Dataset

Now, how do we decide the amount of data to use for training and evaluation. Using more data in training would potentially improve the model’s performance, but it would limit us in how accurate our evaluation is due to the limited evaluation set size. On the other hand, having a larger evaluation set would give us more confidence in our evaluation process’ accuracy, but it might limit the amount and diversity of the data in training.

Depending on the use-case make a prudent choice, we choose a 90–10 split, meaning that the training set comprises 90% of the final dataset while the evaluation set comprises 10%.

TensorFlow Example Object

To optimize the input pipeline, we want to convert each DataFrame row into a TensorFlow Example object. By using Example objects in the input pipeline, we’re able to efficiently feed the data into a machine learning model.

Write Example Data to TensorFlow Records

Now that we’ve completed the function to convert each DataFrame row into an Example object, we can create the efficient input pipeline storage for both the training and evaluation sets. The data storage will be in the form of TFRecords files, which hold serialized Example objects.

The write_tfrecords function (shown below) writes the data from a given DataFrame into a TFRecords file. It uses the create_example function to convert each row of the dataset into an Example object. Each Example object is then serialized and written into the TFRecords file.

We then used the write_tfrecords function to write the training set’s serialized Example data into a TFRecords file called train.tfrecords and the evaluation set’s serialized Example data into a TFRecords file called eval.tfrecords. These files will then be used in the input pipeline for the machine learning model.

Serialize TFRecords

The data is stored as serialized Example objects in TFRecords files. To efficiently parse the Example object in the input pipeline, we need to create an Example spec.

The Example spec is a Python dictionary, mapping feature names to FixedLenFeature objects. For our Store dataset, each of the FixedLenFeature objects has the shape () . This is because each feature contains a single value per data observation.

For both training and evaluation, we require the data to be labelled to calculate the loss for our machine learning model. Since our model is trained to predict Nil Pick , we use the 'Nil Pick' feature as the label for each data observation.

We then parse feature data for a single Example (which represents data for one DataFrame row) using the tf.parse_single_example function.

Training and Evaluation TFRecords Dataset for the Model

We are finally ready to create TensorFlow datasets from the TFRecords files for both training and evaluation.

The TFRecords datasets contain serialized Example objects. Using the Example spec and feature parsing functions, we then convert each serialized Example to a tuple containing the Example’s feature data and label for both training and evaluation data.

The TFRecords dataset’s map function allows us to apply the parsing function ( parse_fn ) to each serialized Example in the dataset. Since the parse_features function takes in two arguments, and map can only be used on functions with one argument, we use a single argument lambda function to wrap around parse_features .

Also, we apply uniform random shuffling with the buffer size of 100000 and configure dataset batches (both for training and evaluation), so that each training/evaluation step contains multiple data observations.