How to speed up data labelling with feature vectors
By: Erwin Van Duijnhoven
Tags: Tensorflow Hub, Image Classification, Active Learning, Labelling, Feature Vector

Large datasets contain loads of valuable information, but you can only use it for supervised learning when all data is labelled. Unfortunately, as you know, labelling can be tedious and slow work. It is not exactly our cup of tea and probably will not be yours either. So, let us speed up the labelling process.

Semi-structured data

At Intrador’s, weakly-labelled data from multiple sources continuously flows into our internal data-enrichment solution. We process thousands of resources per hour and our dataset consists of merely semi-structured data in the form of millions of images of various machines, from tractors and trucks to aircraft and dozers. There is more than just variation in machine type, as some pictures show only machine parts, such as tires, fronts, cabins or the like. It is essential to quickly label those diverse components to keep our dataset clean and optimally useful.

However, we cannot use pre-trained labelling models with standard-specific labels, plus we do not want to train a new one on our company-specific labels, because it would already take far too much time to even prepare for that approach. Luckily, we do not have to.

Intrador’s solution

Instead of training a deep learning machine on company-specific labels, we speed up the process by actively applying predictions during data labelling. More specifically, we use a feature vector to guide the labeller with an analysis of relevance and label suggestions.

The first step is to connect a feature vector to each image. It provides an intermediate representation that can function as direct input by various classification models. Meaning, there is no need to feature the image via convolutional filters in a neural network or such. The feature vector helps training algorithms to screen, filter and structure data quickly.

TensorFlow Hub

We use a pre-trained feature vector instead of training a new one. It saves time and, fortunately, there are good ones available if you know where to look. Well, we perceive that the best place to find such a model is TensorFlow Hub, the self-proclaimed library for the publication, discovery and consumption of reusable parts of machine learning models. It is an excellent open-source initiative.

The right vector

As Big Transfer (BiT) is pre-trained to classify images in large supervised datasets, a BiT set feature vector is an obvious choice. Usually, you use BiT to sort images following the 21,000 ImageNet classes. It often functions as a basic model for transfer learning. We, on the other hand, take it a step back so we can move forward faster: we will directly organise our data with the feature vector’s information.

Note that a Big Transfer feature vector is not the only right fit. Our way to accelerate the labelling process can be executed with any feature vector. Of course, performance depends on the problem you want to solve and the relevance of the chosen feature vector.

Big Transfer M

To optimise calculation speed despite the many dataset images, we use the Big Transfer M-model with the smallest ResNet, the BiT-M R50x1 model. Like all other M-models, it is pre-trained on ImageNet-21k (or Full ImageNet, Fall 2011 release if you prefer). The addition ‘21k’ represents the 21,843 classes on which the model is pre-trained. In comparison, a BiT-s model is trained on the 1k version of ImageNet and therefore identifies only 1,000 classes.

Like all other TensorFlow Hub models, this model is easy to load:

We are running TensorFlow 2.3, to install run ‘pip install -U tensorflow==2.3’

import tensorflow_hub as hub

import tensorflow as tf from PIL import Image

from io import BytesIO

import numpy as np

import pandas as pd

import requests

Download and load with one command:

module = hub.KerasLayer("https://tfhub.dev/google/bit/m-r50x1/1")

Predicting data

Before starting to predict on the data, we load the existing data. Use our two helper functions to gather image data from an image URL:

def url_to_image(url):

response = requests.get(url)

img = Image.open(BytesIO(response.content))

img = img.resize((256,256))

return img def preprocess_image(image):

img = tf.keras.preprocessing.image.img_to_array(image) / 255

return tf.image.convert_image_dtype(img, dtype=tf.float32)



urls = [

...

]

labels = [

...

]

images = [url_to_image(url) for url in urls]

input_tensor = [preprocess_image(image) for image in images]

Note that the model page of TransferFlow Hub describes the input data to be following the common image input conventions, meaning a signature that takes images as input, accepts them as a dense 4-D tensor of d-type float32 and shape [batch_size, height, width, 3], whose elements are RGB-colour values of pixels normalized to the range [0, 1].

Now, run the module function over the input_tensor to predict the feature vectors:

feature_vectors = module(input_tensor).numpy()

By running the NumPy function, the tensor returns a NumPy array, meaning that we again have organised data we can use for further computation.

Using the feature vector

At this point, a feature vector per image describes the contents of its corresponding picture. We use the vectors to compare the contents of the photos and to detect similarities between them. Best part? We do not even have to train a deep learning model that takes the pixel values as input ourselves. And because the feature vector has size 2048 also more basic machine learning models can calculate any difference and similarity.

To illustrate the effectiveness of this high dimensional vector, we look at its two-dimensional representation with the use of dimensionality reduction algorithms such as PCA and TSNE. We see the groups of image labels reappear as clusters. Note that the feature representation of cars and planes are more distinct than cars and their tires.