Using Object Detection for Complex Image Classification Scenarios Part 2: | by Aaron (Ari) Bornstein | Towards Data Science

TLDR; This series is based on the work detecting complex policies in the following real life code story. Code for the series can be found here.
In the last post of the series, we outlined the challenge of a complex image classification task in this post we will introduce and evaluate the Azure Custom Vision Service as a technique for solving our challenge.
Custom Vision Service is a tool for building custom image classifiers. It makes it easy and fast to build, deploy, and improve an image classifier. We provide a REST API and a web interface to upload your images and train.
The Custom Vision Service is a tool for building custom image classifiers, and for making them better over time. For example, if you want a tool that could identify images of “Daisies”, “Daffodils”, and “Dahlias”, you could train a classifier to do that. You do so by providing Custom Vision Service with images for each tag you want to recognize.
Custom Vision Service works best when the item you are trying to classify is prominent in your image. Custom Vision Service does “image classification” but not yet “object detection.” This means that Custom Vision Service identifies whether an image is of a particular object, but not where that object is within the image.
Very few images are required to create a classifier — 30 images per class is enough to start your prototype. The methods Custom Vision Service uses are robust to differences, which allows you to start prototyping with so little data. However, this means Custom Vision Service is not well suited to scenarios where you want to detect very subtle differences (for example, minor cracks or dents in quality assurance scenarios.)
Custom Vision Service is designed to make it easy to start building your classifier, and to help you improve the quality of your classifier over time.
Developers can also export their models to be run on the edge devices such as iOS, Android or the RaspberryPi
The Steps below will walk you through training a model for our challenge pro-grammatically with python with the custom vision service.
The following steps are based on the documentation quick start.
Obtain your training and prediction key by logging into the Custom Vision Service and navigating to account settings as shown below.
Next we need to programatically make a new custom vision service project using the keys your sourced above.
from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClientfrom azure.cognitiveservices.vision.customvision.training.models import ImageUrlCreateEntryENDPOINT = "https://southcentralus.api.cognitive.microsoft.com"# Replace with a valid keytraining_key = '' #<your training key>prediction_key = '' #<your prediction key>trainer = CustomVisionTrainingClient(training_key, endpoint=ENDPOINT)# Create a new projectprint("Creating Beverage Policy Classifier...")project = trainer.create_project("Beverage Policy Classifier")
Creating Beverage Policy Classifier...
Run the following code to create tags for our Valid and Invalid Examples
# Make two tags in the new projectvalid_tag = trainer.create_tag(project.id, "Valid")invalid_tag = trainer.create_tag(project.id, "Invalid")
To add the images we have to the project, insert the following code after the tag creation. This will upload the image with the corresponding tag.
print ("Adding images...")import osfrom multiprocessing.dummy import Pool as ThreadPooldef upload(filepath):    with open(filepath, mode="rb") as img_data:        if "Invalid" in filepath:            trainer.create_images_from_data(project.id, img_data.read(), [ invalid_tag.id ])        else:            trainer.create_images_from_data(project.id, img_data.read(), [ valid_tag.id ])    print('.', end='')    def upload_parallel(filepaths, threads=5):    pool = ThreadPool(threads)    results = pool.map(upload, filepaths)    pool.close()    pool.join()valid_dir = "dataset/Beverages/Train/Valid/"invalid_dir = "dataset/Beverages/Train/Invalid/"valid_paths = [valid_dir + os.fsdecode(image) for image in os.listdir(os.fsencode(valid_dir))]invalid_paths = [invalid_dir + os.fsdecode(image) for image in os.listdir(os.fsencode(invalid_dir))]img_paths = valid_paths + invalid_pathsupload_parallel(img_paths)print("Added {} images, {} valid, {} invalid.".format(len(img_paths), len(valid_paths), len(invalid_paths)))
Adding images.......................................................................................................................................................................................Added 180 images, 90 valid, 90 invalid.
Now that we’ve added tags and images to the project, we can train it. This creates the first iteration in the project. We can then mark this iteration as the default iteration.
import timeprint ("Training...")iteration = trainer.train_project(project.id)while (iteration.status == "Training"):    iteration = trainer.get_iteration(project.id, iteration.id)    print ("Training status: " + iteration.status)    time.sleep(1)
Training...Training status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: TrainingTraining status: Completed
Now lets evaluate the model on the local test dataset that the service has never seen.
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
# Now there is a trained endpoint, it can be used to make a predictionpredictor = CustomVisionPredictionClient(prediction_key, endpoint=ENDPOINT)project_id = project.iddef predict(filepath):    with open(filepath, mode="rb") as test_data:        results = predictor.predict_image(project_id, test_data.read(), iteration.id)    pred  = max(results.predictions, key=lambda x:x.probability).tag_name    true ='Invalid' if "Invalid" in filepath else 'Valid'    print('.', pred , end='')    return (true, pred)        def predict_parallel(filepaths, threads=2):    pool = ThreadPool(threads)    results = pool.map(predict, filepaths)    pool.close()    pool.join()    return zip(*results)test_valid_dir = "dataset/Beverages/Test/Valid/"test_invalid_dir = "dataset/Beverages/Test//Invalid/"test_valid_paths = [test_valid_dir + os.fsdecode(image) for image in os.listdir(os.fsencode(test_valid_dir))]test_invalid_paths = [test_invalid_dir + os.fsdecode(image) for image in os.listdir(os.fsencode(test_invalid_dir))]test_img_paths = test_valid_paths + test_invalid_pathsy_true, y_pred = predict_parallel(test_img_paths)
. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Valid. Invalid. Valid. Invalid. Valid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid. Invalid
from utils import classification_reportclassification_report(y_true, y_pred)
precision    recall  f1-score   support     Invalid       1.00      1.00      1.00        30       Valid       1.00      1.00      1.00        30   micro avg       1.00      1.00      1.00        60   macro avg       1.00      1.00      1.00        60weighted avg       1.00      1.00      1.00        60Confusion matrix, without normalization[[30  0] [ 0 30]]Normalized confusion matrix[[1. 0.] [0. 1.]]
As you can see the custom vision service is a great tool for generating strong models with relatively small amount of data and little work. In the majority of use cases its a go too tool and a great place to get started.Custom Vision Service works best when the item you are trying to classify is prominent in your image.
The service is a go to starting point for any classification task. However there are a couple of items to consider when deploying for production.
What are the cases that the service fails on (It could be that the 2% error case appears 30% in production or are the most important cases to consider for your use case.How many classes/policies are you classifying (The models work better with 2–10 policies )What domain is your task?
What are the cases that the service fails on (It could be that the 2% error case appears 30% in production or are the most important cases to consider for your use case.
How many classes/policies are you classifying (The models work better with 2–10 policies )
What domain is your task?
medium.com
github.com
azure.microsoft.com
The next post in this series will review how to approach this task with the Policy Recognition with Keras CNNs, MobileNet and Transfer Learning subsequent posts will address the following:
Policy Detection with Keras RetinaNet
Training and Computer Vision Models on the Cloud using Azure ML Service
Train a Computer Vision Model on a Remote Cluster with Azure Machine Learning
If you have any questions, comments, or topics you would like me to discuss feel free to follow me on Twitter if there is a milestone you feel I missed please let me know.
Aaron (Ari) Bornstein is an avid AI enthusiast with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.