Using Object Detection for Complex Image Classification Scenarios Part 4: | by Aaron (Ari) Bornstein | Towards Data Science

TLDR; This series is based on the work detecting complex policies in the following real life code story. Code for the series can be found here.
Up until this point we’ve been classifying raw images what if we could use these images to generate targeted features to help us with our advanced classification challenge.
Object detection requires lots of annotated data. Traditionally this is a very manually intensive task, the ideal pipeline couples tagging with model training to enable active learning.
The Visual Object Tagging tool VoTT provides end to end support for generating datasets and validating object detection models from video and image assets.
VoTT supports the following features:
The ability to tag and annotate Image directories or stand-alone videos.
Computer-assisted tagging and tracking of objects in videos using the Camshift tracking algorithm.
Exporting tags and assets to Tensorflow (PascalVOC) or YOLO format for training an object detection model.
Running and validating a trained object detection model on new videos to generate stronger models with a docker endpoint.
The section below will briefly highlight the progression of Region Proposal Object Detection that will be used for this task. For a more in depth tutorial I strongly recommend the following Medium Series by Jonathan Hui
medium.com
The last two posts in this series dealt with image classification. The simplest way to transform an image classifier into an object detector is to use a series of sliding windows of different dimensions across a given image.
If we classify that there is an object we are looking for in a given window, then we can return the dimensions of the captured object.
Unfortunately the sliding window approach, while simple, has a couple of drawbacks. It is slow to evaluate multiple window sizes and it is imprecise since we don’t know in advance what the correct window size should be for each object in our image.
Instead of using sliding windows across an entire image what if we could perform classifications only on the regions of interest that may contain an object. This is the main intuition behind region proposal object detectors. Traditionally we use an algorithm called selective search to propose thes Regions of interests.
While the traditional RCNN approach works well in terms of accuracy, it is very costly to compute since the Neural Network has to be evaluated for each region of interest.
Fast R-CNN addresses this drawback by only evaluating most of the network (to be specific: the convolution layers) a single time per image. According to the authors, this leads to a 213 times speed-up during testing and a 9x speed-up during training without loss of accuracy.
Faster R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Faster R-CNN employs a region proposal network which does not require selective search for candidate region proposals.
RetinaNet is an object detector that builds off the intuition of Faster RCNN it provides feature pyramids and on optimized focal loss that enables faster evaluation time than FasterRCNN and provides a focal loss that helps prevent overfitting the background class. At the time of this article RetinaNet is the current state of the art region proposal network.
For more see:
medium.com
VoTT can be used to generate direct datasets that can be used with Azure Machine Learning to train custom object detection models.
docs.microsoft.com
However since training RetinaNet requires access to N-Series GPU machines, for the purpose of time and to ensure that this tutorial is CPU only, I have taken the liberty of pretraining models. In the next post we will discuss how to train these models with the Azure Machine Learning Service.
# import keras_retinanetimport kerasfrom keras_retinanet import modelsfrom keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_imagefrom keras_retinanet.utils.visualization import draw_box, draw_captionfrom keras_retinanet.utils.colors import label_color# load imagedef evaluate_single_image(model, img_path):    image = read_image_bgr(img_path)    # preprocess image for network    image = preprocess_image(image)    image, scale = resize_image(image)    # process image    start = time.time()    boxes, scores, labels = model.predict_on_batch(np.expand_dims(image, axis=0))    print("processing time: ", time.time() - start)    # correct for image scale    boxes /= scale        return (boxes[0], scores[0], labels[0])def visualize_detection(img_path, model_results):    image = read_image_bgr(img_path)    boxes, scores, labels = model_results    # visualize detections    draw = image.copy()    draw = cv2.cvtColor(draw, cv2.COLOR_BGR2RGB)    for box, score, label in zip(boxes, scores, labels):        # scores are sorted so we can break        if score < 0.5:            break        color = label_color(label)        b = box.astype(int)        draw_box(draw, b, color=color)        caption = "{} {:.3f}".format(labels_to_names[label], score)        draw_caption(draw, b, caption)    plt.figure(figsize=(15, 15))    plt.axis('off')    plt.imshow(draw)    plt.show()# load retinanet modelsoda_model = models.load_model('models/retina_net_soda.h5', backbone_name='resnet50')labels_to_names = {0: 'shelf1', 1: 'shelf2', 2: 'shelf3'}valid_example_path = 'dataset/Beverages/Test/Valid/IMG_4187.JPG'detection_results = evaluate_single_image(soda_model, valid_example_path)visualize_detection(valid_example_path, detection_results)
processing time:  10.065604209899902
Challenge: Can you think of way we could use these boxes to tell that the policy is invalid?
Now that we have an object detection model for finding bottles lets develop a heuristic to determine whether a shelf is properly stocked and test it on two sample images.
Our heuristic will work as follows:
We are going to find the use the min y1 and max y2 of each of the shelf classes
We will confirm that the soda shelf is above the juice shelf which is above water shelf
For each of the max’s we will ensure that there are no other classes that are in between them
def predict_bottles(model_results):    bounds = {}    beverages = {0: [], 1: [], 2: []}    boxes, scores, labels = model_results    for box, score, label in zip(boxes, scores, labels):        # scores are sorted so we can break        if score < 0.5:            break        beverages[label].append(box)            # Find the use the min y1 and max y2 of each of the tag classes    for bev in beverages:        if len(beverages[bev]) == 0:            return False        y1 = min(beverages[bev], key=lambda b: b[1])[1]        y2 = max(beverages[bev], key=lambda b: b[3])[3]        bounds[bev] = {"y1":y1, "y2":y2}     # Confirm that soda is above juice which is above water    if (bounds[0]["y1"] < bounds[1]["y1"]) and (bounds[1]["y1"] < bounds[2]["y1"]):        # For each of the max's we will ensure that there are no other clases that are in between them        for b in bounds.keys():            for bev_type in (set(bounds.keys()) - set([b])):                for bev in beverages[bev_type]:                    if bev[1] > bounds[b]["y1"] and bev[3] < bounds[b]["y2"]:                        return False         return True    else:        return Falsevisualize_detection(valid_example_path, detection_results)predict_bottles(detection_results)
Trueinvalid_example_path = 'dataset/Beverages/Test/Invalid/IMG_4202.JPG'detection_results = evaluate_single_image(soda_model, invalid_example_path)visualize_detection(invalid_example_path, detection_results)predict_bottles(detection_results)
processing time:  8.053469896316528
False
Normally to apply this to our entire dataset we would use batches since we are trying to save memory we will evaluate one image at a time.
from tqdm import tqdm_notebookfrom utils import classification_reportfrom keras.preprocessing.image import ImageDataGeneratory_pred = []y_true = []print("Testing Invalid Cases")img_dir = 'dataset/Beverages/Test/Invalid/'for img_path in tqdm_notebook(os.listdir(img_dir)):    detection_results = evaluate_single_image(soda_model, os.path.join(img_dir, img_path))    y_pred.append(predict_bottles(detection_results))    y_true.append(False)print("Testing Valid Cases")img_dir = 'dataset/Beverages/Test/Valid/'for img_path in tqdm_notebook(os.listdir(img_dir)):    detection_results = evaluate_single_image(soda_model, os.path.join(img_dir, img_path))    y_pred.append(predict_bottles(detection_results))    y_true.append(True)
classification_report(y_true, y_pred)
precision    recall  f1-score   support       False       1.00      1.00      1.00        30        True       1.00      1.00      1.00        30   micro avg       1.00      1.00      1.00        60   macro avg       1.00      1.00      1.00        60weighted avg       1.00      1.00      1.00        60Confusion matrix, without normalization[[30  0] [ 0 30]]Normalized confusion matrix[[1. 0.] [0. 1.]]
We can see that using object detection with a good heuristic can out perform the Custom Vision Service for certain tasks. However it’s important to think about the trade offs and work required to annotate our data and build an object detection model.
medium.com
github.com
azure.microsoft.com
medium.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
The next post in this series will review how to train your own object detection model the Cloud using Azure ML Service with subsequent posts will address deployment.
If you have any questions, comments, or topics you would like me to discuss feel free to follow me on Twitter if there is a milestone you feel I missed please let me know.
Aaron (Ari) Bornstein is an avid AI enthusiast with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.