Bag recognition with Android and TensorFlow using Image classification | by Sumit Mukhija | Towards Data Science

Recently while I was traveling to North-Eastern parts of India, I had to wait for a substantial time for my bag to show up on the airport’s baggage carousel. The area surrounding the carousel was packed with fellow commuters. It was hard to tell my bag apart from the other bags as roughly half the bags looked similar. I had to physically inspect half a dozen bags to ensure that none of them was mine.
I thought someone would have built something to address this problem. I began to look for an existing solution, but I couldn’t find anything. I stumbled upon some of the blogs that demonstrated custom object detection using TensorFlow. I later discovered this incredibly useful resource, based on which I started working towards the solution.
The first thing that I needed was data. I could have clicked a few pictures of my bag, but I decided to capture videos with all the sides and edges of the subject. I extracted individual frames from the videos and handpicked the visually discrete frames. I converted the selected frames to grayscale images. I used ffmpeg. On a command line or terminal, type,
ffmpeg -i video.mp4 -qscale:v 2 image_%03d.jpg
The video.mp4 is the video of my bag and the output images are prefixed by image_. I then needed pictures that were not of my bag. I found a useful Google Chrome extension, named Fatkun. The extension empowers me to download bulk images. I searched for bags on Google Images and downloaded a bunch of those images. Just like the photographs of my bag, I converted the later downloaded pictures to grayscale.
I executed a Python script to convert images to grayscale. The Python script below has a function rescale that takes a directory and converts all the images in that directory to grayscale. The script has one dependency, PIL.
As I was orchestrating my data, there was no risk of an imbalanced data set. I had an equal number of images for my bag and not my bag.
Once I had the images, I downloaded Tensorflow-for-Poets from here. You would need to have TensorFlow installed on your computer. Also, you would need to download PILLOW. You could do both by typing in the commands below.
pip install --upgrade  "tensorflow==1.7.*"pip install PILLOW
In the root directory of Tensorflow-for-poets, I executed,
python3 -m scripts.retrain --bottleneck_dir=tf_files/bottlenecks --how_many_training_steps=500 --model_dir=tf_files/models/ --summaries_dir=tf_files/training_summaries/mobilenet_v2_1.4_224 --output_graph=tf_files/retrained_graph_bags.pb --output_labels=tf_files/retrained_labels.txt --architecture="mobilenet_0.50_224" --image_dir=/Users/sumit/Desktop/carousel/Baggage-Carousel/Labs/data/
The scripts.retrain takes a directory as an input. In my case it was the last part of the command.
image_dir=/Users/sumit/Desktop/carousel/Baggage-Carousel/Labs/data/
The folder looks like this:
The folder named bag holds grayscaled images of my bag. The other folder contains grayscaled pictures of all the other bags that I could get my hands on. The ‘architecture’ argument is the model that I wanted to retrain. I’d encourage everyone to play around with the models. The list of models can be found on here.
The ‘retrained_labels’ is a text file that gets generated and bears the name of the two folders. The two folders behave like two classes. Also, ‘retrained_graph_bags.pb’ is the outcome of the process. Both the files could be found in:
tensorflow-for-poets-2/tf_files
The command should be executed from the root directory of tensorflow-for-poets. When the command is executed, it will initiate the learning process.
Up next, I had to convert the produced graph to a tflite. Tflite or TensorFlowLite is used to deploy machine learning models to mobile devices. The conversion was done by executing the following command
toco — graph_def_file=tf_files/retrained_graph_bags.pb — output_file=tf_files/bagdroid_graph.tflite — output_format=TFLITE — input_shape=1,224,224,3 — input_array=input — output_array=final_result — inference_type=FLOAT — input_data_type=FLOAT
The command above takes the ‘retrained_graph_bags.pb’ as an input and produces a mobile friendly tflite file, named ‘bagdroid_graph.tflite’.
The Android application needed just a few changes. Apart from the cosmetic changes,
I had to copy ‘retrained_labels.txt’ and ‘bagdroid_graph.tflite’ to the assets folder of the application.In ImageClassifier.java, I had to point MODEL_PATH and LABEL_PATH to the correct values.I had to add audio and haptic feedback on detection.
I had to copy ‘retrained_labels.txt’ and ‘bagdroid_graph.tflite’ to the assets folder of the application.
In ImageClassifier.java, I had to point MODEL_PATH and LABEL_PATH to the correct values.
I had to add audio and haptic feedback on detection.
The application on completion was supposed to look better and be able to detect the bag it was trained to detect.
In the end, the application was able to detect my bag, give me audio and haptic feedback.
A pretty obvious problem is that if two bags are identical, the application would identify both the bags. This problem can be mitigated by using NFC chips.
But, the actual problem is that the application often flags random things as my bag. I would love to hear what could I do to minimise the false positives.