Real-time Object Detection Without Machine Learning | by Jamie Bullock | Towards Data Science

Earlier this year Nick Bourdakos, a software developer at IBM posted a series of videos demoing real-time object detection in a web browser. One of his early videos went viral, receiving over 16,000 likes and 900+ comments on LinkedIn. Here’s the original post:
www.linkedin.com
The video shows three bottles (Coke, Pepsi, and Mountain Dew) being recognised by the computer in real-time as they are held up to the camera. When each bottle is detected, it is given a text label and a bounding box is drawn around it. If more than one bottle is held up, the system will correctly label the different bottles.
Nick’s system has now evolved into IBM cloud annotations, but the demo above used TensorFlow.js along with the COCO-SSD deep learning model. SSD, or Single Shot MultiBox Detector, is a widely used technique for detecting multiple sub-images in a frame, described in detail here. This is a task deep learning excels at and these techniques are now so widespread, you probably have a deep learning network in your pocket, running your phone’s object detection for photos or social networking apps.
Inspired by Nick’s post, I decided to challenge myself to explore if similar results could be achieved without the use of machine learning. It struck me that the bottles used in the original demo could be detected based on their colour or other characteristics along with some simple matching rules. This is known as an heuristic approach to problem solving.
Potential advantages of this include:
Ease of development and conceptualisation
Lower CPU and memory use
Fewer dependencies
In terms of CPU and memory, on my i5 MacBook Pro, the IBM Cloud Annotations demo uses over 100% CPU and more than 1.5 Gigabytes of RAM. It also relies on a web browser and some heavy dependencies including Tensorflow, React.js, node.js and COCO-SSD itself.
The rules I set myself are:
Coke, Pepsi and Mountain Dew bottles must be labelled correctlyA rectangle should be drawn around each bottle as it movesMinimal codeNo machine learning techniques!
Coke, Pepsi and Mountain Dew bottles must be labelled correctly
A rectangle should be drawn around each bottle as it moves
Minimal code
No machine learning techniques!
The original demo claims to use only 10 lines of code, however including boilerplate, the current demo is 107 lines of JavaScript. I think under 100 lines is a good aim for this task.
Firstly, I decided to base my project in OpenCV since I have previously used it for work projects, it has relatively easy setup and is designed specifically for computer vision. OpenCV is written in C++ and has bindings in Python and JavaScript. I decided to go with the Python version for convenience.
I started with just recognising a Coke bottle. For this, a naïve solution would be to analyse the colours in a video frame and place a label where coke red is found. One problem here is that depending on the lighting conditions and camera colour accuracy, the bottle label is unlikely to be exactly RGB 244 0 0.
To solve this, we can use a HSV colour representation along with cv::inRange to find colours within the image that are within a given range. Think “shades of red”. This gives us an image mask with all the red coloured areas white and everything else black. We can then use cv::findContours to supply a list of points that define each “red area” within the frame. The basic code will look something like this:
mask = cv2.inRange(hsv, colour.lower, colour.upper)conts, heirarchy = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)biggest = sorted(conts, key=cv2.contourArea, reverse=True)[0]
The third line of code sorts the detected “red” contours and returns the largest one. Done! ...right? Well unfortunately not. Left like this, the program often finds Coke in the image even when there is none.
To deal with this we need an additional heuristic. I found simply excluding any contour smaller than 50×50 worked well enough.
if w < 50 or h < 50:    continue
Finally, for our detection system to work well, we need to exclude colours that are found “inside” other colours. For example both the Pepsi and Mountain Dew labels contain red, which will get detected as Coke unless we exclude it. So we add a special heuristic for Coke that ignores detection if it is within the vertical bounds of another bottle.
if name == "Coke":    if any([contains_vertical(rects[n], rect) for n in rects]):        continue
Putting it all together, here is a working demonstration of the final system.
Clip ID: 374957292
Delivery:application/vnd.vimeo.dash+json
Embed Size:692×389 
Separate AV:true
Dropped Frames:0 / 0 - 0
Playhead / Buffer:0 / 0 / 0
Bandwidth:0(0 Kbps0 Kbps)
On my i5 MacBook Pro this runs smoothly at around 45% CPU with just over 50MB RAM. The full source code comes to 85 lines and is available here.
One of the limitations of this colour-based approach is that it doesn’t place the bounding box around the bottle but only the coloured area. We could define additional rules to consider the colour above or below the detected region, or attempt to guess where the bounding box should be, but the code would quickly become complicated.
Another limitation is that whilst our system can recognise a Coke and a Pepsi bottle at the same time, it can’t detect two Coke bottles. We could add further heuristics to deal with this but I would question if an heuristic approach is the right choice if so much complexity needs to be added.
I have shown that it’s straightforward to build a heuristic detector with accuracy comparable to that of a deep learning-based system for a highly constrained task. Furthermore, the heuristic object detector is conceptually simpler, has fewer dependencies, takes significantly less CPU and uses an order-of-magnitude less memory.
However, the heuristic approach is not as robust or accurate as using deep learning. A deep learning system can trivially recognise multiple instances of the same object at different scales and rotations, depending on how it is trained. It can also do things like recognise partial objects even if key features are missing.
For me, this isn’t a clear win for deep learning and I think there still is a place for an heuristic approach. The more assumptions that can be made about the detection conditions (consistent background and / or scale, constrained object types, distinguishing features such as colour) the more appeal heuristics have. As a developer, I would consider a heuristic based solution if time and resources were tight and the input constraints were clearly defined. If I wanted increased robustness and flexibility, I would opt for machine learning. Both approaches definitely have their place, and it’s a question of choosing the right tool for the job.