Google Objectron — A giant leap for the 3D object detection
By: Jair Ribeiro
Tags: Object Detection, Google, AI, Computer Vision, Artificial Intelligence

bjecrPhoto by Tamara Gak on Unsplash

Google Objectron — A giant leap for the 3D object detection

Google has just announced the launch of MediaPipe Objectron, its mobile technology for real-time detection of 3D objects, enabling the smartphone to recognize the size and orientation of objects.

If there one thing that I love in Google’s research is image analysis. The automatic suggestions of Google Photos or the capabilities of Google Lens are good examples, and the recent news that the company already has its new layer of analysis-ready to understand what is present in each photo.

If we go a little further, to object detection as such, Google also has a few tricks up its sleeve.

Google has just announced the launch of MediaPipe Objectron, its mobile technology for detecting 3D objects in real-time, allowing the smartphone to recognize an object’s size and orientation, opening several application doors.

(MTheiler), CC BY-SA 4.0, via Wikimedia Commons

What is Object Detection?

Humans’ intrinsic ability to rapidly detect, differentiate, and classify objects allows us to make quick decisions regarding what we see.

Detection of objects in images remains one of the most significant challenges in the field of computer vision since objects found in images can be perceived from the most diverse viewpoints and transformations in size and rotation, making it more difficult to detect.

Digital cameras provide good advantages among the critical sensors that AI can use: they can be lightweight, compact, and inexpensive, essential for some environments.

To achieve object detection goals, methods for the extraction of image information, visual attention models, and bio-inspired learning models have been studied to identify and classify vehicles.

Object detection is crucial for good object recognition. A classification technique is to formulate it as a classification problem. Objects are instances with well-defined limits and center, unlike the background.

Methods for detecting and recognizing objects require fewer resources to perform rapid detection due to the increased computational cost. The advanced and complex classifier, such as the CNN — Convolutional Neural Network, is necessary to obtain high accuracy in the Detections.

Detection theories can be used to classify small, low cost or low consumption, electronics, transportation, health, safety, among other areas.

Aphex34, CC BY-SA 4.0, via Wikimedia Commons

The CNN Object Detection problem

The difference between object classification methods (like the networks in the previous section) and object detection methods is that in object detection, you want to:

identify how many objects of a given category are in the image;

objects of a given category are in the image; identify where in the image each object is located.

It is typically not known in advance how many instances of a given object category you will identify in a scene. If you were to build a standard CNN to solve this problem, you would have the following situation: the output size is variable. It depends on how many instances of an object you will find and where they are, which is impossible on a network with an output layer of fixed size. For this, models such as R-CNN and YOLO were developed.

R-CNN: Regions with CNN feature

R-CNN performs segmentation based on the results of object detection. R-CNN initially uses selective search to extract many candidates and then calculates its characteristics through CNN. Finally, it classifies each region using a specific linear classifier, usually SVM (support vector machines). Unlike the networks discussed in the previous section, R-CNN can perform more complex tasks, such as object detection and coarse segmentation of images. An R-CNN can be built on traditional image classification networks, such as AlexNet, VGG, GoogLeNet, and ResNet.

R-CNN solves the problem of multiple exits using a selective search as follows:

1. Generate initial sub-segmentation; we generate many candidate regions

2. Use a greedy algorithm to combine similar regions into larger ones recursively

3. Use the generated regions to produce the final candidate region proposals

R-CNN initially generates around 2000 candidates using the above algorithm based on simple traditional computer vision techniques. From there:

Each candidate is reformatted to a standard size square image; Image is fed to a neural network that generates 4096-dimension feature vectors as output; An SVM classifies the vector of characteristics producing two outputs: a rating an offset indication that can be used to adjust the bounding box.

Disadvantages of the model:

Slow to train: training is in two stages; Very slow to execute: for each image, R-CNN first classifies 2000 sub-images.

Fast-R-CNN: Fast Region-based Convolutional Networks for object detection

In an article the following year, the authors of R-CNN presented a solution that solves the problems of slowness in the approach based on the analysis of 2000 semi-random images of R-CNN: convolutional preprocessing.

Instead of feeding the neural network with candidate images, the entire image is provided to the network to generate a convolutional feature map ( CFM ). This CFM is then used to search for candidate regions, which are reformatted into fixed-size square images through a region-of-interest pooling layer ( RoI pooling ). From the characteristic vector generated for each RoI, a classification is performed with a softmax layer, which predicts the object category and associates it with the origin square’s bounding box.

The graph below shows a comparison between the two approaches and another network from the same time, SPP-Net (Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition) :

Faster-R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Again, the same authors found a solution to make the network even faster was to eliminate the selective search for regions of interest. Similar to Fast R-CNN, the image is used directly as an input to generate a convolutional feature map. In this model, however, instead of performing a selective search on this map, a second, separate neural network is used to predict candidate regions. It is called the Region Proposal Network (RPN). RPN uses a neural mini-network based on a sliding window that analyzes the input image, and the translation is invariant. To avoid excessive proposals, suppression of non-maximums is carried out at this stage.

These candidate regions, like Fast-R-CNN, are reformatted into fixed-size square images through a region-of-interest pooling layer ( RoI pooling ). From the characteristic vector generated for each RoI, a classification is performed with a softmax layer, which predicts the object category and associates it with the bounding box given by the square of origin.

The image below shows a comparison between the three models of the authors and SPP-Net :

Source: http://www.lapix.ufsc.br/ensino/visao/visao-computacionaldeep-learning/deteccao-de-objetos-em-imagens/?lang=en

Single Shot Detection Convolutional Neural Networks for Object Recognition

The above approaches use to focus on the sub-regions of the image to identify objects. The network never looks at the image as a whole. Here we will see approaches that consider the image as a whole.

You Only Look Once: YOLO

In YOLO, a single convolutional network predicts both the bounding boxes and the probabilities of belonging to each detected object’s class. For this, YOLO works as follows:

an image is taken and divided into an SxS grid of cells; using the grid as a reference, m bounding boxes are generated ; bounding boxes with a probability above a threshold are selected and used to locate the object within the image.

Much faster YOLO (45 fps on the authors’ set -> up to two orders of magnitude) faster than contemporary algorithms. Its biggest flaw is inaccuracy with small objects in the image.

Each cell in the grid can predict B bounding boxes ( bbox ) and C class probabilities. A bbox prediction has five components: (x, y, w, h, confidence). The coordinates (x, y) represent the center of the bbox relative to the location of the cell (if the center of a bbox does not fall into a cell, it will not be responsible for it and will not represent it -> cells only have a reference to objects whose center falls into them). These coordinates are normalized to [0, 1]. The bbox (w, h) dimensions are also normalized to [0, 1], relative to the image size.

Suppose you split a 448 × 448-pixel image into a grid of SxS = 3 × 3 cells:

In YOLO, the concept of intersection over union (IoU) plays an important role: the confidence of a prediction in YOLO is given by Pr (Object) * IOU (pred, truth).

Detecting 3D objects in real-time.

While there are ample amounts of 3D data for street scenes, due to the popularity of research into self-driving cars that rely on 3D capture sensors like LIDAR, datasets with ground truth 3D annotations for more granular everyday objects are minimal. To overcome this problem, we developed a novel data pipeline using mobile augmented reality (AR) session data. With the arrival of ARCore and ARKit, hundreds of millions of smartphones now have AR capabilities and the ability to capture additional information during an AR session, including the camera pose, sparse 3D point clouds, estimated lighting, and planar surfaces.

We built a novel annotation tool for AR session data to label ground truth data, allowing annotators to label 3D bounding boxes for objects quickly. This tool uses a split-screen view to display 2D video frames on overlaid 3D bounding boxes on the left, alongside a view showing 3D point clouds, camera positions, and detected planes on the right. Annotators draw 3D bounding boxes in the 3D view and verify its location by reviewing the projections in 2D video frames. For static objects, we only need to annotate an object in a single frame and propagate its location to all frames using the ground truth camera pose information from the AR session data, making the procedure highly efficient.

MediaPipe is Google’s cross-platform framework for creating different data processing pipelines. Objetron is included within this framework as a moving pipeline for real-time 3D object detection.

In other words, Objectron detects objects through 2D images to later estimate their orientation and size through a machine learning model trained on a 3D data set. In other words, it is capable of detecting 3D objects from a single RGB image.

But understanding objects in 3D remains a challenging task due to the lack of large real-world datasets compared to 2D tasks (e.g., ImageNet, COCO, and Open Images).

So, to empower the research community for continued advancement in 3D object understanding, there is a strong need for the release of object-centric video datasets, which capture more of the 3D structure of an object, while matching the data format used for many vision tasks (i.e., video or camera streams), to aid in the training and benchmarking of machine learning models.

Through an encoder and decoder model, Google can convert a 2D input into a 3D output. A single image is enough.

At the basic structure level, the model has an encoder-decoder architecture based on MobileNetV2, Google’s neural networks for classification and visual detection. Using these networks, the 2D projection of the image, and a 3D estimation algorithm, the model can process a 3D output of said object.

To mitigate possible fluctuations in the image, Google has worked under the same detection and tracking framework for its 2D objects. This translates into something that will already sound familiar to you if you know how Google works: image segmentation.

The original image is captured in 2D. Bounding boxes are applied to it for subsequent object detection. The object is detected by Gaussian distribution. The segmentation mask is applied.

Google highlights that the model is so light that it can run in real-time at 26 FPS on a high-end mobile, specifically with the Adreno 650 (the GPU of the recent Snapdragon 865). The new solution is now available to anyone who wants to test it to develop new use cases and applications.

The Dataset

The Objectron dataset is a collection of short, object-centric video clips accompanied by AR session metadata that includes camera poses, sparse point-clouds, and characterization of the planar surfaces in the surrounding environment.

In each video, the camera moves around the object, capturing it from different angles. The data also contains manually annotated 3D bounding boxes for each object, which describe the object’s position, orientation, and dimensions.

The dataset consists of 15K annotated video clips supplemented with over 4M annotated images in the following categories: bikes, books, bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes. It is stored in the objectron bucket on Google Cloud Storage.

Along with the dataset, Google is also sharing a 3D object detection solution for four categories of objects — shoes, chairs, mugs, and cameras.

These models are released in MediaPipe, Google’s open-source framework for cross-platform customizable ML solutions for live and streaming media, which also powers ML solutions like the on-device real-time hand, iris, and body pose tracking.

By releasing this Objectron dataset, Google hopes to enable the research community to push the limits of 3D object geometry understanding and foster new research and applications, such as view synthesis, improved 3D representation, and unsupervised learning.

Dataset Key Features

15000 annotated videos and 4M annotated images

All samples include high-res images, object pose, camera pose, point-cloud, and surface planes.

Ready to use examples in various tf.record formats, which can be used in Tensorflow/PyTorch.

Object-centric multi-views, observing the same object from different angles.

Accurate evaluation metrics, like 3D IoU for oriented 3D bounding boxes.

Photo by Daniel Frank on Unsplash

Conclusion

Humans’ intrinsic ability to rapidly detect, differentiate, and classify objects allows us to make quick decisions regarding what we see.

Detection of objects in images remains one of the most significant challenges in the field of computer vision since objects found in images can be perceived from the most diverse viewpoints and transformations in size and rotation, making it more difficult to detect.

Object detection is crucial for good object recognition. There is a lot of excitement about this google release, especially considering that today there are powerful augmented reality platforms such as AR Core in Google and AR Kit in what refers to Apple.

Video games, applications, photography, video, and more can all benefit from object recognition.

Let’s keep one eye on this technology for future activities and developments by joining the Objectron mailing list and visiting Google’s GitHub page.

Links

References

Google AI Blog — https://ai.googleblog.com/2020/11/announcing-objectron-dataset.html

MediaPipe Objectron — https://google.github.io/mediapipe/solutions/objectron

FLODNet — Detecção e reconhecimento de objetos em dispositivos de baixa especificação: um estudo de caso em classificação de alimentos — https://core.ac.uk/download/pdf/303978504.pdf

Deep Learning:: Object Detection in Images — http://www.lapix.ufsc.br/ensino/visao/visao-computacionaldeep-learning/deteccao-de-objetos-em-imagens/?lang=en