Paper Review — An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection
By: Krut Patel
Tags: Machine Learning, Synthetic Data, Computer Vision, Object Detection, Deep Learning

Stefan Hinterstoisser, Olivier Pauly, Hauke Heibel, Martina Marek, Martin Bokeloh

ArXiv link — https://arxiv.org/abs/1902.09967

Research Problem

Data plays an important role in the performance of a machine learning model. Availability of domain specific datasets is a problem and often data needs to be collected and labelled manually — which is time consuming, expensive and error prone. An inexpensive solution is to generate training data synthetically. This allows generating an infinite amount of labelled training images with large and controlled variations.

Prior Works

A huge number of prior works have suggested techniques for synthesizing training data and a major challenge faced by all of them is — bridging the domain gap between synthesized and real world images. Hence models trained on synthetic data don’t perform well on real world data.

Several techniques like mixing synthetic and real data, using photo realistic rendering and realistic scene composition, using GANs, leveraging architectures with frozen pre-trained feature extractors, or using domain adaptation or transfer learning and domain randomization have been used to close the domain gap. But none of them have shown any signification promise.

Contributions

In this paper, the authors -

Introduce a novel approach for creating synthetic training data for object detection that generalizes well to real data.

Their technique is of domain randomization type — altering the simulated data with non-realistic changes so that reality seems to be just a variation. This helps overcome the domain gap.

Introduce a generation and rendering process that follows a curriculum strategy — ensuring that all objects of interests are presented to the network equally under all possible poses, conditions and complexity.

They demonstrate with experiments that object detectors trained on synthetic training data generated using their approach outperform other object detectors trained on a combination dataset of real data and synthetic data as well as ones trained purely on real data.

They also provide detailed ablation study of their generation pipeline.

Generation Approach

They blend together three image layers — a synthetic background layer, a foreground object layer built using curriculum strategy and lastly a layer of occluders.

Synthetic data generation pipeline (src — https://arxiv.org/pdf/1902.09967.pdf)

1. Background Layer Generation

This layer is generated using a set of 15k textured 3D models. All of them are pre-scaled such that they fit into a unit sphere.

Objects are randomly placed in the background with random pose until the entire background is covered.

The size of the projected background objects is determined based of the size of the foreground object. It is kept within a randomized isotropic scaling range S = [S_min, S_max]. This controls the background objects size so that their projection to the image plane corresponds to the size of the average foreground object size.

The hue value of each object’s texture is randomly changed to diversify backgrounds.

All this steps maximize background clutter, minimize the risk of showing a network the same background image twice, and creating background images with structures similar in scale to foreground objects.

All this in order helps the networks to learn the geometric and visual appearances of objects while minimizing the changes of just learning to distinguish synthetic foreground objects from background objects only based of different object sizes or noise distributions.

2. Curriculum Foreground Layer Generation

A large set of poses are uniformly generated for each foreground object by generating random rotations through recursively dividing an icosahedron — this approach gives an uniformly distributed list of vertices on a sphere and each vertex represents a distinct view of an object defined by two out-of-plane rotations. Also equally sampled in-place-rotation are generated.

The foreground object is rendered based on a curriculum strategy — start with a scale closest to the camera, gradually moving away and at each scale iterating through all possible out-of-plane and in-plane rotations.

The object’s rendering distance is sampled in such a way that it is inversely proportional to the object’s projected size — this makes sure that an approximate linear change in pixel coverage of the object is guaranteed at consecutive scale levels.

An overlap between each pair of foreground objects is allowed up to 30%

Cropping of foreground objects at corners is allowed up to 50%

3. Occlusion Layer Generation

Random objects from the background dataset are placed such that they partially occlude the foreground object — in a range of 10% to 30%

The pose and color of the object is randomized in the same way as background objects.

4. Post-processing and Layer Fusion

Background, foreground and the occlusion layers are fused into one image.

Random light sources with random light color are added.

White noise and blur with a randomly sized Gaussian kernel with a random standard deviation is added.

Experiments

The paper shows detailed info on their experimentation setup along with the results demonstrating that their synthetic data generation technique outperforms state-of-the-art detectors trained on real world datasets.

They show with ablation experiments the benefits of curriculum vs random pose generation, effects of relative scaling of background objects w.r.t. foreground objects, effects of the amount of foreground objects rendered per image, benefits of using synthetic background objects and effects of random colors and blur.

For the synthetic vs real experiments they compare the performance of three Faster R-CNN models with an Inception ResNet feature extractor initialized with pretrained ImageNet weights. The first one trained based of Hinterstoisse et al 2018 method where they freeze the feature extractor layers with pre-trained weights trained on real world data and train only the remaining layers. The second one trained using real data + data augmentation. The third one on synthetic data generated using their approach. The results show that the third one clearly outperforms the other two models as show in figure below

Synthetic vs real experiment (src — https://arxiv.org/pdf/1902.09967.pdf)

3rd Party Code

SynthDet — is an open source project by Unity which is inspired by this paper:

They also have a blog showing their experiments:

Thoughts

This paper’s approach shows a possibility that models trained on synthetic data can match or outperform real world data. This method can be of significant use in applications where large domain specific datasets are not available.