Instance-level Recognition. Introduction, challenges, and recent... | by Kb Pachauri | Towards Data Science

In this blog, I will walk through an introduction to instance-level recognition, use cases, challenges, currently available dataset, and state of the art results (recent winner solutions) on these challenges/datasets.
Instance Level Recognition (ILR), is a visual recognition task to recognize a specific instance of an object not just object class.
For example, as shown in the above image, painting is an object class, and “Mona Lisa” by Leonardo Da Vinci is an instance of that painting. Similarly, the Taj Mahal, India is an instance of the object class building.
Landmark Recognition: Recognize landmarks in images.
Landmark Retrieval: Retrieve relevant landmark images from a large-scale database.
Artwork Recognition: Recognize artworks in images.
Product Retrieval: Retrieve relevant product images from a large-scale database.
Large scale: Most of the current state of the art results of recognition tasks are measured on very limited categories e.g. ~1000 image classes in ImageNet, ~80 categories in COCO. But use-cases like landmark retrieval and recognition, has 200K+ classes e.g. in Google Landmark Dataset V2 (GLDv2), 100K+ classes of the product on Amazon.
Long-tailed: Few popular places have more than 1000+ images but many less know places have less than 5 images in GLDv2.
Intra-class variability: Landmarks are mostly spread across a wide region and have very high intra-class variability as shown in the below image.
Noisy Labels: The success of machine learning models depends on high-quality labeled training data, as the presence of labels errors can greatly reduce the model's performance. These noisy labels as shown in the below image, unfortunately, noisy labels are part of a large training set and need additional learning steps.
Google Landmarks Dataset V2 (GLDv2): Google Landmark Dataset v2 is a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. The below image shows all the details of the landmark dataset
CVPR 2020 AliProducts Challenge: AliProducts Challenge is a large-scale, noisy, and fine-grained product dataset with ~50K categories with ~3M images. This dataset is proposed as competition for studying commodity image recognition problems encountered by world-leading e-commerce companies. This competition is part of the RetailVision workshop RetailVision CVPR 2020 workshop at CVPR 2020.
Amazon Product Challenge: Amazon product dataset is announced in ILR ECCVW 2020 workshop and will be released by 2021 Q1. Images in the dataset have a domain mismatch due to product catalog images are shot by professional and query images are often shot by mobile phone camera and have bad lighting condition with a cluttered background.
Below I will discuss recent winner solutions for landmark recognition and product recognition.
In the landmark recognition challenge, the task is to predict landmark labels from the image and is conducted every year since 2018. The below image shows the progress in the Global Average Precision of this competition.
An ensemble of 7 global descriptors model (SeResNext101, EfficientNet-B3, ResNet152, Res2Net101). The below image shows the setup for the SeResNext101 backbone and for others also we follow similar architecture. Each backbone network is aggregated using Generalized-Mean (GeM) pooling and followed by linear neck (Linear 512, BatchNorm1D, PReLU), before finally feeding into the arc margin head.
ResNet variants (152 & 101) both rely on identifying shortcut connections that skip one or more layers to tackle the vanishing gradient issue. SeResNeXt is a variant of ResNeXt which is an inception net with a shortcut connection and Se refers to the Squeeze and Excitation module added to ResNeXt. Se network improves channel interdependency by adaptively adjusting the weights of feature maps. EfficientNet is a state of the art network for image classification which relies on Auto Machine Learning to find out the best base network and efficient compound scaling to achieve improved results depending on available compute resources.
Generalized Mean Pooling (GeM) in the neck network computes the generalized mean of each channel in a tensor. If pk → ∞, GeM behaves as max-pooling and pk → 1, it behaves as average pooling. With an increase in pk, the contrast of the pooled feature map increases and focuses on salient features of the image.
PReLU: PReLU is a generalization of leaky ReLU, to solve the dying neuron problem which happens when data is not normalized or network weights are not properly initialized.
Arc margin improves on softmax loss to enforce higher similarity for intra-class variation and diversity for inter-class by distributing learned embedding on hypersphere with a radius of s. Below is the pseudo-code of ArcFace loss on MxNet.
Models are trained at different image scales [448x448, 568x568, 600x600, 512x512] using albumentations.
Each model is trained for 10 epochs with cosine annealing scheduler.
The test set of the 2019 competition which was released with labels is used as validation.
As a post-processing step re-ranking was done to penalize non-landmarks images to improve the GAP metric.
Test: Leaderboard test set.
Train: Candidate images to determine labels and confidence.
Non-Landmark: Images with no landmark from the GLDv2 test set.
Calculate the cosine similarity between the test and train image (A)Calculate the average (top-5 or top-10) cosine similarity between the train and non-landmark images. (B)Calculate Ai,j — BjSum the confidence of the same label and pick the highest.
Calculate the cosine similarity between the test and train image (A)
Calculate the average (top-5 or top-10) cosine similarity between the train and non-landmark images. (B)
Calculate Ai,j — Bj
Sum the confidence of the same label and pick the highest.
The most important point of the above solution is the use of the 2019 competition test set as validation for re-ranking as post-processing after inferencing and led to a top leaderboard score of 0.6598 which is ~1.75x better than the 2019 result.
Backbone networks (EfficientNet-B3, EfficientNet-B4, ResNet50, SeResNext50, SeResNext101) are fine-tune with Destruction and Construction Learning (DCL) and Look into Object (LIO) method. The model averaging is used to ensemble all the fine-tune models, achieving a top-1 error rate of 6.27%.
DCL as shown in the below image enhances fine-grained recognition by learning local discriminative regions and features by shuffling local regions. To prevent the network to learn the noisy patterns, an adversarial counterpart is proposed to reject Region Confusion Mechanism (RCM)-induced patterns that are not relevant. For more details kindly check the paper.
LIO as shown in the below image, models structure using self-supervised learning. Object extent learning helps the backbone network to distinguish between foreground and background. Spatial context learning using a self-supervision strengthen structural information for the backbone network. For more details kindly check the paper.
All images are resized to 256x256 then random crop to 224x224 for training and center crop 224x224 for the test. Train data is augmented using
AutoAugmentation
Cutout
Training
All models are trained with an SGD optimizer with manual learning rate decay.
All backbone networks with basic training achieve a top-1 error rate of 20–25%.All backbone networks are fine-tuned with balanced training, achieving a top-1 error rate of 9–12%. The balanced training set includes all the images from validation as well if the number of images in a category is less than 30.All backbone are further fine-tuned using DCL on higher resolution image (448x448), reducing error rate by 1–2% further.All networks are further fine-tuned using accuracy loss as shown in the below image, which is optimized for the top-1 error rate, reducing error rate by ~0.2–0.5%.
All backbone networks with basic training achieve a top-1 error rate of 20–25%.
All backbone networks are fine-tuned with balanced training, achieving a top-1 error rate of 9–12%. The balanced training set includes all the images from validation as well if the number of images in a category is less than 30.
All backbone are further fine-tuned using DCL on higher resolution image (448x448), reducing error rate by 1–2% further.
All networks are further fine-tuned using accuracy loss as shown in the below image, which is optimized for the top-1 error rate, reducing error rate by ~0.2–0.5%.
def acc_loss(y_true, y_pred):    tp = (y_pred, y_true).sum(1)    fp = ((1-y_true)*y_pred).sum(1)    acc = tp/(tp+fp)    return 1 - acc.mean()
Below 11 models are used to calculate final probabilities
Balanced fine-tuned resnet50, seresnext50, seresnext101, efficientnet-b3, efficientnet-b4
DCL fine-tuned resnet50, seresnext50
Accuray loss fine-tunes resnet50, seresnext50, efficientnet-b3
LIO fine-tuned resnet50
Instance-level recognition will unravel the true potential of deep learning technologies for semantic image classification/retrieval for eCommerce, travel, media & entertainment, agriculture, etc. Some of the major building block for an efficient instance-level solution is
Backbone Network Selection (Residual, Squeeze & Excitation, EfficientNet)
Data Augmentation (Albumentation, AutoAugment, Cutout, etc).
Loss function (ArcFace, AccuracyLoss).
Multi-scale processing.
Fine-tuning and post-processing.
Thanks for reading the article, I hope you found this to be helpful. If you did, please share it on your favorite social media so other folks can find it, too. Also, please let me know in the comment section if something is not clear or incorrect.