Latest Computer Vision Trends from CVPR 2019
By: Priya Dwivedi
Tags: Machine Learning, Artificial Intelligence, Trends, Computer Vision, Data Science

Doing cool things with data!

The 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) was held this year from June 16- June 20. CVPR is one of the world’s top three academic conferences in the field of computer vision (along with ICCV and ECCV). A total of 1300 papers were accepted this year from a record-high 5165 submissions (25.2 percent acceptance rate).

CVPR brings in top minds in the field of computer vision and every year there are many papers that are very impressive.

I have taken the accepted papers from CVPR and done analysis on them to understand the main areas of research and common keywords in Paper Titles. This can give an indication of where the research is moving.

The underlying data and code is available on my Github. Feel free to pull this and add your own spin to it.

CVPR assigns a primary subject area to each paper. The breakdown of accepted papers by subject area is below:

Not surprisingly, most of the research is focused on Deep Learning (isn’t everything deep learning now!), Detection and Categorization and Face/Gesture/Pose. This breakdown is quite generic and doesn’t really give good insights. So next I extracted all the words from the accepted paper and used a counter to count their frequency. The top 25 most common keywords were below:

Now this in more interesting. Most popular areas of research were detection, segmentation, 3D, and adversarial training. It also shows the growing research in unsupervised learning methods.

Finally I also plotted the Word Cloud.

You can use my Github to pull top papers by topic as shown below

Papers with research on “face”

I run a Machine Learning Consultancy. Check out our website here. I love to work on computer vision projects. Feel free to contact through the website or email at info@deeplearninganalytics.org if you have an idea that we can collaborate on.

Next in the blog I chose 5 interesting papers from the key areas of research. Please note that I picked select papers that appealed the most to me.

The human visual system has a remarkable ability to make sense of our 3D world from its 2D projection. Even in complex environments with multiple moving objects, people are able to maintain a feasible interpretation of the objects’ geometry and depth ordering. A lot of work has been done in depth estimation using camera images in the last few years but robust reconstruction remains difficult in many cases. A particularly challenging case occurs when both the camera and the objects in the scene are freely moving. This confuses traditional 3D reconstruction algorithms that are based on triangulation.

To learn more about depth images and estimating depth of a scene please check out this blog.

This paper solves this by building a deep learning model on a scene where both the camera and subject are freely moving. See gif below:

Depth estimation on moving people

To create such a model we need video sequences of natural scenes captured by moving camera along with accurate depth map for each image. Creating such a data set would be a challenge. To overcome this, the paper very innovatively uses an existing data set — YouTube videos in which people imitate mannequins by freezing in a wide variety of natural poses, while a hand-held camera tours the scene. Because the scene is stationary and only the camera is moving, accurate depth maps can be built using triangulation techniques. This paper is a very interesting read. It solves a complex problem and is very creative in creating a data set for it.

The performance of the trained model on internet video clips with moving cameras and people is much better than any other previous research. See below:

Model comparison through the paper

You can read the full paper here.

2. BubbleNets: Learning to Select the Guidance Frame in Video Object Segmentation by Deep Sorting Frames

I saw several papers on video object segmentation (VOS). This is the task of segmenting an object in a video provided a single annotation in first frame. This finds applications in video understanding and has seen a lot of research in the last one year.

The location and appearance of objects in video can change significantly from frame-to-frame, and, the paper finds that using different frames for annotation changes performance dramatically, as shown below.

Bubblenets video demo

BubbleNets iteratively compares and swaps adjacent video frames until the frame with the greatest predicted performance is ranked highest, at which point, it is selected for the user to annotate and use for video object segmentation.

BubbleNet first frame selection

A video description of the model is shared on youtube and source code is open sourced on Github.

BubbleNets model is used to predict relative performance difference between two frames. Relative performance is measured by a combination of region similarity and contour accuracy.

It takes as input 2 frames to compare and 3 reference frames. It then passes these through ResNet50 and fully connected layers to output a single number f denoting the comparison of the 2 frames. To perform bubble sort, we start with the first 2 frames and compare them. If BubbleNet predicts that frame 1 has better performance than frame 2 then order of frames is swapped and the next frame is compared with the best frame so far. At the end of processing through the entire video sequence the best frame remains. The figure below shows BubbleNets architecture and process for bubble sort.

Overall the authors show that changing the way the annotation frame is selected with no change to underlying segmentation algorithm results in an 11% increase in perform on the DAVIS benchmark data set.

Bubblenets architecture

3. 3D Hand Shape and Pose Estimation from a Single RGB Image

3D hand shape and pose estimation has been a very active area of research lately. This has applications in VR and Robotics. This paper uses a monocular RGB image to create a 3D hand pose and 3D mesh around the hand as shown below.

3D hand mesh from single image

The paper uses Graph CNNs to reconstruct a full 3D mesh of the hand. Here is a good introduction to the topic of Graph CNNs. To train the network, the authors created a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. Manually annotating the ground truth 3D hand meshes on real-world RGB images is extremely laborious and time-consuming. However, models trained on the synthetic dataset usually produce unsatisfactory estimation results on real-world datasets due to the domain gap between them. To address this issue, the authors propose a novel weakly supervised method by leveraging depth map as a weak supervision for 3D mesh generation, since depth map can be easily captured by an RGB-D camera when collecting real world training data. The paper has rich details on data set, training process etc. Please read through it if this is an area that interests you.

One interesting learning for me was the architecture of the Graph CNN used for mesh generation. The input to this network is a latent vector from the RGB image. It goes through 2 fully connected layers to output 80x64 features in a coarse graph. It then goes through layers of upsampling and Graph CNNs to output richer details resulting in a final output of 1280 vertices.

3D hand mesh model architecture

4. Reasoning-RCNN: Unifying Adaptive Global Reasoning into Large-scale Object Detection

Reasoning RCNN output

Object detection has gained a lot of popularity with many common computer vision applications. Faster RCNN is a popular object detection model that is frequently used. To learn more about object detection and Faster RCNN checkout this blog. However object detection is most successful when number of detection classes is small — less than 100. This paper addresses the large-scale object detection problem with thousands of categories, which poses severe challenges due to long-tail data distributions, heavy occlusions, and class ambiguities.

Reasoning-RCNN does this by constructing a knowledge graph that encodes common human sense knowledge. What is Knowledge Graph? Knowledge graph encodes information between objects such as spatial relationship (on, near), subject-verb-object (ex. Drive, run) relationship as well as attribute similarities like color, size, material. As shown below categories with visual relationship to each other are closer to each other.

Knowledge Graph

In terms of architecture it stacks a Reasoning framework on top of a standard object detector like Faster RCNN. The weights of the previous classifier are collected to generate a global semantic pool over all categories, which is fed into an adaptive global reasoning module. The enhanced category contexts (i.e., output of the reasoning module) are mapped back to region proposals by a soft-mapping mechanism. Finally, each region’s enhanced features are used to improve the performance of both classification and localization in an end-to-end manner. The diagram below shows the model architecture. Please refer to the paper to get more detailed understanding of their architecture.

The model is trained and evaluated on 3 main datasets — Visual Gnome (3000 categories), ADE (445 categories) and COCO (80 categories). The model is able to get 16% improvement on Visual Gnome, 37% on ADE and a 15% improvement in COCO on mAP scores.

Training code will be open sourced at this link. Not available yet.

5. Deep Learning for Zero Shot Face Anti-Spoofing

A lot of progress has been made on Facial Detection in the last few years and now facial detection and recognition systems are commonly used in many applications. In Fact it is possible to build a system that detects faces, recognizes them and understands their emotions in 8 lines of code. See blog here.

However there is also continuous risk of face detection being spoofed to gain illegal access. Face anti-spoofing is designed to prevent face recognition systems from recognizing fake faces as the genuine users. While advanced face anti-spoofing methods are developed, new types of spoof attacks are also being created and becoming a threat to all existing systems. This paper introduces the concept of detecting unknown spoof attacks as s Zero-Shot Face Anti-spoofing (ZSFA). Previous ZSFA works only study 1- 2 types of spoof attacks, such as print/replay, which limits the insight of this problem. This work investigates the ZSFA problem in a wide range of 13 types of spoof attacks, including print, replay, 3D mask, and so on. The image below shows different types of spoof attacks.

Face spoofing can include various forms like print (print a face photo), replaying a video, 3D mask, face photo with cutout for eyes, makeup, transparent mask etc. The paper proposes to use a deep tree network to learn semantic embeddings from spoof pictures in unsupervised fashion. Embeddings here could model things like human gaze. It creates a data set of spoof images to learn these embeddings. During the testing, the unknown attacks are projected to the embedding to find the closest attributes for spoof detection.

Read the paper for more detail about the model architecture for deep tree network and process for training it. The paper is able to create embeddings that separate out live face (True Face) with various types of spoofs. See t-SNE plot below

This paper was awesome. A promising research into tackling a practical problem.

Conclusion

It is fascinating to see all the latest research in Computer Vision. The 5 papers shared here are just the tip of the iceberg. I hope you will use my Github to sort through the papers and select the ones that interest you.

I am extremely passionate about computer vision and deep learning in general. I have my own deep learning consultancy and love to work on interesting problems. I have helped many startups deploy innovative AI based solutions. Check us out at — http://deeplearninganalytics.org/.

You can also see my other writings at: https://medium.com/@priya.dwivedi

If you have a project that we can collaborate on, then please contact me through my website or at info@deeplearninganalytics.org

References: