Study Notes on D&T and AdaScale
By: Ruizhe Li
Tags: Video Processing, Convolutional Network, Computer Vision

Detect to Track and Track to Detect

First, we are going to talk about “Detect to Track and Track to detect”. Along with the development, the procedure of video detection is usually divided into several phases. Complex multi-stage algorithms are proposed. And these algorithms are still getting more and more cumbersome each year. Most algorithms cannot do detection and tracking at the same time. With these background in mind, this paper proposes a model called D&T, which has the following three contributions:

Build an end-to-end ConvNet model for simultaneous detection and tracking

Introduce object co-occurrence features across time to ConvNet during tracking

Achieve high accuracy in both detection and tracking by linking the frame-level detections

Let’s see what the model is like and how it achieves the advantages stated above. Before we get into that, if you are not familiar with R-FCN and RoI Pooling, reading the paper and many other explanatory medium posts would be helpful.

Architecture of Detect and Track (D&T) approach

The above picture shows the model architecture, which basically is built on R-FCN and extended for multi-frame detection and tracking. The model is constructed by two identical R-FCN structures and a correlation module in the middle. The workflow is following specifically.

Two frames in a video, frame at time t and frame at a later time, are input into the model.

Detection: Inputs get through the same R-FCN object detection framework, which is constructed by a feature extractor (i.e. a stack of conv-layers) followed by a region classifier and a regressor in parallel. In this procedure, candidate regions are proposed by an RPN. The feature maps of interested regions get into RoI pooling. Then the pooled features are fed to the classifier (to do object classification) and the regressor (refine bounding boxes).

Tracking: The correlation module takes the intermediate position-sensitive regression maps from both frames as input and outputs the box transformation from one frame to the other. To track multiple objects at the same time, correlation maps are computed for all positions, and RoI pooling is applied to feature maps for track regression.

An example is shown on the bottom below.

Schematic of our approach for two frames at time t and t + τ .

The paper gives a nice example to explain the overall workflow. Here I quote one paragraph: “The inputs are first passed through a fully-convolutional network to produce feature maps. A correlation layer operates on multiple feature maps of different scales (only the coarsest scale is shown in the figure) and estimates local feature similarity for various offsets between the two frames. Finally, position-sensitive RoI-pooling operates on the convolutional features of the individual frames to produce per-frame detections and also on a stack of individual frame-features as well as the between frame correlation features to output regression offsets of the boxes across the two frames (RoI-tracking)”.

The correlation module in the middle of two R-FCN models is actually a regressor using the frame feature maps and correlation feature maps as input to predict the offsets and scales of the object in the frame at a later time.

The loss function of the model has three parts: object classification, bounding box regression, and transformation (offset and scale) of the object between the two frames.

Loss function. The loss in red box is for transformation

Here, we should pay more attention to the term in the red box, which is the loss for transformation. Note that, this term is calculated only for the objects shown in both frames. And the transformation is defined as the relative offsets and log of relative scales of the objects as shown below:

Definition of Transformation

Transformation in parts. Where B stands for a bounding box. (x, y) is the center coordinates of the bounding box. w for width, h for height

Usually, object detection involves the tradeoff between detection accuracy and the number of frames to use due to memory limitations in GPU hardware. In the paper, they use the in-frame tracks to increase precision and reduce the number of frames processed in one iteration. Since video usually contains a lot of redundant information and objects move smoothly as time, it is reasonable to use the predicted tracking information to link detections and build long-term object tubes.

Several tubes (paths) across time are possible in a video. To find the best path, we first define the class-wise linking score, combining detections and tracks. The score is affected by how confident an object belonging to a class in two frames and how much the bounding boxes in two frames are overlapped.

class-wise linking score that combines detections and tracks across time. D for class detection, c for class, t for time, i is an index, T for tracks

The p in the red box stands for the confidence scores for class c in two frames respectively. And the term in the blue box is the pairwise score representing transformation, which is defined below.

Pairwise-score: The pairwise term evaluates to 1 if the IoU overlaps a track correspondence represented by T with the detection boxes represented by D is larger than 0.5.

The optimal path is the one maximizing the scores over the duration T of the video. It is defined as:

There are many methods that can be applied to find the optimal path, e.g. Viterbi algorithm, which is a dynamic programming algorithm. Then, the detection scores in the tube (path) can be re-weighted by adding the mean of the 50% highest scores in the tube, so that the scores are boosted for positive boxes on which the detector fails. Imagine the detector fails to detect an object in the frame at time t, but it manages to detect the same object in frame t-1 and t+1, then the reweighting process would give the false negative detection in frame t a higher score, which could make it a true positive.

In the paper, the model achieves 75.8% mAP (mean average precision) in single image object detection task and 79.8% mAP in video testing.