# Temporal-Guided Label Assignment for Video Object Detection
###### Abstract  
In video object detection, the deterioration of an object's appearance in a single frame brings challenges for recognition; therefore, it is natural to exploit temporal information to boost the robustness of video object detection. Existing methods usually utilize temporal information to enhance features, often ignoring the information in label assignments. Label assignment, which assigns labels to anchors for training, is an essential part of object detection. It is also challenged in video object detection and can be improved by temporal information. In this work, a temporal-guided label assignment framework is proposed for the learning task of a region proposal network (RPN). Specifically, we propose a feature instructing module (FIM) to establish the relation model among labels through feature similarity in the temporal dimension. The proposed video object detection framework was evaluated on the ImageNet VID benchmark. Without any additional inference cost, our work obtained a 0.8 mean average precision (mAP(%)) improvement over the baseline and achieved a mAP(%) of 82.0. The result was on par with the state-of-the-art accuracy without using any post-processing methods. video object detection; temporal-guided label assignment; feature instructing module +
Footnote †: journal:  
negative locations. Some methods, such as RetinaNet [12] and the fully convolutional one-stage object detector (FCOS) [13], rely on prior knowledge such as intersection over union (IOU) and ground truth (gt) box locations for sampling. These methods rely on prior human knowledge and cannot adapt to a variety of objects; therefore, dynamic label assignment strategies have been proposed. GuidedAnchoring [14] and MetaAnchor [15] dynamically change the anchor set during training. Feature selective anchor-free (FSAF) [16] method presents an anchor-free branch for guiding label assignment; however, these methods still somehow depend on prior knowledge. There are other methods that are completely data-driven. Optimal transport assignment (OTA) [17] defines label assignment as an optimal transfer problem (OT) from a global perspective. AutoAssign [18] treats all the locations inside the gt boxes as positives and assigns weights for these locations. In AutoAssign, the weights are the labels. Data-driven methods learn from data to adapt to the variance of objects for label assignment; however, this kind of method is also challenged in video object detection. These labels usually fail or underperform when there is a deterioration in the appearance of objects. As shown in Figure 1, frame \(v\) is a blurred image, and the foreground and background are easily confused, so its positive labels are also unfocused. Frame \(v^{\prime}\) is a clear image whose positive labels are focused. The unfocused labels are not conducive to model training (the model has no bias towards positive and negative samples). Figure 1: The visualization of the positive labels produced by our baseline (without temporal information, which will be introduced in the experiment) for two frames. Frame \(v\) and frame \(v^{\prime}\) belong to the same snippet. We only show the positive labels inside the gt box, and the positive labels outside the gt box are all 0 (i.e., the samples outside the gt box cannot be positive samples). Specifically, we can resort to temporal information to deal with this issue. We operated on labels generated by label assignment in the same way as features, i.e., we enhanced labels in the temporal dimension. When an object's appearance deteriorates, the labels can be adjusted using the labels from other frames containing normal objects; this can correct erroneous or poorly focused labels. Our work proposes a temporal-guided label assignment framework that stabilizes the learning process of region proposal network (RPN). Specifically, we put forward a feature instructing module (FIM), which utilizes the similarity of features at different locations to establish the relationship of the corresponding labels.