Do Different Tracking Tasks Require Different Appearance Models?

1 Introduction . Unlike popular image-based computer vision tasks such as classification and object detection , which are ( for the most part ) unambiguous and clearly defined , the problem of object tracking has been considered under different setups and scenarios , each motivating the design of a separate set of benchmarks and methods . For instance , for the Single Object Tracking ( SOT ) and Video Object Segmentation ( VOS ) communities [ 70 , 29 , 48 ] , tracking means estimating the location of an arbitrary user-annotated target object throughout a video , where the location of the object is represented by a bounding box in SOT and by a pixel-wise mask in VOS . Instead , in multiple object tracking settings ( MOT [ 41 ] , MOTS [ 57 ] and PoseTrack [ 2 ] ) , tracking means connecting sets of ( often given ) detections across video frames to address the problem of identity association and forming trajectories . Despite these tasks only differing in the number of objects per frame to consider and observation format ( bounding boxes , keypoints or masks ) , the best practices developed by the methods tackling them vary significantly . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Though the proliferation of setups , benchmarks and methods is positive in that it allows specific use cases to be thoroughly studied , we argue it makes increasingly harder to effectively study one of the fundamental problems that all these tasks have in common , i.e . what constitutes a good representation to track objects throughout a video ? Recent advancements in large-scale models for language [ 15 , 6 ] and vision [ 24 , 10 ] have suggested that a strong representation can help addressing multiple downstream tasks . Similarly , we speculate that a good representation is likely to benefit many different tracking tasks , regardless of their specific setup . In order to validate our speculation , in this paper we present a framework that allows to adopt the same appearance model to address five different tracking tasks ( Figure 2 ) . In our taxonomy ( Figure 4 ) , we consider existing tracking tasks as problems that have either propagation or association at their core . When the core problem is propagation ( as in SOT and VOS ) , one has to localise a target object in the current frame given its location in the previous one . Instead , in association problems ( MOT , MOTS , and PoseTrack ) , target states in both previous and current frames are given , and the goal is to determine the correspondence between the two sets of observations . We show how most tracking tasks currently considered by the community can be simply expressed starting from the primitives of propagation or association . For propagation tasks , we employ existing box and mask propagation algorithms [ 5 , 61 , 58 ] . For association tasks , we propose a novel reconstruction-based metric that leverages fine-grained correspondence to measure similarities between observations . In the proposed framework , each individual task is assigned to a dedicated “ head ” that allows to represent the object ( s ) in the appropriate format to compare against prior arts on the relevant benchmarks . Note that , in our framework , only the appearance model contains parameters that can be learned via back-propagation , and that we do not experiment with appearance models that have been trained on specific tracking tasks . Instead , we adopt models trained via recent self-supervised learning ( SSL ) techniques and that have already demonstrated their effectiveness on a variety of image-based tasks . Our motivation is twofold . First , SSL models are particularly interesting for our use-case , as they are explicitly conceived to be of general purpose . As a byproduct , our work also serves the purpose of evaluating and comparing appearance models obtained from self-supervised learning approaches ( see Figure 1 ) . Second , we hope to facilitate the tracking community in directly benefiting from the rapid advancements of the self-supervised learning literature . To summarise , the contributions of our work are as follows : • We propose UniTrack , a framework that supports five tracking tasks : SOT [ 70 ] , VOS [ 48 ] , MOT [ 41 ] , MOTS [ 57 ] , and PoseTrack [ 2 ] ; and that can be easily extended to new ones . • We show how UniTrack can leverage many existing general-purpose appearance models to achieve a performance that is competitive with the state-of-the-art on several tracking tasks . • We propose a novel reconstruction-based similarity metric for association that preserves fine-grained visual features and supports multiple observation formats ( box , mask and pose ) . • We perform an extensive evaluation of self-supervised models , significantly extending the empirical analysis of prior literature to video-based tasks . 2 The UniTrack Framework . 2.1 Overview . Inspecting existing tracking tasks and benchmarks , we noticed that their differences can be roughly categorised across four axes , illustrated in Figure 2 and detailed below . 1 . Whether the requirement is to track a single object ( SOT [ 70 , 29 ] , VOS [ 48 ] ) , or multiple objects ( MOT [ 48 ] , MOTS [ 57 ] , PoseTrack [ 2 ] ) . 2 . Whether the targets are specified by a user in the first frame only ( SOT , VOS ) , or instead are given in every frame , e.g . by a pre-trained detector ( MOT , MOTS , PoseTrack ) . 3 . Whether the target objects are represented by bounding-boxes ( SOT , MOT ) , pixel-wise masks ( VOS , MOTS ) or pose annotations ( PoseTrack ) . 4 . Whether the task is class-agnostic , i.e . the target objects can be of any class ( SOT , VOS ) ; or if instead they are from a predefined set of classes ( MOT , MOTS , PoseTrack ) . SOT VOS MO TPoseT MO TS InsDis SOT VOS MO TPoseT MO TS MoCo-v1 SOT VOS MO TPoseT MO TS PCL-v1 SOT VOS MO TPoseT MO TS PIRL SOT VOS MO TPoseT MO TS PCL-v2 SOT VOS MO TPoseT MO TS SimCLR-v1 SOT VOS MO TPoseT MO TS MoCo-v2 SOT VOS MO TPoseT MO TS SimCLR-v2 SOT VOS MO TPoseT MO TS SeLa-v2 SOT VOS MO TPoseT MO TS InfoMin SOT VOS MO TPoseT MO TS BYOL SOT VOS MO TPoseT MO TS DeepCluster-v2 SOT VOS MO TPoseT MO TS SwAV SOT VOS MO TPoseT MO TS BarlowTwins SOT VOS MO TPoseT MO TS DetCo SOT VOS MO TPoseT MO TS VFS Figure 1 : High-level overview of the performance of sixteen self-supervised learning models on five tracking tasks : SOT , VOS , MOT , PoseTracking and MOTS . A higher rank ( better performance ) corresponds to a vertex nearer to the outer circle . A larger area of the pentagon signifies better overall performance of its respective appearance model . Results of a vanilla ImageNet-supervised model are indicated with a gray dashed line as reference . Notice how the best model VFS [ 74 ] dominates on four out of the five tasks considered . Single object , user-specified Multiple objects , detector-specified B o x M as k Po se Task : Single Object Tracking ( SOT ) Datasets : OTB-100 , VOT-16/19 Task : Multi-Object Tracking ( MOT ) Datasets : MOT challenge , KITTI Task : Multi-Object Tracking & Seg . ( MOTS ) Datasets : MOTS , KITTI-MOTS Task : Pose Tracking ( PoseTrack ) Datasets : PoseTrack-2017/2018 Task : Video Object Segmentation ( VOS ) Datasets : DAVIS-2016/2017 Task : Pose Propagation Datasets : JHMDB Class-agnostic Class-agnostic Class-specific Class-specific Class-specific Class-specific Figure 2 : Existing tracking problems and their respective benchmarks differ from each other under several aspects : the assumption could be that there is a single or multiple objects to track ; targets can be specified by the user in the first frame only , or assumed to be given at every frame ( e.g . provided by a detector ) ; the classes of the targets can be known ( classspecific ) or unknown ( class-agnostic ) ; the representation of the targets can be bounding boxes , pixel-wise masks , or pose annotations . Appearance Model AssociationPropagation SOT VOS MOTS PoseTrack MOT L e v e l1 T ra in a b le L e v e l2 T ra in in g -f re e L e v e l3 T ra in in g -f re e Figure 3 : Overview of UniTrack . The framework can be divided in three levels . Level-1 : a trainable appearance model . Level-2 : the fundamental primitives of propagation and association . Level-3 : task-specific heads . Typically , in single-object tasks the target is specified by the user in the first frame , and it can be of any class . Instead , for multi-object tasks detections are generally considered as given for every frame , and the main challenge is to solve identity association for the several objects . Moreover , in multi-object tasks the set of classes to address is generally known ( e.g . pedestrians or cars ) . Figure 3 depicts a schematic overview of the proposed UniTrack framework , which can be understood as conceptually divided in three “ levels ” . The first level is represented by the appearance model , responsible for extracting high-resolution feature maps from the input frame ( Section 2.2 ) . The second level consists of the algorithmic primitives addressing propagation ( Section 2.3 ) and association ( Section 2.4 ) . Finally , the last level comprises multiple task-specific algorithms that make direct use of the primitives of the second level . In this work , we illustrate how UniTrack can be used to obtain competitive performance on all of the five tracking tasks of level-3 from Figure 3 . Moreover , new tracking tasks can be easily integrated . Importantly , note that the appearance model is the only component containing trainable parameters . The reason we opted for a shared and non task-specific representation is twofold . Firstly , the large amount of different setups motivated us to investigate whether having separately-trained models for each setup is necessary . Since training on specific datasets can bias the representation towards a limited set of visual concepts ( e.g . animals or vehicles ) and limit its applicability to “ open-world ” settings , we wanted to understand how far can a shared representation go . Second , we wanted to provide the community with multiple baselines that can be used to better assess newly proposed contributions , and that can be immediately used on new datasets and tasks without the need of retraining . 2.2 Base appearance model The base appearance model φ takes as input a 2D image I and outputs a feature map X = φ ( I ) ∈ RH×W×C . Since ideally an appearance model used for object propagation and association should be able to leverage fine-grained semantic correspondences between images , we choose a network with a small stride of r = 8 , so that its output in feature space can have a relatively large resolution . We refer to the vector ( along the channel dimension ) of a single point in the feature map as a point vector . We expect a point vector xi1 ∈ RC from the feature map X1 to have a high similarity with its “ true match ” point vector xî2 in X2 , while being far apart from all the other point vectors x j 2 in X2 ; i.e . we expect s ( xi1 , x î 2 ) > s ( x i 1 , x j 2 ) , ∀j 6= î , where s ( · , · ) represents a similarity function . In order to learn fine-grained correspondences , fully-supervised methods are only amenable for synthetic datasets ( e.g . Flying Chairs for optical flow [ 16 ] ) . With real-world data , it is intractable to label pixel-level correspondences and train models in a fully-supervised fashion . To overcome this obstacle , in this paper we adopt representations obtained with self-supervision . We experiment both with models trained with approaches that leverage pixel-wise pretext tasks [ 27 , 58 ] and , inspired by prior works that have pointed out how fine-grained correspondences emerge in middle-level features [ 39 , 74 ] , with models obtained from image-level tasks ( e.g . MoCo [ 24 ] , SimCLR [ 10 ] ) .

Summary: This paper proposes using a unified self-supervised framework for various video-related tracking tasks including single-object bounding box (SOT) or mask propagation (VOS) or multiple object bounding box (MOT), mask (MOTS) or keypoints (PoseTrack) association of detections across video frames. The idea proposed in this paper is inspired by the finding that mid-level representations of networks trained either with full-supervision (e.g., for object classification) or via self-supervised learning for a pre-text task are believed to contain information for establishing dense correspondence between objects and instances. The authors propose to use a single backbone pre-trained network to extract fc3/fc4 features for these various tasks and further employ methods based on dense affinity/ discriminative correlation filters/Hungarian matrix, etc to perform the various propagation or association tasks. The main finding that the authors share is that a common task-agnostic mid-level representation learned using ImageNet is competitive with many supervised and unsupervised task-specific algorithms designed for individual tasks.