Do Different Tracking Tasks Require Different Appearance Models?

1 Introduction . Unlike popular image-based computer vision tasks such as classification and object detection , which are ( for the most part ) unambiguous and clearly defined , the problem of object tracking has been considered under different setups and scenarios , each motivating the design of a separate set of benchmarks and methods . For instance , for the Single Object Tracking ( SOT ) and Video Object Segmentation ( VOS ) communities [ 70 , 29 , 48 ] , tracking means estimating the location of an arbitrary user-annotated target object throughout a video , where the location of the object is represented by a bounding box in SOT and by a pixel-wise mask in VOS . Instead , in multiple object tracking settings ( MOT [ 41 ] , MOTS [ 57 ] and PoseTrack [ 2 ] ) , tracking means connecting sets of ( often given ) detections across video frames to address the problem of identity association and forming trajectories . Despite these tasks only differing in the number of objects per frame to consider and observation format ( bounding boxes , keypoints or masks ) , the best practices developed by the methods tackling them vary significantly . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Though the proliferation of setups , benchmarks and methods is positive in that it allows specific use cases to be thoroughly studied , we argue it makes increasingly harder to effectively study one of the fundamental problems that all these tasks have in common , i.e . what constitutes a good representation to track objects throughout a video ? Recent advancements in large-scale models for language [ 15 , 6 ] and vision [ 24 , 10 ] have suggested that a strong representation can help addressing multiple downstream tasks . Similarly , we speculate that a good representation is likely to benefit many different tracking tasks , regardless of their specific setup . In order to validate our speculation , in this paper we present a framework that allows to adopt the same appearance model to address five different tracking tasks ( Figure 2 ) . In our taxonomy ( Figure 4 ) , we consider existing tracking tasks as problems that have either propagation or association at their core . When the core problem is propagation ( as in SOT and VOS ) , one has to localise a target object in the current frame given its location in the previous one . Instead , in association problems ( MOT , MOTS , and PoseTrack ) , target states in both previous and current frames are given , and the goal is to determine the correspondence between the two sets of observations . We show how most tracking tasks currently considered by the community can be simply expressed starting from the primitives of propagation or association . For propagation tasks , we employ existing box and mask propagation algorithms [ 5 , 61 , 58 ] . For association tasks , we propose a novel reconstruction-based metric that leverages fine-grained correspondence to measure similarities between observations . In the proposed framework , each individual task is assigned to a dedicated “ head ” that allows to represent the object ( s ) in the appropriate format to compare against prior arts on the relevant benchmarks . Note that , in our framework , only the appearance model contains parameters that can be learned via back-propagation , and that we do not experiment with appearance models that have been trained on specific tracking tasks . Instead , we adopt models trained via recent self-supervised learning ( SSL ) techniques and that have already demonstrated their effectiveness on a variety of image-based tasks . Our motivation is twofold . First , SSL models are particularly interesting for our use-case , as they are explicitly conceived to be of general purpose . As a byproduct , our work also serves the purpose of evaluating and comparing appearance models obtained from self-supervised learning approaches ( see Figure 1 ) . Second , we hope to facilitate the tracking community in directly benefiting from the rapid advancements of the self-supervised learning literature . To summarise , the contributions of our work are as follows : • We propose UniTrack , a framework that supports five tracking tasks : SOT [ 70 ] , VOS [ 48 ] , MOT [ 41 ] , MOTS [ 57 ] , and PoseTrack [ 2 ] ; and that can be easily extended to new ones . • We show how UniTrack can leverage many existing general-purpose appearance models to achieve a performance that is competitive with the state-of-the-art on several tracking tasks . • We propose a novel reconstruction-based similarity metric for association that preserves fine-grained visual features and supports multiple observation formats ( box , mask and pose ) . • We perform an extensive evaluation of self-supervised models , significantly extending the empirical analysis of prior literature to video-based tasks . 2 The UniTrack Framework . 2.1 Overview . Inspecting existing tracking tasks and benchmarks , we noticed that their differences can be roughly categorised across four axes , illustrated in Figure 2 and detailed below . 1 . Whether the requirement is to track a single object ( SOT [ 70 , 29 ] , VOS [ 48 ] ) , or multiple objects ( MOT [ 48 ] , MOTS [ 57 ] , PoseTrack [ 2 ] ) . 2 . Whether the targets are specified by a user in the first frame only ( SOT , VOS ) , or instead are given in every frame , e.g . by a pre-trained detector ( MOT , MOTS , PoseTrack ) . 3 . Whether the target objects are represented by bounding-boxes ( SOT , MOT ) , pixel-wise masks ( VOS , MOTS ) or pose annotations ( PoseTrack ) . 4 . Whether the task is class-agnostic , i.e . the target objects can be of any class ( SOT , VOS ) ; or if instead they are from a predefined set of classes ( MOT , MOTS , PoseTrack ) . SOT VOS MO TPoseT MO TS InsDis SOT VOS MO TPoseT MO TS MoCo-v1 SOT VOS MO TPoseT MO TS PCL-v1 SOT VOS MO TPoseT MO TS PIRL SOT VOS MO TPoseT MO TS PCL-v2 SOT VOS MO TPoseT MO TS SimCLR-v1 SOT VOS MO TPoseT MO TS MoCo-v2 SOT VOS MO TPoseT MO TS SimCLR-v2 SOT VOS MO TPoseT MO TS SeLa-v2 SOT VOS MO TPoseT MO TS InfoMin SOT VOS MO TPoseT MO TS BYOL SOT VOS MO TPoseT MO TS DeepCluster-v2 SOT VOS MO TPoseT MO TS SwAV SOT VOS MO TPoseT MO TS BarlowTwins SOT VOS MO TPoseT MO TS DetCo SOT VOS MO TPoseT MO TS VFS Figure 1 : High-level overview of the performance of sixteen self-supervised learning models on five tracking tasks : SOT , VOS , MOT , PoseTracking and MOTS . A higher rank ( better performance ) corresponds to a vertex nearer to the outer circle . A larger area of the pentagon signifies better overall performance of its respective appearance model . Results of a vanilla ImageNet-supervised model are indicated with a gray dashed line as reference . Notice how the best model VFS [ 74 ] dominates on four out of the five tasks considered . Single object , user-specified Multiple objects , detector-specified B o x M as k Po se Task : Single Object Tracking ( SOT ) Datasets : OTB-100 , VOT-16/19 Task : Multi-Object Tracking ( MOT ) Datasets : MOT challenge , KITTI Task : Multi-Object Tracking & Seg . ( MOTS ) Datasets : MOTS , KITTI-MOTS Task : Pose Tracking ( PoseTrack ) Datasets : PoseTrack-2017/2018 Task : Video Object Segmentation ( VOS ) Datasets : DAVIS-2016/2017 Task : Pose Propagation Datasets : JHMDB Class-agnostic Class-agnostic Class-specific Class-specific Class-specific Class-specific Figure 2 : Existing tracking problems and their respective benchmarks differ from each other under several aspects : the assumption could be that there is a single or multiple objects to track ; targets can be specified by the user in the first frame only , or assumed to be given at every frame ( e.g . provided by a detector ) ; the classes of the targets can be known ( classspecific ) or unknown ( class-agnostic ) ; the representation of the targets can be bounding boxes , pixel-wise masks , or pose annotations . Appearance Model AssociationPropagation SOT VOS MOTS PoseTrack MOT L e v e l1 T ra in a b le L e v e l2 T ra in in g -f re e L e v e l3 T ra in in g -f re e Figure 3 : Overview of UniTrack . The framework can be divided in three levels . Level-1 : a trainable appearance model . Level-2 : the fundamental primitives of propagation and association . Level-3 : task-specific heads . Typically , in single-object tasks the target is specified by the user in the first frame , and it can be of any class . Instead , for multi-object tasks detections are generally considered as given for every frame , and the main challenge is to solve identity association for the several objects . Moreover , in multi-object tasks the set of classes to address is generally known ( e.g . pedestrians or cars ) . Figure 3 depicts a schematic overview of the proposed UniTrack framework , which can be understood as conceptually divided in three “ levels ” . The first level is represented by the appearance model , responsible for extracting high-resolution feature maps from the input frame ( Section 2.2 ) . The second level consists of the algorithmic primitives addressing propagation ( Section 2.3 ) and association ( Section 2.4 ) . Finally , the last level comprises multiple task-specific algorithms that make direct use of the primitives of the second level . In this work , we illustrate how UniTrack can be used to obtain competitive performance on all of the five tracking tasks of level-3 from Figure 3 . Moreover , new tracking tasks can be easily integrated . Importantly , note that the appearance model is the only component containing trainable parameters . The reason we opted for a shared and non task-specific representation is twofold . Firstly , the large amount of different setups motivated us to investigate whether having separately-trained models for each setup is necessary . Since training on specific datasets can bias the representation towards a limited set of visual concepts ( e.g . animals or vehicles ) and limit its applicability to “ open-world ” settings , we wanted to understand how far can a shared representation go . Second , we wanted to provide the community with multiple baselines that can be used to better assess newly proposed contributions , and that can be immediately used on new datasets and tasks without the need of retraining . 2.2 Base appearance model The base appearance model φ takes as input a 2D image I and outputs a feature map X = φ ( I ) ∈ RH×W×C . Since ideally an appearance model used for object propagation and association should be able to leverage fine-grained semantic correspondences between images , we choose a network with a small stride of r = 8 , so that its output in feature space can have a relatively large resolution . We refer to the vector ( along the channel dimension ) of a single point in the feature map as a point vector . We expect a point vector xi1 ∈ RC from the feature map X1 to have a high similarity with its “ true match ” point vector xî2 in X2 , while being far apart from all the other point vectors x j 2 in X2 ; i.e . we expect s ( xi1 , x î 2 ) > s ( x i 1 , x j 2 ) , ∀j 6= î , where s ( · , · ) represents a similarity function . In order to learn fine-grained correspondences , fully-supervised methods are only amenable for synthetic datasets ( e.g . Flying Chairs for optical flow [ 16 ] ) . With real-world data , it is intractable to label pixel-level correspondences and train models in a fully-supervised fashion . To overcome this obstacle , in this paper we adopt representations obtained with self-supervision . We experiment both with models trained with approaches that leverage pixel-wise pretext tasks [ 27 , 58 ] and , inspired by prior works that have pointed out how fine-grained correspondences emerge in middle-level features [ 39 , 74 ] , with models obtained from image-level tasks ( e.g . MoCo [ 24 ] , SimCLR [ 10 ] ) .

Summary: The paper investigates the possibility of using a single appearance model for all tracking-related tasks/subtasks (the paper proposes a precise classification of these tasks). The authors propose an elegant unified framework with 3 levels: basic appearance model (level 1), propagation and association blocks (level 2), and task-specific heads (level 3). The only level that needs a training phase is level 1, while the other levels are training-free. Another contribution is the introduction of the RSM metric, which is used to measure the similarity between features within the proposed association algorithm. Following a rich experimental section, the authors conclude by giving a negative answer to the question contained in the title of the paper itself.