Forward Propagation, Backward Regression, and Pose Associationfor Hand Tracking in the Wild
Mingzhen Huang1,2 , Supreeth Narasimhaswamy1 , Saif Vazir1,3 , Haibin Ling1 , Minh Hoai1,4
1
Stony Brook University, 2 University at Buffalo, 3 Tulip Interfaces, 4 VinAI Research(ii) Our proposed Handler(i) Current multi-object tracker

t-1tAbstract

0.3

Occluded
hands

0.8

0.9
0.2

We propose HandLer, a novel convolutional architecture that can jointly detect and track hands online in unconstrained videos. HandLer is based on Cascade-RCNNwith additional three novel stages. The first stage is Forward Propagation, where the features from frame t-1 arepropagated to frame t based on previously detected hands
and their estimated motion. The second stage is the Detection and Backward Regression, which uses outputs from theforward propagation to detect hands for frame t and theirrelative offset in frame t-1. The third stage uses an offthe-shelf human pose method to link any fragmented hand
tracklets. We train the forward propagation and backward
regression and detection stages end-to-end together with theother Cascade-RCNN components.
To train and evaluate HandLer, we also contribute
YouTube-Hand, the first challenging large-scale dataset of
unconstrained videos annotated with hand locations andtheir trajectories. Experiments on this dataset and otherbenchmarks show that HandLer outperforms the existing state-of-the-art tracking algorithms by a large margin.
Code and data are available at https://vision.cs.
stonybrook.edu/˜mingzhen/handler/.

1. Introduction
Hand tracking is an important problem in various application scenarios, from gesture and activity recognition
to contact tracing and skill evaluation. One approach for
tracking hands is to consider them as parts of a human bodyand then perform hand tracking based on the tracked humanpose. But pose detection and tracking can be unreliable byitself, especially for people that are partially occluded oroutside the field of view of the camera. Another approachfor hand tracking is to use off-the-shelf tracking methods.
Unfortunately, single-object trackers are not appropriate
for tracking multiple hands, while existing multiple-object
trackers do not work well for hands even though they haveshown impressive performance for tracking pedestrians andvehicles [2, 5, 19, 47, 49, 50, 62]. Hand tracking is difficult
because hands are not ordinary objects, given the extreme

t
0.2
0.8

0.2
0.3

0.1

0.8

0.2

Association

0.1
0.3

0.9

1.0

missed0.8

0.1

Proposals andobjectness scores

0.2

0.3
0.1

0.1

Dropped detection

0.6

Association
Pose

detected

0.9

0.8

Detected hands andconfidence scores

Tracking offsets

Flow maps

Figure 1. We develop an advanced detection and association algorithm for tracking multiple hands. Different with previous methods that estimate probability of a detected hand on the objectness
scores at current frame only, we estimate the probability basedon both the objectness scores at frames t and t-1, and associate
hands across frame via both pose and tracking offsets.

articulation of hands and the frequent interaction of handswith other objects. In a short period of a few frames, thesize, shape, location, and visibility of a hand can change
dramatically and frequently. Many existing multiple-object
trackers use the detection and assoication paradigm. However, hand detection would fail in the presence of motion
blur and occlusion, while hand linking across time is difficult as the size, location, pose, and appearance of a hand
can change drastically. Simultaneously, two different handinstances might look alike, so distinguishing them would
be difficult even for a sophisticated re-identification modulethat has been trained specifically for hands.
In this work, we develop a novel convolutional architecture that can detect and track hands in unconstrained videos.
We name the proposed architecture HandLer, which standsfor Hand Linker. HandLer takes as input two consecutive
video frames at times t-1 and t, and output the detected
hands in frame t as well as their corresponding locationsin frame t-1. The processing pipeline consists of threestages. The first stage is the Forward Propagation, which
propagates features from frame t-1 to frame t based on
the locations of previously detected hands and their estimated movements. The second stage is the Detection and2. Related Work

Figure 2. Representative image sequences from our dataset. The
size, shape, location, appearance, and visibility of a hand canchange drastically and frequently.

Backward Regression that uses outputs from the Forward
Propagation to obtain the hand locations for frame t as wellas their counterparts in frame t-1, and estimate their confidence conditioned on both the objectness scores at framest and t-1 as shown in Fig. 1. This allows us to link handdetections between two frames. Third, we establish correspondence between hand tracklets via pose association.
This is to leverage the fact that hands are undetachable partsof a human body, so we can use pose to recover prematurely
terminated hand tracklets.
Each stage of the proposed processing pipeline has its
benefits. The propagation step and conditional confidence
estimation step are useful for detecting blurry and occluded
hands. The detection step is necessary to account for new
hands in a video and avoid the potential drifting problem
common in methods solely based on propagation. The regression step brings detections at two different times into acommon reference frame for a more reliable linking. The
high-level pose association step avoids frequent ID switch
due to motion blur and occlusion.
We also introduce a new dataset called YouTube-Hand
for developing and evaluating hand tracking algorithms.
YouTube-Hand contains 240 video sequences from diverse
scene categories, including kitchens, mechanical workshops, and gyms. This dataset has 19,728 annotated frameswith 864 unique hand instances. To the best of our knowledge, this is the first large-scale hand tracking dataset containing videos for unconstrained environments with multiple annotated hand trajectories for each video. Fig. 2 showssome representative images from our dataset. We will release this dataset and code for research usage.

Many prior works only focused on hand detection in
static images [6, 8, 9, 13–15, 22, 26–28, 32, 37, 38, 40, 52,55], and works on hand tracking were developed for constrained settings such as laboratory environments and egocentric perspectives. Sridhar et al. [42] proposed a method
to track hands captured using a depth camera. Zhang et al.
[60] proposed a hand tracking solution that predicted a handskeleton of a human from a single RGB camera for AR/VR
applications. Wang and Popović [46] used a single camera to track a gloved hand with an imprinted pattern. Sharp
et al. [39] provided a hand tracking and pose estimation system based on a single depth camera. Mueller et al. [24] developed a 3D hand tracking approach for monocular RGB
videos using a kinematic 3D hand model. Sridhar et al. [43]proposed a method to track hands manipulating objects in
RGB-D videos. However, none of these methods was developed for videos in the wild; they required special markers, depth information, ego-centric perspectives, or scenes
with plain background.
Hands are objects, and we can consider Multiple-Object
Tracking (MOT) methods. A popular MOT approach istracking-by-detection, where an object detector first localizes objects, and then an association method constructs
trajectories. Depending on the association method, we
can categorize MOT methods as offline tracking or online tracking. Given a current frame t, offline methods [35, 56, 57] can use future frames and pose the association as a global optimization method. Meanwhile, most online methods [33, 54, 58, 63] are constrained to use frames
up to frame t only. A typical way to associate detections
over different frames is the Hungarian algorithm [25] withthe affinity costs defined based on the overlaping criterion.
Bewley et al. [4] proposed to predict bounding box movement with Kalman Filter and use Hungarian algorithm forlinking those boxes into tracks. However, this approach
does not work well for unconstrained videos since hands
often move fast, interact, and cross each other. Moreover,
the two-step approach of detecting hands first and then associating them can lead to suboptimal results since the two
steps are not jointly optimized end-to-end.
There are existing methods to alleviate the disadvantagesof the two-step tracking-by-detection paradigm. Bergmann
et al. [2] developed a framework that uses object locationsin the current frame to directly regress their corresponding
locations in the next frames. However, this method onlyuses current frame object locations as region proposals forthe next frame. This method does not work well for tracking
hands since hand locations change drastically over frames.
Zhou et al. [62] proposed a point-based framework for joint
detection and tracking, representing each object by a single point and tracking such points. This method outputs
an offset vector from the current object center to its centerin the previous frame for tracking. However, only using apoint representation does not work well for hands, which
are highly deformable.
There are methods that process multiple frames at thesame time. Feichtenhofer et al. [10] introduced correlation features that represented object co-occurrences acrosstime to generate two-frame tracklets. However, this method
does not work well when an object undergoes heavy occlusions, which is often the case for hands. Peng et al. [30]extended [10] by adding an appearance-based identity attention and proposed an online method to link two-frame tracklets. Wu et al. [51] proposed to generate a re-ID embeddingin each pixel and estimate objects movement offset fromthis embedding. This offset can be used to propagating feature and associating objects. However, those algorithms areappearance-based that does not work well for hands since
the appearance of a hand can change drastically over timeand different hand instances can have similar appearance.
Instead of using correlation features or appearance-based
approaches, our method directly estimates the relative offsets of hands in the previous frame given the hand locations in the current frame. As shown in our experiments,
this makes our hand tracking system more robust to occlusions or motion blur and reduces identity switches withother hand instances.

3. Proposed Method
In this section, we describe our novel method for online
tracking of multiple hands. We illustrate the proposed architecture in Fig. 3. Our method’s core is a convolutional
network that operates on a pair of two consecutive frames at
a time. At time t, the input to the network is a pair of video
frames at time t-1 and t, and the output of the network
are locations and confidence scores of the detected hands inframe t as well as their corresponding locations and confidence scores in frame t-1. We use the estimated locationsof hands in time t-1 to establish the association with theexisting hand tracks, assuming that we have tracked handsin the video until time t-1.
Specifically, given two frames It−1 and It at time t-1
and t, we use a backbone network to obtain their features
Xt−1 , Xt ∈ Rh×w×d . Here, h × w denotes the spatial
size and d denotes the number of channels. We also use
an off-the-shelf pose tracker [29] to obtain pose heatmaps
Pt−1 , Pt ∈ Rh×w×15 corresponding to 15 human joints.
Let Ht−1 ∈ Rh×w denote heatmap for hands that were detected in frame It−1 . We pass features Xt−1 and Xt , pose
heatmaps Pt−1 and Pt , and hand heatmap Ht−1 to the forward propagation stage.

3.1. Forward Propagation
Given features Xt−1 and Xt , pose heatmaps Pt−1 and
Pt , and hand heatmap Ht−1 , the forward propagation stage

estimates a flow map Ft ∈ Rh×w×2 and uses this flow map
to obtain temporally aggregated features Zt ∈ Rh×w×d .
Flow Estimation. To estimate the flow map F t , we propose to use the Flow Estimation Network [12]. The inputs
to this network are multi-scale features Xt−1 and Xt , andthe output is the 2-channeled flow map Ft ∈ Rh×w×2 denoting the motion between frames It−1 and It . The twochannels in Ft corresponds to flows in the horizontal andvertical directions.
We train this Flow Estimation Network end-to-end alongwith other components of the network as follows. Givena pair of hands that have the same ID in frames t-1 andt, we obtain two binary masks Mt−1 , Mt ∈ Rh×w corresponding to two frames. These masks are the ground-truth
binary segmentation maps for the hand in frames t-1 andt, respectively. We then use a bilinear warping function W
proposed by [65] to estimate a binary segmentation map forthe hand at time t: M0t = W(Mt−1 , Ft ). We then definea loss for hand motion as the MSE loss between estimated
M0t and the groundtruth Mt : Lhmo := MSE(M0t , Mt ).
Similarly, we also use the pose heatmap pair (Pt−1 , Pt )to define a loss for pose motion. We first obtain an estimatedpose at time t: P0t = W(Pt−1 , Ft ). We then define a loss
for poss motion as the MSE loss between estimated P0t andthe groundtruth Pt : Lpmo := MSE(P0t , Pt ).
Temporal Feature Aggregation. The output Ft from theFlow Estimation Network is used to aggregate features fromtime t-1 to features from time t. Specifically, we propagate features Xt−1 to features Xt to obtain Zt :Zt = [1+W(Ht−1 , Ft )] Xt +W(Ht−1 Xt−1 , Ft ) (1)In the above equation, is the Hadamard product, Ft isthe estimated flow map from frame t-1 to frame t, and W
is the bilinear warping function.

3.2. Hand detection and backward regression
The second important component of our architecture is
the hand detection and backward regression module. The
input to this module is the propagated feature map Zt alongwith estimated flow map F t . First, a CenterNet [61] will be
used to obtain a dense set of hand proposals at every pixel.
Second, for each proposal we compute: (1) the boundingbox of the hand at frame t, (2) the probability of this bounding box being a hand, (3) the relative offset bounding boxof this hand at frame t-1, and (4) the the confidence ofdetected box and offset box belong to same hand identity.
Tracking-based detection. We observe that for someblurry and occluded hands, our model would yield relative lower confidence scores even though those hands areclearly visible in previous frames. Detections with low confidence scores might be dropped, leading to false negatives.
To address this problem, we formulate the detection probability at frame t to be conditioned on both the objectness

t-1

t-1

Xt-1

Weight-shared DLA

Ht-1

Offsett

Backward
Regression

Zt

TemporalAggregation

F

ForwardPropagation

{ptO}
X

{pt}

t

Detection
Weight-shared DLA

Xt

Ht

Figure 3. Processing pipeline of HandLer. Given input video frames at time t-1 and time t, we first extract their DLA features Xt−1
and Xt . We estimate the flow map O from frame t-1 to frame t, and also obtain a heatmap Ht in frame t from CenterNet [61]. Alongwith heatmap Ht−1 , we aggregate feature as described in Eq. (1) to obtain a feature map Zt . We then extract RoI features from Zt anddetect hands in frame t and also estimate their corresponding offset and probability in the frame t-1 with the backward regression.

scores at frames t and t-1 (shown in Fig. 1). Consider aproposal Ckt at time t and position k and its corresponding detection Dk for the anchor box at location k, we use
P (Dk ) = P (Dk =hand) to denote the probability that Dk
is a hand, and P (Ckt ) = P (Ckt = object) to denote theobjectness probability for the proposal Ckt . The detection
likelihood is formulated as:
P (Dk ) = P (Dk |Ckt )P (Ckt )(2)= P (Dk |Ckt )N
X

P (Ckt |Cjt−1 )P (Cjt−1 ).

(3)jWe further assume that P (Ckt |Cjt−1 ) = 0 if there is no
motion from j to k, and P (Ckt |Cjt−1 ) = P (Ckt ) otherwise.
Thus the detection likelihood becomes:X
P (Dk ) = P (Dk |Ckt )P (Ckt )P (Cjt−1 ),
(4)
j∈Fkt
t
where Fk denotes the set of pixel locations that have mottion vectors pointing to k in the optical flow map F .

will use it to continue the track. Otherwise, we will either
initialize a new hand track for Bti if the detection score p ishigher than a threshold θnew , or discard this detection. Notethat θnew should be higher than θdet to avoid propagating
false positives. In our experiments, we set θdet = 0.6, andθnew = 0.9.
The Hungarian matching process is done as follows. The
inputs to this process are: (1) a set of detected hands, represented by the set of bounding boxes {Dt } in frame t, and(2) a set of active hand tracks, represented by a set of lastbounding boxes of the tracks {Tt−1 }. Note that the lastbounding box of Tt−1 might not be at frame t−1. Following
previous MOT methods, we only remove a hand track fromthe set of active hand tracks if this hand track is not matchedto any new detection for more than σ frames. Given thetwo set of bounding boxes {B O } and {Tt−1 }, we obtainjOi
an affinity matrix M as Mij = (α + pOt )IoU (Bt , Tt−1 ),and use the Hungarian algorithm to find the best set of oneto-one correspondences to maximize the total sum of theaffinity. In our experiments, we set α = 0.1 and σ = 50.

3.3. Hand-track continuation and initialization
We now describe how a newly detected hand is linked
with an existing hand track or used to initialize a new hand
track. Consider a particular detected hand D obtained byrunning the detection module with the input being the twoframes at t-1 and t. Frame t detection Dt is represented
by a quadruple: Dt = (Bt , pt , BtO , pOt ), where Bt is thehand location in frames t, BtO is its corresponding offset
location in frame t-1, pt is the corresponding detection
O
confidence and pO
t is the confidence of Bt and Bt belong
to same hand identity. Note that we only keep a detection
where pt is greater than a detection threshold θdet .
We then use the Hungarian algorithm [25] to match a
detection Dti and also other detections with the set of existing hand tracks. This is a joint optimization process,where the best set of one-to-one correspondences is determined. If Dti is matched with an existing hand track, we

3.4. Pose association
Since hands are undetachable body parts of a human,
we propose to use tracking result to guide our model for
hand motion estimation and tracking. Specifically, we consider the state-of-the-art open source pose tracking algorithm LightTrack [29] and observe that it has a lower recall
than our hand tracker, but most detected poses are generally
accurate. We therefore propose to use LightTrack [29] tohelp estimate motion flows (described in Sec. 3.1) and linka newly detected hand to an existing hand track.
Also, recall that a newly detected hand is represented bya quadruple D = (B, p, B O , pO ). In most cases, this detection will be used to continue a hand track as described inSec. 3.3. In some cases, we will discard D if p is low, andwe will create a new track if either pO is low or there is nomatching hand track for B O . But these actions can lead toa false negative or a false identity switch, so we propose toaddress these problems with pose tracking as follows. First,
given a set of detected hands and a set of wrist locationsof detected poses, we run the Hungarian algorithm to findthe optimal matching, where the matching cost for a handand a wrist is based on their distance. Second, we discard
detections that have low p values and no matching wrists.
Third, we use the procedure described in Sec. 3.3 to linksome detected hands with existing hand tracks. For a detected hand D that has not been linked to any hand track,
we will link it with a hand track T if: (1) D is linked withthe right/left wrist of a pose Pt in frame t; (2) T is linked
with the right/left wrist of a pose Pt−1 in frame t-1; andeither (3a) Pt and Pt−1 are linked via pose tracking, or (3b)the left/right wrist of Pt is linked with another hand D0 thatis linked with the hand track T 0 , which in turn is linked withthe left/right wrist of Pt−1 .

3.5. Loss function
To train this hand detection and regression module, we
optimize the combined loss function: L = Lhmo + Lpmo +O
LRP N + Lclass + Lreg + LOclass + Lreg . Here, LRP N is
the loss for the region proposal network, Lhmo and Lpmo
are flow map loss, and the other terms are for classificationof bounding box or offset regression.

Data source split
Train/test split
Total YouTube PoseTrack Train
Test#Videos
240
200
40
150
90
#Frames
232K
227K
5K 166K
65K
41K
19K 30K
30K#Anno. hands 60K#Trajectories
864
666
198
519
345

Table 1. Statistics of the proposed YouTube-Hand dataset.
Dataset

Scene/camera
Constraints

EgoHands [1]Google glasses
Handseg [20]Color gloves
NYUHands [45]Hands keypoints
ColorHandPose [64] 3D hands keypoints
HandNet [48]Fingertips
GANeratedHands [24] Synthetic
Oxford-Hand [22]Unconstrained
TV-Hand [26]Unconstrained
COCO-Hand [26]Unconstrained
Contact-Hand [27]Unconstrained
100DOH [38]Unconstrained
GTEA [16]Ego-centric
WorkingHands [40] Down-facing cam.
BSL [31]
TV show, segmentation
SynthHands [23]Ego-centric
ICP-PSO [34]Hand keypoints
EpicKitchen [7]Ego-centric, auto-label
VIVA [36]Vehicle-mounted
YouTube-Hand
UnconstrainedHas #Hand Maximum
Video Trajs. #trajs/video

X
X
X
X
X
X
X
X
X

0
0
0
0
0
0
0
0
0
0
0
0
0
2
1
6
1400
45
864

n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
2
1
1
2
4
15

Table 2. Comparing YouTube-Hand with other hand datasets.

4. YouTube-Hand Dataset
We aim to develop a tracker that can track hands in unconstrained scenes, which may contain many people interacting with each other and the other surrounding objects.
For training and evaluation, we needed a dataset of diverse
conditions, but such a dataset did not exist. We therefore
compiled a new dataset containing unconstrained videos
and annotated them with hand locations and trajectories.
Dataset source. We name our dataset YouTube-Hand because the majority of the videos (200 out of 240) were collected from YouTube. Specifically, we scraped 200 videos
from 10 scenarios, namely, casinos, concerts, cooking,dancing, driving, gyms, kids playing, mechanical workshops, sanitizing, and sports. To have a diverse dataset,
we collected different videos from different YouTube uploaders. We manually verified the collected videos to ensure that they were unconstrained and diverse in terms oflighting conditions, camera perspectives, skin tones, andages. We did not collect videos that have copyright marks.
Altogether, we downloaded 200 videos from YouTube,
with 20 videos for each scenario. Additionally, we selected 40 videos from the PoseTracks dataset and annotatedthem. The videos have spatial resolutions from 640×480 to
1920×1080 and frame rate from 24 to 30 fps.
Annotation. For each collected video, we extracted frames
using the original frame rate of the video and annotated ev-

Figure 4. Existing hand datasets are very different from ours. Thisshows some representative images from: VIVA [36] (top left),
EpicKitchen [7] (top right), BSL [31] (bottom left) and SynthHands [23] (bottom right).

ery fifteenth frame. We annotated only those hand instances
whose visible areas’ axis-parallel bounding box had morethan 100 pixels and whose trajectory appear for more than
50 frames. Our dataset was annotated by three annotatorsand subsequently verified by two people.
Train/test split. We split our data into disjoint training andtesting sets. The training set contains 150 videos, randomly
selected from the 200 YouTube videos. The remaining 90
videos are used for testing.
Statistics and comparison with other hand datasets. Table 1 shows the statistics of our dataset. Table 2 compares
our dataset with other existing hand datasets; most of themare for hand detection only, either having no video data or
hand trajectories. Some datasets contain hand trajectories,
but they only have videos for constrained camera settings,
such as ego-centric or in-vehicle mounted cameras. Fig. 4
shows some images from these datasets, which are much
more constrained than our dataset as shown in Fig. 2.

5. Experiments
In this section, we compare our method with various
generic object-tracking methods and hand tracking algorithms. We also perform ablation studies, report qualitative
results, and discuss failure cases.

5.1. Implementation details and evaluation metrics
Architecture Details. We implemented HandLer using Detectron2 [53]. Specifically, we built upon a Cascade-RCNNwith a DLA-34 [11] backbone with a Bi-directional Feature Pyramid Network (Bi-FPN) [44]. The network can betrained end-to-end and the inference speed is 5Hz.
Training Details. The core of HandLer is a network thattakes as input two frames and outputs the linked detections across these frames. The input to the network is notnecessarily a pair of consecutive frames at neither training nor testing time. To handle a wide range of video
frame rates and hand movements, including low frame rate
videos and fast moving hands, we actually sampled training video frames (t0 , t), with varying distance between t
and t0 . Specifically for each t, we used t0 = t − 15k, for1 ≤ k ≤ 5, because the training videos are annotated every
fifteenth frame.
We pre-trained HandLer using static images from the
TV-Hand [26] and COCO-Hand [26] datasets by using thesame static image as both frames t-1 and t. This was toutilize the larger datasets of annotated hands. Subsequently,
we fine-tuned the network on the proposed YouTube-Hand
dataset. For fine-tuning, we optimized the training loss for
12K iterations using SGD, with an initial learning rate of
0.0005 and a batch size of 48. We reduced the learning rateby a factor of 10 after 8K iterations.
Evaluation metrics. We used the standard multiple-objecttracking evaluation metrics [3, 18, 21]: the identification
F1 score (IDF1), the percentage of mostly tracked trajectories (MT), mostly lost trajectories (ML), false positives(FP), false negatives (FN), identity switches (IDs), multiple
object tracking precision (MOTP), multiple object tracking accuracy (MOTA) and higher order tracking accuracy(HOTA). Among these evaluation metrics, MOTA is considered the most important metric to quantify the overall
detection and tracking
performance. MOTA is defined as:
P
t +FPt +IDst )t (FNP, where FNt , FPt , IDst ,MOTA := 1 −t GTt
and GTt are the number of false negatives, false positives,

identity switches, and number of true hands, respectively,
for the frame t.
We found that none of the commonly used MOT metrics
measures the recovery ability; they do not quantify how wella tracker can right the wrong ID switch by reconnecting a
new hand tracklet with a prematurely terminated one. In
particular, while the ID switches (IDs) metric measures thefragmentation of a ground truth trajectory; it would apply
the same penalty to any identity switch, no matter whether
the tracker switches to a new-and-wrong ID or an old-butcorrect ID. For example, the sequences of trajectory IDs(a → b → c) and (a → b → a) would have the same performance in current metrics, but the later is more desirable.
Thus, we introduce a new metric called Longest-TrackletRatio (LTR). For a particular ground truth trajectory that ismatched to multiple predicted tracklets with different IDs,
LTR is defined as the ratio between the length of the longest
predicted tracklet and the length of the entire trajectory. We
will use the average LTR on all trajectories of a test set as
the new performance metric.

5.2. Main Results
Table 3 compares the performance of our hand trackers
with other state-of-the-art MOT tracking methods. TraDes,
CenterTrack and FairMOT were end-to-end trainable
MOT methods, which were trained to detect and track hands
jointly, but they performed relatively poor on hands, perhaps because they were geared towards less deformable andarticulated classes such as pedestrians and vehicles.
We also implemented several tracking-by-detection
methods, where the detection results were provided byHandCNN [26], which is the state-of-the-art hand detection
method. LightTrack used pose tracklets to linking hands.
We first used LightTrack to detect and track human body
joints then associated HandCNN detected hand to a person
based on the distances between the predicted wrist keypoint
and the center of the detected hand bounding box. CenterTrack* was a method where the detection component ofCenterTrack was replaced by HandCNN. MPNTrack was
an offline tracking method, in which a Message Passing
Network (MPN) was used for HandCNN detection association. For all methods, we first pre-trained using static images from TV-Hand and COCO-Hand datasets to improve
the hand detection performance and then fine-tuned themusing the training set of YouTube-Hand.
Based on the those metrics, HandLer outperforms theothers by a wide margin. Fig. 5 shows some representative
results and failure cases by HandLer.

5.3. Ablation Studies
We now present our experiments to study the effectiveness of different components of the proposed architecture.
Effectiveness of HandLer. To study the importance of theMethods
LightTrack [29] (Pose)
FairMot [59]MPNTrack [5] (Offline)
CenterTrack[62]CenterTrack*[62]SORT [4]TraDeS [51]HandLer (proposed)

IDF1↑

MT↑

ML↓

FP↓

FN↓

IDs↓

MOTP↑

MOTA↑

LTR↑

HOTA↑

53.4
41.4
49.0
37.2
57.8
48.3
53.6
70.9

101
96
156
113
137
101
168
218

70
57
66
62
43
72
43
23

6240
2065
5918
2279
3208
2295
3271
2412

12816
12753
11263
12379
10317
12960
9102
5986

1955
3448
1039
3362
1647
1475
1982
712

74.5
76.8
77.0
76.5
79.0
76.7
76.4
79.9

30.8
39.9
40.0
40.7
50.0
44.9
52.7
70.0

48.4
31.3
44.3
27.3
37.5
47.6
44.4
64.3

48.5
39.0
40.7
39.0
49.1
46.1
46.4
59.4

Table 3. Hand tracking performance on the test set of YouTube-Hand. In terms of MOTA, the most indicative MOT metric, HandLer
outperforms other methods by a large margin. In each column, the best result is highlighted in bold, and the second best result is underlined.

out the pose. As can be seen, all those three components areimportant component of HandLer.
FP↓ FN↓ IDs↓ MOTA↑ LTR↑(a) Tracking results by HandLer. This visualizes hand tracking results across twoframes. Hands that belong to the same trajectory are visualized with the same color.

HandLer
2412 5986 712
HandLer w/o forward
3107 6432 761
HandLer w/o backward
2838 6195 1488
HandLer-NP
2875 6169 1256
HandLer-NP w/o forward 3076 6821 1203
HandLer-NP w/o backward 2301 6965 1536

70.0
66.1
65.4
66.1
63.4
64.4

64.3
62.1
58.4
59.0
56.4
52.2

Table 4. Effectiveness of each components of HandLer.

(b) Hand detection and backward regression results. The left and right images
correspond to frames t-1 and t, respectively. The detected hand in frame t and its
corresponding location obtained using backward regression in frame t-1 are shownin magenta color. The detected hands in frame t-1 are visualized in blue and green.

(c) Comparing HandCNN and HandLer. HandCNN fails to detect blurry andoccluded hands. Benefit from our temporal feature aggregation and tracking-based
detection, HandLer can detect those hands.

Robustness to low frame rates. We studied how thetracking performance changed as the frame rate of a video
dropped. For this purpose, we ran HandLer on every Kth frame for various values of K. Specifically, we used
K = 1, 3, 5, 15, which corresponded to 30, 10, 6, and 2
frames per seconds (fps). The results are shown in Table 5.
As can be seen, the MOTA of HandLer did not decreasemuch when the fps was reduced from 30 to 6. Compared
to SORT [4] (with HandLer detection), another tracking
method described in Sec. 5.2, this level of MOTA reduction was relatively small. This demonstrates the robustnessof our linking algorithm across different time gaps.
Tracking

SORT

HandLer

StrideFP↓ FN↓ IDs↓ MOTA↑ FP↓ FN↓ IDs↓ MOTA ↑K = 1 2446 36297 1902 64.9 2412 5986 712
70.0
K = 3 1177 2977 2903 59.2 1099 3525 1284
65.8
K=5915 2297 3301 55.6 906 2861 1468
64.3
K = 15 664 1636 3569 51.2 651 2077 1759
62.7(d) Failure cases from HandLer. The left image shows a case where a hand is notdetected due to heavy occlusions, and the second images shows a case where otherskin areas are mistaken for hands.

Figure 5. Qualitative results on YouTube-Hand dataset.

proposed forward propagation for hand tracking, we trained
a model where there was no forward propagation. Similarly,
we trained a model where there was no backward regression
to frame t-1. In this case, we linked hand detections using
the the Hungarian algorithm with the hand bounding boxes
in frame t. Finally, we trained and tested the model withoutpose. The results are shown in Table 4. We use HandLer torefer to our full model, and HandLer-NP is HandLer with-

Table 5. Performance of tracking algorithms as the frame rateof videos decreases. K is the stride of the tracking algorithm.

5.4. Hand detection
HandLer can also be used for hand detection, as long as
the input is a video. To study the effectiveness HandLer
for detecting hands especially blurry and occluded hands,
we sample a subset of YouTube-Hand that only contains
blurry and occluded hands to test the effectiveness of HandLer for detecting such hands. Here we use the hand keypoints estimation method proposed in [41] to detect hand
keypoints within every ground truth hand box. We claim

that the hand is blurry or occludes if [41] cannot detect
all hand keypoints. Along with YouTube-Hand and VIVA
dataset, we evaluated the performance of various hand detection methods on those three datasets using the VOC average precision metric. Since hand keypoints estimation [17?
] cannot detect hand well for in-the-wild videos, we compared with HandCNN [26], the state-of-the-art hand detection method and summarize the results of these experiments
in Table 6. Moreover, using HandLer as a detector (without linker) would also boost the tracking performance ofother tracking methods, as can be seen in Table 7 for theYouTube-Hand and VIVA datasets. Note that here we only
report the performances of the methods that support tracking with external detections.
DatasetMethodYouTube-Hand Blur&Occ Split VIVA [36]
HandCNN [26]72.4
62.8(13.1% ↓)89.2
HandLer
84.1
76.7(8.8% ↓)95.3

Table 6. Hand detection performance. The colored number isthe percentage of performance dropped on blurry and occluded
hand split comparing on the full set of YouTube-Hand dataset.
Comparing with HandCNN, which runs with around 2fps, ourmethod achieves both efficiency and effectiveness.
IDF1↑

IDs↓

MOTA↑

LTR↑

SORT[4]60.6(+12.3) 1902(+427) 64.9(+20.0) 53.6 (+15.1)
MPNTrack[5] 61.1(+12.1) 1288(+249) 65.2(+25.2) 57.3(+13.0)
CenterTrack[62] 61.3(+24.1) 2167(-1195) 62.7(+22.0) 51.1(+23.8)
LightTrack[29] 71.0(+17.6) 1635(-320) 61.7(+30.9) 65.7(+17.3)

Table 7. Using HandLer as a detector with other MOT methods on YouTube-Hand dataset. The colored number indicts
performance improvement or descent comparing with Table 3.

5.5. Other datasets & tasks
We also evaluate the tracking and detection performanceof HandLer on other datasets: VIVA, BSL. Note that all
methods below use HandLer detection and associating detected hands with their own linker.
The VIVA dataset [36] contains frames sampled from 20
videos captured by ego-centric cameras. It was collected todevelop an algorithm to detect the hands of a driver and apassenger. We used 11 videos for training and the remaining
9 for evaluation. The results are shown in Table 8.
IDF1↑ FP↓ FN↓ IDs↓ MOTA↑
CenterTrack[62]
SORT[4]
MPNTrack[5]HandLer

45.6
44.1
46.2
62.0

341 1287
517 884793 545272 367

79
93
46
58

68.7
72.6
74.7
87.2

Table 8. Comparing different methods on VIVA dataset

The British Sign Language (BSL) dataset [31] contains

6000 frames from BBC TV shows, 296 of them have beenannotated with hand segmentation. All methods reported
in Table 9 were trained on YouTube-Hand training set andthen tested on BSL dataset.

CenterTrack[62]MPNTrack[5]
SORT[4]
HandLer

IDF1↑

FP↓

FN↓

IDs↓

MOTA↑

22.2
11.6
13.6
20.5

65
144
92
60

128
76
82
89

177
64
71
39

45.9
58.8
63.2
72.5

Table 9. Tracking performance on the BSL dataset.

Pose tracking. Since hands are attached to the wrists,
one might wonder if we can track the human pose and thewrists instead. We hypothesize that pose tracking is a difficult problem by itself, its performance is not better than
hand tracking performance. To validate this hypothesis,
we perform experiment on the PoseTrack Split of YoutubeHands. Pose tracking tracks the wrist points, but comparing
point tracking results with bounding box tracking results
is not trivial because MOTA computation are done differently. For a fair comparison, we consider two transformation: (1) Box2Point: represent a bounding box by its center; (2) Point2Box: match a wrist point to a detected hand
by HandLer as explained in Sec. 5.2. Table 10 compares theperformance of HandLer and LightTrack after making thesetransformations.

LightTrack [29]HandLer

Box2Point
60.7
69.6

Point2Box
49.2
61.2

Table 10. Comparing with pose tracking algorithm (LightTrack) on the PoseTrack split. The evaluation metric is MOTA.
Pose tracking is difficult problem by itself, and it does not perform
as well as HandLer.

6. Conclusion and Potential Negative Impacts
We introduced HandLer, a novel convolutional architecture to detect and track hands in unconstrained videos. We
also collected and annotated a large-scale challenging hand
tracking dataset, called YouTube-Hand. This dataset contains videos of hands in unconstrained environments, and it
can be used to develop and evaluate hand tracking systems.
Hand tracking is important in various application scenarios, but there is potential for abuse of the technology toinvade privacy. We will release our implementation for research purposes, but the deployment of this technology needappropriate controls to limit harmful or malicious uses.
Acknowledgements. This work started when the first author was
at Stony Brook University. The work was later supported partially
by DARPA PTG HR00112220001 award. The content of the information does not necessarily reflect the position or the policy of
the Government, and no official endorsement should be inferred.

References[1] Sven Bambach, Stefan Lee, David J. Crandall, and Chen Yu.
Lending a hand: Detecting hands and recognizing activities
in complex egocentric interactions. In Proceedings of theInternational Conference on Computer Vision, 2015. 5[2] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.
Tracking without bells and whistles. In Proceedings of theInternational Conference on Computer Vision, 2019. 1, 2
[3] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics.
EURASIP Journal on Image and Video Processing, 2008:1–
10, 2008. 6[4] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, andBen Upcroft. Simple online and realtime tracking. In Proceedings of the IEEE International Conference on Image
Processing. IEEE, 2016. 2, 7, 8
[5] Guillem Brasó and Laura Leal-Taixé. Learning a neuralsolver for multiple object tracking. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, 2020. 1, 7, 8
[6] Patrick Buehler, Mark Everingham, Daniel P Huttenlocher,and Andrew Zisserman. Long term arm and hand tracking
for continuous sign language tv broadcasts. In Proceedingsof the British Machine Vision Conference, 2008. 2[7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella,
Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide
Moltisanti, Jonathan Munro, Toby Perrett, Will Price, andMichael Wray.
Rescaling egocentric vision.
CoRR,
abs/2006.13256, 2020. 5[8] Xiaoming Deng, Yinda Zhang, Shuo Yang, Ping Tan, Liang
Chang, Ye Yuan, and Hongan Wang. Joint hand detection
and rotation estimation using cnn. IEEE Transactions on
Image Processing, 2018. 2[9] Eng-Jon Ong and R. Bowden. A boosted classifier tree forhand shape detection. In Proceedings of the International
Conference on Automatic Face and Gesture Recognition,
2004. 2[10] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
Detect to track and track to detect. 2017. 3[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016. 6[12] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2017. 3[13] Leonid Karlinsky, Michael Dinerstein, Daniel Harari, andShimon Ullman. The chains model for detecting parts by
their context. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2010. 2[14] Mathias Kolsch and Matthew Turk. Robust hand detection.
In Proceedings of the International Conference on Automatic
Face and Gesture Recognition, 2004.
[15] Pavan. M. Kumar, Andrew Zisserman, and Philip. H. S. Torr.
Efficient discriminative learning of parts-based models. In

Proceedings of the International Conference on Computer
Vision, 2009. 2[16] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2015. 5[17] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, ChuoLing Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv
preprint arXiv:1906.08172, 2019. 8[18] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, PhilipTorr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.
HOTA: A higher order metric for evaluating multi-object
tracking. International Journal of Computer Vision, 129(2):548–578, 2021. 6[19] Wenhan Luo, Junliang Xing, A. Milan, X. Zhang, Wei Liu,
Xiaowei Zhao, and T. Kim. Multiple object tracking: A literature review. arXiv: Computer Vision and Pattern Recognition, 2014. 1[20] Sri Raghu Malireddi, Franziska Mueller, Markus Oberweger, Abhishake Kumar Bojja, Vincent Lepetit, Christian Theobalt, and Andrea Tagliasacchi. Handseg: A
dataset for hand segmentation from depth images. ArXiv,
abs/1711.05944, 2017. 5[21] Anton Milan, Laura Leal-Taixe, Ian Reid, Stefan Roth, andKonrad Schindler. Mot16: A benchmark for multi-object
tracking. 2016. 6[22] Arpit Mittal, Andrew Zisserman, and P. H. S. Torr. Hand
detection using multiple proposals. In Proceedings of the
British Machine Vision Conference, 2011. 2, 5[23] Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt.
Real-time hand tracking under occlusion from an egocentricrgb-d sensor. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017. 5[24] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, andChristian Theobalt. Ganerated hands for real-time 3d hand
tracking from monocular rgb. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2018. 2, 5[25] J. Munkres. Algorithms for the assignment and transportation problems. Journal of the Society of Industrial and Applied Mathematics, 5(1):32–38, March 1957. 2, 4[26] Supreeth Narasimhaswamy, Zhengwei Wei, Yang Wang,
Justin Zhang, and Minh Hoai. Contextual attention for handdetection in the wild. In Proceedings of the International
Conference on Computer Vision, 2019. 2, 5, 6, 8
[27] Supreeth Narasimhaswamy, Trung Nguyen, and Minh Hoai.
Detecting hands and recognizing physical contact in thewild. In Advances in Neural Information Processing Systems, 2020. 5[28] Supreeth Narasimhaswamy, Thanh Nguyen, Mingzhen
Huang, and Minh Hoai. Whose hands are these? hand detection and hand-body association in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2022. 2[29] Guanghan Ning and Heng Huang. Lighttrack: A genericframework for online top-down human pose tracking. In Proceedings of CVPR Workshop on Towards Human-Centric Image/Video Synthesis and the 4th Look Into Person Challenge,
2020. 3, 4, 7, 8[30] Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu,
Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue
Huang, and Yanwei Fu. Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object
detection and tracking. In Proceedings of the European Conference on Computer Vision, 2020. 3[31] Tomas Pfister, James Charles, Mark Everingham, and Andrew Zisserman. Automatic and efficient long term arm andhand tracking for continuous sign language tv broadcasts. In
Proceedings of the British Machine Vision Conference, 2012.
5, 8[32] Pramod Kumar Pisharady, Prahlad Vadakkepat, and Ai Poh
Loh. Attention based detection and recognition of hand postures against complex backgrounds. International Journal ofComputer Vision, 2013. 2[33] Horst Possegger, Thomas Mauthner, Peter M. Roth, andHorst Bischof. Occlusion geodesics for online multi-object
tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014. 2[34] Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian
Sun. Realtime and robust hand tracking from depth. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2014. 5[35] Z. Qin and C. R. Shelton. Improving multi-target tracking
via social grouping. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2012. 2[36] Akshay Rangesh, Eshed Ohn-Bar, Mohan M Trivedi, et al.
Driver hand localization and grasp analysis: A vision-based
real-time approach. In International Conference on Intelligent Transportation Systems, 2016. 5, 8[37] Kankana Roy, Aparna Mohanty, and Rajiv Ranjan Sahay.
Deep learning based hand detection in cluttered environment
using skin segmentation. In International Conference on
Computer Vision Workshops, 2017. 2[38] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey.
Understanding human hands in contact at internet scale. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2020. 2, 5[39] Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, Daniel Freedman, Eyal
Krupka, Andrew Fitzgibbon, Shahram Izadi, and Pushmeet
Kohli. Accurate, robust, and flexible real-time hand tracking. In Proceedings of ACM Conference on Human Factors
in Computing Systems, 2015. 2[40] Roy Shilkrot, Supreeth Narasimhaswamy, Saif Vazir, andMinh Hoai. WorkingHands: A hand-tool assembly dataset
for image segmentation and activity mining. In Proceedingsof the British Machine Vision Conference, 2019. 2, 5[41] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser
Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2017. 7, 8[42] Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, andChristian Theobalt. Fast and robust hand tracking using
detection-guided optimization. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
June 2015. 2[43] Srinath Sridhar, Franziska Mueller, Michael Zollhoefer, Dan
Casas, Antti Oulasvirta, and Christian Theobalt. Real-time
joint tracking of a hand manipulating an object from rgb-d
input. In Proceedings of European Conference on Computer
Vision, 2016. 2[44] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model
scaling for convolutional neural networks. In Proceedings ofthe International Conference on Machine Learning, 2019. 6[45] Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken
Perlin. Real-time continuous pose recovery of human hands
using convolutional networks. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 2014. 5[46] Robert Y. Wang and Jovan Popović. Real-time hand-tracking
with a color glove. ACM Transactions on Graphics, 28(3),2009. 2[47] Zhongdao Wang, Liang Zheng, Yixuan Liu, and Shengjin
Wang. Towards real-time multi-object tracking. arXiv
preprint arXiv:1909.12605, 2019. 1[48] Aaron Wetzler, Ron Slossberg, and Ron Kimmel. Rule ofthumb: Deep derotation for improved fingertip detection. In
Proceedings of the British Machine Vision Conference, 2015.
5[49] Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re-identification. In Proceedings of the IEEE
Workshop on Applications of Computer Vision, 2018. 1[50] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple
online and realtime tracking with a deep association metric.
In Proceedings of the IEEE International Conference on Image Processing, 2017. 1[51] Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming
Yang, and Junsong Yuan. Track to detect and segment: An
online multi-object tracker. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
3, 7[52] Ying Wu, Qiong Liu, and Thomas S. Huang. An adaptive
self-organizing color segmentation algorithm with application to robust real-time human hand localization. In Proceedings of the Asian Conference on Computer Vision, 2000.
2[53] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen
Lo, and Ross Girshick. Detectron2. https://github.
com/facebookresearch/detectron2, 2019. 6[54] Y. Xiang, A. Alahi, and S. Savarese. Learning to track: Online multi-object tracking by decision making. In Proceedings of the International Conference on Computer Vision,2015. 2[55] Xiaojin Zhu, Jie Yang, and A. Waibel. Segmenting handsof arbitrary color. In Proceedings of the International Conference on Automatic Face and Gesture Recognition, 2000.
2[56] B. Yang, C. Huang, and R. Nevatia. Learning affinities anddependencies for multi-target tracking using a crf model. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2011. 2

[57] Yichen Wei, Jian Sun, Xiaoou Tang, and Heung-Yeung
Shum. Interactive offline tracking for color objects. In Proceedings of the International Conference on Computer Vision, 2007. 2[58] Ju Hong Yoon, Chang-Ryeol Lee, Ming-Hsuan Yang, andKuk-Jin Yoon. Online multi-object tracking via structural
constraint event aggregation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,2016. 2[59] Yifu Zhan, Chunyu Wang, Xinggang Wang, Wenjun Zeng,
and Wenyu Liu. A simple baseline for multi-object tracking.
arXiv preprint arXiv:2004.01888, 2020. 7[60] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei
Tkachenka, George Sung, Chuo-Ling Chang, and Matthias
Grundmann. Mediapipe hands: On-device real-time hand
tracking. arXiv preprint arXiv:2006.10214, 2020. 2[61] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850, 2019. 3,
4[62] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.
Tracking objects as points.
arXiv preprint
arXiv:2004.01177, 2020. 1, 2, 7, 8[63] Ji Zhu, Hua Yang, Nian Liu, Minyoung Kim, Wenjun Zhang,and Ming-Hsuan Yang. Online multi-object tracking withdual matching attention networks. In Proceedings of the European Conference on Computer Vision, 2018. 2[64] Xiaojin Zhu, Jie Yang, and Alex Waibel. Segmenting handsof arbitrary color. In Proceedings of the International Conference on Automatic Face and Gesture Recognition, 2000.
5[65] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen
Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2017. 3