No Identity, no problem: Motion through detection for people tracking
Authors: Martin Engilberge
Friedrich Wilke Grosche
Pascal Fua
Venue: Accepted by TMLR ()

Abstract: Tracking-by-detection has become the de facto standard approach to people tracking. To increase robustness, some approaches incorporate re-identification using appearance models and regressing motion offset, which requires costly identity annotations. In this paper, we propose exploiting motion clues while providing supervision only for the detections, which is much easier to do. 

Our algorithm predicts detection heatmaps at two different times, along with a 2D motion estimate between the two images. It then warps one heatmap using the motion estimate and enforces consistency with the other one. This provides the required supervisory signal on the motion without the need for any motion annotations. In this manner, we couple the information obtained from different images during training and increase accuracy, especially in crowded scenes and when using low frame-rate sequences. 

We show that our approach delivers state-of-the-art results for single- and multi-view multi-target tracking on the MOT17 and WILDTRACK datasets.

Published in Transactions on Machine Learning Research (11/2024)
No Identity, no problem: Motion through detection for peo- ple tracking
Martin Engilberge Computer Vision Laboratory, EPFL
martin.engileergeeeepf....
F. Wilke Grosche Computer Vision Laboratory, EPFL
Pascal Fua
puscal @epft.ch
Computer Vision Laboratory, EPFL
Reviewed on OpenReview: https: // opw.:eveee/.........mmmm net/ forum? id= ogEM2H9ΙCK
Abstract
Beacking-by-detecte has become the de facto standard approach people tracking. To increase robustness, some approaches incorporate re-identification using appearance models and regressing motion offsse, which requires costly identity annotations. In this paper, proposc exploiting motion clues while providing supervision only for the detections, which is much easier to do.
Our algorithm predicts detection heatmaps at two different times, along with 2D motion estimate between the two images. It then warps one heatmap using the motion estimate and enforces consistency with the other onc. This provides the required supervisory signal on the motion without the need for any motion annotations. In this manner, we couple the information obtained from different images during training and increase accuracy, especially in crowded scenes and when using low frame-rate sequences.
We show that our approach delivers statro-f--hheaat results for single- and multi-view multi- target tracking on the MOT17 and WILDTRACK datasets.
1 Introduction
Multi-target. tracking has been extensively studied (Ciaparrone al.] 22000] Yilmaz et al. 2006: Smeulders lt al.) 2014), and research has now pivoted towards challenges such as handling crowded scenes, low frame rate sequences, and fast-moving objects.
Most current approaches rely on the tracking-by-detection paradigm (Andriluka et al. 2008 which requires strong object detectors. Linking the detections into trajectories is entrusted to algorithms that do not take the images into consideration anymore, such as Pirsiavash et al. (2011); Berclaz et al. (2011): Wang et al. (2019). Unfortunately, performance suffers crowded scenes, especially when the video frame rafe is low [Bergmann al. 2019). Fig. iillstratte this. Motion prediction can be used alleviate this problem. For example, the popular baacking-by-regressin approach of Zhou al. (020) Xu et al. (2020) does this by predicting people's motion 2D motion offset.
However, motion supervision is often derived from identity annotations, which can be costly. In this paper, we seek to exploit the motion information withou having lo provide identity or motion annolations, relying solely on detection annotations that are much easier to obtain. To this end, we introduce a differentiable approach to constructing future detection heatmap by displacing a current heatmap using a 2D offset map that represents motion. Given a pair of images acquired times and t+ 1, we predict detection heatmaps at both times, along with an offset map. We the offset map warp the heatmap estimated at time to

---

Published in Transactions on Machine Learning Research (11/2024)
200
NOTA IDF1
Delectión
Reconstructed
Differentiable Reconstruction
Nobicc value %
Motion offset
Time t+1
Time
0.075 0.3 0.5
25
FPS
Figure Predicting human motion. Left: We use muSSP Wang et (009 link detections different frame rates. We plot the MOTA and IDFI metrics as function of the frame rate. Below 3FPS, the degradation becomes severe. Right: Our model estimates detection healmap at time & and predicts the motion of objects between and + 1. The offsets are used to warp the heatmap into prediction time ++1 1 and we enforce consistency between that prediction and the one estimated by the network using the image acquired at time + 1.
enforce consistency between the result and the one estimated at time t + 1. In this manner, we couple the information obtained from different images, which increases robustness.
The proposed motion estimation method be used in conjunction with wide variety of detectors and trackers. The supervision for detection can originate from ground-truth detections when trained jointly with detector, or from the output of pre-trained detector, if trained separately. Crucially, in either case, motion offsets are supervised without any additional annotations.
We use the challenging single-view MOT17 dataset Milln et al. 2016 and multi-view WILDTRACK dataset (Chavdarova et al.] 2018) to demonstrate our model's ability to predict accurate human motion without requiring any motion annotations. On MOT17 we outperform the state-of-the-art tracking method Bytetrack (Zhang al., 2022) by significant margin in low framo-rate scenario. On WILDTRACK wo
surpass recent. multi-view detection and tracking lechniques (Engilberge el. al. 2023 Teepe et al. 2024 Cheng et al. 2023). Code can be found at https://github: cwm/ www.la-pelllooolt..eoeee
2 Related Work
Tracking is core task of computer vision and has applications wide range of domains such as surveil- lance, autonomous driving, the medical field the autonomous sales space. Despite having been widely studied (Ciaparrone et al. 2020 Yilmaz et al. 2006 Smeulders et al. 2014), multi-object tracking be challenging. In particular, self-occlusions are frequent in crowded environments and can result in missed detections and identity switches. Furthermore, due to hardware constraints, some applications can only operate at low frame rates, making association across frames difficult. In this section, we briefly review existing approaches and discuss multi-view tracking, which we use validate our approach.
still
2.1
Multiple Object Tracking (MOT).
Most current MOT approaches operate on videos and follow the Baacking-by-deteedee
Andriluka et al.
2008
paradigm. This requires strong object detoctors, such as Faster R-CNN (Ren al. 2015 Girshick
2015)
DPN (Felzenszwalb et al. 2010) or more recently anchor free models such CenterNet (Zhou al.
2019
2

---

Published in Transactions on Machine Learning Research (11/2024)
Given the detections in all frames of video sequence, they must be grouped into trajectories that span the whole video sequence. Many approaches to doing this have been proposed. They can either be online [Breitenstein et al. 2009 Bewley et al.) 2016; Wojke et al.) 2017), where association is done frame-by- frame. or offline by generating tracks given detections throughout the entire video sequence (Berclaz et al., 2006 2011 Wang et al. 2019). Tt. has been shown (Bergmann et al., 2019) that simple association mot combined with strong detectors are able to handle most tracking scenarios and are on par with more complex methods. Similarly, Zhou et al. (2020) propose to use greedy association mechanism combined with an anchor free detector regressing 2D motion offsets. In Voigtlaender et al. (2019), multiple object tracking and segmentation are combined by propagating segmentation masks using optical flow (Dosovitskiy et al. 2015:
Teed & Deng 2020).
Many of these association algorithms rely on a graph formulation. The graph nodes correspond either to all the spatial locations where an object can be present (Fleuret et al.
2008 Berclaz et al. 2011 BenShitrit
ct al.) 2014 or only to detections (Jiang et al., 2007 Tang et al. 2015 Shu ct al. 2012 Benfold Reid 2011
Cheng et al. 2023). The Successive Shortest. Paths (SSP) (Pirsiavash et al.) 2011) good representativ
of this class of techniques. It links detections using sequential dynamic programming. It was later extended
with bounded memory and computation, which enables tracking in even longer sequences (Lenz et al. 2015).
It was further optimized by exploiting the structures of the graphs formulated in multiple objects tracking problems (Wang et al. 2019h. To deal with heavy occlusions, appearance models can be integrated into it.
To make further use of appearance, one can learn a deep association metric on a large-scale person re- identification dataset which enables reliable people tracking in long video sequences (Wojke et al.) 2017). A discriminative appearance feature can also be learned using hard identity mining triplet loss (Ristani & Tomasi 2018), but it requires expensive identity annotations. Motion estimation has also been used in conjunction with deep learning (Liu et al.) 2020; 2022 Engilberge et al.] 2023 when modeling people densities and their evolution over time, but could only model limited motion ranges.
2.2 Multi-View Tracking.
Using multiple viewpoints is an effective way to increase robustness in crowded scenes featuring heavy occlu- sions. Some methods first extract monocular detections before projecting and matching them into a common frame
2016 Flcuret et al. 2008 while others combine view aggregation and prediction (Baqué 2017). Many recent ones (Hou et al., 2020 Song el al.) 2021 Hou & Zheng, 2021 Engilberge et al. Teepe et al. 2024) perform view alignment directly in the feature space. They assume the ground to be flat and project features onto the common ground plane. There, they are spatially aligned and be easily aggregated to produce detection directly on the ground plane. The approach of (Engilberge et al., 2023) leverages properties of the ground plane by predicting weakly supervised human structure motion for tracking. We also leverage the benefit provided by such ground plane representations, but depart from previous work by prodicting motion as an unconstrained 2D offsct, allowing us to model and predict motion of arbitrary length.
(Xu ct al.
et al.
2023
3 Approach
Most recent single- and multi-view tracking approaches rely on simple association methods to link the output of strong single-frame object detectors. Typical constraints imposed on the linking process are limits on displacement length across frames (Engilberge et al. 2023 Bergmann et al., 2019), which are weak when the frame rate is high and inapplicable when it is low. To impose motion constraints while handling low frame rates and the potentially large tame-to-framee displacements they entail, want to avoid such strict limits.
To this end, we introduce the approach depieted by Fig. 2 For all sets of images taken at consecutive time steps and t + 1, we predict probabilities of detections at both time steps and a 2D displacement map from time to t ++1, 1. During training, we provide supervision for the detections and enforce consistency between the detections and displacements, which provides an additional supervisory signal without any additional annotations and increases performance, as will be shown Section 4.2
3

---

Published in Transactions on Machine Learning Research (11/2024)
Detection x
x
Reconstruction Xt+1
X
Motion Offset gu++1
Detection
Distance weight W
Reconstruction. X++1
Figure 2: Details of the proposed differentiable reconstruction from motion Given detection map at time t and an offset map capturing motion of objects between time and time + we reconstruct the detection map at time 1. Fach reconstructed pixel a weighted sum of the detection of the previous time step, the weights are derived from the distance between the reconstructed location and the expected position of the previous locations after being moved by the offset. The example above illustrates the reconstruction for three locations, for the bottom one it is mainly unaffected by the offset and the distance weight is therefore disc decreasing as the distance to the reconstructed location increases. For the middle reconstruction, the offset shows that the object has moved away from that location, therefore the corresponding weight for that location will be small. For the top reconstruction, the offset arrives al that location, therefore the contribution from the starting point of the offset will be high.
3.1 Formalism
We denote I an image drawn from camera e E [1, CJ at acquisition time t E [1, Here C' 2 is the number of cameras and T 2 length of the vidco sequence. Let P be pairs of consocutive images taken at times and / ++1 by camera c. We define as Pt, the set of all pairs of frames taken time / and (
For each camera c, let K. be the intrinsic camera matrix and let R and te be the camera rotation and translation parameters which can be obtained by calibrating the cameras (Tsai 1987]. As in many other will reason in ground plane that is common to all
works (Hou et al. 2020 Engilberge views. We assume the ground to be flat and set its z-coordinate to O. We can now define homography H. = K. Re]tt]] that relates points in the image plane e to corresponding points in the ground plane. The assumption of flat ground plane valid in most crowd tracking scenarios but the approach can be generalised to complex topographies. The generalisation is performed by replacing the homography with a non-lincar mapping that can be precomputed. We denote the ground plane as 2D grid of size w h. We [0...w] and y [0... h.] the set of all locations in the ground plane.
al. 2023),
define G ((x,v)
For every pair of frames Pt, our network predicts heatmap X E [0, 1 of the same dimension the ground plane. This heatmap represents the probability that someone is present at given ground plane location at time 6. We also predict a corresponding displacement map (4++1 E 22xxxh that indicates where a person located at a given place at time is likely to be at time t+ 1. Each value of At#+1 a 2D vector that represents the z and y components of the displacement of the person at that location. A1++1 can be used compute heat map X as discussed in Section 3.2 These estimated values can then be compared to ground truth binary maps X and X++1 that denote the uct presence of someone any given location.

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
0
==1.1
0.8
A=2
Weight Value w())
A=5
1,0.5
0.6
0.4
0.2
0.0
0
2
3
4
5
Distance [px]
Figure Reconstruction weight function During the reconstruction from detection and offset, the contributions of previous detections are reweighted using the function plotted above. With a value of X, 0.8 location 2 pixels away from the reconstructed location (distance accounting for motion offset) has weight of and is fully added to the reconstruction that location. By varying X, control the trade-off between reconstruction accuracy and differentiabilltty. Note that when A = 5 the weight value is only one when the pixel distance is smaller than 0.5 (represented by the dashed, black line), during reconstruction this means that the detection in the previous time step only contributes to a single location in the next one.
3.2 Motion-Based Supervision
Given a frame P., most existing methods compute X and X++1 independently and are trained to make them as similar as possible to X and X gt By contrast, our approach estimates and 44+4 and uses them infer X+1, In effect, force the estimated motion be consistent with the observed probabilities of presence, which provides supervisory signal on the predicted motion without having to provide motion annolations. Note that when repeat the process on PL+1, we also force the X directly predicted by the network to be similar to x Thus, we also use the annotations as effectively as the earlier methods.
To implement this, the inference of X from X and 41+1 must be differentiable. To this end, we imple- mented the following algorithm that derives a value any given location by spatially global interpolation. Let 2t and 8 be probabilities of presence at location of X and X++1
We write
a5t1 = (((x:(((
(1)
IE
1
W(I
(2)
where 0/.0+1 is the predicted displacement location i of 11++1 and d is the Euclidean distance between coordinates and Note that and j represent locations in the ground plane and consist of pairs (z, y) of pixel coordinates.
W(I) is a weight function that decays with distance I and Ay is a hyper-parameter that controls the speed of the decay. In Fig. g we plot this function for different Xp. During training, lower values of A, make convergence casier by allowing contribution from locations that are further apart. However, incrcasing X, makes the reconstruction more accurate. For this reason 7 initialized with small value during training, which is then increased time the precision of the predicted offset improves.
5

---

Published in Transactions on Machine Learning Research (11/2024)
We can now define loss function
T.mot Σ /oot (P*)
mmot (P") = |X
gt
whose minimization will enforce consistency between estimates of the probability of presence and the pre- dicted motion.
3.3 Training Loss Function
When training our model, we minimize the composite loss function
L L + Ldet + Lno + Ase
(4)
where Lmot the motion- aware loss of Eq. 3 While Laet is detection loss and Ln, and Lee are regularization losses weighted by the scalars An and Ase which describe below.
3.3.1 dett Detection loss.
Detection heatmaps at time t and + are supervised using ground-truth detections. We write
Ldet = Σ ldet (P'),
(5)
leet
X 22 + +|| 1
3.3.2 Lob Forward/Backward Loss.
For any t, we can reverse the order of the images in the frame P. and, instead of estimating A1++1 estimate A411. In other words, we can reverse the motion and, if the network is appropriately trained, we should obtain a motion that approximately the opposite the original one. To enforce this, we define
Lsb = Σ lab (P').
(6)
ln (P')
(1.011 - 05+1,0)2
IEG
where j + 6. gf,
3.3.3 Iye: Motion Spatial Extent Loss.
In practice, the ground truth maps are obtained by pasting Gaussians kernels where people are and zeros elsewhere. As result, individual detections appear as Gaussian peaks. Since we train our networks to approximate these ground-truth maps, the same can be said about X To reliably move such peaks from their location at time to that at time + 1, the predicted motions should have roughly the same spatial extent. To encourage this, we define
Lee = Ei
1se (P*)
Lee (P')
Σ STD (o : :iENNt).
N
where STD is the function computing the standard deviation, and N, the neighborhood of ground truth point p with the same radius the ground truth Gaussian peaks.

---

Published in Transactions on Machine Learning Research (11/2024)
Motion From Detection
Motion In Tracker
BBoxes
Gaussian
Reconstructed
Frame Detections
Motion offsets
Motion Predictor
map
Differentiable
Reconstruction
Detector
X
X1TT
A
R++1
A tt++1
xt+1
fo
Motion
offset
Motion Aware Tracker
Joint Motion and Detection
View Feat.
Ground Feat
Motion offset
H
Reconstructed map
Feature Extractor
Scene Aggregator
Differentiable
Reconstruction
P
G
A c+++
Xt+1
H
Object Trajectories
feo
F++1
G
fe,
X
Detection map
Figure 4: Network Architectures. Our approach to motion prediction is flexible and usable in conjunction with various detectors/ ttracorrssstaersss.e.r Trainable components have blue background, while static modules have white background. Top Left: Single-view setup where the detector's serves as the primary training signal for the motion predictor. Bottom Left: Multi-view with simultaneous training for delection and motion. ResNet feature extractor produces feature maps F. These are projected onto a common ground via view homographics H. resulting in G. These are combined, and passed scene aggregator producing an initial detection heatmap X and a motion offset map A4,+1 which a differentiable reconstruction module to compute the next frame's detection map X++1, Right: The predicted motion can be leveraged by trackers to predict higher quality trajectories.
3.4 Model Architecture and Training
We use two different network architectures depicted by Fig. one for the single-view case and the other for the multi-view one.
3.4.1 Single-Viee View Model.
We extend the state-of-the-art YOLOX architecture GG et al. 2021 to process a pair of frames, P and to output an additional displacement map, A Since YOLOX returns bounding boxes, we convert them Gaussian heatmaps X and X++1 ,hey are then concatenated and fed through a ResNet (He al,, 2016) parametrized by weight 0, which outputs 2D displacement map 44++1 Finally, the displacement map and the heatmap X are fed to the prediction module of Section 3.2 to estimate X
3.4.2 Multi-View Model.
is similar in design to the ones of Hou al. (2020 Engilberge al. (2023). As shown in Fig. 4 it takes as input the images of P and uses a ResNet He et al. (2016) parametrized by weight d to extract features from all images from all cameras at both time steps independently. The feature maps are then projected onto a common ground plane using their respective image to ground plane homography H., as defined Soction 3.1 The results are concatenated and processod by a scene aggregator parametrizod by weight 0, that outputs a detection probability map X and a 2D displacement map A4++1 Both are fed to the prediction module of Section 3.2 estimate X++
The network

---

Published in Transactions on Machine Learning Research (11/2024)
3.4.3 Training
For both model X X++1, and A are used to compute the losses Lmot Eq. and Lse of Eq. 7 To compule A++1 and the loss Lab of Eq. [ we simply reverse the temporal order of the images P The training of the two models only differs with respect to Ldet In the single view case, YOLOX is pretrained and kept frozen, and thercfore Ldet is ignored. These losses are summed into the full training loss L of Eq. that we minimize over our training set to set the network weights 0.
3.5
Inference and Track Creation
At inference time, we our now trained network and obtain X and At++ for each time
3.5.1
Multi-view association
For the multiview model we perform non-maximum suppression in each X and use k-means clustering on the confidence of the remaining maximum (with k 2) to separale true delections from noise. To link these detections into trajectories, method that operates on graph whose nodes are the detections connected by temporal edges that connect detections at different times. These edges are weighted as follows. An edge connecting locations and j at times 4 and l2 respectively is taken to be
use MuSSP (Wang et al.) 2019) that does this using min-cost flow
((e c<c((ti-tal-1) goad(t.s) W.,.
Where d(i,j) is the Euclidean distance between the two locations and j. or, da and om hyperparameters that control the contribution of the temporal, spatial and motion-based distances between the detections, respectively. Wm tta (211 is a new term used to model motion based distances wetween
detections. Specifically Wm measure the distance between location i at time t and the expected position of the detection at location j after applying the estimated motion defined by 0/2-2=1
3.5.2 Single-view association
In our single-view pipeline, we retain the original post-processing step implemented by YOLOX. For track association, we modify ByteTrack (Zhang al.] 2022). First, we modify the detection association step utilize ground plane IoU instead of image plane IoU. Secondly, we replace the Kalman filter used in ByteTrack for motion estimation with the learned predicted motion 1 output by our extended YOLOX.
4 Experiments
In this section, we empirically validate our approach. After defining the experimental protocol, we first evaluate the quality of the predicted motion. Subsequently, we assess the complete multi-view detection and tracking pipeline on the WILDTRACK dataset (Chaydarova et al. 2018), and the singlc-view pipeline on MOT17 [Milan et al. 2016). Finally, we conduct multiple ablations to explain the contribution each component to the overall performance.
4.1 Experimental Setup
4.1.1
Datasets
We evaluate our approach on WILDTRACK Chavdarova et al. 2018 and MOT17 MMilan et al. 2016
WILDTRACK is a calibrated multiview multi-person detection and tracking dataset featuring 7 view- points focusing on an area of 12 x 36m² in the real world. It contains 400 synchronized frames per view with resolution of 1080 1920 pixels. Each person annotated with a bounding box. As in Engilberge et al.) 2023)) we discretize the ground plane such that one cell corresponds 20cm in the real world yielding a ground plane map of size 180 x 80 cells.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Tracking performance on WILDTRACK. Our model significantly outperforms existing base- lines across all tracking metrics. The performance values for MVDet and MVDeTr are taken from Engilberge et al. 2023), while those for other methods are sourced from their respective papers. For our method, we report both the best result and the average result over three runs.
WILDTRACK dataset
Model
MOTA
MOTP
IDF1
IDP
IDR
DeepOcclusion+KSSP
10
69.6
73.2
83.8
65.0
10
72.2
78.4
84.4
73.1
ReST
[2]
81.6
85.7
MVDet
24
muSSP
80.6
0.80
79.4
79.2
79.6
Tr
muSSP
89.4
0.58
90.7
90.5
90.9
BarlyBird
!E!
89.5
92.3
MVFlow
16
91.3
0.57
93.5
92.7
94.2
Ours mean
91.7=1..2
0.56=0.02
93.9+0.22
93.0=1.2
94.9#0.1
Ours best
91.9
0.54
94.1
93.2
95.0
MOT17 is a single-viow, multi-person detection and tracking dataset comprising sequenees captured by both static and moving cameras. The sequence lengths vary from 450 to 1500 frames, with frame rates ranging from 14 30 fps, totaling 15,948 frames. Video resolutions are either 1080x1920 pixels or 680 pixels, and each sequence contains up to 222 tracks. Every individual is annotated with bounding box. For our method, utilize the groundplane homographies H. determined in Dendorfer et al. (2022) to map into the ground-plane. Since the lest set is private we follow the train/ val splil of Zhou et al. (0020)
4.1.2 Implementation Details.
Our modes are implemented in Pytorch Paszke et al. 2019 and trained on a single NVIDIA A100 GPU,
2021 described in Section 3.4 We use the cxisting implementation of MMDetection (Chen et al. 2019) to develop our training pipeline. To augment the input images, we follow the strategy outlined in Ge et al. 2021). To accommodate pairs of frames as input, the detections are first extracted for the two frames using a frozen YOLOX, the detection are converted to gaussian heatmaps of 512 channels, first channel is a gaussian mask, while the remaining 512 channels are use provide crop features of each detoction. ResNet-50 (He et al., 2016) is used to extract feature for each heatmaps independently. The number of channels in the first layer of the ResNet is modified from to 513 accordingly. The resulting features are concatenated along their channel dimension before being fed into two convolutional blocks consisting of ReLu & & Hinton 2010 convolutional layer. The last convolutional layer output a 2 chamnels displacement map. The model is trained on MOT17 following the approach in Milan et al. (2016), with modifications as described in Section 3.4 Training images are supplied with random intervals between them to encourage the model to accurately learn motion behavior at various frame rates.
Single- View Model The detector is modified YOLOX GG et al.
Multi- View Model The feature extractor ResNet--00 (He et al. 2016) whose last. four layers have been removed such that outputs feature maps containing 256 channels. The input images are augmented using a random crop strategy with a probability of 0.5. The scene aggregator a multi-scale module, it comprises four scales. Between each scale the feature resolution is halved. Each scale consists of four blocks of 'convolutional layer batch normalization (Toffe & Szegedy 2015) ReLu (Nair & Hinton 2010) : The outputs of the four scales are bilinearly interpolated back to their original dimension. After concatenation, they go through a final 1 1 convolutional layer with output channels. One channel goes through a sigmoid function to produce the detection probability map. The other are taken to be the motion offsets. The model trained on WILDTRACK using the Adam optimizer (Kingma Ba, 2015) with lcarning rate of 0.0001 and a batch size of one. The learning rate is halved after epochs 20, 40 and 60.
Memory optimization The differentiable reconstruction model of Section 3.2 aggregates the entire input detection map to reconstruct. each location of the output. In practice this becomes too memory intensive
9

---

Published in Transactions on Machine Learning Research (11/2024)
MOTA Low Framerate MOT17
Tracking performance with low frame rate on MOT17 validation. We also evaluate our approach in a monocular setting. We use the MOT17 dataset and modify YOLOX and Bytetrack to predict and use motion.
Table 2:
0.70
MOTA
0.65
0.60
MOT17 val dataset
Model
FPS
Sct
MOTA
MOTP
IDFI
ByteTTrcck
0.75
val 51.1
84.6
55.0
Ours
0.75
val
58.9
84.8
58.3
5.06.0 Framerate (FPS)
ByteTraak 1
2
val
59.1
59.6
Ours
2
val
65.5
84.4
62.9
Figure 5: Tracking results in low FPS scenario. In low frame rate scenarios our model outperforms the ByteTraac baseline in term of MOTA. The lower the frame rate the higher the performance gap, showing the benefit of the pro- posed approach.
Center
in
30
val
61.1
ByteeTrack
62
30
val
77.0
84.1
77.5
Ours
30
val
76.6
84.1
76.1
when the input/output resolutions increase. Therefore the reconstruction step is implemented with a sliding window, where the window size can be set to any value, making memory consumption fixed and not dependent use window size of 59 pixels, meaning that, at most
heatmap resolution. For all experiments the reconstruction step handle motion offset of length 59/2 29.5 pixels. Which is larger than any groundtruth motion.
The reconstruction hyperparameter A, is initialized to 0.8, during training it is in- creased at the end of every epoch by 0.08 until it caps out at 5. Additionally when computing the detection loss on roconstruction, the ground truth heatmap is also passed through the roconstruction module with a zero offset to have the same imprecision (scaled up Gaussian peak linked to the value of X on both the ground truth and prediction. The hyperparameters in the loss L are set to An 0.05 and Age = 1. A summary of the hyperparameters can be found in Table A.6.
dyperparameters
4.1.3 Metrics.
To gauge the quality of the final trajectories, we report the same CLEAR MOT metrics (Kssturi et al. 2009) as in previous works Chavdarova, al. (2018); Engilberge al. (2023). They include Multiple Object Detection Accuracy (MODA), Multiple Object Tracking Accuracy (MOTA). and Multiple Object Tracking Precision (MOTP). We also report identity preservation metric IDF1 RRitani et al. 2016 computed from Identity Precision (TDP) and Tdentity Recall (TDR). Since we are concerned with motion and tracking people, MOTA, MOTP, and IDF1 are the most significant because they quantify the quality of trajectories, instead of that of detections in individual frames like MODA. We use the py-motmetrics and MMDetection (Chen et al. 2019 libraries to compute these metrics.
Since a specificity of our approach is predict 2D motion offsets, we also seek to quantify how good they are. To this end, we compute the L1 distance between predicted and the ground truth offsets. For non- zero ground truth offsets, we also compute the average angular error and norm between ground truth and prediction. We will refer to these measures as L1, Angle. and Norm.
4.2
Comparative Results
4.2.1 Tracking evaluation
In Table we compare the CLEAR MOT metrics Kasturi al. 2009) achieved by our method with those yielded by other state-of-the-art methods. On WILDTRACK, our approach significantly outperforms all others. For MOT17, results can be found in Table 2 Our approach surpasses the original YOLOX
10

---

Published in Transactions on Machine Learning Research (11/2024)
ByteTrack at most frame rates, thus demonstrating the advantage of learning to represent motion. In low frame rate scenarios, it significantly outperforms the baselines: at 2FPS, MOTA is improved by more than 10%. We further illustrate the ability of our single view method in low frame rate scenario in Fig. 5
4.2.2 Motion evaluation
To evaluate, the quality our motion offsets, we compare them to those produced by two additional baselines. First, since our motion offset representation shares similarities with optical flow, we used RAFT (Teed & Deng 2020), a network pro-trained to estimate optical flow, to estimate the flow in each one of the viewpoints. The resulting flows are projected onto the ground plane and averaged. Second, we trained version of our model using full supervision by skipping the reconstruction and instead directly applying an L2 loss between predicted and the ground-truth offset. In other words, unlike in our normal approach, we provide full motion supervision. The groundtruth motions are derived from identity annotations present in WILDTRACK. We report the results in Table 3 Our weakly supervised outperforms the RAFT basclines by significant margins and comes close to the fully supervised method on both datasets. In Fig. g we provide qualitative result WILDTRACK show this visually. The supplementary material contains video results illustrating the robustness of our motion prediction method even in low frame rate scenario.
Table Motion offsct evaluation We evalu- ate 2D motion offset according to three metrics, L1 error, Angular error (in degree) and Norm error. Our model outperforms the optical flow baseline by a large margin. It even performs com- petitively against the fully supervised methods, especially in terms of norm error.
Table 4: Ablation results WILDTRACK We conduct an ablation study three com- ponents of the system, the spatial extent term (Eq. (7), the temporal consistency term (Eq. 16 and the reconstruction step (Eq.
Losses
Metries
Lyoo
Liee
Lpb
MODA f
L1 4
Angle J
WILDTRACK dataset
90.7
0.87
82.0
model
L1
Angle + Norm
89.6
0.59
36.2
RAFT
43
1.06
48.8
2.07
v
90.0
0.65
43.8
Supervised
0.55
32.5
0.90
v
91.3
0.59
35.0
Ours
0.58
34.8
0.92
91.8
0.58
34.8
4.3 Ablation Study
To understand which components of the model contribute to the overall performance, we conduct an ablation study on the WILDTRACK dataset. We test three components our model, the smoothing term Lye, the temporal consistency term Lns of our loss both defined in Section 3.3 and the reconstruction from motion defined in Section 3.2] To disable the reconstruction from motion, smoothing and temporal consistency we simply omit the losses.
As it can be seen in Table adding Lmot greatly improves the motion metrics at the cost of slightly degrading the MODA one. This was to be expected since the same model has to predict motion on top of the original detection. Further adding Lse and Lib only has beneficial effect both in terms of MODA, L1, and angle error. The best score is obtained when the three elements are combined.
We also conduct an ablation study on the benefit of using motion offset in the muSSP (Wang et al. 2019) association algorithm. We modify Eq. ( by removing the last term Wm and hence ignore the predicted motion offset. When muSSP is executed without motion offset MOTA drops by 0.5 points and IDF1 0.9 by points compared to the last row of Table [ This confirms the usefulness of integrating motion offset inside association algorithms.
4.4 Limitations
One limitation is tied the formulation of Lmot- It has multiple local minima, any motion flow that correctly maps the detection from one timestep to the next would yield zero loss. In practice and when combined
11

---

Published in Transactions on Machine Learning Research (11/2024)
Ours
Groundtruth
Figure 6: Visualizing the Motion Offsets. We visualize the detection and the motion offset predicted by our model on the left and the ground truth on the right. The green squares mark the position of the person time and the blue square the position at time + 1. The green arrows connecting them denote the ground truth motion. The offset map is visualized by regularly drawing arrows corresponding to the motion offiset. Qualitatively the predicted detection and motion offset are very close to the ground truth ones. The quantitative results reported in Table confirm this.
with Lee and Ly we observe that the model converges to predicting the real motion. Additional results are provided in the appendix to showcase this.
A second limitation of the proposed differentiable reconstruction from motion is its memory cost. The memory consumplion grows quadratically with respect to the dimension of the space it is reconstructing: To mitigate this difficulty, the implementation relies on sliding window mechanism, as described in Section 1.1 This fixes the required amount of memory but also limits the maximum range of motion to half the size of the window. In practice it hasn't been a problem, but it could one for very high resolution scenarios that feature long-range motions. In such scenario memory would be bottleneck anyway, and parallelizing the workload across multiple GPUs would be the way overcome this limitation.
5 Broader Impact
Our rescarch contributes to the field of multi-object tracking by proposing motion prediction method that reduces the need for costly identity annotations, relying only on detection supervision. This development has significant implications for real-world applications such surveillance, autonomous driving, and public safety, where robust tracking is crucial but manually labeling motion data is prohibitively expensive. By improving tracking performance challenging low wwwframo-rate scenarios and crowded environments, this work has the potential to enhance the efficiency and accuracy of systems that monitor human activity, making them more accessible and scalable.
However, the deployment of such technologies must be done with care, as they could raise privacy concerns if used for mass surveillance other invasive purposes. It is important that future applications of this work consider othical implications and adhere to strict guidelines to ensure the protection of individual privacy and mitigate risks of misuse.
6 Conclusion
We have proposed a network to predict both delection heatmaps and motion offsets given only detection supervision. To this end, we use the consistency between the predicted heatmaps and offsets as a supervisory signal. We have demonstrated that this increases performance challenging single- and multi-view people- tracking scenarios. The effect of this approach is particularly noticeable in low frame-rate scenarios, where conventional tracking-by-detectioo approaches are most. likely to fail.
12

---

Published in Transactions on Machine Learning Research (11/2024)
References
[L] M. Andriluka, S. Roth, and B. Schiele. Soople-Tracking-yyy By-Detection and People-Detection By- Tracking. In Conference on Computer Vision and Pattern Recognition, June 2008.
[2] P. Baqué, F. Flcuret, and P. Fua. Deep Occlusion Reasoning for Multi-Camera Multi- Target Detection.
In International Conference on Computer Vision, 2017.
[3] B. Benfold and I. Reid. Stable Multi-Target Tracking in Real-Time Surveillance Video. In Conference Computer Vision and Pattern Recognition, 2011.
[4] H. BenShitrit, J. Berclaz, F. Fleuret, and P. Fua. Multi- Colti-Commodity Network Flow for Tracking Multiple People. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36($):1614-1227, 2014.
[5] J. Berclaz, F. Flouret, and P. Fua. Robust People Tracking with Global Trajoctory Optimization. Conference Computer Vision and Pattern Recognition, 2006.
[6] J. Berclaz, F. Fleuret, E. Türetken, and P. Fua. Multiple Object Tracking Using K-Shortest. Paths Optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(11):1806-18) 1819, 2011.
[7] P. Bergmann, Mcinhardt, and L. Lcal-Taixe. Tracking Without Bells and Whistles. In Conference on Computer Vision and Pattern Recognition, 2019.
[8] A. Bewley, Z. Ge, L. Ou, F. Ramos, and B. Uperoft. Simple Online and Realtime Tracking, In International Conference on Image Processing, 2016.
[9] M.D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier and L.Van Gool. Robust Tracking By- Detection Using Detector Confidence Particle Filter. In International Conference on Computer Vision, pp. 1515 1522, 2009.
[10] T. Chavdarova, P. Baqué, S. Bouquet, A. Maksai, C. Jose, L. Lettry, P. Fua, L. Van Gool, and F. Fleuret. The Wildtrack Multi- Caltii-Caerraa Person Dataset. In Conference on Computer Vision and Pattern Recognition, 2018.
[11 Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint ar Xiv: 1906.091661... 2019.
[12] C.-C. Cheng, M.-X. Qiu, C.-K. Chiang, and S.-H. Lai. Rest: A Reconfigurable Pratial-Temport Graph Model for Multi-Camera Multi-Object Tracking. In International Conference on Computer Vision, 2023.
[13] G. Ciaparrone, F. L.. Sánchez, S. Tabik, L. Troiano, R. Tagliaferri, and F. Herrera. Deep Learning in Video Multi-Object Tracking: A Survey. Neurocomputing, 2020.
[14] P. Dendorfer, V. Yugay, A. Osep, and L. Leal-Taixé, Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Term Multi-Object Tracking?, 2022.
[15] A. Dosovitskiy, P. Fischer, E. Ilg. Hausser, C. Hazrbas, V. Golkov, Smagt, D. Cremers, and T. Brox. Flownet: Learning Optical Flow with Convolutional Networks. In International Conference on Computer Vision, 2015.
[16] M. Engilberge, W. Liu, and P. Fua. Multi-View Tracking Using Weakly Supervised Human Motion Prediction. In IEEE Winter Conference on Applications of Computer Vision, 2023.
[17 M. Engilberge, H. Shi, Z. Wang, and P. Fua. Two-Level Data Augmentation for Calibrated Multi- View Detection. Tn TEEE Winter Conference Applications of Computer Vision, 2023.
13

---

Published in Transactions on Machine Learning Research (11/2024)
[18] P.F. Felzenszwalb, R.B. Girshick, D. McAllester, and D. Ramanan. Object Detection with Discrimina- lively Trained Part Based Models. IEFE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627 1645, 2010.
[19] F. Fleuret, J. Berclaz, R. Lengagne, and P. Fua. Multi-Camera People Tracking with a Probabilistic Occupancy Map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):267-282. February 2008.
20 Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Excooding YOLO Series in 2021, 2021.
[11] R. Girshick. Fast R-CNN. In International Conference Computer Vision, 2015.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
[23] Y. Hou and L. Zheng. Multiview Detection with Shadow Transformer (And View-Coheren Coherent Data Aug- mentation). In Proccodings of the 29th ACM International Conference on Multimedia, pp. 1673-1682 2021.
[24] Y. Hou, L. Zheng, and S. Gould. Multiview Detection with Feature Perspective Transformation. In European Conference on Computer Vision, pp. 1-18, 2020.
[25] S. Ioffc and C. Szegedy. Batch Normalization: Accelorating Deep Network Training by Reducing Internal Covariale Shift. In International Conference Machine Learning, 2015.
[26] H. Jiang, S. Fols, and J.J. Little. Lincar Programming Approach for Multiple Object Tracking. In Conference on Computer Vision and Pattern Recognition, pp. 1-8, June 2007.
[27] R. Kasturi, D. Goldgof, P. Soundararajan, V. Manohar, J. Garofolo, M. Boonstra, V. Korzhova, and J. Zhang. Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol. TEEE Transactions Pattern Analysis and Machine Intelligence,
31 31(2: 2009.
[28] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Represontations, 2015.
[29] P. Lenz, A. Geiger, and R. Urtasun. Followme: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation. In International Conference on Computer Vision, pp. 4364 4372, December 2015.
330] W. Liu, M. Salzmann, and P. Fua. Estimating People Flows to Better Count Them in Crowded Scenes. In European Conference Computer Vision, 2020.
[31] W. Liu, N. Durasov, and P. Fua. Leveraging Superviino for Cross-Domain Crowd Counting. In Conference on Computer Vision and Pattern Recognition, 2022.
[32] A. Milan, L. Leax-Taixe, Reid, S. Roth, and K. Schindler. Mot16: A Benchrnrkk for Multi-Object Tracking. In arXiv Preprint, 2016.
[33] V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In International Conference on Machine Learning, 2010.
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An Imperative Style, High-Porformance Deep Learning Library. In Advances in Neural Information Processing Systems, 2019.
[35] H. Pirsiavash, D. Ramanan, and C. Fowlkes. Globally-Optimal Greedy Algorithms for Tracking a Variable Number of Objects. In Conference on Computer Vision and Pattern Recognition, pp. 1201-
1208, June 2011.
14

---

Published in Transactions on Machine Learning Research (11/2024)
[36] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN:
Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems, 2015.
[37] E. Ristani and C. Tomasi. Features for Multi-Target Multi- Multi-Camera Tracking and Re-Identificatio. Conference Computer Vision and Pattern Recognition, 2018.
(38] E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi. Performance Measures and a Data Set for Mulgi-Target, Multi-Camera Tracking. In European Conference on Computer Vision, 2016.
[39] G. Shu, A. Dehghan, O. Oreifej, E. Hand, and M. Shah. Part-Based Multiple-Person Tracking with Partial Occlusion Handling. In Conference on Computer Vision and Pattern Recognition, 2012.
[40] A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah. Visual Tracking: An Experimental Survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(7):1442-1444 1468, July 2014.
[11] L. Song, J. Wu, M. Yang, Q. Zhang, Y. Li, and Yuan. Stacked Homography Transformations for Multi-View Pedestrian Detection. In Conference Computer Vision and Pattern Recognition, 2021.
[42] S. Tang, B. Andres, M. Andriluka, and B. Schiele. Subgraph Decomposition ion for Mulgi-Target Tracking. In Conference on Computer Vision and Pattern Recognition, pp. 5033-5041, 2015.
[43] 7. Teed and J. Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In European Conference on Computer Vision, 2020.
[44] T. Teepe, P. Wolters, J. Gilg, F. Herzog, and G. Rigoll. Earlybird: Early-Fusion for Multi View Tracking in the Bird's Eye View. In Conference on Computer Vision and Pattern Recognition, 2024.
[45] R.Y. Y. Tsai. A Versatile Cameras Calibration Technique for High Accuracy 3D Machine Vision Metrology Using Off-Tf--Sheff TV Cameras and Lenses Journal of Robotics and Automation, 3(4):323-344 344, 1987.
[46] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sckar, Andreas Geiger, and Bastian Leibe. Mots: Multi-Object Tracking and Segmentation. In Conference on Computer Vision and Pattern Recognition, 2019.
[47] C. Wang, Y. Wang, Y. Wang, C.T. Wi, and G. Yu. muSSP: Efficient Min-Cost. Flow Algorithm for Multi-Object Tracking. In Advances in Neural Information Processing Systems, pp. 423-432, 2019.
[48] N. Wojke, A. Bewley, and D. Paulus. Simple Online and Realtime Tracking with Deep Association Metric. In International Conference Image Processing, 2017.
[49] Y. Xu, X. Liu, Y. Liu, and S.C. Zhu. Multi-Viee People Tracking via Hierarchical Trajectory Compo- sition. In Conference on Computer Vision and Pattorn Recognition, pp. 4256-4265, 2016.
[0] Y. Xu, A. Osep, Y. Ban, R. Horaud, L. Leal-Taixe, and X. Jlameda-Pineda. How to Train Your Deep Multi-Object Tracker. In Conference on Computer Vision and Pattern Recognition, 2020.
[1] A. Yilmaz, O. Javed, and M. Shah. Object Tracking: A Survey. ACM Computing Surveys, 38(4). December 2006.
[22 Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. ByteTraak:: Multi- Object Tracking by Associating Every Detection Box, 2022.
[3] K. Zhou, Y. Yang, A. Cavallaro, and T. Xiang. Omni-Scale Feature Learning for Person Re- Identification. In International Conference on Computer Vision, 2019.
54] X. Zhou, D. Wang, and P. Krähenbühh. Objects Points. In arXiv Preprint, 2019.
[55] X. Zhou, V. Koltun, and P. Krähenbühl. Tracking Objects as Points. Tn European Conference on Computer Vision, 2020.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
Appendix organized as follows: Appendix provides a detailed evaluation of the predicted displacement and additionnal analysis of the single-view model. Appendix B offers additional quantitative results on MOT17. In Appendix E present visualizations of the motion offsets. Appendix details tracking results for additional dataset, Multiview.X. The supplementary archive also contains discussion about broader impact of the work Section 5 video examples discussed in Appendix G and the code described in Appendix H
A
A Further Analysis of Single-Vies View Model
A.1 Comparison with Kalman Filter
The original Byte'Track algorithm uses a Kalman Filter to estimate the motion of objects between framess We aim to assess the contribution of the Kalman Filter to BytcTrack's performance and compare it to our proposed motion estimation mechod.
Bytelrack
0.75
Ours
0.70
MOTA (%)
0.65
0.60
0.55
0.50
0.4
0.7
1.0
2.0
3.0
5.06.07.5
10.0 15.0
30.0
Framerate (FPS)
In our adaptation of ByteTrack, we substitute the Kalman Filter with the motion offset. predicted by our method. This comparison demonstrates that the Kalman Filter exerts a negligible influence on BraeTrack, offering slight enhancements at high frame rates while providing no benefit or even impairing performance lower frame rates. In contrast, our approach markedly enhances performance low frame rates.
Figure A.1: Comparison with Kalman Filter:
To this end, present results various frame rates for modified version of ByteTrack where the Kalman Filter has been removed. In other words, new detections are directly matched to the head of existing trajectories.
The results of this modified model can be found in Figure Fig. A.1 Surprisingly, the motion estimated through the Kalman Filter only has a marginal effect on the tracking performance. At high frame rates (> 7.5 FPS), MOTA slightly increases, while at lower frame rates, it has no effect or detrimental. Our weakly supervised method for motion estimation performs significantly better than the Kalman Filters.
A.2 Additional Motion Baselines
As discussed in the limitations section of the main article, the proposed method for training a motion predictor with detoction supervision can have multiple global minima. Any displacement map that maps all motion at timestep & a unique detection timestep + would result in Lmot 0. In practice, when combined with the smoothing term Lae and the time consistency term Lep,, we observe that the model
16

---

Published in Transactions on Machine Learning Research (11/2024)
converges to predicting the actual motion. We argue that, considering time consistency and given the provided data, it is easier for the model to learn the true motion than any other solution.
3 of the main paper already demonstrate the quality of the predicted displacements, we propase to compare the motion estimated by our model to two other baselines that could minimize the loss /mmot 0.
While the results in Table
0.75
Ours
Motion
0.70
MOTA (%)
0.60
0.55
0.50
0.4
0.7
1.0
2.0 3.0
5.06.07.5
10.0 15.0
30.0
Framerate (FPS)
Figure A.2: A Simple Motion Baseline: We compare our motion prediction to a basic baseline where the motion from one timestep to the next is estimated by identifying the nearest detection. Utilizing this motion in place the Kalman filter in ByteTrackk results in the green curve depicted above. Surprisingly, this straightforward baseline outperforms ByteTrrack at low FPS; however, it is disadvantageous high FPS. Our methods outperforms this baseline in both scenarios.
A.2.1 Motion to Nearest
First, we will test a bascline that estimates motion such that maps a detection time the nearest detection at time + The results for this baseline can be found in Fig. A.2
Surprisingly, such simple baseline outperforms Byte'Track at low frame rate regimes (<3 FPS) and un- derperforms at higher FPS. When comparing the proposed motion estimation method to this baseline, the weakly supervised motion greatly outperforms the baseline. This would tend to confirm that the proposed model not morely learning to map detections to their nearest neighbors.
A.2.2 Motion from Bipartite Matching
One could argue that the previous baseline does not guarantee Lmot to be zero, since multiple detections can be mapped towards the same detection, which is indeed a valid concern. Therefore, we propose a second baseline that guarantees reaching a global minimum of Tmmot- We suggest estimating motion using bipartite matching method match detections from time with those from time + 1, utilizing the Hungarian algorithm.
The results be found in Fig. A.3 Contrary to expectations, the bipartite matching baseline performs worse than the nearest neighbor approach. This is likely due to the one-to-one matching constraint, which, when applied to noisy detections, can lead to detections being wrongly matched with distant ones, resulting in large motion errors. Even adding a cutoff distance does not significantly improve performance. heles,, the bipartite matching bascline slightly beneficial very low frame rate scenarios (<2 FPS) but detrimental otherwise. When compared the proposed approach, its performance is significantly lower, confirming that the proposed model is not learning form of bipartite matching but is more likely capturing the true motion.
17

---

Published in Transactions on Machine Learning Research (11/2024)
ByteTack
0.75
Motion from Bipartite Matching
0.70
MOTA (%)
0.65
0.60
0.55
0.50
0.4
2.0
5.06.0 10.0 15.0
30.0
Framerate (FPS)
Figure A.3: Motion from Bipartite Matching: ByteTracc employs the Hungarian algorithm associate detections between timesteps using similarity score computed via the Intersection over Union (IoU) of bounding boxes. This approach fails when the motion is significant and the intersection is null. To address this, we propose a straightforward baseline utilizing the Hungarian algorithm to estimate the motion between detections at two timesteps. Similar to the previous method, this estimated motion replaces the Kalman filter in ByteTrack. The results, represented by the green curve, indicate a marginal benefit at very low frame rates (<2 FPS) and are otherwise detrimental.
Additional qualitative results for both baselines can be found Fig.
A.7
A.3 Tracking on the Groundplane
Our method utilizes the ground plane to predict motion and associate detections at low frame rates. To assess its contribution the overall performance of our model, we propose comparing it. with a ground plane-adapted version of ByteTrack.
The association step in BraeTrack is modified as follows: Detections are projected onto the ground plane: The bottom center of the bounding box is projected onto the ground plane using the frame homography H. This ground point serves as the center of a square ground bounding box, with side length of 25. These ground bounding boxes are used to compute similarity scores, which replace ByteTrack's original similarity scores. Results for this baseline can be found in Fig. A.4
Below 5 FPS, BytcTrack Ground significantly outperforms BrtcTrack, which is expected since operating on the ground plane mitigates the effects of perspective projection, thereby making motion and speed uniform across the scene and simplifying association. Above 5 FPS, it performs worse than ByteTrack, which may be attributed to noise in the detections and inaccuracios in ground planc projection.
Overall, our method significantly outperforms the ByteTrack Ground baseline across all frame rates. The performance gap between the two reflects the superiority our motion estimation.
A.4 Detection Features
In the single-view scenario, our motion prediction model is trained solely using the detection map. To enrich the scene representation, the input map can be augmented by appending convolutional features corresponding to each detection onto their channel dimension. To achieve this, we utilize a pretrained OSNet (53) to extract features with dimensionality of 512 for cach detection bounding box. It's important note that these features are solely provided input the model and are not utilized to measure the similarity between detections.
18

---

Published in Transactions on Machine Learning Research (11/2024)
ByteTack
0.75
BybelTack Ground
0.70
MOTA (%)
0.65
0.60
0.55
0.50
0.4
2.0 5.06.0 10.0 15.0
30.0
Framerate (FPS)
Working on the ground plane significantly simplify tracking. In the image plane, individuals appear move faster when closer to the camera due to perspective elfects. By negaling this perspective effect, trajectories across the scene become more uniform. This uniformity is especially advantageous at low frame rates, where motion is more pronounced. However, at higher frame rates, this approach can be slightly counterproductive, likely due to calibration inaccuracies or noise in the detection process.
Figure A.4: Effect of Tracking on the Ground Plane:
As demonstrated in Table A.1 the inclusion of detection features, regardless of the frame rate, enhances the tracking performance of our motion prediction network.
A.5 Time Interval
Given that our model processes pairs of frames and aims to predict accurate motion across a broad range of frame rales, explore different strategies for sampling training pairs. In Table A. present results from training pairs sampled with intervals ranging from [1 2[ to [s 13[. These findings indicate that intervals more closely aligned with the target frame rate lead to superior outcomes. Consequently, for all our experiments, we select the training sampling interval based on the cesired target frame rate.
We evaluate two additional elements our training process: the interval between sampled frame pairs during training and the inclusion detection features the input map. Incorporating detection features appears advantageous across all scenarios. Moreover, the training frame interval achieves optimal results when it closely mirrors the inference regime. A frame interval of 1 corresponds to 30 FPS, whereas interval of 10 equates to 3 FPS.
Table A.1: Ablation Results on MOT17:
Model component.
FPS
Metrics
Frame interval
Detetection features
MOTA 1
MOTP 1
IDF1 f
[1 2
2
59.4
84.3
60.6
[ 2
2
61.3
84.6
61.9
[8 13
2
65.5
84.3
58.0
13
30
75.7
84.1
73.2
[ 13
30
76.6
84.1
76.1
[ 2
30
76.6
84.1
76.0
1 2
/
30
76.9
84.1
77.6
19

---

Published in Transactions on Machine Learning Research (11/2024)
A.6 Faster R-CNN detector
Since all single-view results a pre-trained YOL.Ox detector, we propose evaluate approach with a less performant detector: Faster R-CNN (36).
Table A.2: Tracking performance with Fastcr-RCNN detector. We also evalu- ale our approach in a monocular when com- bined with a two stage detector Baster--CNN,
MOTA vs. Low Framerate on MOT17 Faster R-CNN Detector
SyeeTine
0.65
Duts
0.60
Best results with Faster R-CNN detector are underlined
MOTA (%6
0.55
MOT17 val dittsee
Model
FPS
Set
MOTA
MOTP
IDF1
ByseTrack
52
0.75
vil
51.1
84.6
$5.0
Ours
58.9
58.3
0.45
ByteTrack
M9
FRCNN
0.75
val
45.8
80.9
51.2
Ours
FRCNN
0.75
val
48.7
80.9
53.0
0.4
0.7
1.0
2.0 3.0
Framerate (FPS)
5.06.07.5 7.5 10.0
15.0
30.0
ByteTtack
59.1
Ours
val
65.5
84.4
62.9
ByteTrack
x2
FRCNN
val
50.0
80.6
54.6
Figure A.5: Tracking results in low FPS scenario. When using Faster-RCNN object detector the proposed approach outperform the Bytetack approach. The lower the frame rale the higher the performance gap, showing the benefit of the proposed approach.
Ours
FRCNN
53.5
80.5
67.8
Bytelltack
[8
30
val 77.0
84..1
77.5
ByteTrack F9
FRCNN
val 66.2
80.4
71.0
Ours
FRCNN
val
65.7
80.4
67.8
Quantitative results The results are shown in Table A.2 and Fig. A.5 As expected, both ByteTrack + Faster R-CNN and Our + Faster R-CNN perform worse than their counterparts using YOLOx. Nonetheless, our approach, oven when combined with Faster R-CNN, surpasses Byte + Faster R-CNN at most frame rales. These results are consistent with those obtained using the YOLOx detector and rate that the proposed approach performs reliably across different object detectors.
A.7 Memory Consumption and Runtime
In addition to tracking performance, we provide an analysis of the memory usage of our approach for both training and inference the MOT17 dataset. Results can be found in Table A.3
During training, the memory consumption of the differentiable reconstruction step grows quadratically with respect to the dimension of the space being reconstructed. In our implementation, this is managed using a sliding window mechanism. During inference, the reconstruction step is not needed, and the memory overhcad of our approach is minimal.
Table A.3: Memory Consumption Training and inference GPU memory consumption for ByteTTrack (YOLOx) and our method (motion prediction). Tests were during inference and 2 during training. When training method, YOLOx is kept frozen, its memory footprint is not included. During inference, the memory consumption of BrtcTrack (YOLOx) is added to the memory footprint of our method.
NVIDIA V100 with batch size of I
Model
Phase
Memory
Byte'Track
Training
29.4 GiB
ByteTrack
52
Inference
1.7 GiB
Ours
Training
5.2 GiB
Ours
Inference
1.7 0.7 GiB
20

---

Published in Transactions on Machine Learning Research (11/2024)
B
Additional MOT17 results
Duc to the private nature of the test set and the need to conduct experiments at various frame rates, results in the main paper are conducted exclusively on a validation set, obtained by splitting the original training data in half. The first half is used for training and the second one for validation. For more information about the data split, see 152)
For completeness, we provide results at 30FPS on the private lest set. Results can be found Table A.4 The results the
are consistent with those on the validation set and are on par with the original BytcTrack (52)- Note that at 30FPS, the overlap of consecutive objects in trajectory is large and the motion is small. In such scenarios, there is little benefit in using motion; nonetheless, our method is able to preserve the original performance.
Table A.4: Tracking performance on MOT17 test with private detectors. At high FPS, predict- ing motion is of limited interest, nonetheless our method obtains results on par with the original ByteTrack.
MOT17 val dataset.
Model
FPS
Set
MOTA
MOTP
IDF1
CenterTrrack
55
30
test
67.8
64.7
ByteTraak
52
30
test
77.1
80.7
74.4
Ours
30
test
76.7
80.8
72.4
C MOT 20 Results
We provide results for our single view model on the MOT20 datasets.
MOTA vs. Low Framerate on MOT20
ByciTrakk Dus
Table A.5: Tracking performance with low frame rate on MOT20 validation. We also evaluate our approach in monocular setting. We use the MOT20 dataset and modify YOLOX and Byletrack to predict and use motion.
MOTA (%6)
0.5
0.4
MOT20 dataset.
0.3
Model
FPS
Set
MOTA
MOTP
IDF1
ByteTrack
0.75
val 23.1
79.0
24.9
0.2
Ours
0.75
val
30.9
79.0
25.0
0.4
0.7
1.0
2.0 3.0
Framerate (FPS)
5.06.0
10.0
15.0
30.0
ByteTrack
val 51.2
78.8
45.9
Ours
val 55.5
78.8
45.8
Figure A.6: Tracking results in low FPS scenario. In low frame rate scenarios model outperforms the Byte- Track bascline in term of MOTA. The lower the frame rate the higher the performance gap, showing the benefit of the proposed approach.
ByteTrack
30
val
70.7
78.6
70.5
Ours
val
70.6
78.6
69.8
MOT20 is a single-view, multi-person detection and tracking dataset comprising sequences captured by both static and moving cameras. The sequence lengths vary from 429 to 3315 frames, with frame rates of 25 fps, totaling 13410 frames. The sequence represent busy scene and contains up to 1211 tracks. Every individual is annotated with bounding box. For method, utilize the groundplane homographies H.
Results obtained using MMDetection implementation of BrteTrack
21

---

Published in Transactions on Machine Learning Research (11/2024)
determined in Dendorfer al. (14 to map into the ground-plane. Since the test set is private follow the train/val split of MMDetection.
Quantitative results Results can be found in Table A.5 and Fig. A.6 Our approach surpasses the original YOLOX ByleTrack most frame rales, thus demonstrating the advantage of learning to represent motion. In low frame rate scenarios, significantly outperforms the baselines: 2FPS, MOTA is improved by more than 10%. Those results are in line with the results on MOTI7 and show that the proposed approach performs consistently across datasets with various crowd densities.
D Hyperparameter Details
In this section, we provide details about the hyperparameters used in our approach. These hyperparameters were carefully selected through experimentation to achieve the best performance.
Table A.6: Hyperparameters and their default values
Hyperparameter
Description
Single View
Multi View
Learning rate X,
Adam learning rate
0.001
0.0001
Trade off reconstruction accuracy accuracy/differentie diiferentttttibbltyy
0.8
0.8
Xxb
Weight of forward backward loss Weight of Spatial extent loss
0.05
0.05
Ase
The table in Table A.6 lists the hyperparameters along with their corresponding values. Note that for most hyperparameters the value is the same in the single and multiview use cases, it is also kept the same across datasets.
E
Qualitative Results
We present qualitative results for various scenes from MOT17 in Fig. A.7 When compared with the two aforementioned baselines, motion-to-nearest detection and bipartite matching, our proposed method more accurately approximates the ground truth motion, despite being trained solely with detection supervision.
F
MultiviewX Results
F.1
Motion Offsets
We also evaluate the predicted motion offsets on the MultiviewX dataset. Results are presented in Table A.7
Table A.7: Motion offset evaluation We evaluate 2D motion offset according to three metrics, L1 error, Angular error (in degree) and Norm error. Our model outperforms the optical flow baseline by a large mar- gin. It even performs competitively against the fully supervised methods, especially in terms of norm error.
Tracking performance v.ewX. Our model outperforms existing baseline on the metrics related to trajectories by a significant mar- gin. Performance numbers for MVFlow (16) was gen- erated from publicly available code.
Table A.8:
Multi-
MultiviewX dataset.
MultiviowX datasct
Model
MOTA
MOTP
IDF1
IDP
IDR
model
L1
Angle
Norm
MVFlow 16
66.2
0.87
50.2
56.6
45.1
RAFT
4
2.56
50.3
2.84
Ours
81.1
0.55
55.1
54.4
55.8
Supervised
1.26
12.6
1.58
Ours
1.44
19.4
1.59
22

---

Published in Transactions on Machine Learning Research (11/2024)
GroundTruth
Ours
Hungarian
Nearest
Figure A.7: Qualitative Results MOT17: We compare our motion prediction model against two baseline methods: the Nearest Neighbor and the Hungarian algorithm. In both cases, the motions predicted by model more closely align with the ground truth. Best viewed when zoomed in.
F.2 Tracking
Previous methods have utilized the MultiviewX dataset solely for evaluating multiview detection tasks. We extend its include tracking evaluation. Employing the same evaluation protocol as in the WILD- TRACK dataset, we report findings in Table A.8 Utilizing publicly available code, establish baseline for the MVFlow method (16). Our model demonstrates significant improvement MVFlow the MultiviewX datasct.
G
Video Results
The supplementary archive illustrate our model's capability in motion prediction, relying solely on detection data for supervision. Included is the video wild_ hilddoffsstt.m offsettmm4, highlighting our model's offectiveness on the WILDTRACK dataset. The video features detection and offset maps projected onto the ground plane from the perspective of camera. 1.
G.1 Low frame rate Results
The video wild extended low low_fps.mpl illustrates our model's capabilities in a low frame rate setting, We augmented the original WILDTRACK dataset's frame rate from fps to 10 fps by adding frames through
23

---

Published in Transactions on Machine Learning Research (11/2024)
interpolation and adjusting the ground truth annotations accordingly. A specialized model was trained this enriched dataset. For both the training and evaluation phases, we used frame pairs with 15- frame interval, yielding an effective frame rate of 3 fps. Our model demonstrates robust motion prediction capabilitics even at this reduced frame rate.
H Code
We provide the Torch implementation of the differentiable reconstruction slep in our approach. The code to reproduce some of the experiments is available at httpp//ww....o//o/bba//////////d
24


Reviews:
[{"review_id": "ZjXLFSXJ0k", "reviewer": "Reviewer_81nZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The method is simple and easy to understand. Although the memory usage is quadratic with number of heatmap pixels forcing the method to be used in a sliding window manner.\n- The Table 3 \"motion offset evaluation\" experiment is particularly nice in the light of TMLRs focus on solid experimentation backing the claims made in the paper. In this experiment they try to use motion offsets directly predicted by RAFT (optical flow model) instead of teaching the model to predict those. They also use ground truth motion offsets to have a supervised baseline. Their method is much better than the RAFT based ablation and comes close to the supervised baseline.\n\n### Weaknesses\n- The manuscript is scoped to cover 'object' motion but all the experiments are scoped to cover 'people' motion which is a subset. Even though the method is not specialized for people, because of the limited experiments, the title and the text need to be adjusted to reflect this more limited scope.\n- The experiments demonstrate that the proposed motion offset based method performs competitively against methods that use more supervision eg. identity supervision. But this is not the same as showing that the need for identity supervision is circumvented, which is the idea I got after reading the introduction. In order to demonstrate that the need for identity supervision is circumvented, one needs to run the ablation \"Ours + Identity supervision\" and show that this is roughly on par with \"Ours\". Alternatively it must be made clearer that this method does not intend to replace identity supervision, which may as well be orthogonal to the contributions of this paper.\n- Are the FPS vs metric value plots in figure 1 and figure 5 aligned? It will help the reader if these two plots are made equivalent in terms of metrics, loss, datasets, models etc. That way the reader can understand, as to what extent the problem initially motivating the manuscript has been solved.\n- There are no error bars reported for any of the results. In case this is computationally too expensive, please at least provide error bars for the numbers for \"Ours\" in Table 1.\n- Please clarity where the ground truth offsets for section 4.2.2 are obtained from. Are these part of the dataset?\n- The proposed method includes several loss terms and many hyper-parameters. A table summarizing all the hyper-parameters will greatly assist readers trying to benefit from this manuscript. I cannot confirm whether all the hyper-parameters have been documented in the manuscript without such a table.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript presents a method to train models for multi-object tracking in multi-view data. They build on the tracking by detection paradigm: A detector generates heatmaps for the objects of interest (people in this manuscript) and a post processing helps create tracks out of a sequence of these heatmaps.\n\nKey idea: In addition to training a detector, they also predict motion offsets between time-steps, which are used in the loss function as follows. They use the spatial transformer networks strategy to differentiably warp the detections using the predicted motion offsets. These warped detection heatmaps are evaluated against the ground truth heatmaps from the future time-step.\n\nThe main claims are:\n1. Doing so circumvents the need for identity or other non-detection annotations which are typically very expensive to obtain.\n2. Doing so also makes the model more robust to frame rate reduction (in other words, the model can handle larger displacements between time-steps).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Already documented in the previous section.", "broader_impact_concerns": "People tracking is a very sensitive topic prone to misuse by state and non-state powers. The authors themselves acknowledge the sensitive application of surveillance. A broader impact section is needed for this paper.\n\nAdditionally, in my humble opinion, it is high time the literature in this field started documenting model fairness. For example, underlying detection models are not always equally accurate across race, skin colour, sex, hairstyle, glasses etc. How does that effect the tracking performance? Is tracking performance worse or better for certain groups or intersections thereof? \n\nThis paper studies model robustness with changing frame rate. This is a good step forward. But robustness of these models in other dimensions would be helpful in addressing broader impact concerns. For example effect of weather, indoors vs outdoors, amount of sunlight etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EOm1e1k2Ja", "reviewer": "Reviewer_XLcJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method follows an intuitive and practically convincing path to using the detection models for multi-object tracking.\n2. The authors present the results in both single-view and multi-view settings, which are helpful for broader use of the community.\n\n\n1. The claim \"In this paper, we propose exploiting motion clues while providing supervision only for the detections, which is much easier to do.\" can be potentially misleading. Given two frames with multiple targets presented, the identity of objects is required to associate the ground truths on time step t and t+1. In this sense, not just \"supervision only for the detections\" is required because only bounding boxes without identities are not enough to train the model.\n2.  The claim \"In this manner, we couple the information obtained from different images during training and increase accuracy, especially in crowded scenes and when using low frame-rate sequences.\" may need more evidence to be supported. To be precise, the experiments to demonstrate the effectiveness of the proposed method are conducted on WILDTRACK and MOT17. Neither of them is usually considered as \"crowded scene\". In contrast, in the community of multi-object tracking, some other datasets are used typically to demonstrate the effectiveness of tracking in crowded scenes, such as MOT20[1] and DanceTrack[2].\n3. Though novelty is not a main concern when considering a TMLR submission, this paper's idea is very much similar to CenterTrack [Zhou et al. 2020]. CenterTrack also provides experiments on KITTI and Nuscenes, which are helpful to be added especially considering that they are better fit to study multi-object tracking in `low frame-rate sequences` cases.\n4. The splitting of MOT17 train-val, following CenterTrack [Zhou et al. 2020], has been extensively discussed and considered biased because the training and validation subsets are from the same set of videos. I would encourage the authors to test the results on the MOT17-test set by submitting them to the evaluation server. Moreover, this raises a concern about overfitting: Compared to ByteTrack, which does not use learnable modules for association, the trained modules in the proposed method are more likely to overfit MOT17-train and achieve an unfair advantage when evaluating on MOT17-val. Given the limited baseline methods used for MOT17 experiments, the results on this benchmark are not convincing to me.\n\nReference:\n[1] https://motchallenge.net/data/MOT20/\n[2] Sun, Peize, et al. \"Dancetrack: Multi-object tracking in uniform appearance and diverse motion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to use a detection model to solve multi-target tracking tasks. The key idea is to predict the offset between a pair of image frames to associate the targets presented on both images. The model can reconstruct the object's position at future frames with the detection results and the offset map predicted between frames. The paper's key contribution is to construct such a method upon existing detectors and demonstrate the effectiveness of the proposed paradigm. A set of corresponding implementations, including loss functions, network architecture, and experiments are presented. The results on single-view and multi-view multi-object tracking datasets are provided to help understand the significance of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. To demonstrate the effectiveness of the proposed method, experiments on standard multi-object tracking benchmarks are required. To have strong evidence about the performance for tracking in `crowded scenes`, I would recommend adding the experiments on DanceTrack and MOT20 datasets. Previous methods, such as ByteTrack [Zhang et al. 2022] and OC-SORT[3] have provided publicly available YOLOX detector models and weights on these datasets so it would be easy to achieve fair comparison by reusing the detector.\n2. The splitting of MOT17 train-val sets introduces potentially biased results. I suggest adding experiments on MOT17-test set.\n3. The difference of the proposed method and the long-standing CenterTrack should be explicitly discussed. Given the very similar idea and practice, the experiments on related datasets can be added, especially considering that they bring extra benefit of demonstrating the performance under low frame-rate cases.\n\n\nReference:\n[3] Cao, Jinkun, et al. \"Observation-centric sort: Rethinking sort for robust multi-object tracking.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "broader_impact_concerns": "Multi-object tracking datasets involve human face, which is a long-standing legacy issue in this community. Such potential privacy issues should be paid attention to and discused.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BD57Nk4N9e", "reviewer": "Reviewer_Nt6h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Supervision approach is very clever - warping between object detection heat maps at two time steps is an innovative way to take advantage of video data and powerful object detection algorithms to create a supervisory signal for the goal, motion prediction\n- Loss functions and architecture are described clearly. Overall, the method is pretty well-communicated\n- Experimental setup is well-designed and supports the contributions of the paper\n- Impact is convincing\n  - Improved results on crowded scenes and low frame rates are demonstrated\n  - Ability to take advantage of existing data that doesn't have complex ground-truth annotations without paying extra costs to collect annotations\n\n### Weaknesses\n- System requires high-quality initial detections. While we do have good tools for object detection, it would be interesting to see how this fares under various conditions where object detection is harder (not just crowds, for example), or where less performant detectors are being used\n- The approach is somewhat complex beyond just loss, particularly with elements like the dual-pathway architecture. While the ablations on different parts of the loss are worthwhile, they could be extended to other aspects beyond loss. \n- Memory intensiveness could use more concrete documentation and quantification. The efficacy in weak data regimes is a main selling point, so while memory intensiveness is not the same as needing costly annotations, potential users might benefit from the budget prediction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for predicting object motion using only detected objects as a supervisory signal. This eliminates the need for motion and identity annotations that are costly and hard to find for existing data. The method involves generating detection heat maps at two different time steps along with a 2D motion estimate between the two time steps. The motion estimate is then used to warp one heat map to match the other, and the resulting consistency between the two heat maps acts as the supervisory signal. \n\nThe paper introduces the method and conducts a series of experiments on the MOT17 and WILDTRACK datasets. These allow for experiments on both single- and multi-view inputs, and the dual-pathway network can accommodate both. The results show that this method outperforms state-of-the-art methods particularly in settings with low frame rates, constituting another data limitation that this method can help mitigate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor\n- Some more ablations as described above, and some more results to quantify the usability of this system for those who need it\n- Experiments with poorer object detectors (a small set)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend that the paper should be accepted and are happy with the revisions. There are, to the best of my knowledge, no pending concerns that would warrant a revision since the authors have addressed all of the reviewer concerns in their response to the reviews. I therefore recommend accepting the paper as-is (i.e., with the authors simply updating the paper so that the changes are no longer in red).
Decision: Accept (poster)