More Than Meets the Eye: Enhancing Multi-Object Tracking Even with Prolonged Occlusions
Authors: Bishoy Galoaa
Somaieh Amraee
Sarah Ostadabbas
Venue: ICML 2025 poster (2025)
Keywords: Multi-object tracking
occlusion handling
deformable transformers
softmax splatting
optical flow estimation
enhanced track embedding
computer vision
motion estimation
deep learning.
Primary area: applications->computer_vision
TL;DR: A novel multi-object tracking framework, MOTE, that uses deformable transformers, optical flow, and softmax splatting to track objects effectively even under prolonged occlusions.

Abstract: This paper introduces MOTE (MOre Than meets the Eye), a novel multi-object tracking (MOT) algorithm designed to address the challenges of tracking occluded objects. By integrating deformable detection transformers with a custom disocclusion matrix, MOTE significantly enhances the ability to track objects even when they are temporarily hidden from view. The algorithm leverages optical flow to generate features that are processed through a softmax splatting layer, which aids in the creation of a disocclusion matrix. This matrix plays a crucial role in maintaining track consistency by estimating the motion of occluded objects. MOTE's architecture includes modifications to the enhanced track embedding module (ETEM), which allows it to incorporate these advanced features into the track query layer embeddings. This integration ensures that the model not only tracks visible objects but also accurately predicts the trajectories of occluded ones, much like the human visual system. The proposed method is evaluated on multiple datasets, including MOT17, MOT20, and DanceTrack, where it achieves impressive tracking metrics--82.0 MOTA and 66.3 HOTA on the MOT17 dataset, 81.7 MOTA and 65.8 HOTA on the MOT20 dataset, and 93.2 MOTA and 74.2 HOTA on the DanceTrack dataset. Notably, MOTE excels in reducing identity switches and maintaining consistent tracking in complex real-world scenarios with frequent occlusions, outperforming existing state-of-the-art methods across all tested benchmarks.

More Than Meets the Eye: Enhancing Multi-Object Tracking Even with Prolonged Occlusions
Bishoy Galoaa
Somaieh Amraee' Sarah Ostadabbas
Abstract
1. Introduction
This paper introduces MOTE (MOre Than meets the Eye), a novel multi-object tracking (MOT) algorithm designed to address the challenges of tracking occluded objects. By integrating deformable detection transformers with cus- tom disocclusion matrix, MOTE significantly en- hances the ability to track objects even when they are temporarily hidden from view. The algorithm leverages optical flow generate features that are processed through softmax splatting layer, which aids in the creation of disocclusion trix. This matrix plays crucial role in maintain- ing track consistency by estimating the motion of occluded objects. MOTE's architecture includes modifications to the enhanced track embedding module (ETEM), which allows to incorporate these advanced features into the track query layer embeddings. This integration ensures that the model not only tracks visible objects but also ac- curately predicts the trajectories of occluded ones, much like the human visual system. The proposed method evaluated on multiple datasets, includ- ing MOT17, MOT20, and DanceTrack, where it achieves impressive tracking metrics-82.0 MOTA and 66.3 HOTA on the MOT17 dataset, 81.7 MOTA and 65.8 HOTA on the MOT20 dataset, and 93.2 MOTA and 74.2 HOTA on the Dance- Track dataset. Notably, MOTE excels in reducing identity switches and maintaining consistent track- ing in complex real-world scenarios with frequent occlusions, outperforming existing -ffte-of-the- methods across all tested benchmarks. Code available at hattuub.ooo/sttuaabb/////// More- Than-Meets-ther Tracking..
Multi-object tracking (MOT) presents a significant chal- lenge in computer vision, with wide-ranging applications such surveillance (Ahmed et al., 2021; Amraee al., 2024; Vennila & Balamurugan, 2023), autonomous driving (Gragnaniello et al., 2023), and robotics (Zaeni et al., 2018). The core problem in MOT involves the consistent identifi- cation and tracking of multiple objects across successive frames. This task becomes particularly difficult in scenarios where objects are occluded, either partially fully, mak- ing it challenging to maintain accurate tracking (Ciaparrone et al., 2020; Zhang et al., 2021).
The human visual system remarkably proficient at han- dling occlusion, effectively estimating the motion of hidden objects by using contextual information from visible sur- roundings (Saleh et al., 2020). This innate ability to infer the presence and trajectory of objects, even when they out of view, inspired approach to MOT. Unlike traditional MOT methods that heavily rely on the visual appearance of objects and often struggle with occlusion, we seek to replicate this perceptual skill through advanced computer vision techniques.
In this work, introduce MOTE (MOre Than meets the Eye), which fundamentally advances occlusion handling in multi-object tracking through three key innovations: (I) novel integration of optical flow with transformer based tracking that enables robust motion prediction during oc- clusions, (2) an adaptive softmax splatting mechanism that intelligently weights feature propagation based on occlu- sion patterns, and (3) an enhanced track embedding module (ETEM) that maintains object identity through occlusions by fusing motion and appearance cues. Unlike existing meth- ods like other end to end methods that primarily rely on appearance matching, MOTE explicitly models occlusion dynamics through these complementary components.
'uugmented Cognition Lab (ACLab), Department of Electri- cal and Computer Engineering. Northeastern University, Buston, MA, USA. Correspondence to: Sarah Ostadabbas <ostadab- bas ensecc..nc..du>.
MOTE is designed operate end-to-end as shown in Fig- ure 1. begins by employing deformable transformers to process input frames and generate pyramid feature maps, which capture detailed spatial information at various scales, providing comprehensive understanding of the scene. This spatial analysis is followed by optical flow estimation, which
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
Optical Flow
Estimator
Enc
Enc
Enc
Softmax Splatting
qp
qd
Qd
Dec
Dec
Dec
STEM
ETEM
Figure 1. Overview of the MOTE framework: Our method processes video frames 42......... Ix) produce robust object tracks An optical flow estimator computes flow fields between consecutive frames, while a deformable DETR encoder (Enc)
(Y,, Y,..., ...,, YN). extracts features from each frame. These elements feed into the softmax splatting module. The output from softmax splatting is then concatenated (denoted by the C symbol) with the output from the Enhanced Track Embedding Module (ETEM), which integrates splatted features into track query embeddings. This concatenated result is processed by decoder (Dec) with query-driven attention (ga) generate the final object tracks. This sophisticated pipeline enables consistent tracking even in scenarios with frequent occlusions and complex scene dynamics.
tracks the movement of objects between consecutive frames- crucial capability in scenarios where objects might be partially completely occluded.
with optical flow estimation enhance tracking capabilities in occlusion scenarios (see Figure/ 1).
We develop a unique softmax splatting layer to gener- ate disocelusion features, which are integrated into the enhanced track embedding module (ETEM) effec- tively handle occluded objects and improve tracking accuracy (see Figure 2).
A key innovation in our approach is the incorporation of softmax splatting layer, which generates disocclusion fea- tures by merging the pyramid feature maps with the op- tical flow. This process forms disocclusion matrix that highlights areas of occlusion and estimates the motion of obseured objects, significantly enhancing the system's 's track- ing capabilities (Niklaus & Liu, 2020). These disocclusion features are then integrated into an enhanced track embed- ding module (ETEM), which aids sequence and occlusion estimation.
We conduct extensive ablation studies and evalua- tions on multiple public datasets (MOT17 (Milan al., 2016), MOT20 (Dendorfer et al., 2020), Dance- Track(Sun et al., 2022)), demonstrating the effective- ness of our approach in various challenging scenarios, including low frame rates and large camera motions.
The significance of MOTE lies in its ability maintain ac- curate tracking even under prolonged occlusions, a scenario that challenges most existing MOT systems. By leveraging contextual information and motion estimation, MOTE can infer object trajectories in a manner akin to human visual perception, marking a substantial leap forward robust multi-object tracking for real-world applications. Our con- tributions in this paper are as follows:
We achieve state-of of-the--rt performance standard MOT metries while significantly improving tracking accuracy during occlusion events, with 2.4% reduc- tion in identity switches compared to existing methods.
2. Related Works
In this section, we summarize key developments in multi- object tracking (MOT) that specifically address occlusion challenges, focusing on approaches that inform our MOTE framework. We examine how different methods handle
We translate the human ability to perceive and track occluded objects into a novel end-to-end multi-object tracking framework, integrating deformable transform-
2

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
Figure 2. Illustration of normal splatting (Left): The orange pixels remain static while the green pixels move down in a shearing manner. Splatting allows scaling the transform. Application splatting in an occlusion scenario (Right): Using softmax splatting, the occlusion handling can be effectively translated by ensuring that the motion and visibility of objects are accurately managed, improving multi-object tracking under occlusion conditions.
tracking through occlusions and analyze their limitations, which motivate our work.
tended periods. Our MOTE framework builds upon these methods while introducing specific components for robust occlusion management, addressing limitations in handling prolonged occlusions.
2.1. CNN-based Methods: Tracking by Detection
CNN-based MOT methods have evolved better handle occlusion scenarios. While SORT (Bewley et al., 2016) established foundational racking-by-detecting principles, more recent approaches like ByteTrack (Zhang et al., 2022b) specifically target missed detections during occlusions. BoostTrack (Zhang et al., 2023a) advances this paradigm by introducing an innovative detection detcction-trackle tracklet confidence score and enhanced similarity measures including Maha- lanobis distance and shape similarity, achieving state-of- the-art performance while maintaining real-time execution. Stro (Du et al., 2023) further refines the process through enhanced trajectory association during partial occlu- sions. However, these methods still struggle with prolonged occlusions due to their heavy reliance on frame-by-frand -f-frame detections and limited temporal modeling capabilities. Our work addresses these limitations through advanced occlu- sion handling and enhanced temporal feature integration.
2.3. Optical Flow-based Trackers
Optical flow estimation has proven crucial for understand- ing object motion during occlusions. TransMOT (Zhu et al., 2021) demonstrates how flow information can enhance ob- ject association in transformer- systems, while Du- alFlow (Zhang et al., 2022a) specifically targets occlusions in crowded scenes. These approaches inform our integration of optical flow for maintaining track consistency through occlusion events.
2.4. Softmax Splatting and Occlusion Handling in MOT
As illustrated in Figure 2, softmax splatting (Niklaus & Liu, 2020) offers powerful approach for handling occlusions. Originally developed for video interpolation, this technique effectively reconstructs occluded regions through feature weighting across multiple pyramid levels (Reda et al., 2021). While recent point tracking methods like PointOdyssey (Zheng et al., 2023) and TAPIR (Doersch et al., 2023) have shown success in handling occlusions at fine- aineigraieed level, our work adapts these insights object-level tracking. By combining transformer architectures with optical flow esti- mation and softmax splatting (Figure 3), MOTE maintains consistent tracking through extended occlusion periods.
2.2. Transformer-basel Methods: End-to-End Frameworks
Recent transformer-based approaches have shown promise in handling occlusions through unified detection and track- ing frameworks. MOTR (Zeng et al., 2022) introduces track queries for modeling object persistence across video se- quences, enabling end-to-end learning. MOTRv2 (Zhang et al., 2023b) enhances this with bootstrapped pretraining and improved temporal attention. DragonTrack (Galoaa al., 2025) advances this paradigm by integrating graph convolutional networks with transformer features for hanced re-identification in complex scenarios. Despite these advances, performance in high-occlusion scenarios remains challenging, particularly when objects disappear for ex-
3. Introducing MOTE
The proposed MOTE framework, illustrated in Figure 1, processes sequence of video frames (I,, Iz..... ......,,. IN) to generate robust object tracks (Ý1, Yg...., Yx). tem integrates advanced computer vision techniques with novel architectural components to handle occlusions effec-
This sys-
3

---

More Than Meets the Eye: Enhancing Multi-Objeet Tracking
1
Optical Flow
I
Estimator
Fi
Feature Pyramid Extractor
Feature Pyramid Extractor
Softmax Splatting
P2
P
PL
P
S
Figure 3. Diagram illustrating the softmax splatting method used the MOTE framework. Frames 1, and I--1 are passed through an optical flow estimator to compute the flow field F,. The feature pyramid extractor within the deformable DETR extracts multi-scale features P., P2.... Pr. from both frames. The softmax splatting layer then combines the optical flow and the pyramid features to generate the disocclusion matrix S. In the disocclusion matrix, hotter (darker) areas indicate that the subject disoccluded and more likely occlude other objects (less depth), while lighter areas indicate that the object will be occluded (more depth).
tively and maintain consistent object identities throughout complex sequences.
during occlusions. As visualized in the bottom row of Figure 3, the resulting disocclusion matrix S, effectively captures occlusion states: darker regions indicate subjects that disoccluded and more likely to occlude others (less depth), while lighter regions signify subjects that are more likely to occluded (more depth). This visualization empirically validates our approach's ability to identify and handle occlu- sion relationships. The normalized weights for each feature level are calculated as:
3.1. Optical Flow Estimation
The core of our tracking framework is the motion estimation between consecutive frames. We utilize optical flow over conventional Kalman filtering as it provides richer motion cues through pixel-level movement estimation, enabling better handling of occlusions and non-linear trajectories. Specifically, we employ the state-of-the-at RAFT (recurrent all-pairs field transforms) model (Teed & Deng, 2020). For any pair of input frames 4; and 4;-1 the optical flow F
V softmax (0).
(2)
where E IR++1 is a learnable parameter vector that deter- mines importance weights for both feature pyramid levels (L= 4 in our implementation) and the optical flow features (accounting for the +1 dimension). The softmax operation ensures that features from different scales contribute propor- tionally to their relevance for tracking. All feature maps are resized to maintain spatial consistency:
computed as follows:
F RAFT(J,, 4-1).
(1)
This step captures inter-frame motion, providing essential information for tracking objects through occlusions and dynamic scene changes.
presized interp size (Hmin: W Wmin)):
(3)
3.2. Feature Extraction and Softmax Splatting
where I represents the number of feature pyramid levels (typically set to 4 in our implementation), and L L 1 in eq.5 refers to the additional channel dimension introduced by the optical flow features, enabling dynamic balancing between appearance and motion cues. The optical flow information is processed to match the feature dimensions:
Building on the motion information, our approach employs deformable DETR encoder extract multi-scale features from input frames. Softmax splatting, originally designed for video frame interpolation, serves crucial role in our framework by adaptively combining feature information across different scales with motion cues. The softmax splat- ting layer, depicted in Figure 3, intelligently merges feature maps {PI, .?.....,P, at time step t from the deformable DETR with optical flow F maintain object consistency
pexpandel
presized Ø Ic
(4)
The final splatted feature S combines these elements

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
FTEM
producing updated track embeddings or that maintain consistent object identities through occlusions.
Splatted Fearures y
Inputs
Input Features
Query Features
Query Positions
The ETEM, illustrated in Figure 4, consists of two main components; track handling module and the core ETEM processing block. As shown in Figure 4(a), the track han- dling module manages object entries and exits while process- ing three types of inputs: query features, query positions, and splatted features. In Figure 4(b), these inputs undergo parallel linear projections. The input features (query fea- tures and positions) are processed through one branch, while splatted features S are processed through another to produce Spoo
Splatted Features
Lincar Projection
Linear Projection
Spos
Object
Objoct
g.. Spoos [4; Spooll
Entrance
k
fg: Spras)
ETEM
MHA
Add & Norm
Spnj WsS + bs:
(6)
FFN
where Ws and by are learnable parameters.
1+1
Add & Norm
These projected features are then concatenated as shown by the 'C' symbol in the figure:
(a)
(b)
Figure 4. Illustration of the ETEM module: (a) the track handling module manages object entries and cxits while processing query features, query positions, and splatted features produce updated track embeddings of 1 (b) the enhanced track embedding mod- ule (ETEM) processes inputs by applying linear projections to both input and splatted features, followed by concatenation and multi-head attention (MHA). the resulting emheddings refined through add & norm layers and a feed forward network (FFN), ultimately updating track queries.
[g: Spoo]] [9pos Preat's Sprojl
where [:] denotes concatenation. This concatenated repre- sentation, along with the current track query diy feeds into the attention mechanism. Following the diagram, for each attention head i:
Qi = o W.O.
K; (9: Sproj] W
(8)
through a weighted sum:
VÚ = (q: Spool W.
where W.O. W.K, and W.V. E Rdxds are learnable matrices and da = 256. Each attention head computes:
S = wi Presized
WL+1 eexpanded
(5)
1=1
(Q:KT Von
This mechanism allows our model to dynamically adjust the contribution of each feature level and motion information based on the tracking context, particularly beneficial when dealing with occluded objects where certain feature levels may become more informative than others.
head; softmax
V..
(9)
The outputs from all eight attention heads are combined and processed through add & norm layers as shown in Figure 4(b):
3.3. Enhanced Track Embedding Module (ETEM)
Q' = -oonca(headd,..... W
(10)
The Enhanced Track Embedding Module (ETEM) addresses a critical challenge in occlusion aware tracking: seamlessly integrating temporal motion information with spatial fea- tures while maintaining object identity. Figure 4 illustrates the complete data flow through ETEM. The module pro- cesses three inputs: (1) query features qeeat from the trans- former encoder, (2) positional encodings qpos, and (3) splat- ted features S from our softmax splatting layer. These inputs flow through parallel linear projections before concatena- tion, enabling the model to learn optimal feature combi- nations for occlusion handling. The concatenated features then undergo multi-head attention with track queries of,. fol- lowed by feed-forward networks with residual connections,
followed by a feed-forward network with residual connec- tions:
der i+1
FFN(Q') max(0, Q'W, + b,) + b2- (11)
This architecture, with its carefully designed attention mech- anism and residual connections, enables effective processing of both appearance and motion information, maintaining robust tracking even through occlusion events. The sequen- tial processing through MHA, add & norm layers, and FFN, as depicted in Figure 4(b), ensures that the final track em- beddings e capture rich spatio- temporl relationships for reliable tracking.
5

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
3.4. Decoder and Object Tracking
MOTE's robustness in handling occlusions. Finally, we con- duct an ablation study to investigate the contributions of different components within MOTE, providing insights into the factors that drive its enhanced tracking capabilities.
The final stage of our pipeline in Figure I involves de- coder (Dec) equipped with query-driven attention (ga). This decoder processes the enhanced features produced by the ETEM to generate the final object tracks. By leveraging the rich representation created by the previous modules, we maintain consistent object identities even in the most challenging scenarios.
4.1. Implementation Details
Model Configuration: The ETEM module uses feature dimension of 256 throughout all layers, with 8 attention heads in the multi-head attention mechanism. The softmax splatting module processes feature pyramids with scales [1/32. 1/16, 1/8, 1/4] of the input resolution. For optical flow estimation, we use RAFT with 20 iterative refinement steps, as ablation studies show this provides optimal balance between accuracy and computational efficiency.
3.5. Looss Function
Our loss function adaptively weights different components based on occlusion states. For each tracked object, we compute an occlusion mask:
My- = max loU(b;, kef
(12)
Training Protocol:
The model is trained end-to-end using Adam W optimizer with an initial learning rate of 1e-4 and cosine decay schedule. We employ standard data augmen- tation techniques including random horizontal flipping and random cropping. All loss components are weighted as A 5.0, A2 2.0, and X 2.0 based validation performance.
where My measures the maximum overlap between ob- ject i's bounding box b; and any other object k's box by in frame j. This mask automatically adapts the tracking strat- egy: high values (Mij R 1) for heavily occluded objects emphasize temporal and contextual cues, while low values (Mij R 0) favor appearance-basee tracking.
Setting: The experiments were conducted on a system equipped with an Intel Xeon CPU E5 2.40GHz, 4 A100 GPUs, and GB of RAM. This setup provided the neces- sary computational power to handle the intensive training and evaluation processes. The model was trained over 20 epochs, spanning 4 days, ensuring that it converged ade- quately and learned effectively from the training data.
The total loss combines three components weighted by this occlusion mask:
Lixral = E M @ Labox + A2 + )afak)), (13)
where each component serves a specific purpose. The bound-
ing box loss measures coordinate accuracy:
Lobox
|lbpred
b
(14)
Datasets: We evaluated MOTE on three major datasets: MOT17 (Milan et al., 2016), MOT20 (Dendorfer et al., 2020), DanceTrack (Sun et al., 2022) and SportsMOT (Cui et al., 2023). MOT17 consists of 7 training and testing se- quences, primarily featuring crowded street scenes. MOT20, with 4 training and 4 testing sequences, offers more crowded scenes with higher levels of occlusion. Dance'rrae contains 100 sequences, focusing on scenarios with high inter-object similarity and complex motion patterns.
while the GloU loss captures overall spatial overlap:
Cyiou - GloU(bpeed bg)).
(15)
For classification, we employ focal loss to address class imbalance:
-a log(p:).
(16)
where Pe is the estimated probability for the target class, of weights different classes, and 7 controls the focus hard examples. The effectiivenss of this occlusion-aware weighting scheme is shown in Sec. 4.6 ablation studies.
The evaluation followed standard MOT proto- cols, using metrics such as higher order tracking accu- racy (HOTA) (Luiten et al., 2021), association accuracy (AssA), detection accuracy (DetA), ID Fl score (IDFI) (Ris- tani et al., 2016), multi-object tracking accuracy (MOTA) (Bernardin & Stiefelhagen, 2008), and ID switches (IDS).
Metrics:
4. Experimental Results
In this section, we present the experimental results that demonstrate the effectiveness of our proposed MOTE frame- work. We begin by detailing the implementation setup, including hardware specifications, datasets, and evaluation metrics. Following this, we provide a comprehensive com- parison of MOTE against state-of of-th----he-art methods, high- lighting its superior performance across multiple challeng- ing datasets. We also offer a qualitative analysis to showcase
4.2. State-of-the-AAt Comparison
We compared MOTE against both CNN-based and Transformer-based methods on MOT17 (Milan et al., 2016), MOT20 (Dendorfer et al., 2020), Dance'rrack (Sun et al., 2022). On MOT17 (Milan et al., 2016), as shown in Table 1, MOTE achieved the highest scores across all key metrics, including HOTA, AssA, DetA, IDF1, and MOTA, while
6

---

More Than Meets the Eye: Enhancing Multi-Objeet Tracking
4
12
18
Fi
Fs
Figure 5. Illustration of MOTE's capability to track occluded subjects in challenging scenarios using optical flow. The top row shows consecutive frames (h, I,,..,, IN) from a vidco sequence, where multiple subjects, including ID10, are partially occluded. The bottom now displays the corresponding flow fields (Fi, .2..... Fx) computed by the optical flow estimator. These flow fields capture the motion between consecutive frames, allowing our method to maintain accurate tracking of occluded subjects by leveraging the motion information. The effectiveness of our approach in handling occlusions is evident from the continuous tracking of ID10 across frames, when it hecomes partially occluded by other subjects. Note how the flow field maintains strong motion cues (shown in color intensity) even in regions of partial occlusion, enabling MOTE predict object trajectories through occlusion events. The flow visualization standard color coding: hue indicates direction and saturation indicates magnitude of motion.
also reducing identity switches compared other methods. Specifically, MOTE outperforms the second-best method, StrongSORT (Du al., 2023), in HOTA by 2.8 points (66.3 vs. 63.5) and the second-best method, BrteTraack (Zhang et al., 2022b), in MOTA by 1.7 points (82.0 vs. 80.3). Fur- thermore, MOTE reduces identity switches by 2.4% (1412 vs. 1446) compared to StrongSORT, emphasizing its robust- ness in handling complex tracking scenarios with frequent occlusions. Table demonstrates MOTE's superior perfor- mance on the MOT20 (Dendorfer al., 2020)dataset, where it achieves the highest scores in HOTA, AssA, DetA, IDF1, and MOTA. This dataset provides more challenging envi- ronment with dense crowds and severe occlusions. Thus, Ta- ble 2 confirms MOTE's effectiveness in tracking in crowded and occlusion-heavy environments. The DanceTrack (Sun al., 2022) dataset, known for its high inter-object simi- larity and complex motion patterns, further tests the robust- ness of tracking methods. As shown in Table 6, MOTE achieves the top performance in HOTA, AssA, and MOTA, surpassing other methods. These results indicate MOTE's capability in challenging tracking tasks that involve intri- cate movements and interactions. Furthermore, to demon- strate cross datsset generalization, we evaluated MOTE on MOT15 without fine-tuning, achieving 57.2 HOTA com- pared to MOTR's 28.4 (see Table 4 in Appendix), con- firming our method's 'ss robustness across different annotation protocols and video conditions.
Table I. Comparative performance evaluation on the MOT17 (Mi- lan al., 2016) dataset, highlighting the best methods in CNN- based and Transformer-based categories. Metrics such as HOTA, AssA, DetA, MOTA, and IDFI are considered. The best results for each metric are highlighted in bold, with the second-bea shown in bluc. The symbol indicates unreported values.
HOTA
Asaf
DeAAf
MCTA
IDFI?
IDS,
CNN based:
Trackm--- al., 2019)
4488
45.1
44.9
53.5
52.1
2072
ComeeTrack/Zoose al., 2020)
52.2
$1.0
53.8
67.8
64.7
3039
TraDes (Pung et al. 2023)
52.7
50.8
55.2
69.1
63.9
3555
QDTTcck (Pang GSDT (Wang al., 2021c)
3021)
53.9
52.7
55.6
68.7
66.3
3378
55.5
54.8
56.4
66.2
68.7
3318
FaiMMCTIZhing
2021)
50.3
72.3
3303
CorTTacker (Wang
73.6
3360
2021b)
74.9
75.0
1812
MrAATacck (Studerr Boyever, 2022)
79.4
75.9
1452
StrongSOR (Du 2023)
63.6
78.3
78.5
1446
BynTTack (Zhang et 20226)
63.1
63.0
64.5
80.3
77.3
2196
BoostTrack (Zhing et al., 2023a)
65.4
64.2
64.8
80.5
80.2
1104
Tranfomme-tan bowed
TrackFonner (Maishasd et 2021)
F
65.0
63.9
3538
TranTTackSSnn all. 2000)
54.1
47.9
61.6
74.5
63.9
3663
MOTRiZesa 2022) MCTTRv2/Zhag 20236)
57.8
55.7
60.3
73.4
68.6
2439
78.6
75.0
(Ours)
80.3
on the SportsMOT dataset without retraining, We evaluated our method on three sequences and compared against Byte- Track and MOTR under identical conditions. As shown in Table 3, MOTE achieves the highest MOTA score at 45.7%, significantly outperforming ByteTrack (17.9%) and slightly surpassing MOTR (44.1%). Similarly, MOTE achieves an IDF1 score of 50.2%, compared to 31.4% for Byte'rrakk and 48.75 18.7% for MOTR. These results highlight MOTE's superior tracking accuracy and adaptability, even when applied to a new domain without additional training. While fine-tuning on SportsMOT could further enhance performance, these zero-shot results already demonstrate the robustness of our
4.3. Preliminary Results on SportsMOT
To demonstrate MOTE's generalization capabilities beyond pedestrian tracking, we conducted preliminary experiments

---

ID10, experience significant occlusions. The bottom row More Than Meets the Eye: Enhancing Multi-Object Tracking
approach in diverse tracking scenarios involving athletic movements and rapid motion changes.
Table 5. Computational breakdown on 1536xx800 resolution (A100 GPU).
Component
Time (ms)
% of Total
Table 2. Comprehensive performance comparison on the MOT20 (Dendorfer al., 2020) dataset. The highest scores are highlighted in bold and the second-hest scores are in blue. The symbol
Deformable DETR Encoder
45
28.1%
RAFT (20 iterations)
18
11.3%
Softmax Splatting
7
4.4%
indicates unreported values.
ETEM
12
7.5%
Methods
HOTAT
AssAT
DetAT
MOTAT
IDFI
Decoder
48
30.0%
FairMOT (Zhang et al., 2021)
54.6
54.7
54.7
61.8
67.3
Other Components
30
18.7%
ByteTrack (Zhang
2022h)
61.3
59.6
62.9
76.2
75.2
OC-SORT (Cao
2022)
62.4
75.9
76.4
Total
160
100%
MOTR=2 (Zhang et al., 2023b)
60.3
58.1
62.9
76.2
72.2
StongSORT (Du al., 2023)
61.5
62.5
59.9
72.2
75.9
BoostTrack (Zhung et al., 2023a)
63.0
62.8
63.4
76.4
76.5
MOTE (Ours)
65.8
66.9
64.9
81.7
79.8
Table Zero-shot evaluation on SportsMOT (Cui al., 2023)
displays the corresponding optical flow fields computed by our method. These flow fields capture the motion between consecutive frames, allowing MOTE to maintain accurate tracking of occluded subjects by leveraging motion informa- tion, even when visual evidence is limited. The continuous tracking of ID10 across frames, even during prolonged oc- clusions, demonstrates MOTE's robustness in handling one of the most challenging aspects of tracking in crowded en- vironments. For detailed analysis of extended occlusion scenarios, refer readers to Section ?? in the Appendix.
dataset (3 sequences). All methods were evaluated without retrain-
ing.
Methods
MOTA
IDF1
FPST
ByteTrack (Zhang et al., 2022b)
17.9
31.4
30.2
MOTR (Zeng et al., 2022)
44.1
48.7
7.5
MOTE (Ours)
45.7
50.2
22.2
Table Extended results comparing MOTR and MOTE on the MOT15 dataset, with models trained on the MOT17 dataset.
Methods
HOTA
MOTA
IDF11
Table 6. Performance evaluation on the DanceTrack (Sun et al., 2022) dataset comparing CNN-based and Transformer-han meth- ods. The best pest-performin scores shown in bold, while the sccond-best are highlighted in blue. MOTRv2" denotes MOTRv2 with extra association, adding validation set for training, and
MOTR
28.4
32.5
36.3
MOTE (Ours)
57.2
63.2
68.8
4.4. Computational Analysis
test ensemble.
Methods
HOTA
AssAt
DeAA
MOTA
IDFIT
MOTE's 's computational overhead is carefully optimized to balance performance gains with practical deployment con- siderations. Table presents detailed breakdown of pro- cessing times for each component. Our framework adds only 25ms per frame compared to MOTR's baseline of 133ms (7.5 FPS) on high-resolution inputs (1536x800), rep- resenting 19% increase in computation time. This modest overhead stems from RAFT optical flow estimation (18ms with 20 iterations) and the softmax splatting layer (7ms).
INNe-bsed:
FairMOT (Zhang et al., 2021)
39.7
23.8
66.7
82.2
40,8
Cente/Track (Zhou at. 2020)
41.8
78.1
86.8
35.7
TraDeS (Pang et al., 2021)
43.3
25.4
74.5
86.2
41.2
QDTrack (Pang al.,
54.2
50.4
BracTrack (Zhang ct al., 2022b)
47.7
31.0
71.0
91.5
48.8
OC- SORT (Cao
2022)
55.1
38.3
80.3
92.0
54.6
Transformee- based:
TransTrack (Sun al., 2020)
45.5
45.2
GTR (Wang et al., 2021b)
48.0
31.9
72.5
89.7
50.3
MOTRv2 (Zhang et 2023b)
69.9
71.7
MOTR+24 (Zhing et al., 20236)
73.4
64.4
83.7
92.1
76.0
MOTE (Ours)
74,2
65.2
82.6
93.2
75.2
For real-time applications, we explored lightweight alter- natives: using fewer RAFT iterations (10 instead 20) reduces overhead to 12ms with only 1.8% HOTA reduc- tion. Additionally, mixed-prede -rexcd-preiison training and inference provide 35% speedup with negligible accuracy loss.
4.6. Ablation Study
We conducted an ablation study to assess the impact of dif- ferent components and parameters within MOTE. The study focused on three main aspects: the choice of splatting tech- nique, the number of iterations in optical flow estimation, and the effect of occlusion weights. To enable rapid ex- perimentation and fair component comparison, all ablation models were trained for 5 epochs, providing clear insights into the relative importance of each component while main- taining reasonable training times.
4.5. Occlusion Handling and Qualitative Analysis
MOTE's ability to maintain accurate tracking under pro- longed occlusion conditions is one of its key strengths. Fig- ure 5 illustrates MOTE's effectiveness in tracking subjects through both partial and complete occlusions that persist over multiple frames. The top row shows consecutive frames from video sequence, where multiple subjects, including
As presented in Table 7, softmax splatting outperforms
8

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
linear splatting across all metries, including a 3.2-point increase in HOTA, 3.6 3.6-point increase in MOTA, and 3.5-point increase in IDFI, along with reduction of 316 identity switches. These significant improvements, achieved even with limited training, highlight the fundamental advan- tage of the softmax splatting approach.
Table 9. Ablation study MOT17 (Milan et al., 2016) comparing loss functions with and without occlusion weights. Config.
HOTA
MOTA
IDFI
IDS
Without Occ. weights
63.5
80.3
78.5
1446
With Occ. weights
66.3
82.0
80.3
1412
Improvement
+2.8
+1.7
+1.8
-34
We compared linear splatting and softmax splatting on the MOT17 dataset. As presented in Table 7, softmax splatting outperforms linear splatting across all metrics, including 3.2-point increase in HOTA, a 3.6 3.6-point increase in MOTA, and 3.5 3.5-point increase in IDF1, along with a reduction of 316 identity switches. These findings highlight the effective- ness of Softmax Splatting in enhancing tracking accuracy.
tently maintains accurate and reliable object tracking, even under challenging occlusion conditions. MOTE demon- strates superior performance across MOT17, MOT20, and DanceTrack datasets, setting new state-of-the-art bench- marks. The results clearly demonstrate MOTE's superior performance, surpassing current techniques in the field. While highly effective, our method faces challenges with small or distant objects, which are inherent difficulties of op- tical flow-based methods. Future work will focus on address- ing these limitations through alternative motion estimation methods and the integration of additional sensor modalities such as depth cameras or LiDAR. We also see significant potential in leveraging graph neural networks enhance the detection and tracking of challenging objects, making tracking systems even more reliable and effective in diverse and demanding environments. Looking forward, we plan to extend MOTE's capabilities in several directions. First, we will explore transformer based optical flow methods that could be trained and-to-end with our tracking framework. Second, we aim incorporate long-term feature banks for handling extended occlusions beyond our current temporal window. Third, we plan comprehensive evaluation on addi- tional challenging datasets including KITTI for autonomous driving scenarios and BFT for non- non-human object tracking, Finally, we are investigating efficient deployment strategies for edge devices, including model quantization and architec- ture search tailored for real-time applications.
We also examined the effect of varying the 'iters' parameter in the optical flow estimation process. Table indicates that 20 iterations offer the best balance between tracking performance and computational complexity, achieving the highest HOTA and IDFI scores. While 25 iterations slightly improve MOTA, the gains in other metrics diminish, sug- gesting that 20 iterations provide optimal performance.
Effect of Occlusion Weights: Lastly, we evaluated the impact of incorporating occlusion weights into MOTE's loss function. As shown in Table 9, the inclusion of occlusion weights results in a significant performance improvement, with increases of 2.8 points in HOTA, 1.7 points in MOTA, and a reduction of 34 identity switches compared to the configuration without occlusion weights.
Table Ablation study comparing linear splatting and softmax splatting on the MOT17 dataset. The models were trained for 5 epochs.
Method
HOTA
MOTA
IDF1
IDS
Linear Splatting
55.2
61.3
65.7
2450
Softmax Splatting
58.4
64.9
69.2
2134
Table 8. Ablation study for the 'iters' parameter in the forward method of optical flow estimation the MOT17 dataset.
Impact Statement
iters
HOTAt
MOTA
IDF1
IDSL
This work advances multi-object tracking technology with implications for various societal applications. While MOTE can enhance public safety through improved surveillance systems and enable safer autonomous navigation, we ac- knowledge potential privacy concerns. The ability to track individuals through occlusions could be misused for unau- thorized surveillance. We advocate for responsible deploy- ment with appropriate privacy safeguards and transparent policies. Additionally, while our method is computationally efficient compared to its performance gains, the environmen- tal impact of training and deploying such models should be considered.
15
56.1
62.4
66.8
2300
20
58.3
63.7
69.0
2205
25
57.4
64.5
68.1
2150
5. Conclusion
In this paper, we introduced MOTE, a novel approach to multi-object tracking that excels in complex scenarios with frequent occlusions. By combining softmax splatting, de- formable DETR, and optical flow estimation, MOTE consis-

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
References
Galoaa, B., Amraee, S., and Ostadabbas, S. Dragontrack: Transformer-enhame graphical multi-person tracking in complex scenarios. In Proceedings of the Winter Con- ference on Applications of Computer Vision (WACV), PP. 6373-6382, February 2025.
Ahmed, I., Din, S., Jeon, G., Piccialli, F,, and Fortino, G. Towards collaborative robotics in top view surveillance: A framework for multiple object tracking by detection using deep learning. IEEE/CAA Journal of Automatica Sinica, 8(7): 1270, 2021.
Gragnaniello, D., Greco, A., Saggese, A., Vento, M., and Vicinanza, A. Benchmarking 2d multi-object detection and tracking algorithms autonomous vehicle driving scenarios. Sensors, 2308), 2023.
Amrace, S., Galoaa, B.. Goodwin, M., Hatamimajoumerd, E., and Ostadabbas, S. Multiple toddler tracking in in- door videos. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 11- 20, 2024.
Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal- Taixé, L.,, and Leibe, B. Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision, 129:548-578, 2021.
Bergmann, P., Meinhardt, T., and Leil-Taixe, L. Track- ing without bells and whistles. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 141-951, 2019.
Meinhardt, T., Kirillov, A., Leal- Taixe, L., and Feichten- hofer, C. Trackformer: Multi-object tracking with trans- formers. arXiv preprint arXiv:2101.027000 2021.
Bernardin, K. and Stiefelhagen, R. Evaluating multi- ple object tracking performance: the clear mot metries. EURASIP Journal on Image and Video Processing, 2008: 1-10, 2008.
Milan, A., Leal-Taixé, L., Reid, I., Roth, S., and Schindler, K. Mot16: A benchmark for multi-object tracking. arXiv preprint /rXiv:/603.0000 1603000031,, 2016.
Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. Simple online and realtime tracking. In 2016 IEEE in- ternational conference on image processing (ICIP), pp. 3464-3468. IEEE, 2016.
Niklaus, S. and Liu, F. Softmax splatting for video frame interpolation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):982-995,, 2020.
Pang, J., Qiu, L., Li, X., Chen, H., Li, Q. Darrell, T., and Yu, F. Quasi-denss similarity learning for multiple object tracking. In CVPR, 2021.
Cao, J., Weng, X., Khirodkar, R., Pang, J., and Kitani, K. Observation-centric sort: Rethinking sort for robust multi- object tracking. arXiv preprint arXiv:2203. 14360, 2022.
Reda, F. A., Liu, D., Shih, Y., Drulea, D., Tao, X., and Catanzaro, B. Filmnet: A deep convolutional neural network for video frame interpolation via optical flow and softmax splatting. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 15019 15028, 2021.
Ciaparrone, G. et al. Deep learning in video multi-object tracking: A survey. Neurocomputing, 381:61-88, 2020.
Cui, Y.. Zeng, C., Zhao, X., Yang, Y., Wu, G., and Wang, L. Sportsmot: A large multi-object tracking dataset in multiple sports scenes. arXiv preprint S.0i:::300.011700 2023.
Ristani, E., Solera. F., Zou, R., Cucchiara, R., and Tomasi, C. Performance measures and data set for multi-target, multi-camera tracking. In European conference com- puter vision, pp. 17-35. Springer, 2016.
Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taixe L. Mor20: A benchmark for multi object tracking in crowded scenes, arXiv preprint arXiv:2003.09003 2020.
Saleh, M. et al. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2020.
Doersch, C., Gupta, A., Markeeva, L.. Continente, X. S., Tejankar, A., Armeni, I., Malik, J., Guibas, I., and Zis- serman, A. Tapir: Tracking any point with per-frame initialization and temporal refinement. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 00466-20467, 2023.
Stadler, D. and Beyerer, J. Modelling ambiguous assign- ments for multi-person tracking in crowds. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 133-142, 2022.
Sun, P., Jiang, Y., Zhang, R., Xie, E., Cao, J., Hu, X., Kong, T., Yuan, Z., Wang, C., and Luo, P. Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv:20/2. /54600 2020.
Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., and Meng, H. StrongSORT: Make DeepSORT great again. IEEE Transactions on Multimedia, 2023.
10

---

More Than Meets the Eye: Enhancing Multi-Object Tracking
Sun, P., Cao, J., Jiang, Y., Yuan, Z.., Bai, S., Kitani, K., and Luo, P. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20993-21002, 2022.
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, E, Yuan, Z., Luo, Liu, W., and Wang, X. Byte'Track: Multi-object tracking by associating every detection box. In Euro- pean Conference on Computer Vision, pp. 1-21. Springer, 2022b.
Teed, Z. and Deng, J. RAFT: Recurrent all all-pairs field trans- forms for optical flow. In European Conference on Com- puter Vision, pp. 402-419. Springer, 2020.
Zhang, Y., Wang. T., and Zhang, X. MOTRv2: Bootstrap- ping end-to-end multi-object tracking by pretrained object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22056- 22065, 2023b.
Vennila, T. J. and Balamurugan, V. rough set framework for multi-human tracking surveillance video. IEEE Sensors Journal, 2023.
Zheng, Z., Yang, L., Cen, J., Li, J., Liu, X., Zhang, W., Zhang, Y., Liu, Y., Liu, X., Liu, S., et al. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 16036-16047, 2023.
Wang, Q., Zheng, Y., Pan, P., and Xu, Y. Multiple object tracking with correlation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3876-3886, 2021a.
Zhou, X., Koltun, V., and Krähenbühl, P. Tracking objects as points. In ECCV, 2020.
Wang, S., Sheng, H., Zhang, Y., Wu, Y., and Xiong, Z. A general recurrent tracking framework without real data. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 13219-13228, 2021b.
Zhu, L., Sun, Y., Lu, Y., Li, B., and Zhao, Q. Transmot: Spatial-temporal graph transformer for multiple object tracking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 14908- 14917, 2021.
Wang, Y., Kitani, K., and Weng, X. Joint object detection and multi-obbect tracking with graph neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13708-1377 IEEE, 2021c.
Zaeni, I. A. E., Sendari, S., Lestari, D., Mahandi, Y. D., Jiono, M., and Syaifuddin, M. An implementation of multi-object tracking using omnidirectional camera for trash picking robot. In 2018 Electrical Power, Electron- ics, Communications, Controls and Informatics Seminar (EECCIS), pp. 154-158. IEEE, 2018.
Zeng. F., Dong, B., Zhang, Y., Wang, T., Zhang, X., and Wei, Y. MOTR: End-to-end multiple-object tracking with transformer. In European Conference on Computer Vision, pp. 659-675. Springer, 2022.
Zhang, K., Wang, J., Chen, X., and Yang, Y. Dualflow: Integrating raft-based optical flow with appearance mod- els for multi-object tracking, In Proceedings of the IEEE/CVF Winter Conference Applications of Com- 2022a.
puter Vision, pp.
Zhang, W. Wang, J., Li, Y., and Zhou, Boosttrack: Boosting multi-object tracking through adaptive feature selection and track refinement. In Proceedings of the IEFF/CVF International Conference on Computer Vision, 2023a.
Zhang, Y., Wang, C., Wang, X., Zeng, W., and Liu, W. Fair- MOT: On the fairness detection and re-identification in multiple object tracking. IJC,, pp. 1-19, 2021.


Reviews:
[{"review_id": "krxTssVXq0", "reviewer": "Reviewer_ZziK", "summary": "This paper propose leveraging optical flow with soffmax splitting to estimate the motion of occluded objects. Together with the proposed enhanced track embeddgins module (ETEM), the model, i.e. MOTE, achieves state-of-the-art (SOTA) performance on various multiple object tracking (MOT) benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback on our MOTE framework. We appreciate your recognition of our method's performance improvements and effective use of optical flow and memory mechanisms.\n\nQualitative analysis: We understand your concern about limited qualitative results. While Fig. 5 demonstrates our approach's effectiveness, we agree that additional visualizations would strengthen our case. We have prepared more examples showing MOTE's performance on challenging occlusion scenarios and will include these in the supplementary video materials.\n\nDataset selection: We evaluated MOTE on three diverse benchmarks (MOT17, MOT20, and DanceTrack) as well as the MOT15 dataset in the extended results. The DanceTrack dataset offers particularly challenging scenarios with complex motion patterns and high inter-object similarity. We appreciate your suggestion regarding Bird Flock Tracking (BFT) and SportsMOT datasets. To address this, in the last few days, we conducted a preliminary experiment on SportsMOT without retraining and evaluated our method on a sample of three sequences. We also compared our approach with ByteTrack and MOTR under the same conditions. We plan to expand to BFT in future research applications to diversify the tracked object types.\n\nPreliminary results on SportsMOT: Our preliminary experiments indicate that MOTE achieves the highest MOTA score at 45.7\\%, significantly outperforming ByteTrack, which achieves only 17.9\\%, and slightly surpassing MOTR at 44.1\\%. Similarly, in terms of IDF1 score, MOTE achieves 50.2\\%, compared to 31.4\\% for ByteTrack and 48.7\\% for MOTR. These results highlight MOTE's superior tracking accuracy and adaptability, even when applied to a new dataset without additional training. While fine-tuning on SportsMOT could further enhance performance, these results already demonstrate the robustness of our approach in diverse tracking scenarios.\n\nOptical flow Iterations vs. HOTA: The apparent paradox where more iterations (25) improve MOTA but harm HOTA can be explained by the balance between detection and association accuracy. At 20 iterations, we achieve an optimal balance between efficiency and tracking performance. At 25 iterations, improved flow estimation increases detection accuracy (reflected in MOTA) but introduces over-smoothing that reduces feature distinctiveness (affecting association accuracy in HOTA). This trade-off demonstrates the importance of careful parameter tuning in tracking systems.", "claims_and_evidence": "The proposed softmax splatting method is validated by Fig. 5 and equation 1 to 5.\n\nThe proposed ETEM is supported by equation 6 to 11.\n\nAll results are further validated by ablation studies tables, i.e. Table 4 to 6.", "methods_and_evaluation_criteria": "The methods are evaluated on MOT17, MOT20, and DanceTrack with the standard metircs, i.e. HOTA, ASSA, and DetA.", "theoretical_claims": "The theorectical claims are fine and validated.", "experimental_designs_or_analyses": "The experimental desgins are good and the analyses are fruitful.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strenghts:\n- The performance improvements over 3 popular datasets are good and demonstrates method's superiority.\n- The usage of optical flow and new memory mechanism is reasonable and effective.\n\nWeakness:\n- THe qualitative analyses are limited. For MOT, it requires more qualitative results to see the effectness of the method, especially for the occluded objects.\n- Despite that the three datasets results are provided, but there are some more challenging datasets such as Bird Flock Tracking (BFT) and SportsMOT. Those are more advanced and should be evaluated on along with newer baselines.", "other_comments_or_suggestions": "See the weakness.", "questions_for_authors": "The explanation of Table 5 looks not convincing enough. Why the optical flow estimation improves over iteration and yet harms the HOTA metirc?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SpyX2iDtE4", "reviewer": "Reviewer_XwuH", "summary": "The paper introduces MOTE, an end-to-end multi-object tracking framework that integrates optical flow estimation and softmax splatting to robustly handle prolonged occlusions.\n\n## update after rebuttal\nThe authors did not provide a detailed FLOPS analysis, leaving key computational efficiency aspects unaddressed. Based on other reviews, I agree that the description of inter-module data flow and the design of comprehensive ablation experiments remain insufficient. Therefore, I am modifying my score to \"Weak Accept.\"", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work and thoughtful questions.\n\nComputational costs: Our MOTE framework adds only 25ms of additional processing time per frame compared to the baseline MOTR method's 133ms inference time (7.5 FPS) on high-resolution (1536x800) inputs, representing just a 19\\% increase in total computation. This modest overhead comes from two main components: the RAFT implementation with 20 iterations (requiring $\\sim$18ms) and the softmax splatting layer (adding $\\sim$7ms). We believe this represents a reasonable trade-off considering the significant performance improvements demonstrated in our results. For real-time applications where speed is critical, we've explored lightweight flow estimators that reduce the total overhead to $\\sim$12ms with only a 1.8\\% HOTA reduction.\n\nFast motion handling: We've implemented adaptive flow resolution scaling that detects rapid motion and applies higher-resolution flow estimation selectively. Our occlusion masking mechanism (Eq. 12-13) also addresses this by weighting tracking components based on occlusion states. As shown in Table 6, the occlusion weighting mechanism significantly improves performance in complex scenarios, leading to better results in challenging cases involving rapid motion and occlusions.\n\nFusion strategies:  We compared softmax splatting with linear splatting as an alternative approach. As shown in Table 4, softmax splatting consistently outperformed linear splatting with a 3.2-point increase in HOTA (58.4 vs. 55.2), a 3.6-point increase in MOTA (64.9 vs. 61.3), and a 3.5-point increase in IDF1 (69.2 vs. 65.7). The softmax splatting mechanism is particularly effective for preserving motion information during occlusions compared to the simpler linear approach.", "claims_and_evidence": "MOTE is the first end-to-end tracking framework that successfully integrates optical flow and softmax splatting to handle prolonged occlusions, surpassing other methods. The experimental results on MOT17, MOT20, and DanceTrack datasets show improved performance.", "methods_and_evaluation_criteria": "The proposed method combines deformable DETR for multi-scale feature extraction with RAFT-based optical flow estimation. •\tEvaluation is performed on standard MOT datasets using common metrics (MOTA, HOTA and IDF1)", "theoretical_claims": "The paper does not focus on new theoretical proofs; it is primarily experimental and engineering-driven.", "experimental_designs_or_analyses": "The experimental design is sound, with comparisons on multiple datasets and a comprehensive ablation study.", "relation_to_broader_scientific_literature": "The work builds on transformer-based tracking (e.g., MOTR) and integrates ideas from optical flow-based approaches and video frame interpolation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \n1. Innovative combination of optical flow and softmax splatting within an end-to-end framework.\n2. Comprehensive experimental evaluation and ablation studies\n3. Significant performance improvements on standard benchmarks.\n\nWeaknesses: \n1. The method may incur higher computational costs due to optical flow estimation, which might pose challenges for real-time applications.\n2. Sensitivity to extremely rapid motion or complex interactions is acknowledged but not deeply analyzed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you provide more details on how the increased FLOPS due to optical flow estimation impact real-time performance? \n2. Have you considered any mechanisms or additional experiments to address the potential sensitivity of the optical flow module in extremely fast motion or highly complex interaction scenarios?\n3. Did you explore other fusion strategies besides softmax splatting", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rd1fSaMlRE", "reviewer": "Reviewer_bRBd", "summary": "This paper presents MOTE, a novel multi - object tracking (MOT) algorithm aiming to solve the problem of tracking occluded objects. It combines deformable detection transformers, optical flow estimation, and softmax splatting. By leveraging optical flow to generate features and using a softmax splatting layer to create a disocclusion matrix, MOTE can estimate the motion of occluded objects. The enhanced track embedding module (ETEM) in its architecture helps maintain object identity during occlusions. MOTE is evaluated on multiple datasets such as MOT17, MOT20, and DanceTrack. It achieves high tracking metrics, outperforming existing state - of - the - art methods, especially in reducing identity switches and handling complex occlusion scenarios. Ablation studies are conducted to verify the effectiveness of different components. However, there are some limitations, like the lack of ablation experiments on different component combinations and unclear data flow details among modules.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback on our MOTE framework.\n\nID switches: You raise an important point. The videos show some ID switches in extremely challenging scenarios with prolonged, complete occlusions. These represent edge cases where even our approach struggles. Our primary focus was on handling prolonged occlusions, and our quantitative results (Tables 1-3) demonstrate significant improvements in ID switch reduction overall (1412 vs. 1446, 2.4\\% fewer than previous methods). We've reported ID switch metrics throughout our evaluations (Tables 1,4,5,6), providing transparency about our model's performance in this aspect. ID switching remains an open research challenge, and we've been transparent about current limitations while demonstrating substantial progress.\n\nData flow clarity: We apologize for any lack of clarity in describing module interactions. The flow proceeds as follows: 1) Optical flow estimation between frames, 2) Feature extraction via deformable transformers, 3) Softmax splatting to generate disocclusion features, 4) ETEM integration of these features with track queries, and 5) Final object tracking via the decoder. We'll improve our description to make these interactions clearer.\n\nComponent integration: Our approach to component integration is guided by careful ablation studies. While computational constraints limited testing all combinations, Tables 4-6 provide evidence of each component's contribution. The integrated tests confirm that the full integration provides 3.2\\% better HOTA than any subset. Specifically, our softmax splatting approach enables the extraction of disocclusion features, providing the model with perceptual understanding of subjects under prolonged occlusion scenarios. This perceptual capability represents a significant advancement over previous methods that struggle with occlusion handling.\n\nThe key innovation in our approach is how softmax splatting interacts with ETEM in complex scenarios. Splatting provides weighted feature propagation that preserves motion information during occlusions, while ETEM integrates these features with appearance cues to maintain consistent tracking. This synergy enables MOTE to handle occlusions more effectively than methods that rely on either motion or appearance alone.", "claims_and_evidence": "The claims in the submission are mostly supported by clear evidence. The proposed MOTE algorithm shows excellent performance on multiple datasets, and the ablation studies effectively verify the contributions of different components. For example, the comparison between softmax splatting and linear splatting demonstrates the superiority of softmax splatting in enhancing tracking accuracy. However, the evidence regarding the method's effectiveness in handling occlusions may be challenged due to the obvious ID switches in the supplementary materials' videos.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem at hand. The integration of deformable transformers, optical flow, and softmax splatting is innovative and suitable for multi - object tracking in occlusion scenarios. The evaluation on multiple datasets with standard MOT metrics like HOTA, MOTA, and IDF1 is reasonable. However, the lack of ablation experiments on different component combinations limits the comprehensiveness of evaluating the method. Also, the unclear data flow details among modules may affect the understanding and reproducibility of the method.", "theoretical_claims": "There are no complex theoretical proofs in the paper that require checking. However, the proposed approach is based on established concepts in computer vision, such as optical flow and transformers. The combination of these concepts seems reasonable, but a more in - depth theoretical analysis of how the different components interact and why they work effectively could strengthen the theoretical foundation.", "experimental_designs_or_analyses": "The experimental designs are generally sound. The ablation studies on individual components like the splatting technique, the number of iterations in optical flow estimation, and the effect of occlusion weights are well - designed and provide valuable insights. However, as mentioned before, the lack of ablation experiments on different component combinations is a limitation. Also, the evaluation of the method's performance in handling occlusions could be more comprehensive, considering the ID switch issues in the supplementary materials.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the broader scientific literature. The paper builds on existing works in multi - object tracking, especially those addressing occlusion challenges. It improves upon CNN - based and Transformer - based methods by integrating optical flow and softmax splatting. The use of softmax splatting for occlusion handling is also related to previous research in video interpolation. However, the paper could further discuss how its approach differs from and improves upon these related works in more detail.", "essential_references_not_discussed": "There are no essential references that are clearly missing from the paper. The paper comprehensively reviews the relevant literature in the field of multi - object tracking and occlusion handling. However, it could explore more deeply some recent research trends and their potential implications for the MOTE algorithm.", "other_strengths_and_weaknesses": "Strengths:\n1、The innovative combination of multiple techniques effectively addresses the occlusion problem in multi - object tracking, which is a significant contribution to the field.\n2、The comprehensive experimental evaluation on multiple datasets and the ablation studies enhance the credibility of the method.\n3、The paper is well - structured, making it easy to follow the research.\nWeaknesses:\n1、The lack of ablation experiments on different component combinations limits the understanding of the interactions between components.\n2、The unclear data flow and interaction details among modules make it difficult to fully understand and reproduce the method.\n3、The ID switch problem in the supplementary materials' videos may cast doubt on the method's effectiveness in handling occlusions.", "other_comments_or_suggestions": "1、Conduct ablation experiments on different component combinations to better understand the synergies between components.\n2、Provide more detailed explanations of the data flow and interaction among modules, including the specific operations and data format changes.\n3、Re-evaluate the method's performance in handling occlusions, considering the ID switch issues in the supplementary materials.", "questions_for_authors": "Can you explain the reasons for the obvious ID switches in the videos provided in the supplementary materials? How do you plan to address this issue in future research? If the ID switches are due to limitations in the current method, it may significantly affect the practical application of MOTE, and I may lower my evaluation of the paper.\nCould you elaborate more on the potential interactions between different components in MOTE? For example, how does the softmax splatting module interact with the ETEM module in more complex scenarios? A better understanding of these interactions could strengthen the theoretical and practical value of the paper, and it may lead to a more positive evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L8vxr6WznM", "reviewer": "Reviewer_urni", "summary": "This paper presents MOTE , a novel multi-object tracking  algorithm designed to tackle the persistent challenge of tracking occluded objects. MOTE introduces a unique approach by integrating deformable detection transformers with a custom disocclusion matrix, which significantly improves the ability to track objects even when they are temporarily hidden from view. The algorithm utilizes optical flow to generate features, which are processed through a softmax splatting layer to create the disocclusion matrix. This matrix is very important in maintaining track consistency by estimating the motion of occluded objects. It is important to note that state-of-the-art performance has been achieved on multiple datasets.\n\n## update after rebuttal\nThank you for the authors' response, which has addressed some of my concerns.  Importantly, this paper achieve high performance, which is an encouragement for end-to-end MOT. Additionally, I hope the code could be made open-access to advance research in the multi-object tracking (MOT) field. I keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work and insightful comments.\n\nComputational complexity: MOTE's end-to-end processing time is $\\sim$45ms per frame on an A100 GPU, compared to $\\sim$20ms for MOTR and $\\sim$15ms for ByteTrack. The additional overhead comes primarily from optical flow estimation ($\\sim$18ms) and softmax splatting ($\\sim$7ms). We've explored optimization strategies including fewer optical flow iterations and model pruning that reduce computation by 40\\% with only 2.3\\% HOTA drop.\n\nKITTI results: We appreciate your suggestion regarding evaluation on the KITTI dataset. This would indeed be beneficial for exploring more challenging interactive scenarios involving both vehicles and pedestrians. While we focused our current evaluation on MOT17, MOT20, and DanceTrack datasets, we plan to conduct evaluation on KITTI in future work to further validate our approach in diverse scenarios.\n\nReal-world improvements: For practical deployment, we identify three key areas for improvement: 1) Model optimization for computational efficiency, 2) Enhanced handling of fast motion through adaptive resolution scaling, and 3) Integration with long-term feature banks to handle extended occlusions. We're actively working on these directions.\n\nImpact Statement: We apologize for this oversight and will include a comprehensive impact statement addressing both the benefits (improved surveillance and autonomous navigation safety) and potential concerns (privacy implications and computational resource requirements).\n\nExtremely fast motion: While our current implementation may face challenges with extremely rapid motion, our adaptive flow resolution approach has shown promising results in preliminary testing. This is particularly important in scenarios like sports tracking, where sudden rapid movements are common.", "claims_and_evidence": "Yes, the claims made in the submission are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-suited for multi-object tracking .", "theoretical_claims": "Yes, i have checked the theoretical Claims of the proposed method.", "experimental_designs_or_analyses": "Yes, i have checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "Prior work in MOT has long struggled with occlusion handling, often relying on heuristic methods or appearance-based features that fail in complex scenarios. MOTE addresses this by introducing a novel disocclusion matrix and optical flow. This paper has led to a significant improvement in the performance of end-to-end methods and also demonstrates new trends for the future of multi-object tracking.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1.This paper significantly improves tracking performance under occlusion challenges by leveraging deformable transformers and a disocclusion matrix, which sounds quite innovative.\n\n2.The paper introduces an ETEM module, further enhancing the model's robustness in occlusion scenarios.\n\n3.The proposed method achieves state-of-the-art (SOTA) performance on multiple tracking datasets. End-to-end object tracking is a future trend, and this paper further demonstrates the potential of end-to-end object tracking.\n\nWeaknesses:\n\n1.The paper employs optical flow estimation, which may significantly increase computational complexity and reduce speed. This could pose a bottleneck in real-world applications.\n\n2.The authors should analyze the computational complexity of their method compared to other approaches, at least to identify potential optimization directions and further promote the development of end-to-end multi-object tracking.\n\n3. The paper lacks an Impact Statement.", "other_comments_or_suggestions": "See Weaknesses", "questions_for_authors": "Has the author tested the performance on the KITTI dataset, and how does it compare with other methods? Additionally, if this method is applied to real-world scenarios, what further improvements are needed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces an innovative approach by integrating deformable transformers with optical flow estimation to advance multi-object tracking, particularly in managing occlusions. The proposed method shows significant improvements in performance compared to existing techniques, especially under challenging conditions such as heavy occlusions.
The authors have effectively addressed the reviewers' concerns, leading all reviewers to agree on a "weakly accept" rating for the paper. ACs concur with this positive assessment and recommend acceptance of the paper.
Decision: Accept (poster)