Object Tracking by Hierarchical Part-Whole Attention
Authors: Jinkun Cao
Jiangmiao Pang
Xinshuo Weng
Rawal Khirodkar
Kris M. Kitani
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-object tracking
transformer
visual representation

Abstract: We present in this paper that hierarchical representations of objects can provide an informative and low-noisy proxy to associate objects of interest in multi-object tracking. This is aligned with our intuition that we usually only need to compare a little region of the body of target objects to distinguish them from other objects. We build the hierarchical representation in levels of (1) target body parts, (2) the whole target body, and  (3) the union area of the target and other objects of overlap.  Furthermore, with the spatio-temporal attention mechanism by transformer, we can solve the tracking in a global fashion and keeps the process online.  We design our method by combining the representation with the transformer and name it Hierarchical Part-Whole Attention, or HiPWA for short. The experiments on multiple datasets suggest its good effectiveness.  Moreover, previous methods mostly focus on leveraging transformers to exploit long temporal context during association which requires heavy computation resources. But HiPWA focuses on a more informative representation of objects on every single frame instead. So it is more robust with the length of temporal context and more computationally economic. 

Under review as a conference paper at ICLR 2023
OBJECT TRACKING BY HIERARCHICAL ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
We present in this paper that hierarchical representations of objects can provide informative and low-noisy proxy to associate objects of interest in multi-object tracking. This is aligned with our intuition that we usually only need to compare little region of the body of target objects to distinguish them from other objects. We build the hierarchical representation in levels of (1) target body parts, (2) the whole target body, and (3) the union area of the target and other objects of over- lap. Furthermore, with the spatio-temporal attention mechanism by transformer, we can solve the tracking in a global fashion and keeps the process online. We de- sign our method by combining the representation with the transformer and name it Hierarchical Part- Whole Attention, or HiPWA for short. The experiments on mul- tiple datasets suggest its good effectiveness. Moreover, previous methods mostly focus on leveraging transformers to exploit long temporal context during associa- tion which requires heavy computation resources. But HiPWA focuses informative representation of objects on every single frame instead. So it is more robust with the length of temporal context and more computationally economic.
INTRODUCTION
How to represent the visual existence of an object in a discriminative fashion is a core question of computer vision. In this paper, we propose a hierarchical part-whole representation to represent the visual existence of objects. We adopt multi-object tracking as the application area since the distin- guishable appearance feature is critical avoid the mismatch among target objects when tracking across frames. To gather and process the visual information from different levels, we combine the hi- erarchical part-whole representation with the attention mechanism from transformers to summarize distinguishable and discriminative visual representations for objects of interest.
In the task of multi-object tracking, given a bounding box to localize objects of interest, how should we recognize the major object within the box and distinguish it from the background and other ob- jects, especially some also having partial existence in the box? We believe the visual specificity of one object comes from three perspectives: the compositional, the semantic and the contextual. The compositional suggests the salient and unique visual regions on an object, such as a hat on a pedestrian whose color is different from all others in the same image. With salient visual com- position attached to an object, we can track it across frames even without seeing its full body. The semantic visual information is the commonly adopted one in modern computer vision such as a tight bounding box or instance segmentation mask. It defines the occupancy area of the object with the bond between its visual existence and semantic concept. Finally, contextual visual information de- scribes the surroundings of an object. It helps to distinguish an object via contrast. For example, the bounding box might contain pixels from the background and secondary objects. However, tight bounding box offers a strong underlying prior when combined with visual context: an object whose parts span across the boundary of the bounding box should not be the major object of this bounding box. Being the secondary object or not object of interest, it should be regarded noise when we generate a distinguishable visual representation for the major subject in the bounding box. The analysis above shows each level has its value to represent object discriminatively. Motivated by the insight, we propose to represent an object by a three-level hierarchy: body parts, full body, and the union area including objects with overlap. We summarize it as a (Part-Body-UU union"" hierarchy.
With the hierarchy constructed, an ideal path to solving the target association in multi-object tracking is to leverage the salient information within the body area and discard mismatch by eliminating the

---

Under review as a conference paper at ICLR 2023
noise revealed by the contextual contrast. Without requiring more fine-graine's data annotation, propose to use transformers to process the hierarchical representation as the attention mechanism can discover important visual information. So, by combining the hierarchical visual representation and attention-based feature fusion, we finally propose our method as Hierarchical Part-Whole Attention, or HiPWA for short. In this work, we build baseline model following this design and demonstrate its effectiveness in solving multi-object tracking problems. Through experiments on multiple multi- object tracking datasets, the proposed method achieves comparable or even better performance than the state-of-the transformer- basssrr methods with a more lightweight implementation and better time efficiency during training and inference.
2
RELATED WORKS
2.1 REPRESENTING OBJECTS BY PARTS
The most commonly used object representation for multi-object tracking is bounding boxes. How- ever, the bounding box is noisy by containing background pixels and pixels from secondary objects. On the other hand, our life experience demonstrates that, in many scenarios, it is not necessary to ob- serve the full body of objects specify an object visually and tracking targets by the distinguishable parts on is usually more efficient. Therefore, researchers also have been studying object detection and tracking with more fine- fine-grained representation. A common way is to use pre-defined certain parts on target bodies, such as only human head (Sundararaman et al., 2021; Shao et al., 2018), human joints (Andriluka et al., 2018; Xiu al., 2018) or even every pixel (Voigtlaender al., 2019; Weber et al., 2021). However, all these choices require more fine-grad tne-grained data annotation beyond bounding boxes and more fine- fine-grained perception modules beyond just normally available object de- tectors. In the contrast, the part-whole hierarchy we construct requires no additional annotations and we still solve tracking tasks at the granularity of bounding boxes. The idea of modeling objects with different levels is inspired by the hierarchical modeling of the human body (Marr, 2010) by David Marr when he explains how to construct the visual structure of an object from primal sketch to 2.5 sketch and further 3D representation. His classic three levels of visual information processing sys- tem concludes this in a higher-level: the computational, the algorithmic, and the implementational. A similar theory is also introduced by Fodor & Pylyshyn (1988) as the semantic, the syntactic, and the physical. Compared to these cognitive theories aiming to model general visual representation, the three perspectives we propose to recognize an object and distinguish it from others (the composi- tional, the semantic and the contextual) only apply to the specific problem of generating an effective visual descriptor to represent the objects of interest.
2.2 TRANSFORMER-BANE
MULTI-OBJECT TRACKING
Transformer (Vaswani et al,, 2017) is originally proposed for natural language processing. It shows a powerful capacity for information representation and processing. Later, DETR (Carion et al., 2020) introduces the transformer to the area of visual perception for object detection. It models object strategy by DETR
detection as solving a bipartite matching problem. Given that the matching- is quite similar to the target matching in the task of multi-object tracking, it is intuitive to further mi- grate transformer to this area. TransTrack (Sun et al., 2020) is the first work using the transformer to solve the MOT problem but it does not invent any association strategy by transformers. A concurrent work TrackFormer (Meinhardt et al., 2021) takes a further step to use the cross attention in trans- former decoder in the stage of association by query passing. On the other hand, VisTR (Wang et al., 2021c) proposes a novel global association scheme upon transformer where a video clip of multiple frames is forward into the transformer the same time to associate objects within the clip. More recently, many works (Zhou et al., 2022; Zeng et al., 2021) follow the global association scheme in either training or inference and achieve good performance. A key to their success is process the information over a long temporal period, which can be hardly handled without the transformer. GTR (Zhou et al., 2022) makes baseline model of using only appearance in associating objects and removing some secondary modules such as positional encoding and learnable object query. How- ever, a downside of processing multiple frames as a batch by the transformer is the high requirement of computation resources. It has become a common practice to train the model at least 100 GPUs (Zhou et al., 2022; Sun et al., 2020; Zeng et al., 202.) or even 100 GPUs (Cai et al., 2022). These methods usually suffer from significant performance drop if only limited computation resource is available. This is because they usually make improvements to association performance by taking advantage of long temporal window and gathering more visual context within it. In
2

---

Under review as a conference paper at ICLR 2023
frame CHAPKIS
frame L+T
CHAPKIS
WIP
WOR
CHAPKIS
Images
Parts
Body
Union
Object Detector
Objects
Feature Encoder
Hierarchical Part-Whole Attention
Tokens
concat
Global Association
Part feature
Token
Global Association Matrix
pyramid
Tk
The pipeline of our proposed method. Our method follows the scheme and conducts association in a global fashion. Our proposed hierarchical feature attention module fuses the features from three levels, i.e., object parts (compositional), object bodies (seman- tic) and union area of objects with overlap (contextual) The output features serve tokens in the following transformer decoder for the global association.
Figure 1:
this work, we focus on building a more computation and wemory-efficieen visual representation for objects from the scope of a single frame instead. This scheme is flexible be integrated with transformers and more robust to short time windows during object association.
METHOD
In this section, we introduce the method we propose to leverage hierarchical part-whole visual representation with the attention mechanism from the transformer for multi-object tracking. Sec- tion 3.1, we describe the overview structure of our method using global association. Then, in Sec- tion 3.2, we dive into the details of our proposed part-whole attention module. Finally, we talk about the details of training and inference by HiPWA in Section 3.3.
3.1 GLOBAL ASSOCIATION
Before the transformer is introduced into this area, people usually solve multi-object tracking in frame-by-frane fashion where the association is performed only two frames. Recently, the trans- former shows the advantage to gather and process information from multiple steps in parallel. To leverage this advantage, previous methods (Wang al., 2021c; Zhou et al., 2022) propose to per- form association in a video clip instead of just two frames. Therefore, the spatio-temporal attention capacity of the transformer leads to new global association fashion. We follow this scheme in our proposed method. The overall pipeline of HiPWA is shown in the left-hand half of Figure 1. Now, we explain the three stages of
Detection and Feature Extraction. Given video clip of T frames, i.e., T ++, 1,.,, t T),
we have the corresponding images I = {I*, _++1 (t+t1...,,*** ['TT]. Given a detector model, we could derive the detections of the object category of interest on all frames in parallel, noted as O {O' .o., Ox* }. N is the number of detections on the frames and t; E T S N) is the time step where the i-th detection, i.e., O is detected. Then, we generate the representations of each detected object and note them F = (Fi, F2,..., Fx). The most commonly adopted solution is to use the backbone output on the object area as the representation features while we adopt our proposed hierarchical whrl- representation instead whose details are to be introduced soon.
Token Generation by Hierarchical PholeWhole Attention. After being projected into vectors, the hierarchical representations of detections become tokens dett (Tkdet, Tkgeet ,..., Tk-det).

---

Under review as a conference paper at ICLR 2023
which are also terms as "'ojectt query" in previous works (Sun et al., 2020; Meinhardt et al., 2021). Concatenating the tokens makes Q E RNxD, where D is the feature dimension. If we aim to associate the new-coming detections with existing trajectories, we also need the tokens to repre- sent the existing M trajectories, i.e. T {Tk] The ..... Tk J. The transformer has shown good power to generate more discriminative feature tokens for trajectories by iterative query pass- ing (Zeng et al., 2021) or long-timg feature buffering (Cai et al., 2022). But to make our method simple, we directly project the hierarchical representation of objects on existing trajectories to rep- resent the trajectories. Given a historical horizon H to backtrack the objects on the previous time steps of a trajectory, we represent a trajectory, Thy with the "track query" Q E HR The track query is the combination of the feature tokens of detections within the historical horizon on the corresponding trajectory.
Global Association. By cross-attention, we could get the association score between the set of detections and the trajectory Th as S(Q Quat) E H In practice, because we aim to associate between all M trajectories and N detections, we perform the cross-attention on all object queries and track queries at the same time, namely S(Q"hi, Quéet) E R By averaging the score on the H frames selected from the historical horizon, we could finally get the association score between detections and trajectories as S E MRN, Then, we need to make sure that a trajectory will never be associated with more than one object from the same frame. We normalize the association scores between a trajectory and objects from the same time step by softmax. So the normalized association score between the j-th trajectory and the i-th detection is exp(S,..)
P(M
-M = 1| Qtii = ExE(1......)) 1 It& = ==)exp(S,,.))
(1)
where the binary indicator function [ts, t.] indicates whether the i-th detection and the k-th detection are the same time step. M e (M))xN is the final global association matrix. Its dimension is of (M + 1) x N because each detection can be associated with an "empty trajectory" to start new track in practice. The query of the "empty trajectory" is represented by query randomly drawn from previous unassociated tokens during training. Also, after the association, unassociated trajectories will be considered absent on the corresponding frames. In such fashion, we can train over large set of detections and trajectories in parallel and also conduct inference in an online manner by setting O as the set of the detections only from the new-coming frame.
3.2 HIERARCHICAL PART- WHOLE ATTENTION
Finally, we come to the details about constructing hierarchical part-whole visual representations. We name this process hierarchical whrt- attention as we use the attention mechanism to gather and process information from different levels in the hierarchy, which is illustrated in the right-hand half of Figure 1. We design this representation because we think there are three perspectives to describe the existence of an object and its identification over other objects: the compositional, the semantic, and the contextual. Correspondingly, we think the body part patches, the full object body, and the union of the occupancy of objects with interaction provide the knowledge from the three perspectives respectively. The insight behind this module is what we would like the most to deliver in this work.
Hierarchy Construction. We represent a detected object by a quintuple, i.c., O [z. y, w, ft, c). where the first four values describe its bounding box and e is the detection confidence. So its body area is B- = [xr, y, x + W. y + h.. Next, we divide the body into multiple sub-regions (parts). By default, similar to what ViT (Dosovitskiy et al., 2020) does upon images, we divide the bounding boxes into 2 x 2 bins, making set of body parts as P = [Pi, Ps, Ps. P.). On the other hand, from a global scope, there are other targets interacting with which are highly likely to mismatched with O in the association stage. We crop the union area enclosing O and all other targets having overlap with it. We note the union area as U. Till now, we have derives the part-whole hicrarchy {P, B, U} in a very straightforward way.
Feature Fusion. Given the whole hierarchy, we have to fuse the features from different levels to get the final feature tokens for association. With feature encoder, we extract the CNN features from them as Fp E IR BR4Cx FR E Cx and Fg E RCCH We simply concatenate the features from the first two levels as Fr.n E R5CxHxW Then, by two-layer
4

---

Under review as a conference paper at ICLR 2023
projection network, gain projected features Vp+B R We also apply the projection to the union area features and get Vy E RD. Finally, we perform cross aross-attention between Vp+B and Vo and forward the output an MLP network to get the tokens of shape R5x0, Before being forwarded to the global association stage, the tokens would he projected to the uniform dimension of D.
3.3 TRAINING AND INFERENCE
The method we implement is a baseline model without complicated designs on "queries". We simply use the hierarchical part-whole features of detected objects to serve as the representations of both detections and trajectories. And during training, we can associate between detections in the sampled video clips or between detections and existing trajectories. These two schemes of associations thus are implemented online, we only perform association between detections from the new-coming frame and existing trajectories. We realize this by iterating sliding window with the stride of one frame.
the same and share all model modules. During inference, to keep the process
Training. We train the association module by maximizing the likelihood of associating detections belonging the same ground truth trajectory as expressed in Eq. But Eq. 1 happens step i; only. To speed up training, we calculate the association score on all T frames of the sampled video clip at the same time and maximize the likelihood of the association aligned with the ground truths globally in the time window. The objective thus turns to
time
(+T
II P(M q=t
J. = 1|Q Qtai),
(2)
where T/ is the ground truth index of the detection which should be associated with the j-th tra- jectory on the time step q. Therefore, by traversing the association of all trajectories, the training objective becomes the negative log-liiel loss
t+T
I Lasso ΣΣ $=1
DogP(Mase = 1|Qde Qua)).
J.7.
On the other hand, trajectories can also be absent on some time steps because of occlusion or target disappearance. So similar to the practice of DETR (Carion et al., 2020) for detection and GTR (Zhou et al., 2022) for tracking, Eq. 3 has included the situation of associating trajectory with "empty". Moreover, the main reason why mismatch happens is the features of objects of different identities be- ing indiscriminative. Therefore, encourage the representations from objects of different identities to be distinguishable, we design a feature discrimination loss in the form of triplet loss as
Np
Lisat = max (0, min ||At(((FP.))..)) f(FB) ((LE (LE ((F))) ((FB)l +a)), (4)
u=1
where f()) the shared projection layers to project CNN features to feature vectors and Np the number of part patches (Np 4 in our default setting). Att(-.-) is the operation of cross attention the margin to control the distance between positive and negative pairs. Fn and FP. (1 < S Np) are the extracted features of the body area and the part sub- regions explained already. FlS is the CNN features of the background area in the union box. We obtain the background features by setting the pixels of B in the area of U to be 0 and forward the processed patch of the union area into the feature encoder. We design Eq. 4 to encourage the projection network to pay more attention to the salient area on the body of target objects while less attention to the background area when processing the hierarchical part-whole representations. Also, it encourages the features of the background area in the union box, which probably belongs to another object target, to be distinguishable from the body features. This can be expected to decrease the chance of mismatch between neighboring objects. Finally, the training objective is
to generate attended features.
Lnsso + Lieat + Laet;
(5)
where Laet is an optional detection loss. In our default implementation, we finetune the detector at the same time when training the association modules.
Inference. We adopt the traditional diding-windon style to realize online influence. With win- dow size T = 24 and stride 1, we start from the first frame of the input video. On the first frame, every detection is initialized original trajectory. In each time window, would generate
5

---

Under review as a conference paper at ICLR 2023
trajectories by detections within it. Then we use the association score in Eq. to associate these tra- jectories with existing trajectories outside this time window. By averaging the detection-trat trajectory alongside detections of a trajectory, ative value serves as the entries in the cost matrix for the association assignment. And we adopt Hungarians matching to make sure the one-to-one mapping. Only when a pair of trajectories has the association score higher than a threshold B = 0.3, they are eligible to be associated. All unassociated detections on the new-coming frames will start new tracks.
get the trajectory-trajectee
association scores, whose neg-
4 EXPERIMENTS
4.1 EXPERIMENT SETUPS
Datasets. We conduct quantitative experiments on multiple multi-object tracking datasets, includ- ing MOT17 (Milan et al., 2016), MOT20 (Dendorfer et al., 2020) and DanceTrack (Sun et al., 2021). We focus on pedestrian tracking this paper so pedestrian the only category of objects of interest on all datasets. MOT17 and MOT20 are the classic and popular datasets in the area of pedestrian tracking but their scales are relatively small and have no official validation sets. DanceTrrack, on the contrary, is a recently proposed dataset that is of much larger scale and provides an official valida- tion set with no overlap with the training set. Dance focuses on the scenarios where targets are in the foreground Track mainly contains videos where targets have heavy occlusion, complex motion patterns, and similar appearances so it provides good platform to study the robustness of the tracking algorithm.
detection is not considered as the bottleneck as it is on MOT20. And Dance-
Evaluation Metries. The popular CLEAR evaluation protocol (Bernardin & Stiefelhagen, 2008) is based Longle-frame-vee matching between the ground truth and predictions. This makes the metric emphasize single-frame detection quality rather than cross-frame association performance. MOTA, the main metric of CLEAR protocol, is also biased to the detection quality. To provide a more accurate sense of association performance in tracking, we mainly adopt the more recent HOTA (Luiten et al., 2021) metric set where the metric is calculated by the video-level associa- tion between ground truth and predictions. In the set of metrics, AssA emphasizes the association performance, and DetA stresses on the detection quality. HOTA is the main metric by taking both detection and association quality into consideration.
Implementation. We follow the common practice (Sun et al., 2020; Zeng et al., 2021; Cai et al,, 2022) to use ResNet-50 (He et al., 2016) as the backbone network, which is pretrained on Crowdhu- man (Shao et al., 2018) dataset first. Though advanced detector (Zhang et al., 2021a) is demonstrated as a key to boosting tracking performance, we want our contribution to be more from the improve- ment of the association stage. Therefore, on MOT17, we follow the practice of another transformer- based global association tracking method GTR (Zhou et al., 2022) to use the classic CenterNet Zhou et al. (2019; 2020) as the detector and all training details are aligned with it to make fair comparisons with this close baseline method. The CenterNet detector is pretrained together with the backbone on Crowdhuman to align with the common practice on this dataset. For the fine-tuning of association modules, we use a 1:1 mixture of MOT17 and Crowdhuman for MOT17. We fine-tune with only the MOT20 training set for evaluation on MOT20. For Dance Track, we use its official training set as the only training set during finetuning. The image size is set to be 1280 x 1280 during training and the test size is 1560 for the longer edge during the test. During finetuning, the detector head is also fine- tuned as mentioned already. The training iterations Dance Track. We use BiFPN (Tan et al., 2020) for the feature upsampling. For the implementation of the transformer, we follow the practice of Zhou et al. (2022) to use stack of two layers of "Linear+ ReLU" as the projection layers and one-layer encoders and decoders. We use Adam (Loshchilov & Hutter, 2017) optimizer for training whose base learning rate is set to be 5e-5. The length of the video clip is T = 8 for training and T = 24 for inference in a sliding window. We use 4 x V100 GPUs as the default training device following some previous practice (Zhou et al., 2022; Zeng et al., 2021) but we will see that even using only one RTX 3090 GPU for training, our method can also achieve good performance. The training on MOT17 or MOT20 takes only 4 hours and the training on DanceTrace takes 11 hours.
set to be 20k
and 80k on
4.2 BENCUMARK RESULTS
We benchmark our proposed method with existing methods now. The results on the MOT17-test dataset are shown in Table 1. HiPWA achieves the highest HOTA score among all transformer-
6

---

Under review as a conference paper at ICLR 2023
Table 1:
Results on MOT17 test with the private detections. Bold numbers indicate the overall
best result and underlined numbers are the best transformer- results.
Tracker
HOTAT
AssAt
MOTAT
IDFIt
FP(10') ) FN(10'X X IDsL
Fragl
FairMOT (Zhang et 2021b)
99.3
58.0
73.7
72.3
2.75
11.7
3,303
8,073
Semi-TCL (Li 2021)
59.8
59.4
73.3
73.2
2.29
12.5
2,790
8,010
CSTrcck (Liang et al, 2020)
59.3
$7.9
74.9
72.6
2.38
11.4
3,567
7,668
(Wang ol.,
1,824
QDTrack (Pang et al., 2021)
$3.9
52.7
68.7
66.3
2.66
14.7
3,378
8,091
MAA (Stadler Beyerer, 2022)
62.0
79.4
75.9
3.73
1,452
2,202
ReMOT (Yang at. 2021)
59.7
57.1
77.0
72.0
3.32
9.36
2,853
5.304
PermaTr (Tokmakov al, 2021)
55.5
53.1
73.8
68.9
2.90
11.5
3,699
6,132
MOT (Chu 2021)
61.7
76.7
75.1
3.62
2.346
7,719
BracTrack (Zhang et al,, 2021a)
63.1
62.0
80.3
77.3
2.55
8.37
2,196
2,277
Transformer-bases Methods
TransC (Xu et 2021)
54.5
49.7
73.2
62.2
2.31
12.4
4,614
9,519
TransTrk (Sun 2020)
54.1
75.2
63.5
5.02
8.64
3,603
4,872
MOTR (Zeng et at.
57.2
55.8
71.9
68.4
2.11
13.6
2,115
3,897
TrackFormer (Meinhard:
2021)
65.0
7.44
12.4
3,528
GTR (Zhou al, 2022)
59.1
57.0
75.3
75.1
2.68
10.9
2,859
McMOT (Cai at,,, 2022)
56.9
55.2
72.5
69.0
3,72
11.5
2,724
HiPWA (Ours)
75.4
75.7
2,45
10,8
2,879
3,029
Table 2: Results on MOT20 test with the private detections. Bold numbers indicate the overall best result and underlined numbers are the best transformer-bes based results.
Tracker
HOTAЛ
AssAt
MOTA
IDF11
FP(10*
PN(10*)
IDs
FaiMOO (Zhang et al., 2021b)
54.6
54.7
61.8
67.3
10.3
8.89
5,243
CSTrack (Liang at, 2020)
54.0
54.0
66.6
68.6
2.54
14.4
3,196
GSDT (Wang ot al., 2021b)
53.6
52.7
67.1
67.5
3.19
13.5
3.131
RelationT
2021)
4,243
MAA (Stadler& & Beyorer, 2022)
57.3
55.1
73.9
71.2
2.49
10.9
1,331
ByteTrack (Zhang et at., 2021a)
61.3
59.6
77.8
75.2
2.62
8.76
1.223
OC-SORT (Cao 2022)
62.1
62.0
75.5
75.9
1,80
10.8
913
Transformer-basee Methods
TransCt (Xu et al, 2021)
43.5
37.0
58.5
49.6
6.42
14,6
4,695
TransTik (Sun et ol., 2020)
48.5
45.2
65.0
59.4
2.72
15.0
3,608
McMOT (Cai et ol., 2022)
54.1
55.0
63.7
66.1
4,79
1,938
HIPWA (Ours)
53.0
51.1
63.8
64.4
3.64
13.7
3348
based methods. But our method only achieves a comparable MOTA score with TransTrack (Sun et al., 2020) and GTR (Zhou et al., 2022), suggesting the superior part of HiPWA does not lie in the detection stage. The higher AssA score of our method also demonstrates superior association performance.
MOT20 a challenging dataset by containing scenes crowded pedestrian flows. We report the re- sults the MOT20-test set in Table 2. Though HiPWA shows better performance than MeMOT (Cai et al., 2022) on MOT17, its performance is inferior on MOT20. This probably related to the heavy and frequent occlusion on MOT20. It is common MOT20 that large portion of pedestrians' bodies is occluded for long time. the occlusion period is longer than the horizon of associating existing trajectories and new-coming detections, HiPWA will be likely to fail. On the other hand, the much longer temporal buffer object appearance history maintained by MeMOT turns out more effective such scenarios. However, we note that we design HiPWA with the main goal demon- strating the hierarchical part-whole representation and choosing the most naive implementation for association heads make a computationally efficient baseline model. In the contrast, MeMOT requires 8 A A100 GPUs for training to support the long- time history context buffering (22 frames v.s. frames by HiPWA uses COCO (Lin et al., 2014) dataset as the additional pretraining data.
Next, we come to the benchmark on the DanceTrack dataset in Table 3. HiPWA achieves compa- rable performance with the best transformer-basee methods. The association of HiPWA is inferior to MOTR (Zeng et al., 2021). MOTR has carefully designed global association and optimization modules. The global collective loss and query interaction module to propagate information frame by frame proposed by MOTR show good effectiveness. However, inference speed is much slower due to the heavy architecture. For example, training on MOT17 takes MOTR 2.5 days for MOTR on 8x V100 GPUs while only 4 hours on 4x V100 GPUs for proposed method. And the inference speed is 6.3FPS for MOTR while 17.2FPS for our method on the same machine (V100 GPU). Compared to the close baseline GTR (Zhou et al., 2022), HiPWA
side-effect, its training and

---

Under review as a conference paper at ICLR 2023
Jaylen
Kullan
Khilam
BAN
DANCE
BANC CE DOW
29
Figure 2: The upper line shows the results by HiPWA on three randomly sampled frames of a video in the DanceTrack-ters Track-test set. The video is challenging to show camera motion, heavy occlusion, non- linear motion, and the crossover among targets at the same time. The bottom line shows results on MOT20-test video where the pedestrians in the crowd and heavily occluded.
achieves a more significant gap of outperforming on DanceTrrak. Such an observation suggests our proposed phrt-whole hierarchical representation can be more powerful when the occlusion is heavy.
Given the results shown on the three benchmarks, we have demonstrated the effectiveness of our proposed HiPWA to be comparable to the state-of-the-art transformer-based multi-object tracking algorithms with a lightweight design. It builds a new baseline for future research this line of works. The commonly adopted techniques of query propagation and iteration (Meinhardt et al., 2021; Sun et al., 2020; Zeng et al., 2021), deformable attention (Sun et al., 2020; Cai et al., 2022) and long-time feature buffering (Cai et al., 2022) are all compatible to be integrated with HiPWA.
Table 3: Results on DanceTrack test set. Bold numbers indicate the overall best result and underlined numbers are the best transformer-basee results.
Tracer
HOTAT
DeAt
AssAt
MOIAT
IDFIT
ConterTrack (Zhou et al., 2020)
41.8
78.1
22.6
86.8
35.7
FaiMOOT (Zhang 2021b)
66.7
QDTrcck (Pang al., 2021)
45.7
72.1
29.2
83.0
44,8
TraDes (Wa et al., 2021)
74.5
ByteTrac (Zhang al., 2021a)
47.3
71.6
31.4
89.5
52.5
OC SORT (Cao al., 2022)
55.7
81.7
38.3
92.0
54.6
Transforme based Methods
Trans Sun ol, 2020)
43.3
75.9
27.5
88.4
45.2
MOTR (Zeng 2021)
54.2
73.5
40.2
79.7
51.5
GTR (Zhou al., 2022)
48.0
72.5
31.9
84.7
50.3
HiPWA (Ours)
52.1
76.3
35.8
86.1
4.3 ABLATION STUDY
Though we provide the results multiple benchmarks show the efficiency and effectiveness of our proposed method, there are many variables in the design. We now ablate their contributions to the overall performance of HiPWA. Many previous works the multi-object tracking community follow the practice of CenterTram (Zhou et al., 2020) on MOT17 (Milan et al., 2016) to use the latter half of training video sequences as the validation set. However, this makes the ablation study on the validation set not always convincing because the data distribution of the training set and validation set is too close and the performance gap reflected on the validation set might shrink or even disappear on the test set. Therefore, turn to DanceTrack (Sun et al., 2021) for the ablation study instead where an independent validation set provided and of a much larger scale than previous MOT datasets.
In Table 4 and Table 5, we study the influence of video clip length in the training and inference stage respectively. The result suggests that training the association model with longer video clips
8

---

Under review as a conference paper at ICLR 2023
Table 4: Results of using different length of
Table 5: Results of using different length of
video clip during training.
video clip during Inference.
HOTA!
DetAt
AssAt
IDFIt
HOTAT
DeAt
AssA
MOIAT
IDF11
47.8
70.0
32.8
81.1
8
47.5
69.8
32.5
80.1
8
48.1
70.2
33.2
80.6
50.3
47.9
70.1
32.9
81.4
50.6
48.7
70.0
34.0
80.3
51.7
48.1
70.2
33.2
80.6
50.3
82.6
32.8
81.2
Table Results on DanceTrack validation set to study the contribution from each level our hierarchical representations to the association performance.
HOTA!
DeiAf
AssA
MOTAT
IDF1
Body
45.7
69.5
30.3
81.6
48.1
Body
Part
46.3
69.5
30.7
80.0
48.
Body Union
47.3
70.1
32.0
Body Part Union
48.1
70.2
33.2
80.6
50.3
Table 7: Results on DanceTrack validation set with different configurations for multiple training device choices. HiPWA shows good performance given limited computation resources.
Trinning Device
Trin len
Image Size
HOTA1
DetAf
AssAt
MOTA
IDFI
1x RTX 3090-24GB
1280
81.1
Ix V100-32GB
1560 x 1560
48.0
70.8
32.6
82.4
50.1
4x V100-32GB
1280 x 1280
70.2
80.6
50.3
can continuously improve performance. Limited by the GPU memory, we cannot increase the video clip length to longer than 12 frames here. On the contrary, during the inference stage, increasing the sliding window size does not significantly influence the tracking performance.
The hierarchical part-whole representation the the main contribution of our proposed method. Considering that the hierarchical representation gathers information from three levels (Part, Body, Union), we study the contribution each of them Table 6. Compared to only using the features extracted from the bounding box (body) area, hierarchical representation achieves performance improvement of 2.4 points of HOTA and 2.9 points of AssA. On the challenging DanceTTrach dataset, such improvement can be considered significant when they share the same detections. Also, inte- grating the features of the union area shows better effectiveness than solely integrating the features of body parts. This probably because the cross attention between object body and union areas can provide critical information to compare object targets with their neighboring objects, which can prevent potential false association among them. On the other hand, the information about body parts is already contained in the object's body features. By concatenating the part features and body features, we can't introduce previously missing information pieces significantly.
Finally, we aim to build baseline model for future research in this area, we hope the proposed method is more accessible and computationally economic. We try different parameter configurations in Table 7. Even with only single RTX 3090 GPU for training and inference, its performance is still quite close to our default setting which requires 4 x V100 GPUs. We hope this makes the notorious computation barrier of transformer-batt based methods not that terrible anymore.
CONCLUSION
In this paper, we propose to build discriminative hicrarchical part-whole representations as the visual descriptor for objects in multi-object tracking. The representation is built upon only bounding box annotation and in three levels: Part, Body, and Union. They are designed to provide visual specificity of the object from the compositional, semantic, and contextual perspectives respectively. We further propose use attention in the transformer to gather and process the visual features. The combination of these two aspects makes our method, namely Hierarchical Part- Whole Attention and HiPWA for short. The results multiple datasets demonstrate its efficiency and effectiveness.. We hope the study of this paper can provide new knowledge in the visual representation of objects and an advanced baseline model to solve multi-object tracking problems.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mykhaylo Andriluka, Umar lqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5167-5176, 2018. 2
Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008: 2008. 6
Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong. Wei Xia, Zhuowen Tu, and Stefano Soatto. Memot: Multi-object tracking with memory. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8090-8100, 2022. 2, 4, 7,
Jinkun Cao, Xinshuo Weng, Rawal Khirodkar, Jiangmiao Pang, and Kris Kitani. Observation- sort: Rethinking sort for robust multi-object tracking. arXiv preprint arXiv: 14360, 2022.. 7,
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020. 2,
Peng Chu, Jiang Wang, Quanzeng You. Haibin Ling, and Zicheng Liu. Transmot: Spatial- temporll graph transformer for multiple object tracking. arXiv preprint urXiv:2104.00199 2021.
Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé. Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1tt1:20100.119299 2020. 4
Jerry A Fodor and Zenon Pylyshyn. Connectionism and cognitive architecture: critical analy- sis. Cognition, 28(1-2):3-71, 1988. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 6
Wei Li, Yuanjun Xiong, Shuo Yang, Mingze Xu, Yongxin Wang, and Wei Xia. Semi-tel: Semi- supervised track contrastive representation learning. arXiv preprint arXiv:2107.02396 2021. 7
Chao Liang, Zhipeng Zhang. Yi Lu, Xue Zhou, Bing Li, Xiyong Ye, and Jianxiao Zou. Re- thinking the competition between detection and reid in multi-object tracking. arXiv preprint arXiv:2010.1 12138, 2020. 7
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014. 7
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1rXiv:1711.05101, 2017. 6
Jonathon Luiten, Aljosa Osep. Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: higher order metric for evaluating multi-object tracking. International journal of computer vision, (99(2):588577) 2021. 6
David Marr. Vision: A computational investigation into the human representation and processing of
visual information. MIT press, 2010.
Tim Meinhardt, Alexander Kinillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer:
Multi-object tracking with transformers. arXiv preprint arXiv:2101.0222 2021. 2, 4, 7,8
10

---

Under review as a conference paper at ICLR 2023
Anton Milan, Laura Laix- lan Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv 00831, 2016. 6, 8
Jiangmiao Pang. Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi- dense similarity learning for multiple object tracking. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 164-173, 2021. 7, 8
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint arXiv:/805.00123, 2018. 2,
Daniel Stadler and Jürgen Beyerer. Modelling ambiguous assignments for multi-person tracking in crowds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 133 142, 2022.
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012. 15460, 2020. 2, 4, 6, 7,8
Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dance- track: Multi-object tracking in uniform appearance and diverse motion. arXiv preprint arXiv:2111.14690 2021. 6,8
Ramana Sundararaman, Cedric De Almeida Braga, Eric Marchand, and Julien Pettre. Tracking pedestrian heads in dense crowd. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3865-3875, 2021.
Mingxing Tan, Ruoming Pang, and Quoe Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10781-10790, 2020. 6
Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon. Learning to track with object per- manence. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 10860-10869, 2021. 7
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017. 2
Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceed- ings of the ieee/cvf conference on computer vision and pattern recognition, pp. 7942-7951, 2019.
2
Shuai Wang. Hao Sheng, Yang Zhang, Yubin Wu, and Zhang Xiong. A general recurrent track- ing framework without real data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13219 13228, 2021a. 7
Yongxin Wang, Kris Kitani, and Xinshuo Weng. Joint object detection and multi-object tracking with graph neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13708-1371 13715. IEEE, 2021b. 7
Yuqing Wang, Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the JEEF/VVF Conference on Computer Vision and Pattern Recognition, pp. 8741-8750, 2021c. 2,
Mark Weber, Jun Xie. Maxwell Collins, Yukun Zhu, Paul Voigtlaender, Hartwig Adam, Bradley Green, Andreas Geiger. Bastian Leibe, Daniel Cremers, et al. Step: Segmenting and tracking every pixel. arXiv preprint arXiv: 102. 1859, 2021. 2
Jialian WI, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan. Track detect and segment: An online multi-object tracker. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 12352-12361, 2021.

---

Under review as a conference paper at ICLR 2023
Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu. Pose flow: pose tracking. arXiv preprint arXiv:1802.009100 2018.
Efficient online
Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Alameda- Pineda. Transcenter: Transformers with dense queries for multiple-object tracking. arXiv preprint 1rX1v:2103.15145, 2021. 7
Fan Yang, Xin Chang. Sakriani Sakti, Yang Wu, and Satoshi Nakamura. Remot: A model-agnostic refinement for multiple object tracking. Image and Vision Computing, 106: 104091, 2021. 7
En Yu, Zhuoling Li, Shoudong Han, and Hongwei Wang. Relationtrack: Relation-aware multiple object tracking with decoupled representation. arXiv preprint TrXiv:2105.04322 2021. 7
Fangao Zeng, Bin Dong Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiplc-object tracking with transformer. arXiv preprint arXiv:2105.03247, 2021. 2, 4, 6, 7,
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xing- gang Wang. Bytetrack: Multi-object tracking by associating every detection box. arXiv preprint UrXiv:2/10.0666 2021a. 7,8
Yifu Zhang, Chunyu Wang, Xinggang Wang. Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairesss of detection and re-identification in multiple object tracking. International Journal of Computer Vision, 129(1 1):3699-3087, 2021b. 7,8
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.
Objects as points. arXiv preprint
1XXi::904.008500 2019. 6
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. European Conference Computer Vision, pp. 474-490. Springer, 2020. 6, 8
Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8771-8780 2022. 2, 2022, 5,6,777, 7,7,8
12


Reviews:
[{"review_id": "ukZoP46B2M", "reviewer": "Reviewer_rS4y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed modification for multiple object tracking framework based on GTR[Zhou et.al 2022]. Instead of use the feature representation from detection bounding boxes. This paper proposed to use split bounding boxes and surrounding image patch if there are other objects located close to the current object, which aims to tackle occlusion and avoid identity switch cases.\n\nThe proposed methods performs comparable with state-of-the-art methods on MOT17, MOT20 and DanckTrack benchmark. ", "summary_of_the_review": "This paper proposed Part-Whole Attention module for GTR framework for multiple object tracking task. The proposed module are clearly described and experiments demonstrated the effectiveness. \n\nMore experiments on TAO dataset should be included to compare with its baseline method GTR. Discussions of the structure design for Part-Whole Attention module should be added", "strength_and_weaknesses": "Transformer based tracking framework demonstrated promising performance. Research effort to improve those framework should be encouraged. This paper tried to explicitly guide the model to learn local object patches and surrounding object patches to improve performance. Experiments demonstrated the proposed part-whole attention improves tracking performance, both in the ablation study and on MOT17 and DanceTrack comparing to GTR.\n\n- How was the structure in part-whole attention module designed? Did the author try other designs? Any insights?\n- Given this paper is based on GTR, experiments should conduct on TAO dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oEccdgVEckS", "reviewer": "Reviewer_YxoG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a hierarchical representation of objects for multi-object tracking. The hierarchical representation consists of 3 levels: part, whole and union of overlapped objects. The proposed approach demonstrated good performance on multiple pedestrian/dance public datasets. ", "summary_of_the_review": "Overall, this work shows somewhat creativity, but only considers the appearance features can be insufficient to solve challenging tracking cases.\n", "strength_and_weaknesses": "The strength of the paper is the fusing of the three levels of representation for multi-object tracking associations. The weaknesses of the paper are some of the notations and areas are not clearly presented.\n1. The notations of M and N in section 3.1 are not defined when appearing in the first and second part. This causes confusion when reading the paper.\n2. N should be T in 1<=i<=N in the third page.\n3. In page 4, \"we can tract the CNN features from them\", what is the CNN architect to extract the features?\n4. In page 6 Inference section, \"whose negative value serves as the entries in the cost matrix\". The cost value should not be some negative values. \n\nAnother weakness of the approach is the lack of motion feature in the association. In the association, only appearance features are considered. This can lead to association errors when two objects have similar appearances (similar color of clothes) even they are farther apart in the space. In the Detection and Feature Extraction section, detection features are extracted from a video clip of T frames and then the detection from each one of the T frames is associated with the trajectories. There can be significant motions in the T frames and appearance would be hard to handle the discriminability of the objects.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "77jvaCc_1F", "reviewer": "Reviewer_hWAz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting 'Hierarchical Part-Whole Attention' for multi-object tracking. The proposed module is integrated with transformer network and achieves good performance (comparable or even better results than SOTA mot trackers). The overall training efficiency is also good, i.e., 4 hours on 4*v100 GPUs, while other Transformer based trackers may need days. This paper is well-written and organized, and I believe it will be a good baseline for future works to compare and in-depth development on this framework. ", "summary_of_the_review": "the idea is reasonable, \nthe framework is not very complicated for MOT. \nthe training time is acceptable, but the inference is fast; \nthis paper is well-written and easy to follow. ", "strength_and_weaknesses": "strength: \n1. the idea of Hierarchical Part-Whole representation of target object in MOT seems interesting; \n2. the combination of Hierarchical Part-Whole Attention and Transformer works well on existing benchmark datasets; \n3. the paper is easy to follow and understand. \n\n\nweakness: \n1. the source code will be released or not is unclear. it is an interesting idea, but the implementation is also complicated. If the code is not available, maybe it is hard for other researchers to follow. \n2. the running efficiency is not real-time. i.e., not fast enough for practical applications. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject