Towards Category Unification of 3D Single Object Tracking on Point Clouds
Authors: Jiahao Nie
Zhiwei He
Xudong Lv
Xueyi Zhou
Dong-Kyu Chae
Fei Xie
Venue: ICLR 2024 poster (2024)
Keywords: 3D single object tracking
Category-unified model
Point clouds
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Category-specific models are provenly valuable methods in 3D single object tracking (SOT) regardless of Siamese or motion-centric paradigms. However, such over-specialized model designs incur redundant parameters, thus limiting the broader applicability of 3D SOT task. This paper first introduces unified models that can simultaneously track objects across all categories using a single network with shared model parameters. Specifically, we propose to explicitly encode distinct attributes associated to different object categories, enabling the model to adapt to cross-category data. We find that the attribute variances of point cloud objects primarily occur from the varying size and shape (e.g., large and square vehicles v.s. small and slender humans). Based on this observation, we design a novel point set representation learning network inheriting transformer architecture, termed AdaFormer, which adaptively encodes the dynamically varying shape and size information from cross-category data in a unified manner. We further incorporate the size and shape prior derived from the known template targets into the model’s inputs and learning objective, facilitating the learning of unified representation. Equipped with such designs, we construct two category-unified models SiamCUT and MoCUT. Extensive experiments demonstrate that SiamCUT and MoCUT exhibit strong generalization and training stability. Furthermore, our category-unified models outperform the category-specific counterparts by a significant margin (e.g., on KITTI dataset, $\sim$12\% and $\sim$3\% performance gains on the Siamese and motion paradigms).

Published as a conference paper at ICLR 2024
TOWARDS CATEGORY UNIFICATION OF 3D SINGLE OBJECT TRACKING ON POINT CLOUDS
Jiahao Nie', Zhiwei He Xudong Xueyi Zhou', Dong-Kyu Chae?, Fei Xie School of Electronics and Information, Hangzhou Dianzi University, China
Sccool of Computer Science, Hanyang University, Korea
MME Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University, China (jhnie, zwhe, (xxudong)@@hiee edu. cn, jaffe031 @sjtu. edu. cn
(hokyee dongkyu dongkyyjehanyangg... (eheeeaagaaa.....
ABSTRACT
Category-specifie models are provenly valuable methods in 3D single object track- ing (SOT) regardless of Siamese or motion-centric paradigms. However, such over-specialized model designs incur redundant parameters, thus limiting the broader applicability of 3D SOT task. This paper first introduces unified mod- els that can simultancously track objects across all categories using single net- work with shared model parameters. Specifically, we propose to explicitly encode distinct attributes associated to different object categories, enabling the model to adapt cross-category data. We find that the attribute variances of point cloud objects primarily occur from the varying size and shape (e.g.. large and square vehicles v.s. small and slender humans). Based on this observation, a novel point set representation learning network inheriting transformer architec- ture, termed AdaFormer, which adaptively encodes the dynamically varying shape and size information from cross-category data in a unified manner. We further in- corporate the size and shape prior derived from the known template targets into the model's inputs and learning objective, facilitating the learning of unified repre- sentation. Equipped with such designs, we construct two category-unified models SiamCUT and MoCUT. Extensive experiments demonstrate that SiamCUT and MoCUT exhibit strong generalization and training stability. Furthermore, our category-unified models outperform the category-specific counterparts by sig- nificant margin (e.g., on KITTI dataset, ~ 12% and ~3% performance gains on the Siamese and motion paradigms).
design
INTRODUCTION
Artificial general intelligence (AGI), an emerging concept the field of artificial intelligence (AI), aims to achieve versatile cognitive abilities akin to human intelligence. AGI is envisioned as an intellectual system capable of managing a wide range tasks. Recently, with the advancement of deep learning technology, increasing efforts (Ghiasi et al., 2021; Girdhar et al., 2022) have been devoted to exploring general vision models that can simultancously address diverse vision tasks.
As fundamental task in computer vision, 3D single object tracking (SOT) LiDAR point clouds holds significant potential for various application, such as autonomous driving, mobile robotics, and augment reality (Xie al., 2023; Lv et al., 2021; Zhang & Tao, 2020). Given point cloud sequence, with an arbitrary object in the first frame serving as a template target, the goal of tracking to search for this target in subsequent successive frames. Currently, 3D SOT methods can be mainly divided into two paradigms: Siamese paradigm (Giancola et al., 2019; 9; Qi et al., 2020; Hui al., 2021; Zhou et al., 2022; Nie al., 2023b) and motion-centric paradigm (Zheng et al., 2022a). The former utilizes an appearance matching mechanism to determine seed points, subsequently inferring the target's position based on these points. The latter considers the predicted result from the previous frame as the target prior, and then infers the target's 's relative displacement between the previous and current frames. Both paradigms have achieved outstanding performance. However, regardless of the paradigm followed, existing methods commonly perform the tracking task for each object category independently, as illustrated in Fig. 1 (a). This requires the models to learn and evaluate individual datasets specific to each category. leading to limited generalization and redundant parameters. Therefore, a question naturally arises: it possible to track objects across all categories by unified model with shared parameters?

---

Through a series of comprehensive empirical studies, discover that training models directly with cross-category data leads to suboptimal generalization and training stability, i.e., tracking perfor- Published as a conference paper at ICLR 2024
Cur Model
Podkstian Modil
Addereer
Yon
Modal Eopt
Learing Otgedes
Cydist
Hodel
(a) Previous category specific models
(b) Our category unified models
Figure 1: Comparison between different tracking models. In previous category category-specific models (a), multiple networks are required to perform individual tracking task for each category. In contrast, our category- models (b) can simultaneously track objects across all categories using single network with shared parameters.
mance is relatively unstable after the model converges (detailed in Section 4.3). This phenomenon can be primarily attributed to three key factors incurred by the changing size and shape of cross- category objects. First, existing tracking models (Qi et al., 2020; Shan et al., 2021; Zheng et al., 2021: Hui et al., 2022; Nie et al., 2023c) employ a point set network (Qi et al., 2017a:b) as their backbone, where receptive field of each point is expanded via group operation. However, such a group operation typically acts on a 3D sphere with fixed radius, resulting in fixed receptive field that is not adaptable to changing size and shape information embedding; Second, search region of current frame is generated by enlarging predicted target area from previous frame by fixed dis- tance. This imposes an inconsistent proportion of foreground and background within the search regions for different object categories (i.e., more background interference for smaller objects, less for larger objects): Third, offset learning is widely available in both the Siamese and motion-centric paradigms (Qi et al., 2020; Zheng al., 2022a), and there is considerable gap in the distribution of offset outputs across cross-category objects due to the different sizes and shapes. A similar problem occurs in the distribution of positive and negative samples. Consequently, different categories of data exhibit distinct learning objectives.
Based on the aforementioned observations, we introduce SiamCUT and MoCUT, which can simul- taneously track objects across all categories using a single network with shared model parameters, as shown in Fig. (b). Our unified models consist of three core designs: unified representation network, model inputs and learning objective. We first design novel point set representation net- work, termed AdaFormer that serves as the representation network for our unified tracking models. Our AdaFormer block consists of a deformable group rector-attention sub-block to aggregates infor- mation over variable range of receptive fields. To be specific, this sub-block incorporates a group regression module, which learns adaptive group configurations to accommodate shape- and size- changed geometric information from cross-category data. Additionally, a vector-attention mech- anism (Zhao et al., 2021) is utilized to facilitate feature interaction of points within the resulting groups Then, in order to form unified model inputs, we leverage the known size and shape of the template target guide the generation of search regions. These search regions generated by cropping 3D point cloud scene at a scale that corresponds to the area occupied by the template tar- get. As result, the unified inputs maintain consistent proportion of foreground and background, effectively mitigating the impact on feature learning caused by variations in background distribu- tions across different object categories. Furthermore, we associate the predicted objects' offsets in 3D space with their length, width and height, thereby unifying model's learning objective and effectively enhancing the stability and generalization of the proposed unified models.
A comprehensive set of experiments are conducted on two challenging benchmarks, including KITTI (Geiger et al., 2012) and NuScenes (Caesar et al., 2020). The experimental results demon- strate that our category-unified models can allow for the tracking of all object categories using a single network. Moreover, it is important to note that our models not only exhibit stable gener- alization, but also outperform category-specific counterparts across all categories. Next, our key contributions can be summarized as: 1) We propose two category-unified models SiamCUT and MoCUT based on Siamese and motion- 3D SOT paradigms, respectively. To the best of our knowledge, this is the first work to unify tracking across all object categories. 2) We design a novel point set network, termed Fdrorr, which encodes geometric information of different object cat-
2

---

Published as a conference paper at ICLR 2024
egories in unified manner, enabling the proposed unified models to adapt cross-category data. In addition, we further introduce unified model inputs and learning objective, which facilitate the learning of unified representation and thereby enhancing the generalization and stability of our mod- els. 3) To verify our unified models, extensive experiments are conducted on KITTI and NuScenes datasets. Our SiamCUT and MoCUT demonstate excellent performance with strong generalization and training stability. Furthermore, given that our category-unified models perform better than exist- ing category-specifie models, we may enlighten the community to perform 3D SOT by cross-category training and testing instead of a per-category basis.
2
RELATED WORK
2.1
SNTEGORY-SPECIHER TRACKING
In recent years, object tracking has been divided into two categories: multiple object tracking (MOT) and single object tracking (SOT). Depending on different data patterns, tracking models can be further categorized into those based on 2D camera images or 3D LiDAR point clouds. 2D MOT aims to simullaneoussly estimate identities and bounding boxes of an indeterminate number of objects with pre-defined categories. Tracking-by-detectee 2022: 2023b) has witnessed significant success by associating detection results from consecutive frames. Nevertheless, owing to the variations in semantic information of different object categories, the detector and association components of the model cannot be shared across categories. Thus, 2D MOT models are inherently category-specific, requiring specific datasets. For instance, MOT17 (Milan et al., 2016) and MOT20 (Dendorfer al., 2020) tailored to track multiple pedestrians in crowded streets. 3D MOT (Zhu et al.. 2022: Ding et al., 2023) shares with 2D MOT a comparable task definition and model structure. Differently, 3D MOT is primarily geared towards tracking vehicles or pedestrians in autonomous driving road scenarios (Geiger et al., 2012; Caesar et al., 2020). 3D SOT focuses on tracking a single object in a point cloud sequence of continuous frames, where object is pre-categorized into predefined set of categories within relevant datasets like KITTI and NuScenes. Current methods (Giancola et al., 2019; Qi et al., 2020; Zheng et al., 2021; Shan et al., 2021; Hui et al., 2021; Zhou et al., 2022: Zheng et al., 2022a; Hui et al., 2022; Nie et al., 2023c;d;b) for 3D SOT typically adopt Siamese or motion-centric paradigms to perform tracking on per-category basis. Despite the demonstrated success, category-specife tracking results in redundant parameters. Therefore, unifying categories for tracking is a hot issue.
paradigm (Zhang et al., 2022; Wu et al., 2023a;
2.2 CATEGORY-UNIFIE TRACKING
Image-based based 2D SOT can be classified into the scope of generic object tracking (GOT) (Zheng et al., 2022b), which is designed to track an arbitrary object given in the initial frame of a video sequence, regardless of its category (Javed et al., 2022). In contrast to MOT, which relies the extraction of category-level semantic information for distinguishing object categories and subsequent data asso- ciation, 2D SOT predominantly adopts Siamese matching paradigm (Bertinetto al., 2016; Li et al., 2018; Zhang et al., 2020; Nie et al., 2022b;a; 2023a; Cui et al., 2022). Given the inherent advantage of Siamese representation networks that are free of learning category-specific semantic information, 2D SOT can effectively track arbitrary objects using a single model.
eategory-unifiee 3D SOT. Nevertheless, directly applying techniques (Chen et al., 2021; Xie et al.. 2022) from 2D SOT for category unifi- cation in 3D SOT poses significant challenge. In 2D SOT, objects are shaped into standardized size through scaling and resizing images, without impacting the color and texture information of the objects. In contrast, such normalization operations in 3D SOT seriously distort the geometric information of point clouds. Moreover, LiDAR point clouds are usually sparse, textureless, and se- mantically incomplete, making it difficult to encode generic features like highly-structured images.
Inspired by the concept of GOT, we target at developing
2.3 DYNAMIC GROUPING
To achieve category-uniffed tracking, the model is required to encode varying geometric proper- ties, such as size and shape of different object categories in a unified manner. However, existing point set networks (Qi et al., 2017a;b) utilized in 3D SOT face limitations attributed to the fixed receptive field resulting from size-fixed grouping. In the context of detection, some dynamic group- ing operations (Mao et al., 2021; Yang et al., 2022; Wang et al., 2022b;a) have been proposed. E.g., DBQ-SSD (Yang et al., 2022) assigns suitable receptive field for each point based on the correspond- ing point features. CAGroup3D (Wang al., 2022a) introduces class-aware local group strategy
3

---

Published as a conference paper at ICLR 2024
to capture object-level shape diversity. Motivated by these approaches, our AdaFormer integrates deformable group regression module to learn adaptive groups for diverse object categories.
METHODOLOGY
3.1 OVERVIEW
Task Definition, Given a category-known template target denoted as P )- along with its corresponding 3D bounding box (BBox) Be = (æt, Vt, 24, Wt, h4, L4, 0e) in the initial frame, 3D SOT aims to locate this object within search region p = {p:)e and yield a tracking BBox B, (I.: Vs. zg. 2s 0.) frame by frame. Here, N2 and N, denote the number of points for template and search region, respectively. (z, y, 2) and (w, I) represent the center coordinate and size, while 0 is the rotation angle around up-axis. Note that, due to the consistent size of given target across all frames, only 4 parameters (x. y, 2, () are required be predicted for Bg.
Revisiting Siamese and Metion-centric Paradigms. To predict the parameter set (ar, y, 2, 0), exist- ing approaches adopt either Siamese or motion-centric paradigms to design category-specific models that are trained using data corresponding to individual category. Siamese network based models typ- ically extract geometric features of template and search region using a Siamese backbone (Qi al., 2017b). The extracted features are then fused to generate seed points for subsequent target localiza- tion. The localization process is exccuted via 3D region proposal network (RPN), as introduced in VoteNet (Qi al., 2019). This RPN structure serves to offset the seed points towards the target's 3D center and predicts a rotation angle and a classification score for each point, thereby yielding M 3D proposals {< Mi, 2i, (.) and corresponding proposal-wise scores {s The pro- posal with the highest score is considered as the tracking result. On the other hand, motion-centric models directly concatenate the point clouds Pj-1 = [ from the previous frame with the
point clouds Pi = {pr N. set network. Áfter that, PointNet (Qi et al., 2017a) is employed to infer the relative movement (Art, Ayt, Azt, A0,) of the target to be tracked. In summary, both the Siamese and motion-centric paradigms adhere a general structure characterized as "feature extraction + point offsetting", which can be formulated as:
{p of the current frame, and then segment the foreground points by a point
(Iss Ye: Zz, 0,) = Sof miamoss/mmtiion (p (p1))))),
(1)
where F and O represent feature embedding and offset embedding, respectively.
Problem Description. In this paper, we propose to explore category-unified model for 3D SOT. However, a significant challenge emerges when attempting to train models directly using cross- category data. Our investigation reveals that the feature learning network F sisemssn/motion encoun- ters difficulty in adapting to diverse object categories with varying size and shape attributes. More- over, the inherent attributes variations of cross-category data are associated with distinct learning objectives, thereby posing distraction for the offset learning module Ooffset- Therefore, we view the category-unified tracking two fundamental sub-problems, i.e., the learning of unified feature embedding and a unified offset embedding.
Proposed Solution. To tackle these challenges, we first propose novel point set network (AdaFormer, Section 3.2), which effectively encodes shape- and size-changed geometric informa- tion from cross-category in unified manner. To facilitate the learning of unified feature embedding, we further design unified model inputs (Section 3.3), which ensures that the bregrrouddbacckgrone ratios within the input data remain invariant across different object categories. In addition, uni- fied learning objective (Section 3.4) is devised, including the consistent numerical distribution of predicted targets and the balanced distribution of positive and negative samples. To this end, we construct two category-unified tracking models, i.e., SiamCUT and MoCUT.
3.2
UNIFIED REPRESENTATION NETWORK: ADAFORMER
The overall architecture of our AdaFormer, as illustrated in Fig. 2, is constituted by a series of cascaded subsample operators and AdaFormer blocks. The core design is deformable group vector- attention, which aims to extract unified feature representation for cross-category objects. More concretely, it utilizes a group regression module to leam deformable groups to enable dynamically adaptive receptive fields, and then performs feature interaction of points within these deformable groups via pector-attention mechanism. The details presented below.
4

---

Published as a conference paper at ICLR 2024
N 3
Resure Emneeknng
Sabemmpl
A daForme Block
Sabssngg
daForrer Block
Sulbeamgh ple
A dbFormer Block
Input
Stage
Slape2
Stape3
Channel
Default Grougs
Deformable Groups
BatchNorm
Vidio Attentioon
Gooup Regresstion
Transform
K.
Group
Deformable Group
V setoo-AA tiention
Transfomation Matiis Parameters
Sunggte
BatchNorm
Center Sample (0000)
Figure 2: Overall architecture of AdaFormer. The proposed unified representation network shares a similar three-stage hierarchical structure with existing point set network (Qi al., 2017b) used in 3D SOT, consisting of series of subsample operators and AdaFormer blocks. Our representation network is empowered to learn dynamic groups through a deformable group vector-attenie sub- block, thereby enabling variable range of receptive fields.
Deformable Group Vector-A Attention Given input points {a [ Us. = = along with their corresponding features {f.)= we treat these points as 3D centers and generate a set of spheres with fixed radius r. which serve as default groups {9]]" To enable dynamically adaptive receptive fields for objects across different categories, we introduce a group regression module. This module regards the default groups references and applies projection transformation each group, resulting in deformable groups (g J== with varying sizes and shapes. Specifically, the projection transformation is defined as a combination of scaling and rotation along the zyz axis, involving a scaling matrix Tg and three rotation matrices Tru: Tr, and T,,:
Sy
0
T,
0
8y
Tr
cos O
sin O2 cos 0r
0
0 sin O₂
cos fg
0
sin By
cos 0z
sin 0=
T,
0
1
Tr.
sin O2
cos Og 0
sin Oy
0
cos Og
where the scaling parameters Sy. and the rotation parameters oz, fy. 6g need to be pre- dicted create transformation matrices. Towards this goal, our group regression module predicts 6-dimensional vectors (4: [Str, 1 82, Oz,, Oy. 12 for the default groups (9] E.g. the vector 1, for j-th group 9j can calculated as:
t, MLP (Averagggggoo (fff:
[fac]e
where le is the number points gathered within the j-th group 9j from the previous layer, and ty learned based on prior information of the features and coordinates of the points within this group. To this end, final transformation matrix calculated by sequentially multiplying the derived scaling matrix and rotation matrices:
T= Tr, X Tr, X Tr.
(4)
By using Eq. 4, a series of transformation matrices (T; )2_ are generated. For the j-th group 9j. we project the coordinates of all input points {c: = [xi, Vi, 2:] a to new 3D coordinate space using j-th transformation matrix Ty:
y T, x
Mi, (I,,
Afterwards, a unit sphere is used to gather & points that are closest to the center coordinate [xj, My zd within this sphere, thereby forming a new group 9 By this way, we obtain a series of deformable groups (g y 1= which offer dynamic receptive fields tailored to diverse object categories.
5

---

Published as a conference paper at ICLR 2024
Pointclood scene
Pereground point
Background point
Podcstaa
Poderitan
(a) Model imputs of existng methods
(b) Our unifted model tmpurs
Figure 3: Comparison between different search regions, i.e, model inputs. The previous methods (a) generate search regions by expanding the predicted result of previous frame by fixed 3D distance, while our method (b) expands by a scale to the width, height and length target objects.
To form a unified feature representation, we utilize vector-attention to facilitate feature learning and interaction of points within each group g Unlike traditional self-attention, vector-attention not only preserves the permutation invariance of point clouds but also effectively models both channel and spatial information interactions. In detail, we regard the features of center point within g as query, while the features of other points as key and value. Additionally, position embedding is obtained by:
Pj = uplcs (c)) E g
(6)
where up represents a two-layer MLP. Leveraging the query, key, value and position embedding, the process of feature learning and interaction for gffef can be mathematically expressed as:
f100 VA(gj, D)) Softmax(p((s,(4)) we wk(kky +D))) (w)) + D2)
where Wg. Wh, We and y denote linear layers. The original features fj are transformed into new features fyae by integrating information within the deformable receptive field.
3.3 UNIFIED MODEL INPUT
In the tracking procedure, to ensure that the search area of cach frame contains the target object to be tracked while avoiding the inclusion of excessive irrelevant background information, a common practice in existing methods (Qi et al., 2020; Zheng et al., 2022a; Nie et al., 2023b) is expand the area occupied by the predicted result of the previous frame by a certain distance to generate search regions. Consequently, representation network needs simultaneously learn geometric informa- tion of objects and distinguish between foreground and background points. In fact, background information inevitably introduced during tracking. However, such a practice leads to inconsistent foreground- foreground-backgroundd ratios of search regions, especially for different categories of objects. As shown in 3 (a), relatively large car and relatively small pedestrian exhibit significantly different background information, which distracts the representation network.
similar issue occurs with single object tracking based on 2D images. To generate unified model inputs for performing generic object tracking arbitrary objects, 2D SOT methods (Bertinetto et al., 2016; Li et al., 2018; Chen et al., 2021) typically obtain the region of interest by adding a consistent margin context p centered on the target object. Subsequently, the region is resized to constant rectangle A² using a resizing operator s:
A = s(w + p) s(h +
where w and h denote the width and height of the target object. The amount of margin context 2" For point cloud tracking, however, employing similar operators will distort ge- ometry structure of point clouds. Fortunately, benefiting from the proposed representation network AdaFormer is able adapt to changing shape and size information. Therefore, instead of focusing on the amount of background context, we aim to unify the -arerouuddbbaakoouudd ratios. To achieve this, we expand the area of the predicted result of the previous frame by adding a scale of width, height and length to its size:
p is set with
X (w + a w) x (h + a h) x (1)
where denotes the scale factor, and we can form unified model inputs with consistent foreground- background ratios, as shown in Fig. 3 (b). This eliminates interference with the learning of our unified representation network.

---

Published as a conference paper at ICLR 2024
3.4 UNIFIED LEARNING OBJECTIVE
In addition to achieving unified feature representation through our proposed AdaFormer network, unified learning objective also matters for sategory-unifen models. As illustrated in Eq. I, the track- ing model involves point offsets. However, different object categories possess distinct prediction targets and exhibit varied positive-negative sample distributions, which prevents a category-unified model from effectively tracking all categories of objects.
To address these problems, we propose to develop a consistent numerical distribution of predicted targets, as well a balanced distribution of positive and negative samples. Specifically, we first correlate the specific offset values along zyz axis in 3D space with the width w, height h and length I of the target object:
[Ax, Ay, A=] = [u. h] [ae le'.y',2) [2']
(10)
where wo h and I are known, given by the template target in the initial frame, and the prediction targets [Ar, Ay, A=] are transformed into [s'.v'= = As a result, we obtain the unified numerical distribution of prediction targets, i.e., offsets for objects across different categories.
Then, to balance positive and negative samples and enable a unified aesitive-negativy sample distri- bution during the training phase, we introduce shape-aware labels. Different from existing methods that define points at a certain distance from the target center as positive samples and other points as negative samples, causing an imbalanced distribution of positive and negative samples for cross- categories data, our method defines the points within a cube scaled of the target object as positive samples:
if ya. yo: 24) E R B0xx otherwise, c { O
(11)
where 3 is the scale factor. Combining these two designs, we achieve unified learning objective, which facilitates the training of our eategory-unified models.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTING
Datasets. To evaluate our category-unified models, we utilize two popular datasets KITTI (Geiger et al., 2012) and NuScenes (Caesar et al., 2020) to conduct comprehensive experiments. KITTI contains 21 training sequenees and 29 test sequences. Due to the inaccessibility of the test labels, split the training sequences into training and testing sets following previous works (Giancola et al., 2019; Zheng et al., 2022a). NuScenes is a more challenging and large-scale dataset, which contains 700 and 150 scenes for training and testing, respectively.
Evaluation Metrics. Following common practice in previous methods (Giancola et al., 2019; Qi et al., 2020), we measure Success and Precision metrics in One Pass Evaluction (OPE) manner (Wu et al., 2013) to evaluate the tracker. Success calculates the intersection over union (IOU) between the predicted BBox and the ground truth BBox, while Precision calculates the distance between the centers of the two BBoxes.
Implementation Details. Here, propose unified representation network, model inputs and learning objective. To construct category-unified models, we integrate these three components into existing classic trackers P2B (Qi et al., 2020) and M (Zheng al., 2022a), following Siamese and motion-centrie paradigms. To be fair comparison, we use consistent parameters with setting of P2B and M2Track, including model epprr-paraeeters and training parameters. In addition, the scale factors and / brought by our methods are set to 1.0 and 0.4, respectively. All experiments are conducted using these parameters if not specified.
4.2 COMPARISON WITH STATE-OF-THE-AA METHODS.
Results on KITTI. Tab. compares the proposed category- unified methods with other state-of-the art sategory-specific methods on the KITTI dataset. Our unified models exhibit excellent tracking performance, confirming the feasibility and potential of category-unified model design. MoCUT achieves average Success and Precision of 65.8% and 85.0% by using a single model, only slightly lower than the latest CXTrack. In addition, compared to the baseline methods P2B and M Track, our SiamCUT and MoCUT not only significantly reduce the number of parameters, but also effec- tively track objects of different categories using single network. Moreover, we obtain improved
7

---

Published as a conference paper at ICLR 2024
performance across all categories, attributed to the proposed unified components. Notably, the per- formance improvements for categories with relatively small data, including Pedestrian, Van and Cyclist, higher than Car category with relatively large data. This is due the inherent ben- efit of cross-category training, where an increase training sample size will contribute to enhanced performance.
Table 1: Performance comparisons with ofate-of-the-art methods KITTI dataset. Bold and underline represent our category-unitie methods and the corresponding baselines. " "and" " refer to Siamese and motion-centric paradigms, respectively.
Method
Source
Car [6,424]
Pedestrian [6,088]
Van [1,248]
Cyclist [308]
Mean (14.068)
SC3D (Giancola al. 2019)
CVPR2019
41.3/57.9
18.2/37.8
40.4/47..
41.5 47.5
31.2/48.5
P2B (Qi et al., 2020)
CVPR2020
8
28.7 49.6
40.8 48.4
12.1 / 44.7
42.4 60.0
MLVSNet (Wang al., 2021)
ICCV2021
56.0/74.0
34.1 /61.1
52.0/61.4
34.4/44.5
45.7 / 66.6
PIT (Shan et al., 2021)
IROS2021
67.8/81.8
44.9/720
43.6/
37.2/47.
55.1 74.2
V2B (Hui at. 2021)
NeurIS20021
70,5/813
48.3/73.5
50.1 /
40.8 40.8/49.7
58.4/75.2
PTTR (Zhou et 2022)
CVPR2022
65.2/77.4
50.9/8166
52.5/61.8
65.1 /90.5
57.9 78.2
TAT (Lan et 2022)
ACCV2022
72.2/83.3
57.4/84.4
69.2/69.2
74.2/93.5
64.7 82,8
STNer (Hui al. 2022)
ECCV2022
72.1/84.0
49.9/77.2
58.0/
/335/93.7
61.3/ 80.1
DMT (Xia et 2023)
T-ITS2023
66.4/79.4
4
53.3/65.6
79.4
55.1 75.8
GLTT
2023c)
AAAI2023
6822/2211
52.478.88
52.6/62.9
68.9/92.1
79.3
OSP2B (Nic ot 2023b)
LICA12023
67.5/82.3
53.6/85.1
56.3/66.2
65.6/90.5
60.5 82.3
CXTrack (Xu al., 2023)
CVPR2023
69.1/81.6
67.0/915
60.0/7188
74.2/94.3
67.5 85.3
Sync Track (Ma ct 2023)
ICCV2023
78.3
54.7
60.37
73.1 /93.8
64.1 / 81.9
M (Zheng el al., 2022a)
CVPR2022
65.5 880.8
61.5 /8.2
53.8 /3.7
23.2 /93.5
83.4
SianCUT
Ours
58.1/73.9
48.2/76.3
63.1
36.7 / /47.4
$4.0 74.6
MoCUT
Ours
67.6/80.5
63.3/90.0
76.7
85.0
Results on NuScenes. Tab. 2 presents the comparison results on the NuScenes dataset, which pro- vides large-scale and more challenging scenes for 3D SOT. Our methods demonstrate consistent per- formance advantages, when the benefit of cross-category training is absent, owing to the large-scale data in most categories, which further validates the effectiveness of our proposed category- models. Additionly, we observe that SiamCUT shows a larger performance improvement compared to the baseline than MoCUT, in both the KITTI and NuScenes datasets. This phenomenon can be attributed to the fact that the Siamese paradigm relies on sophisticated shape and size information embedding modules for tracking. while the motion-centri paradigm directly infers relative dis- placement between two frames. The latter is relatively simpler and less susceptible to the changes in shape and size information when dealing with cross-category data.
Table 2: Performance comparisons with state-of-the-an methods nuScenes dataset. Bold and underline represent our category-unified methods and the corresponding basclines.
Method
Car [54]
Padlestrian [33,227]
Truck [13.587]
Thailer [3,352]
Bus [2,953]
Mean [1 [1,278)
SC3D (Giancola et al., 2019)
22.31/2135
11.29 12.65
30.67/ 27.73
35.28 / 28.12
29.35 24.08
20.70/ 22.70/22..0 22.20
P2B al., 2020)
38.81 18
28.39 / 52.24
42.95 41.59
4 /0.00
32.45/27.41
36.48 /45.08
PTT al., 2021)
41.22/ 45.26
19.33/ 32.03
50.23/ 48.56
41.70/46.50
39.40/ 36.70
36.33/410
PITR (Zhou al., 2022)
51.89/ / /8.61
29.90/ 45.09
45.30 44.74
45.87/ 38.36
43.14/37.74
44,50/ 44.50/52.07
GLT-T (Nic al., 2023c)
48.52/544. 54.29
31.74/ 36.49
52.74/51.43
52.00
44.55/ 40.69
44.42/ 54.33
MTTreck (Zheng al. 2022a)
53.85 / 65.09
32.10/ 60.72
$7.36/ 59.54
57.61 / 58.26
51.39/51.44 51.44
49.32 62.73
SianCUT (Ouns)
40.96 44.91
31.42/ 53.80
53,91/ 52.65
63.29 63.29/58.00 58.21
41.03/ 38.01
40.41/ 48.54
MoCUT (Ours)
57.32/ 66.01
33.47 / 63.12
61.75 / 64.38
60,90 / 61.84
57.39 / 56.07
51.19/64.66
4.3 ABLATION STUDIES
In this section, we present a series of ablation studies, including the proposed components within our unified models and the analysis of generalization and stability. Following previous works (Qi et al., 2020; Zheng et al., 2022a), all ablated experiments are conducted on the KITTI dataset.
Components of Unified Models. To give better understanding of our unified category- unified models, we investigate the impact of the designed unified components to the tracking performance. As reported in Tab. 3, removing any component will lead to a significant decrease in average per- formance for the Siamese paradigm. In addition, for the motion-centric paradigm, although it has a simpler structure and does not overly rely on size and shape information for tracking, our uni- fied components still effectively enable the model to uniformly process cross-category data, thus enhancing performance. More detailed analysis of these components can be referred to Appendix.
Generalization and Stability Analysis. As illustrated in Fig. 4. the proposed unified components enable the model to learn generalizable features by explicitly encoding distinct shape and size infor- mation in unified manner, thus guiding performance improvements across all categories. Besides,
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation studies for different components of Siamese and motion-centric paradigms KITTI dataset. The last rows represent the full category-unifie models.
Unified Representation
Learinng Objective
Padestrian [6,088]
Cyelist [308]
Paradigm
Network: AdaFormer
Model Inputs
[6,424]
[1.248]
[14,068]
56.3/ 72.4 72.4
33.2/60.4
57.0/68.6
42.2/43.5
/
Siamese
73.9
38.0/66.7
34.1/44.1 61.6/7288
49.1 70.1
57.5 / 72.8
44.5/
61.2/ 70.4
35.6/ 44.5
71.7
58.1 73.9
48.2 / 76.2
63.1/740
36.7 / 47.4
54.01 74.6
65.2/78..
61.9/88. 88.6
59.5 74.4
74.1 92.7
63.5 83.0
Motion
67.5 80.4
63.07 89.3
63.8 77.9
76.6/ 83.4
65.5 84.2
66.9 80.1
63.2 88.7
62.4/77.0
74.31 93.0
65.1
67.6 80.5
63.3 / 90.0
64.5 78.8
76.7 / 94.2
65.8 85.0
Sasue Ruai
Utalked
12ach
Egeck
Figure 4: Generalization and stability comparisons of with and without the proposed unified com- ponents. We plot "performance v.s. epoch" curves on four categories, and include error bands calculated by running the corresponding experiments three times using different random seeds.
we observe that the blue curves and error bands after 20 epochs (the model tend to converge after 20 epochs), show relatively large jitter in the case of "without unified components". This instability ascribed to the fact that the model is disturbed by cross-category data. Fortunately, our unified components alleviate this instability to some extent.
4.4 INFERENCE SPEED
Computational efficiency is critical aspect in practical applications. To evaluate the tracking speed of our proposed category-unifice models, we adopt a standard implementation (Qi et al., 2020; Nie et al., 2023c) and calculate the tracking speed by counting the average running time of all frames in the Car category of the KITTI dataset. SiamCUT and MoCUT run at real-time speed of 36.4 and 47.9 frame per second (Fps), respectively single NVIDIA 3070Ti GPU, including 7.2/8.6 for point cloud preprocessing, 19.8/12.0 ms for network forward computation, and 0.5/0.3 ms for post-processing.
5 CONCLUSION
We propose SiamCUT and MoCUT, two category-unified models to address 3D single object track- ing (SOT) on point clouds task. For the first time, our methods achieve the unification network architecture and cross-category data training and testing, allowing to simultaneously track objects across all categories using a single network with shared parameters. Extensive experiments demon- strate that both SiamCUT and MoCUT perform better than existing category- counterparts on two challenging KITTI and NuScenes datasets. We hope our work will inspire further exploration of dategory-unifier tracking architecture for the 3D SOT task.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (*MSIT) (0188IAA577055499 and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government Artificial Intelligence Graduate School Program (Hanyang University) ). *Ministry of Science and ICT. This work was supported by the National Natural Science Foun- dation of China under Grant 62376080, the Zhejiang Provincial Major Research and Development Project of China under Grant 2023C01242, the Central Guiding Local Science and Technology De- velopment Fund Projects of China under Grant 2023ZY 1008, and the Zhejiang Provincial Key Lab of Equipment Electronics.
(MSIT)
REFERENCES
Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Tomr. Fully- convolutional siamese networks for object tracking. In Computer Aaion-ECC'S 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14, pp. 850- 865. Springer, 2016.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-1631, 2020.
Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer track- ing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8126-8135, 2021.
Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13608 13618, 2022.
Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé. Mor20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003: 2020.
Shuxiao Ding, Eike Rehder, Lukas Schneider, Marius Cordts, and Juergen Gall. 3dmotformer: Graph transformer for online 3d multi-object tracking. arXiv preprint arXiv:2308.00 06635, 2023.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 0364-336. IEEE, 2012.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung- Yi Lin. Multi-task self- self-training for learning general representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8856-8865, 2021.
Silvio Giancola, Jesus Zarzar, and Bernard Ghanem. Leveraging shape completion for 3d siamese tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1359-1368. 2019.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16102-16112, 2022.
Le Hui, Lingpeng Wang, Mingmei Cheng, Jin Xie, and Jian Yang. 3d siamese voxel-to-bev tracker for sparse point clouds. Advances in Neural Information Processing Systems, 34:28714-28777 2021.
Le Hui, Lingpeng Wang, Linghua Tang, Kaihao Lan, Jin Xie, and Jian Yang. 3d siamese trans- former network for single object tracking point clouds. In Computer Vision-ECCS 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part II, pp. 293-310. Springer, 2022.
10

---

Published as a conference paper at ICLR 2024
Sajid Javed, Martin Danelljan, Fahad Shahbaz Khan, Muhammad Haris Khan, Michael Felsberg, and Jiri Matas. Visual object tracking with discriminative filters and siamese networks: a survey and outlook. IEEE Transactions Pattemn Analysis and Machine Intelligence, 2022.
Kaihao Lan, Haobo Jiang, and Jin Xie. Temporal- -awar aware siamese tracker: Integrate temporal context for 3d object tracking. In Proceedings of the Asian Conference on Computer Vision, pp. 399-414 2022.
Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with stamese region proposal network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8971-8980 2018.
Xudong Lv, Boya Wang. Ziwen Dou, Dong Ye, and Shuo Wang. Leenet: Lidar and camera self- calibration using cost volume network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2894 2901, June 2021.
Teli Ma, Mengmeng Wang, Jimin Xiao, Huifeng Wu, and Yong Liu. Synchronize feature ex- tracting and matching: A single branch framework for 3d object tracking. arXiv preprint arXiv:2308. 12549, 2023.
Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid r- cnn: Towards better performance and adaptability for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2723-2732, 2021.
Anton Milan, Laura Leal-Taixé, lan Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark 2016.
for multi-object tracking. arXiv preprint
Jiahao Nie, Zhiwei He, Yuxiang Yang, Mingyu Gao, and Zhekang Dong. Learning localization- aware target confidence for siamese visual tracking. IEEE Transactions on Multimedia, 2022a.
Jiahao Nie, Han Wu, Zhiwei He, Mingyu Gao, and Zhekang Dong. Spreading fine fine-grained prior knowledge for accurate tracking. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 3269))61 2022b.
Jiahao Nie, Zhekang Dong, Zhiwei He, Han Wu, and Mingyu Gao. Faml-rt: Feature alignment- based multi-level similarity metric learning network for two-stage robust tracker. Information Sciences, 632:529-542, 2023a.
Jiahao Nie, Zhiwei He, Yuxiang Yang. Zhengyi Bao, Mingyu Gao, and Jing Zhang. Osp2b: One- stage point-to-box network for 3d siamese tracking. In Proceedings of the Thirty-Second Interna- fional Joint Conference on Artificial Intelligence, pp. 1285-1293, 2023b.
Jiahao Nie. Zhiwei He, Yuxiang Yang. Mingyu Gao, and Jing Zhang. Git-t: Global-local transformer voting for 3d single object tracking in point clouds. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1957-1965, 2023c.
Jiahao Nie, Zhiwei He, Yuxiang Yang, Xudong Lv, Mingyu Gao, and Jing Zhang. Glt-t++: Global- local transformer for 3d siamese tracking with ranking loss. arXiv preprint arX1v:2304.00242, 2023d.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277-9286, 2019.
Charles Ruizhongtai Qi, Li Yi, Hao Su. and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
11

---

Published as a conference paper at ICLR 2024
Haozhe Qi, Chen Feng, Zhiguo Cao, Feng Zhao, and Yang Xiao. P2b: foint-to-box network for 3d object tracking in point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6329-63233 6338, 2020.
Jiayao Shan, Sifan Zhou, Zheng Fang, and Yubo Cui. Ptt: single object tracking in point clouds. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1310-1316. IEEE, 2021.
module for 3d
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li. Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. Advances in Neural Information Processing Systems, 35:29975- 2022a.
Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1110-1119, 2022b.
Pan Wang, Liangliang Ren, Shengkai Wu, Jinrong Yang, En Yu, Hangcheng Yu, and Xiaoping Li. Implicit and efficient point cloud completion for 3d single object tracking. IEEE Robotics and Automation Letters, 8(4):1935-1942, 2023.
Zhoutao Wang, Qian Xie, Yu-Kun Lai, Jing Wu, Kun Long, and Jun Wang. Mlysnet: Multi-level voting siamese network for 3d visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1001-110, 2021.
Han Wu, Jiahao Nie, Zhiwei He, Ziming Zhu, and Mingyu Gao. One-shot multiple object tracking in uav videos using specifci ine-grained features. Remote Sensing, 14(16):3853, 2022.
Han Wu, Jiahao Nie, Ziming Zhu, Zhiwei He, and Mingyu Gao. Learning task-specific discrimi- native representations for multiple object tracking. Neural Computing and Applications, 35(10): 7761-7777. 2023a.
Han Wu, Jiahao Nie, Ziming Zhu, Zhiwei He, and Mingyu Gao. Leveraging temporal-awan fine- grained features for robust multiple object tracking. The Journal of Supercomputing. 79(3):2910- 2931, 2023b.
Yi Wu, Jongwoo Lim, and Ming- Hsuan Yang. Online object tracking: A benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2411 1411-24188 2013.
Ming-Hssun
Yan Xia, Qiangqiang Wu, Wei Li, Antoni B Chan, and Uwe Stilla. A lightweight and detector- free 3d single object tracker on point clouds. IEEE Transactions on Intelligent Transportation Systems, 2023.
Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, and Wenjun Zeng. Correlation- aware deep tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition (CVPR), pp. 8751-8760, June 2022.
Fei Xie, Lei Chu, Jiahao Li, Yan Lu, and Chao Ma. Videotrack: Learning to track objects via video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22826-22835, June 2023.
Tian-Xing Xu, Yuan-Chen Guo, Yu-Kun Lai, and Song-Hai Zhang. Cxtrack: Improving 3d point cloud tracking with contextual information. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1084-1093, June 2023.
Jinrong Yang, Lin Song, Songtao Liu, Weixin Mao, Zeming Li, Xiaoping Li. Hongbin Sun, Jian Sun, and Nanning Zheng. ssq Dynamic ball query for efficient 3d object detection. arXiv preprint arXiv:2207.10940 2022.
12

---

Published as a conference paper at ICLR 2024
Jing Zhang and Dacheng Tao. Empowering things with intelligence: survey of the progress, challenges, and opportunities in artificial intelligence of things. IEEE Internet of Things Journal, 3(10):779-78177, 2020.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In European Conference on Computer Vision, pp. 1-21. Springer, 2022.
Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor-free tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 771-787. Springer, 2020.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 16259-16268, 2021.
Chaoda Zheng, Xu Yan, Jiantao Gao, Weibing Zhao, Wei Zhang, Zhen Li, and Shuguang Cui. Box- aware feature enhancement for single object tracking on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13199-13208, 2021.
Chaoda Zheng, Xu Yan, Haiming Zhang, Baoyuan Wang, Shenghui Cheng, Shuguang Cui, and Zhen Li. Beyond 3d siamese tracking: A motion-centric paradigm for 3d single object tracking in point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3111-8120, 2022a.
Chaoda Zheng, Xu Yan, Haiming Zhang, Baoyuan Wang, Shenghui Cheng, Shuguang Cui, and Zhen Li. Beyond 3d siamese tracking: motion-centric paradigm for 3d single object tracking in point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8111-8111 8111-8120, June 2022b.
Changqing Zhou, Zhipeng Luo, Yueru Luo, Tianrui Liu, Liang Pan, Zhongang Cai, Haiyu Zhao, and Shijian Lu. Pttr: Relational 3d point cloud object tracking with transformer. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8531-8540, 2022.
Ziming Zhu, Jiahao Nie, Han Wu, Zhiwei He, and Mingyu Gao. Msa-mot: Multi-stage association for 3d multimodality multi-object tracking. Sensors, 22(22):8650, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A
MORE IMPLEMENTATION DETAILS
Model Architecture The presented vategory-unifenn models SiamCUT and MoCUT are built upon P2B (Qi et al., 2020) and MrTrack (Zheng et al., 2022a) frameworks, respectively. We employ the proposed AdaFormer the feature extraction network for both models. In addition, unified model in- puts and learning objectives are integrated into the models, introducing no additional computational overhead during the inference phase. The remaining components of our category-unifie uniifed models adhere to the structures utilized in P2B and MrTrack.
Training & Inference. Our category-unified models are trained end-to-end manner, using a Tesla A A10 GPU. Following common practice (Qi et al., 2020; Zheng et al., 2022a; Hui et al., 2022; Xu et al., 2023), we conduct separate training and testing on the KITTI (Geiger et al., 2012) and NuScenes (Caesar et al., 2020) datasets. To ensure a fair comparison, training sets of all categories within a specific dataset and then tested on the corresponding test sets. We further perform Dataset (Sun et al., 2020). The training and inference hyper-parameter for SiamCUT and MoCUT remain consistent with P2B and MaTrack, such optimizer, learning rate, and batch size.
train models the
evaluation of the model pre-trained from KITTI on the Waymo Open
B MORE ANALYSIS AND EXPERIMENTS
Here, we first introduce the motivation of our category-unified models in Section B.1. Then we provide the detailed experiments and analysis of the proposed components within our category- unified models, including unified representation network AdaFormer, model inputs and learning objective in Section B.2. These experiments are conducted using Siamese paradigm (Qi et al., 2020) on the KITTI (Geiger al., 2012) dataset. Finally, we present state-of- the-art comparison Waymo Open Dataset (Sun et al., 2020) and computational cost analysis in Section B.3 and B.4.
B.1 MOTIVATION ANALYSIS
Our goal is to develop a category-agnostc model capable of tracking any object regardless of its category. However, different object categories possess distinct attributes, such shape, size, and motion state, which pose challenges to achieving category unification. As shown in Fig. 5, we vi- sualize the length, width, and relative motion (Az, Ay, 2 Ad) of two adjacent frames of all data on Car and Pedestrian category from KITTI (Geiger et al., 2012) dataset. It can be observed that cars exhibit much larger sizes than pedestrians, with different length-to-wwi ratios. Moreover, the motion states of cars and pedestrians display significant differences, particularly in the ryy-llane dis- placement. In addition, we visualize the distribution of background distractors around pedestrians and vehicles, as shown in Fig. 6. The density of distractors surrounding pedestrians is consider- ably higher than that around vehicles. Considering these factors, we think that handling objects of different categories with diverse sizes, shapes, and motion states uniformly is pivotal for realizing category- category-unified unified tracking.
10
1.50
1.00
-10
-15
0.25
Lengh
Figure 5: Numerical statistics of object's length, width and relative motion (Ar, Ay, Az, A0) of two adjacent frames Car and Pedestrian categories from KITTI dataset.
14

---

Published as a conference paper at ICLR 2024
Datractors on Car
Dittnctors on Podestiinn
Intra-distractors on Car
Intra distratton on Pedestriam
1195
22.3%
78.7%
IL.%%
93.55
15.0%
Disuctioo Leadl
1.0%
118%
10.0%
12.0
45%%
Figure 6: Numerical statistics of distractors and intra-class distractors on Car and Pedestrian cat- egories from KITTI dataset. We plot four quantity levels: 0, 1, 2, and 23, with respect to the proportion of the corresponding data size.
B.2 DETAILED EXPERIMENT AND ANALYSIS UNIFIED COMPONENTS
Unified Representation Network: AdaFormer. Our unified representation network AdaFormer incorporates a group regression module for learning deformable groups to enable adaptive receptive fields for various object categories, along with a vector-attention mechanism facilitate feature interaction of points within these deformable groups, ultimately forming a category-unified feature representation. Tab. presents an ablation study understand the two sub-components. Benefit- ing from the adaptive receptive fields achieved by the group regression module, our representation network can learn geometric information of various object categories in a unified manner. Conse- quently, when this module is removed, average performance drops by 6.9% and 7.0% in terms of Success and Precision, respectively. It's noteworthy that the most obvious performance degradation occurs in the Pedestrian category. This is due to the relatively small training samples for the Pedes- trian category and the significant differences in shape and size compared to other object categories. To visually understand of how the group regression module works, we provide some visualizations of deformable groups on the Car and Pedestrian categories, as shown in Fig. 7. In addition, when removing the vector-attention mechanism, we employ a feature propagation operator in existing backbone network (Qi et al., 201 7a;b) to substitute it. Tab. 4 demonstrates that the vector-attention mechanism plays a crucial role in promoting the learning of a unified feature representation.
Fixed groups for pedeestraa
Fixed groups for car
0.25
1.0
4004
0.5
0.0
2.30
-10
0.75
-1.0
-0.5
04
Sa La 20 0.4
0.5
0.2
Deformable groups for podestria
Deformable groups for car
1.00
035
5.50
16
4.00
D.
6.
-0.
4.50
-1.0
+5.5
-1.0
3.6
0.5
0.
Figure 7: Comparison of groups on the Car and Pedestrian categories. We plot four fixed groups and deformable groups from the first layer in PointNet++ (Qi et al., 2017b) and our AdaFormer network, respectively, using different colors.
15

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study of unified representation network. Success Precision are used for evalua- tion. Bold denote the best performance.
Group Regression Module
Vector Attention Mechanism
Pedestrian [6,088]
Van [1,248]
Cyclist [308]
[6,424]
[14,068]
X
56.3 72.4
33.2/ 60.4
57.0/ 68.6
32.2/43.5
45.9/6322
56.5 72.7
62.2
59.4
32.3/44.0
67.1
57.7 73.8
44.9/71.2
618/72.0
35.3/46.1
52.1/ 72.0
73.9
48.2/76.2
63.1/749
36.7/47.4
Unified Model Input. The scale factor a is an important hyper-parameter our unified model inputs. Hence, we conduct an ablation experiment using different values to determine the optimal setting for this parameter. As presented in Tab. 5, our method is not sensitive to the scale factor within reasonable range of values, i.e., when this parameter is set in the range from 0.8 to 1.4. Nevertheless, excessively large value will introduce noise, whereas overly small value will ignore valuable information, both leading to significant performance degradation.
Table 5: Performance using different scale factor a. Success Precision are used for evaluation.
Bold denote the best performance.
Scale Factor
Car [6,424]
Pedestrian [6,088]
Van [1,248]
Cyclist [308]
Mean [14,068)
0.6
/ 68.1
42.3 70.3
48.9 57.4
31,2/43.0
/7.66
$7.5 73.2
48.4/76.7
63.0 74.7
47.2/45.0
53.7/74.3
73.9
48.2/76.2
63.1/74.9
36.7 44.4
54.0
1.2
38.3/73.77
48.0/76.1
62.8/ 74.7
36.1 44666
53.8 74.2
1.4
73.1
75.6
62.0/
35.0
53.3 73.66
1.6
55.8
32.4/
56.4/ 66.2
30.2/42.5
45.2 64.2
Car Offect Pedestiian Offer
Car Offset Pedesician clisat
0.30
0.25
0.20
0.82 7
-025
0.25
0.25
0.50
2.50
-075
-0.5
0.
2.0
Un fied Car Offeet
Unfiled Ca- Offect
Un ried Rediedr an Offset
Unit Pedestria Offer:
unflied Pedeale an Othe:
0.4
0.
Z
2.0
00
-02
-04
-2.6
-28
0.25
-1.0
0.5
-3.0
-10
22
0.5
-0.5
X
1.0
0.0
55
1.00
2.35
1.0
Figure 8: Comparison of offsets between Car and Pedestrian categories. The upper and lower rows represent the cases of without and with unified prediction target design, respectively.
Unified Learning Objective. The unified learning objective involves a consistent numerical distri- bution of predicted targets and balanced distribution of positive and negative samples. To investi- gate their contributions, we conduct ablation experiments and report the ablation results in Tab, 6. Firstly, illustrated in the upper row of Fig. 8, different object categories exhibit significant varia- tions in offset targets, distracting the model. However, by unifying the offset targets across the three coordinate axes zyz based on length, width, and height information (as shown in the lower row), the offset targets of diverse object categories converge within a common numerical space, thereby resulting in improved performance.
16

---

Published as a conference paper at ICLR 2024
In addition, employ shape-aware labels to define positive and negative samples, which further enhances the tracking performance by 1.2% and 1.4% in average Success and Precision, as shown in Tab. 6. The scale factor 3 controls the uniform ratio of positive and negative samples. When the parameter value is set too small, it leads to a scarcity of positive samples, especially at the beginning of training, making it difficult for the model to converge. Conversely, setting this value too large results in an overabundance of positive samples, posing challenge for the model to distinguish the most accurate ones. Therefore, we further conduct an ablation experiment to determine the optimal value for this parameter. According to Tab. 7, we set the scale factor 33 to 0.4 in our main experiment.
Table 6:
Ablation study of unified learning objective. Bold denote the best performance.
Success / Precision are used for evaluation.
Unified Prediction Target
Unified Positive Negaative Sample
Car [6.424]
Pedesirian [6,088]
Van [1,248]
Cyclist [308]
Mean [14,068]
x
57.5
44.5 44.5/ 72.1
61.2/ 70.4
35.6/44.5
51.8/ 71.7
4 57.6 77.6
61.47 70.8
35.8/44.9
32.1/72.0
78.1/73.7
4
62.8 74.4
35.9/46.6
52.8 73.2
58.1 733.9
48.2 /
63.1 / 74.9
36.7/47.4
54.0 / 74.6
Table 7:
Performance using different scale factor B. Success Precision are used for evaluation. Bold denote the best performance.
Scale Factor 3
Car [6,424]
Pedestrian [6,088]
Van [1,248]
Cyclist [308]
Mean [14,068]
0.2
$1.4 65.8
26.6 46.9
38.4 445.0
49.6 41.8
39.1 55.3
55.2 71.1
34.1/61.3
48.2
31.8 44.3
45.0/ 65.1
0,4
58.1 73,9
48.2 76.2
63.1/74.9
36.7 /47.4
54.0 / 74.6
0.5
58.3 74.1
4
65.7/
/7.3
54.0 74.5
72.6
44.2/73.1
68.8/67.7
34.7 /4.3
50.2/71.8
67.6/67.0
/9.1
/5.3/
/
46.7 66.2
B.3 RESULTS ON WAYMO OPEN DATASET
To validate the generalization ability of our proposed category-unifice models. we conduct an eval- uation by applying the models trained on KITTI dataset to the Waymo Open Dataset (WOD) (Sun et al., 2020). We select six state-of-the-an methods that have reported performance on WOD for comparison. Different from existing methods like CXTrack (Xu et al., 2023) and T'Track (Zheng et al., 2022a) require pre-trained models on both Car and Pedestrian categories from KITTI for tracking vehicle and pedestrian on WOD, our SiamCUT and MoCUT achieve category unification. Therefore, a single model is capable of simultaneously tracking both vehicles and pedestrians. As reported in Tab. 8, our MoCUT consistently outperforms other methods across all subsets, including easy, medium, and hard subsets, for both Vehicle and Pedestrian categories. Compared to baseline category-speee specific methods P2B and Mrackk, formance, achieving improvements in various subsets. These results demonstrate that our proposed category-unified models SiamCUT and MoCUT not only generalize well across diverse categories but also exhibit a certain level of generalization to datasets with varied data distributions.
category-unifet models exhibit competitive per-
Table 8: Performance comparisons with state-of-the-a methods on Waymo Open Dataset. Bold and underline represent our category-unified methods and the corresponding baselines.
Vehicle
Pedestria
Method
Easy [67,832]
Medium [61,252]
Hard
Mean [185,731]
Easy [85,280)
Medium [82,253]
Hard [74,219]
Mean [241.752]
[56,647]
P2B (Qi al., 2020)
$7.1 65.4
52.0760.7
47.9/585
52.6/61.7
18.17 30.8
17.8/30.0
09.3/2933
/ 30.1
BAT (Zheng et. al., 2021)
61.0 68.3
53.3
48.9/57.8
54.7 62.7
19.3 32.6
17.8/293
17.2 28.3
18.2 30.3
V2B (luui el al., 2021)
61.5/71.5
55.1
52.0/62.
57.6
27.9/
/2.5/36.2
20.1
/3.9
(Tui al., 2022)
72.7
$7.5/66.0
54.6
59.7 68.0
29.2 45.3
24.7/38.2
22.2/ 35.8
25.5/39.9
CXTrack (Xu
2023)
73.9/71.1
54.2/62.7
57.1 /
55.3
24.7
26.3/44.4
30.7
MPTrack (Zheng et al., 2022a)
68.1 75.3
58.6 / 66.6
55.4 64.9
61.1 6603
35.5 /5.2
30.7/448.4
29.3 45.9
32.01 49.7
SiamCUT (Ours)
66.0
60.8
59.1
53.0
23.4
20.7
21.4/31.5
MoCUT (Ours)
75.0
59.4 66.9
57.1 66.3
61.9 69.7
36.5 / 54.8
31.8/488.9
29.5 45.4
32.4/ 49.9
B.4
COMPUTATIONAL COST ANALYSIS
In practical applications, computational cost analysis has always been a critical metric to evaluate the performance of trackers. As shown in Tab. 9, we analyze the model complexity, computational
17

---

Published as a conference paper at ICLR 2024
overhead and inference speed by counting parameters, floating and latency/speed, respectively. The parameters and FLOPs for all methods are manually counted, if feasible. The latency and speed for baseline methods P2B (Qi et al., 2020) and M2Track (Zheng et al., 2022a) are computed our workstation to ensure a fair comparison. The other methods utilize reported results from relative references (Giancola et al., 2019; Zheng al., 2021; Zhou al., 2022; Nie et al., 2023b;e; Wang et al., 2023; Xu et al., 2023). Compared to the baseline methods, our category-unified models demonstrate performance improvement with tolerable computational overhead.
per second (FLOPs)
Table 9: Computation cost comparisons with state-of-the-an methods. Bold and underline represent our category-unified methods and the corresponding baselines.
Method
Parameters
FLOPs
Latency
Speed
Hardware
KITTI Performance
SC3D (Giamcula el al., 2019)
6,46 M
19.82 G
2
GTX 1080Ti
32.1 488.5
P2B al. 2020)
1.34 M
G.30
20.8
48 Fins
RTX 3070TT
42.4 60.0
BAT (Zheng et al., 2021)
1.48 M
2.77
57 Fps
RTX 2080
51.2/72.8
PTTR (Shan al.,
M
2.1
Fps
Tesla V100
37.9/78.2
OSP28 (Nie et al., 2023b)
1,70 M
G
29.5 ms
34 Fips
GTX 10801i
60.5 82.3
(Nic 2023c)
3.87
Fiss
GTX
PCET' (Wang et al., 20233
30.5 ms
33 Fips
GTX 1080Ti
64.8 81.3
CrTkk (Xu
2023)
M
4.63
Fps
RIX 3090
85.3
M2Track (Zheng et al., 2022a)
2.24 M
G
ms
63 Fis
RTX 3010T
62.9 83.4
SiamCUT (Ours)
2.06 M
4.51
27.5 ms
Fps
RIX 307011
54.0
McCUT (Ours)
2.34 M
3.27
20.9
48 Fps
RTX 3070TT
65.8 05.8
C COMPARISON WITH CATEGORY-SPECIFIE MODELS
To further demonstrate the potential of our category-unified models, we integrate the proposed uni- fied components, including unified representation network AdaFormer, model inputs and learning objective into existing tracking methods. We select some classic trackers, such as PTT (Shan et al., 2021), 1), PTTR (Zhou et al., 2022) and OSP2B (Nie et al., 2023b) to report the results, as presented in Tab. 10. These unified components not only empower category- speecfff trackers to track objects across all categories, but also enhance overall tracking performance, which proves the effectiveness and promise of proposed components.
Table 10: Performance comparisons on the KITTI dataset. "Improvement" refers to the performance gain our category-unified models over the corresponding category-specifi counterparts. and refer to Siamese and motion-centric paradigms, respectively.
Method
Car [6,424)
Pedestrian [6,088]
Van [1,248]
Cyclist [308]
Mean [14,068]
Category specilic
2020)
56.2/ 72.8
40.8/48.4
/4.7
Category unified P2B (Ours)
58.1 / 73.9
76.2/76.2
/
36.7 /4..4
54.0/ 74.6
Improvement
1.9/1
19.55/
22.33/122.5
1.6//13.
11.67 14.6
selegorr--spccife PTT (Shan al. 2021)
67.8/ 81.8
44.9/72.0
43.6/52.5
37.2/47.3
55.17742
Calegory unified PTT (Ours)
67.67 82.1
49.2/77.4
65./7770
37.5 3
58.8 76.4
Improvement
1022/1
1..3/144
1
1,.3/
1
Eelegory-specifie PTTR (7hou al., 2022)
65.2/ 77.4
50.9/8166
52.5/6188
65.1 /90.5
$7.91 78.2
Category-unifeed PITR (Ours)
68.3 80.1
33.7/84.3
64.2//55.6
66.8/93.2
61.6/8188
Improvement
13.1/12.7
128/1 2.5
117711 13.8
1 1./1/22
1.7.1 3.6
Calegory specific OSP3R Category-uniffed OSP2B (Ours)
2023b)
56.3/66.2
67.5 82.8
/
68.7/79.3
65.4/91.2
62.3 84.4
Improvement
1.07
1 1.5/1
12.4/1 13.1
1.22/1
118/1
Categoory specific MrTrack (Zheng 2022a)
65.5 80.8
68.5/
53.8/70.7
73.2/93.5
62.9 83.4
Cutegory-unified Track (Ours)
67.6/ 80.5
63.3 90.0
64.5/78.8
76.7/94.2
65.8 83.0
Improvement
0.1
1 1.88//
9.77/18.1
13.5//11.
1.971
18


Reviews:
[{"review_id": "AyA4DiCRhr", "reviewer": "Reviewer_5oVT", "summary": "Existing 3D single object tracking (SOT) approaches mainly focus on category-specific model training and evaluation. Inspired by general 2D SOT, this paper proposes to use category-unified model for 3D SOT. Specifically, the paper proposes a point set network named AdaFormer to encode geometric information of different object categories in a unified manner. To further boost the performance, the unified model inputs and learning objective are introduced to facilitate the learning of unified representation. To verify the effectiveness, two category-unified models SiamCUT and MoCUT based on Siamese and motion-centric 3D SOT paradigms are proposed. Experiments on KITTI and NuScenes datasets show that the proposed approaches gain much performance improvements over the baselines.", "questions": "- The proposed approach can track objects across all categories using a single network with shared parameters. But as I mentioned above, there is no training details about the unified training. Is the proposed only trained on the full KITTI and then test on it (the same for Nuscenes)? or the combination of KITTI and Nuscenes are used? Please all more illustration in the paper.\n- The main concern in this paper is about the lack of Waymo dataset evaluation and missing recent approaches for comparison. Please also include the latest references below for comparison, in order to better verify the effectiveness of the proposed approach.\n\n[1] Temporal-aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking. ACCV 2022.\n[2] 3D Siamese Transformer Network for Single Object Tracking on Point Clouds. ECCV 2022.\n[3] CXTrack: Improving 3D Point Cloud Tracking with Contextual Information. CVPR 2023.\n[4] A Lightweight and Detector-Free 3D Single Object Tracker on Point Clouds. IEEE Transactions on Intelligent Transportation Systems. 2023.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper written&&organization is good, which is easy to follow.\n- I think the problem solved in this paper is valuable to the 3D SOT community, since current approaches mainly need to train multiple models corresponding to various training categories in the dataset in order to achieve higher performance. This paper shows competitive category-specific results by only learning a unified representation model (although I do not see the authors claim any training details about their unified training, e.g., training their models on all category samples on KITTI and then test it on category-specific KITTI).\n- The paper is technically sound, which solves the above problem progressively by proposing multiple modules.", "weaknesses": "- The proposed approach can track objects across all categories using a single network with shared parameters. But as I mentioned above, there is no training details about the unified training. Is the proposed only trained on the full KITTI and then test on it (the same for Nuscenes)? or the combination of KITTI and Nuscenes are used? Please all more illustration in the paper.\n- The main concern in this paper is about the lack of Waymo dataset evaluation and missing recent approaches for comparison. Please also include the latest references below for comparison, in order to better verify the effectiveness of the proposed approach.\n\n[1] Temporal-aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking. ACCV 2022.\n[2] 3D Siamese Transformer Network for Single Object Tracking on Point Clouds. ECCV 2022.\n[3] CXTrack: Improving 3D Point Cloud Tracking with Contextual Information. CVPR 2023.\n[4] A Lightweight and Detector-Free 3D Single Object Tracker on Point Clouds. IEEE Transactions on Intelligent Transportation Systems. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFHa3YQuYr", "reviewer": "Reviewer_vKoh", "summary": "Previous 3D single object tracking models are all category-specific, incurring redundant parameters. In this paper, the authors propose to unify the different categories in a single model. A novel point cloud representation learning network based on transformers, named AdaFormer, is proposed to encode the dynamically varying shape and size information from cross-category data in a unified manner. Moreover, the authors construct two unified models following previous Siamese and motion-centric paradigms to compare. Performance gains validate the effectiveness of proposed method.", "questions": "Are there any inference speed and FLOPs comparison for proposed method?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The motivation is great, category-unified model designing is meaningful and significant to SOT task and trying to unify them is of novelty.\n2. The model achieves good performance on KITTI and nuScenes.", "weaknesses": "No obvious weakness from my perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEZ1J4jwX4", "reviewer": "Reviewer_zjK7", "summary": "This paper introduces an Adaformer to dynamically extract features of different categories with diverse object sizes. To this end, it can adapt to LiDAR-based 3D SOT tasks for unified training. Albeit the idea is relatively common, it can achieve the target of unified training with performance improvement.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is the first work to unified train all categories on the 3D SOT task.\n- The proposed Adaformer can effectively learn adaptively ball region relative to different categories, which agrees with the motivation of this paper.\n- The paper is well-written.", "weaknesses": "- You should discuss relative works on your key idea (e.g., encoding shape- and size-changed geometric information). \n   Relative works include [a] (adaptive region learning), [b] (dynamic ball-query selection for size, dynamic foreground and background learning), [c], etc.\n\n   [a] Pyramid r-cnn: Towards better performance and adaptability for 3d object detection.\n\n   [b] DBQ-SSD: Dynamic Ball Query for Efficient 3D Object Detection.\n\n   [c] RBGNet: Ray-Based Grouping for 3D Object Detection.\n\n- The comparison of latency should include other methods that have been reported, e.g., [c], [d], etc. In addition, the recent SOTA SOT methods should also be included.\n\n   [c] Beyond 3d siamese tracking: A motion-centric paradigm for 3d single object tracking in point clouds.\n\n   [d] Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking.\n\n- Why not verify on Waymo dataset？\n\n- Can Adaformer be extended to general detection？\n\n- Can you conduct other statistical analysis to reveal diverse size of different categories to further support you motivation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)