3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds
Authors: Le Hui
Lingpeng Wang
Mingmei Cheng
Jin Xie
Jian Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Point Cloud
3D Single Object Tracking

Abstract: 3D object tracking in point clouds is still a challenging problem due to the sparsity of LiDAR points in dynamic environments. In this work, we propose a Siamese voxel-to-BEV tracker, which can significantly improve the tracking performance in sparse 3D point clouds. Specifically, it consists of a Siamese shape-aware feature learning network and a voxel-to-BEV target localization network. The Siamese shape-aware feature learning network can capture 3D shape information of the object to learn the discriminative features of the object so that the potential target from the background in sparse point clouds can be identified. To this end, we first perform template feature embedding to embed the template's feature into the potential target and then generate a dense 3D shape to characterize the shape information of the potential target. For localizing the tracked target, the voxel-to-BEV target localization network regresses the target's 2D center and the z-axis center from the dense bird's eye view (BEV) feature map in an anchor-free manner. Concretely, we compress the voxelized point cloud along z-axis through max pooling to obtain a dense BEV feature map, where the regression of the 2D center and the z-axis center can be performed more effectively. Extensive evaluation on the KITTI tracking dataset shows that our method significantly outperforms the current state-of-the-art methods by a large margin. Code is available at https://github.com/fpthink/V2B.

3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds
Le Hui Lingpeng Wang', Mingmei Cheng, Jin Xie", Jian Yang" PCA Lab, Nanjing University of Science and Technology, China hui, calpwang, chengmm, csjxie, csjyang)onjust... edu. en
Abstract
3D object tracking in point clouds is still challenging problem due the sparsity of LiDAR points in dynamic environments. In this work, we propose Siamese voxel-to-BER BEV tracker, which can significantly improve the tracking performance sparse 3D point clouds. Specifically, it consists of a Siamese shape-aware feature learning network and voxel-to-BEV target localization network. The Siamese shape-aware feature learning network can capture 3D shape information of the object to learn the discriminative features of the object so that the potential target from the background in sparse point clouds can be identified. To this end, we first perform template feature embedding to embed the template's feature into the potential target and then generate dense 3D shape to characterize the shape information the potential target. For localizing the tracked target, the voxel-to- BEV target localization network regresses the target's 2D center and the z-axis center from the dense bird's eye view (BEV) feature map in an anchor-free manner. Concretely, compress the voxelized point cloud along z-axis through max pooling to obtain a dense BEV feature map. where the regression the 2D center and the 2-axis center can be performed more effectively. Extensive evaluation the KITTI and nuScenes datasets shows that our method significantly outperforms the current state-of-the-ant methods by large margin. Code is available https: //github. com/ fpthink/
Introduction
Object tracking an essential task computer vision and has been widely in various applications, such as autonomous vehicle, mobile robotics, and augmented reality. In the past few years, many efforts [33, 2, 12, 64] have been made on 2D object tracking from RGB data. Recently, with the development of 3D sensor such as LiDAR and Kinect, 3D object tracking [37, 72, 53, 28, 42] has attracted more attention. Lately, some pioneering works [22, 18, 52] have focused on point cloud based 3D object tracking. However, due to the sparsity of 3D point clouds, 3D object tracking on point clouds is still challenging task.
Few works are dedicated to 3D single object tracking (SOT) with only point clouds. As a pioneer, SC3D [21] is the first 3D Siamese tracker that performs matching between the template and candidate 3D target proposals generated by Kalman filtering [22]. Furthermore, a shape completion network is used enhance shape information of candidate proposals sparse point clouds, thereby improving the accuracy of matching. However, SC3D cannot perform the end-to-end training, and consumes
Equual Contributions, "Corresponding authors.
Le Hui, Lingpeng Wang, Mingmei Cheng, Jin Xie, and Jian Yang are with PCA Lab, Key Lab of Intelligent Perception and Systems for Hiigh-iimensionll Information of Ministry Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology, China.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

much time when matching exhaustive candidate proposals. Towards these concerns, Qi et al. [52] proposed an end-to-end framework termed P2B, which first localizes potential target centers in the search area via Hough voting [48], and then aggregates vote clusters to generate target proposals. Nonetheless, when facing sparse scenes, P2B may not be able to track the object accurately, or even lose the tracked object. On the one hand, it adopts random sampling to generate initial seed points, which further exacerbates the sparsity of point clouds. On the other hand, it is difficult to generate high-quality target proposals on sparse 3D point clouds. Although SC3D has enhanced shape information of candidate proposals, the low-quality candidate proposals obtained from sparse point clouds still degrade tacking performance.
As shown in Fig. 1, we count the number of points on KITTI's cars. It can be found that 51% of cars have less than 100 points, and only 7% of cars have more than 2500 points. When facing sparse point clouds, it difficult to distinguish the target from the background due to the sparsity of point clouds. Therefore, how to improve the tracking performance in sparse scenes should be considered. Our intuition consists of two folds. First, enhancing shape information of target will provide discriminative information to distinguish the target from the background, especially in sparse point clouds. Second, due to the sparsity of the point cloud, it is difficult regress the target center in 3D space. We hence consider compressing the sparse 3D space into a dense 2D space, and perform center regression in the dense 2D space improve tracking performance.
7%
13%
51%
100
[101, 200]
2201, 300]
[301, 400)
18%
[401, 2500]
> 2300
Number of Points
Figure 1: Statistics of the number of points on KITTI's cars. Cars are colored in red.
In this paper, we propose novel Siamese voxel-to-BEV (V2B) tracker, which aims to improve the tracking performance of 3D single object tracking, especially in sparse point clouds. We illustrate our framework in Fig. 2. We first feed the template and search area into the Siamese network to extract point features, respectively. Then, we employ the global and local template feature embedding to strengthen the correlation between the template and search area so that the potential target in the search area be effectively localized. After that, introduce shape-aware feature learning module to learn the dense geometric features of the potential target, where the complete and dense generated. Thus, the geometric structures of the potential target can
point clouds of the target be captured better so that the potential target can be effectively distinguished from the background in the search area. Finally, we develop a voxel-to-BEV target localization network to localize the target in the search area. In order to avoid using the low-quality proposals on sparse point clouds for target center prediction, we directly regress the 3D center of the target with the highest response in the dense bird's eye view (BEV) feature map, where the dense BEV feature map is generated by voxelizing the learned dense geometric features and performing max- max-pooling along the z axis. Thus, with the constructed dense BEV feature map, for sparse point clouds, our method can more accurately localize the target center without any proposal.
In summary, we propose a novel Siamese voxel-to-BEV BEV tracker, which can significantly improve tracking performance, especially in sparse point clouds. We develop Siamese shape-aware feature learning network that can introduce shape information to enhance the discrimination of the potential target in the search area. We develop voxel-to-BEV target localization network, which can accurately detect the 3D target's is center in the dense BEV space compared to sparse 3D space. Extensive results show that our method has achieved new state-of thethe results on the KITTI dataset [20], and has a good generalization ability on the nuScenes [6] dataset.
2 Related Work
2D object tracking. Numerous schemes [5, 22, 34, 74, 1] have been presented and achieved impressive results in 2D object tracking. Early works are mainly based on correlation filtering. As a pioneer, MOSSE [4] presents stable correlation filters for visual tracking. After that, correlation- based methods use Circulant matrices [25], kernelized correlation filters [26]. continuous convolution filters [13], factorized convolution operators [12] to improve tracking performance. In recent years, Siamese- baaee methods [16, 24] have been more popular in the tracking field. In [2], Bertinetto et

---

al. proposed SiamFC, a pioneering work that combines naive feature correlation with a fully- convolutional Siamese network for object tracking. Subsequently, some improvements [85, 68, 75, 82, 76] are made to Siamese trackers, such as combining with region proposal network [17, 39, 80, 65] or an anchor-free FCOS detector [11], using a deeper architecture [38] or two-branch structure [23], exploiting attention [67, 84] or self-attention [7], applying triplet loss [14]. However, these methods are specially designed for 2D object tracking, so they cannot be directly applied to 3D point clouds.
3D single object tracking. Early 3D single object tracking (SOT) methods focus on RGB-D information. As pioneer, Song et al. [59] first proposed a unified 100 RGB-D video dataset, which opened up new research direction for RGB-D tracking [3, 43, 29]. Based on RGB-D D information, 3D SOT methods [60, 45, 53] usually combines techniques from the 2D tracking with additional depth information. However, RGB-D tracking also relies on RGB information. and it may fail when the RGB information is degraded. Recent efforts [46, 69] begin to use LiDAR point clouds for 3D single object tracking. Among them, SC3D [21] is the first 3DD Siamese tracker, but it is not an end-to-end framework, Following it, Re-Track [18] a two-stage framework that re-tracks the lost objects of the coarse stage in the fine stage. Lately, Qi et al. [52] proposed P2B, which solves the problem that SC3D cannot perform end-to-end training and consumes a lot of time. P2B uses VoteNet [48] to generate proposals and selects the proposal with the highest score as the target. Based on P2B, to handle sparse and incomplete target shapes, BAT [83] introduces a box-aware feature module to enhance the correlation learning between template and search area. Nonetheless, when facing very sparse scenarios, VoteNet used in P2B and BAT may be difficult to generate high-quality proposals, resulting in performance degradation.
3D multi-object tracking. Most 3D multi-object tracking (MOT) systems follow the same schemes with the 2D) multi-object tracking systems, but the only difference is that 2D detection methods are replaced by 3D detection methods. Most 3D MOT methods [73, 55, 30] usually adopt tracking-by- detection schemes. Specifically, they first use a 3D object detector [57, 58, 56] to detect numerous objects of each frame, and then exploit the data association between detection results of two frames to match the corresponding objects. To exploit the data association, early works [54] use handcrafted features such as spatial distance. Instead, modern 3D trackers use motion information that can be obtained by 3D Kalman filters [47, 10, 71] and learned deep features [78, 81].
Deep learning on point clouds. With the introduction of PointNet [49], 3D deep learning on point clouds has stimulated the interest of researchers. Existing methods can be mainly divided into: point-based [51, 40, 63, 9], volumetrie-based [50, 44], graph-based [70, 36, 35, 66, 8, 27], and view- based [61, 62, 77] methods. However, volumetric-basee and view-based methods lose fine-grained geometric information due to voxelization and projection, while graph-based methods are not suitable for sparse point clouds since few points cannot provide sufficient local geometric information for constructing a graph. Thus, existing 3D tracking networks [21, 18, 52, 83) are point-based methods.
3 Method
Our work is specifically designed for 3D single object tracking in sparse point clouds. An overview of our framework is depicted in Fig. 2. We first present the Siamese shape-aware feature learning network to enhance the discrimination of the potential target in search area (Sec. 3.1). We then localize the target by voxel-to-BEV target localization network (Sec. 3.2).
3.1
Siamese Shape-Aware Feature Learning Network
3.1.1
Template Feature Embedding
Suppose the size of the template is N, and the size of the search area M (generally, M N). Before template feature embedding, we first use the Siamese network to extract point features of the template and search area, denoted by P = {p, N and Q {q, | The Siamese network consists of template branch and detection branch. In order to reduce the inference time of the network, we just use PointNet++ [51] as the backbone and share parameters. It can be replaced with a powerful network such as KPConv [63]. We then employ template feature embedding to encode the search area by learning the similarity of the global shape and local geometric structures between the template and search area. The illustration of template feature embedding is shown in the left half of Fig. 3.
3

---

lengloee fedure
Eapape
temp
X feature
Point
Shupes waae Feature Learning
head
center
pooling
saamd
TTsek & zot besd
suchi
Point
Torglate Fasture Libbediing
oxel
Pixel
3-2206 hood
3
Siamese Network
Q-M C searth area feature
M x (C 3)
Network
2D Network
Siamese Shape-Aware Feature earning Network
Voxel-to-BEV Target Localization Network
Figure 2: The architecture of V2B. Given template and search area, we first use Siamese network to obtain template and search area features. We then perform template feature embedding and shape-aware feature learning to enhance the ability to distinguish the target from the background. Finally, we perform voxel-to-BEV target localization to detect the 3D object center from the BEV.
Template global feature embedding. We use the multi-layer perceptron (MLP) network to adaptively learn the correlation between the template and search area. The similarity between the template and search area is formulated as:
(I) where p; q characterizes the difference between the two feature vectors and w.j BCC is the correlation weight between two points. The global shape information of the template is given by:
w = foor (Pr. 9,) MLP q₂), Vp, E P.qj
[emb P1, P2....... .......... PN) MLP(, MAX (pr Wiil), Vg, Q
1.2.....N
where MAX represents the max pooling function and Wi is the correlation weight. The obtained q, C considers the similarity between the template and search area, and characterizes the global shape information of the target through the max pooling function.
Template local feature embedding. To characterize the local similarity between the template and search area, we first obtain the similarity map by computing cosine distance between them. The similarity function faim is written
P 9j
8ij f Sam (pi, Qj)
Vp: P.g. E Q
(3)
|p:112 1
where 8t indicates the similarity between points and j. We then assign each point in the search area with its most similar point in the template, which is written as:
qj MLP((q, Skj Pk. Pk. xxl),, k argmax {feim (p.. 9,),, Vq, E Q
(4)
i=1.2....N
where & indicates the index of the maximum value of similarity, and skj, 2y are the corresponding maximum value and 3D coordinate, respectively. [ represents the concatenation operator. We hence obtain the embedded feature q, E Rº of j-th point in the search area after using MLP. Finally, we concatenate the obtained global and local feature maps to obtain an enhanced feature map F
= ML. ( J.
3.1.2 Shape-Aware Feature Learning
Due to the sparse and incomplete point clouds of the potential target in the search area, we employ shape-aware feature learning learn dense geometric features of the target, where the dense and complete point clouds of the target can be obtained. It is expected that the learned features from the generated dense point clouds can characterize the geometric structures of the target better.
Dense ground truth processing. To obtain the dense 3D point cloud ground truth, we first crop and center points lying inside the target's ground truth bounding box in all frames. We then concatenate all cropped and centered points to generate a dense aligned 3D point cloud, denoted by = { {x. ) 2048 where æ; is the 3D position, and we fix the number of points to 2048 by randomly discarding and duplicating points.
Shape information encoding. We depict the network structure in the right half of Fig. 3. Suppose the input point feature E MC that has been embedded with the template information. Before generating dense and complete point cloud of the target, we first use gate mechanism to enhance

---

Template Feature Embedding
Shape-Aware Feature L Learning
M.X.C.
EiN
MIL
MLP
PrAt geoling
N X.C
N X X
X.C
MLF
outilation
coostieates giobel
MEP
2010 x
2048
pooling
K similacit
loca Iesture
2040
subtrruction
distance
multplicationn
gather
conceranation
sigmold
A-moamsst points
Figure 3: The architecture of template feature embedding and shape-aware feature learning.
the feature of the potential target and suppress the background in the search area, which is written as: F (FWT b) F
(5)
where F' E MxC is the enhanced feature map, a is the sigmoid function, and is the element wise product. Besides, W E RIxC' is the weight to be learned. It is expected that the potential target can provide more information for generating a dense and complete point cloud of the target. Once we obtain the enhanced feature map, we execute the feature expansion operation [79] to enlarge the feature map from M x C' to 2048 x C, i.c., the number of points increases from M to 2048. Then, we capture the global shape information and local geometric structure of the potential target to generate the complete and dense 3D shape. On the one hand, we exploit the max pooling combined with fully connected layers to capture global shape information. On the other hand, we adopt EdgeConv [70] to capture local geometric information of the target. After that, we augment the local feature of each point with the global shape information, yielding a new feature map of size 2048 x 2C. Finally, we adopt MLP to generate 3D coordinates, denoted by X = { network, we follow [21, 15] and use Chamfer distance (CD) loss to enforce the network generate realistic 3D point cloud, The Chamfer distance measures the similarity between the generated point cloud and dense ground truth, which is given by:
22048 To train the shape generation
Lshape Σ min || -,,11++
Σ
min//= - 22112 2,Ex
2,EX 2,
Ø
By minimizing the CD loss, we can learn dense geometric features of the potential target in the search area by generating a dense and complete point cloud of the target. Note that shape information encoding is only performed during training and will be discarded during testing. Thus, it does not increase the inference time of object tracking in the test scheme.
Although SC3D [21] also uses shape completion to encode shape information of the target, the template completion model in SC3D cannot recover complex geometrie structures of the potential target well due to limited templates and large variations of the potential target in the search area. However, our method is a complex point cloud generation method that learns the target complction model from the samples of search areas with the gate mechanism to enhance the feature of the potential target and suppress the background in the search area. addition, the template completion model in SC3D only employs PointNet [49] to extract point features of sparse point clouds, while target completion model constructs a global-loca local branch to extract global shape features and local geometric features of sparse point clouds.
3.2 Voxcl-to-BEV Target Localization Network
In order to avoid using the low-quality proposals on sparse point clouds for target center prediction, we develop simple yet effective target center localization network without any proposal to improve the localization precision in sparse point clouds.
3.2.1 Dense BEV Feature Map Generation
In order to improve the localization precision in sparse point clouds, we utilize the voxelization and max-pooling operation to convert the learned discriminative features of sparse 3D points into the dense bird's eye view (BEV) feature map for the target localization, as shown in the right half of Fig. 2. We first convert the point features of the search area into volumetric representation by
5

---

averaging the 3D coordinates and features of the points in the same voxel bin. Then, we apply stack of 3D convolutions on the voxelized feature map to aggregate the feature of the potential target in the search area, where the voxels lying on the target can be encoded with rich target information. However, in the sparse volume space, due to the large number of empty voxels, the differences between the responses in the voxelized feature map might not be remarkable. Thus, the highest response in the feature map is difficult to distinguish from the low responses, leading to the inaccurate regression of the 3D center of the target, including the z-axis center. By performing max-pooling on the voxelized feature map along the z-axis, we can obtain the dense BEV feature map, where the low responses in the voxelized feature map be suppressed. Thus, compared to the voxelized feature map, we can more accurately localize the 2D center of the target with the highest response in the dense BEV feature map. The response of the 2D center (i.e., max-pooling feature along the 2-axis) in the BEV feature map actually contains the geometric structure information of the potential target while the responses of other points in the BEV feature map do not. In addition, we apply stack of 2D convolutions on the dense BEV feature map to aggregate the feature so that the potential target can obtain sufficient local information in the BEV feature map. Thus, with the constructed dense BEV feature map, for sparse point clouds, our method can more accurately localize the target center without any proposal.
3.2.2 Target Localization in BEV
Inspired by [19], we develop a simple yet powerful network to detect the 2D center and the z-axis center based on the obtained dense BEV feature map. As shown in the right half of Fig. 2, it consists of three heads: 2D-center head, offset & rotation head, and z-axis head. The 2D-center head aims to localize 2D center of target on the z-y plane, and z-axis head regresses the target center of the z-axis. Since the 2D center of 2D grid is discrete, we also regress the offset between it and the continuous center. Thus, we use an offset & rotation head to regress offiset plus additional rotation.
Target center parameterization. Given the voxel size v and the range of the search area [xxmin, Tmaz), (Ymiin: Ymaz ) in x-V plane, we can obtain the resolution of the BEV feature map by H Emax -mmin| + 1 and W = |Muuce MMuia| + 1, where [-] is the floor operation. Assuming the 3D center (z, y. x) of the target ground truth, we can compute the 2D target center c = (Cz, Cy) in x-y plane by C₂ = g-#mmn and Cy M-Bmin Besides, the discrete 2D center ė = (čzy 2,)) is defined by c22 - [c]] and 2, = LcyJ.
2D-center head. Following [19], we obtain the target center's 's ground truth H E Hx For the pixel (i,j) in the 2D bounding box, if â and j Cy. the Hij = 1, otherwise 1 2+I where d represents the Euclidean distance between the pixel (1. f) and the target center (C,,čy) For any pixel outside the 2D bounding box. Hay is set to O. In the training phase, enforce the predicted map H E xxx1 to approach the ground truth H by using Focal loss [41]. The modified Focal loss formulated as:
Leenter = = "IM,
= 11-(11111)) log (ha))+++( * (1-(11944) (Plr)) log(1 -He)) (7)
where [(cond.) is the indicator function. If cond. is true, then I(cond.) 1. otherwise O. Besides, we empirically set ax = and 3 = in all experiments.
Offiset & rotation head. Since the continuous 2D object center is converted into the discrete one by floor operation, we consider regressing the offset of the continuous ground truth center. To improve the accuracy of regression, we consider square area with radius r around the object center. Here, we also add rotation regression. Given predicted map Ô E R xxx x3, where 3-dim means the 2D coordinate offset plus rotation, the error of offset and rotation is expressed as:
Cofs= = Aym-r
Oa - (c ö (Ax, (Ax,
(8)
where č and e mean the discrete and continuous position of the ground truth center, respectively. Besides, 0 indicates the ground truth rotation angle and [ is the concatenation operation.
z-axis head. We directly regress the z-axis location of the target center from the BEV feature map. Given a predicted map 2 E R Hx x1, we use L1 loss to compute the error of z-axis center by:
L₂ = 2

---

where è is the discrete object center, and 2 is z-axis center's ground truth.
The final loss of our network is follows: Liotal X Shhape >2(Ceenter Lass) + X3h2. where λ₁ X2, and A3 are the hyperparameter for shape generation, 21D center and offset regression, and 2-axis position regression, respectively. In the experiment, set A 10-6, 6, X2 1.0, and A = 2.0.
4 Experiments
4.1 Experimental Settings
Datasets. For 3D single object tracking, we use KITTI [20] and nuScenes [6] datasets for training and evaluation. Since the ground truth of the test set of KITTI dataset cannot be obtained, we follow [21, 52] and use the training set to train and evaluate our method. contains 21 video sequences and 8 types of objects. We use scenes 16 for training, scenes 17-18 for validation, and scenes 19-20 for testing. For nuScenes dataset, we use its validation set to evaluate the generalization ability of our method. Note that the nuScenes dataset only labels key frames, so report the performance evaluated on the key frames.
Evaluation metrics. For 3D single object tracking, we use the Success and Precision defined in the one pass evaluation (OPE) [32] to evaluate the tracking performance different methods. Success measures the IOU between the predicted and ground truth bounding boxes, while Precision measures the eтror AUC of the distance between the centers of two bounding boxes.
Implementation details. Following [52], we set the number of points N = 512 and M = 1024 for the template and search area by randomly discarding and duplicating points. For the backbone network, we use slightly modified PointNet++ [51], which consists of three set-abstraction (SA) layers (with query radius of 0.3, 0.5, and 0.7) and three feature propagation (FP) layers. For each SA layer passed, the points will be randomly downsampled by half. For the shape generation network, we generate 2048 points. The global branch is the max pooling combined with two fully connected layers, while the local branch only uses one EdgeConv layer. We use a two layer MLP network to generate 3D coordinates. For 3D center detection, the voxel size is set to 0.3 meters volumetric space. We stack four 3D convolutions (with stride of 2. 1, 2, 1 along the z-axis) and four 2D convolutions (with stride of 2, 1, 1, 2) combined with the skip connections for feature aggregation, respectively. For all experiments, use Adam [31] optimizer with learning rate 0,001 for training, and the learning rate decays by 0.2 every epochs. It takes about 20 epochs to train our model to convergence.
Training and testing. For training, we combine the points inside the first ground truth bounding box (GTBB) and the points inside the previous GTBB plus the random offset as the template of the current frame. To generate the search area, we enlarge the current GTBB by 2 meters and plus the random offset. For testing, we fuse the points inside the first GTBB and the previous result's 's point cloud (if exists) as the template. Besides, we first enlarge the previous result by 2 meters in current frame, and then collect the points lying in it to generate the search area.
4.2 Results
Quantitative results. We compare our method with current state-of-thee-ar methods. including SC3D [21], P2B [52], and BAT [83]. The quantitative results are listed in Tab. 1. For the KITTI [20] dataset, we follow [21, 52] and report the performance of four categories, including car, pedestrian, van, and cyclist, and their average results. As one can see from the table, our method is significantly
better than other methods on the mean results of four categories. For the car category. our method even improve the Success from 60.5% (BAT) to 70.5% (V2B). For the nuScenes [6] dataset, we directly apply the models, trained on the corresponding categories of the KITTI dataset, to evaluate performance on the nuScenes dataset. Specifically, the corresponding categories between KITTI and nuScenes datasets are Car, Pedestrian indesttrian=PPeessttiiia -Pedestrian, Van-xTruck, and Cyclist-Bicyclle respectively. It can be seen that V2B can still achieve better performance the mean results of all four categories. The quantitative results on the nuScenes dataset further demonstrate that our V2B has good generalization ability adapt different datasets.
Quantitative results on sparse scenes. To verify the effectiveness of our method for object tracking in sparse scenes, we count the performance of SC3D, P2B, BAT, and our V2B in sparse scenes of the
7

---

Table 1: The Success/Precisionn different methods on the KITTI and nuScenes datasets. "Mean" indicates the average results of four categories.
Method
Car
Pedestrian 6088
Van 1248
Cyclist 308
Mean 14068
Dataset
Frame Number
6424
SC3D [21]
41.3 57.9
18.2 37.8
40.4 / 47.0
41.5 / 70.4
31.2 48.5
P2B [52]
56,2
28.7 49.6
40.8 / 48.4
32.1 /4.7
42.4 60.0 60.0
KITTI
BAT [83]
60.5 77.7
42.1/
52.4 67.0
33.7/45.4
51.2 72.8 18.475555
V2B (ours)
70.5/81.3
48.3 73.5
00.1/58.0
40.8 49.7
Method
Car 15578
Pedestrian 8019
Truck 3710
Bicycle 501
Mean 27808
Dataset
Frame Number
SC3D [21]
25.0 / 27.1
14.2 16.2
16.2
25.7 / 21.9
17.0/ 18.2
21.8 23.1 23.1
P2B [52]
29.2
15.9 22.0
21.5/16.2
20.0
25.3 25.3
nuScenes
BAT [83]
24.5
17.3 24.5
19.3 15.8
17.0/ 18.8
20.5 2 23.0
V2B (ours)
31.3/
17.3
17.3/23.4 23.4
21.7 16.7
22.2 19.1
29.0
KITTI dataset. Specifically, we filter out sparse scenes for evaluation according to the number of points lying in the target bounding boxes in the test set. Specifically, the conditions for the sparse scenes are: I 150 (car), < 100 (pedestrian), I 150 (van), and < 100 (cyclist), respectively. For the four categories, the number of selected frames are 3293 (car), 1654 (pedestrian). 734 (van), and 59 (cyclist), respectively. In Tab. 2, we report the results of Success and Precision. As one can see from the table, our V2B achieves the best performance on the mean results of all four categories. Note that when switching from sparse frames (Tab. 2) to all types of frames (Tab. 1), SC3D and P2B suffer from a performance drop on the mean results of four categories. The worse tracking performance of SC3D and P2B on large amounts of sparse frames leads to the inaccurate template updates on the consecutive dense frames. Thus, SC3D and P2B cannot obtain better tracking performance on the dense frames. Although SC3D uses template shape completion, due to limited template samples and large variations of the potential target in the search area, it cannot accurately recover the complex geometric structures of the target in the sparse frames, which poses challenges localizing the potential target with sparse points. On the contrary, our V2B employs the proposed shape-aware feature learning module to generate dense and complete point clouds of the potential target for the target shape completion, leading to more accurate localization of the target in the sparse frames. Compared with BAT, our V2B achieves the performance gain of 2% the mean results of all four categories from sparse frames to all types of frames. Therefore, the comparison results can demonstrate that our V2B can effectively improve the performance of single object tracking sparse point clouds.
Table 2: Comparison of Success/Precissoon of different methods on the sparse scenarios.
Method
Car
Pedestrian
Van 734
Cyclist 59
Mean 5740
Frame Number
3293
1654
SC3D [21]
37,9 / 53.0
20.1 /2.0
36,2 / 48.7
50.2 / 69.2
32.7 / 49.4
P2B [52]
56.0 70.6
33.1/58.
41.1 / 46.3
24.1/28.3
47.2 63.5
BAT [83]
75.5
48.3 77.1
41.5 / 47.4
/3.5
54.3 71.9
V2B (ours)
64.7 / 77.4
50.8 / 74.2
46.8 / 55.1
30.4/37.2
58.0 73.2
Visualization results. As shown in Fig. 4, we plot the visualization results of P2B and our V2B on the car category. Specifically, we plot a couple sparse and dense scenarios on the car category of the KITTI dataset. It can be clearly seen from the figure that compared with P2B, our V2B can track the targets more accurately both sparse and dense scenes. Especially sparse scenes, compared with P2B, our V2B can track the targets effectively. The visualization results can demonstrate the effectiveness of our V2B for sparse point clouds.
4.3 Ablation Study

---

Timelinel (Eumel
T=5
T=10
=
T
T
Timeline2 (Eracie).
T
T 90
T T20
T 150
T 180
P2B
V28
Ground Truth
Figure 4: Visualization results of sparse (the first-row sequence) and dense (the second-row sequence) scenes on the car category. The green boxes are ground truth bounding boxes. The red boxes are the objects tracked by our V2B, while the blue boxes are the objects tracked by P2B. In addition, we mark the points of cars to red for better identification.
Template feature embedding. We study the impact of template feature embedding on tracking performance. As shown in Tab. 3, we report the results of the car category in the KITTI dataset. can be seen that without using the template feature embedding (dubbed "w/o template feature"), the performance will be greatly reduced from 70.5 / 81.3 to 63.9 / 73.9 by large margin. addition, only using the local branch or global branch cannot achieve the best performance. Since template feature embedding builds the relationship between the template and search area, it will contribute to identify the potential target from the background in the search area. Therefore, when the template feature embedding is absent, the performance will be greatly reduced, which further demonstrates the effectiveness of the proposed template feature embedding for improving tracking performance.
Table 3: The ablation study results of different components on the car category.
Module
Success/Precision
w/o template feature
63.9 73.9
local temple
68.0/ 79.2
global temple
68.8 80.0
w/o shape information
67.6 78.2
local geometric
68.6 / 79.3
global shape
69.6 80.3
default setting
70.5 / 81.3
Shape-aware feature learning. For sparse point clouds, further introduce shape-aware feature learning to enhance the ability to distinguish the potential target from the background in the search area. As shown in Tab. we conduct experiments demonstrate the effectiveness of the shape information. It can be seen from the table that without using shape-aware feature learning module (dubbed "w/o shape information"), the performance will reduce from 70.5 81.3 to 67.6 / 78.2. In addition, only using the local geometric branch or global shape branch cannot achieve the best performance. Since the shape generation network can capture 3D shape information of the object to learn the discriminative features of potential target so that it can be identified from the search area.
Toxel-to-BEV target localization. Different from SC3D [21] and P2B [52], our V2B adopts another route localize potential target in object tracking. SC3D performs matching between the template and the exhaustive candidate 3D proposals to select the most similar proposal as the target. P2B and BAT use VoteNet [48] to generate 3D target proposals, and select the proposal with the highest score as the target. However, when facing sparse point clouds, it hard generate high-quality proposals, so these methods may not be able to track the object effectively. Our V2B is an anchor- free method that does not require generating numerous 3D proposals. Therefore, our method can overcome the above concern. In order to prove this, we use VoteNet instead of voxel-to-BEV target localization to conduct experiments in the KITTI dataset. In Tab. 4, we report the results different detection methods in the sparse scenarios. Likewise, filter out sparse scenes in the test set for
Table
Comparison of different detection schemes on the sparse scenarios.
Module
VoteNet
Voxel-to-BEV
Car
56.9 / 72.0
64.7 / 77.4
Pedestrian
50.8 / 74.2
Van
30.7 39.0
46.8 / 55.1
Cyclist
/3.9/30.00
30.4/ 37.2

---

evaluation according to the number of points (refer to the setting of Tab. 2). It can be found that the results of VoteNet are lower than that of voxel-to-BEV target localization, which further demonstrates the effectiveness of our method in sparse point clouds.
Different voxel sizes. We compress the voxelized point cloud into BEV feature map for subsequent target center detection. Since the scope of object tracking is a large area, the size of voxel will affect the size of BEV feature map, thereby affecting the tracking performance. We hence study the impact of different voxel sizes on the tracking performance. Specifically, we consider four sizes, including 0.1, 0.2, 0.3, and 0.4 meters. The Success/Precision results of the four sizes are 52.5 62.2 (0. mm), 68.1 / 79.4 (0.2m), 70.5 81.3 (0.3m), and 69.1 80.0 (0.4m), respectively. When the voxel size is set to 0.3 meters, we achieve the best performance. larger voxel size will increase the sparsity and cause the loss of the target's s details. A smaller voxel size will increase the size of the BEV feature map, thereby increasing the difficulty of detecting the center.
Template generation scheme. Following [21, 52, 83], we study the impact of different template generation schemes on tracking performance. As shown in Tab. 5, we report the results of four schemes on the car category in the KITTI dataset. It can be seen from the table that our V2B outperforms SC3D, P2B, and BAT in all schemes by large margin. Compared with these methods, our V2B can yield stable results on four template generation schemes, which further demonstrates that our method can consistently generate accurate tracking results in all types of frames.
Table 5: The results of different template generation schemes of different methods in the car category.
Scheme
SC3D [21]
P2B [52]
BAT [83]
V2B (ours)
The First GT
31.6/44.4
46.7 66.7/59)
51.8 / 65.5
67.8 / 79.3
Previous result
35.7
53.1/ 68.9
79.2/
70.0 / 81.3
The First GT & Previous result
34.9/49.8
56.2
60.5
70.5 / 81.3
All previous results
51.3/57.9
31.4/66.8
55.8/71.4
69.8 / 81.2
5 Conclusion
In this paper, we proposed Siamese voxel-to-BEV (V2B) tracker for 3D single object tracking on sparse point clouds. In order to learn the dense geometric features of the potential target in the search area, we developed Siamese shape-aware feature learning network that utilizes the target completion model to generate the dense and complete targets. In order to avoid using the low-quality proposals on sparse point clouds for target center prediction, we developed a simple yet effective -BVV target localization network that can directly regress the center of the potential target from the dense BEV feature map without any proposal. Rich experiments on the KITTI and nuScenes datasets have demonstrated the effectiveness of our method on sparse point clouds.
Acknowledgments
This work was supported by the National Science Fund of China (Grant Nos. U1713208, 61876084).
References
[1] Luca Bertinetto, Jack Valmadre, Stuart Golodetz, Ondrej Miksik, and Philip HS Torr. Staple: Complementary learners for real-time tracking. In CVPR, 2016.
[2] Luca Bertinetto, Jack Valmadre, Joao Henriques, Andrea Vedaldi, and Philip Torr Follytcoovolutiinall Siamese networks for object tracking. In ECCV, 2016.
[3] Adel Bibi, Tianzhu Zhang, and Bernard Ghanem. 3D part-based sparse tracker with automatic synchronization and registration. In CVPR, 2016.
[4] David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui. Visual object tracking using adaptive correlation filters. In CVPR, 2010.
[5] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "Siamese" time delay neural network. In NeurIPS, 1993.
10

---

[6] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903. 2019.
[7] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR. 2021.
[8] Mingmei Cheng, Le Hui, Jin Xic, and Jian Yang. SSPC-Net: Semi-supervised semantic 3D point cloud segmentation network. AAAI, 2021.
[9] Mingmei Cheng, Le Hui, Jin Xie, Jian Yang, and Hui Kong. Cascaded non-local neural network for point cloud semantic segmentation. IROS, 2020.
[10] Hsu-kuang Chiu, Antonio Prioletti, Jie Li, and Jeannette Bohg. Prohabilistic 3D multi-object tracking for autonomous driving. arXiv preprint arXiv:2001.05672 2020.
[1I] Janghoon Choi, Junseok Kwon, and Kyoung Mu Lee. Visual tracking by tridentalign and context embedding. In ACCV, 2020.
[12] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, ECO: Efficient convolution operators for tracking. In CVPR, 2017.
[13] Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, and Michael Felsberg. Beyond correlation filters: Learning continuous convolution operators for visual tracking. In ECCV, 2016.
[14] Xingping Dong and Jianbing Shen. Triplet loss in Siamese network for object tracking. In ECCV, 2018.
[15] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3D object reconstruction from a single image. In CVPR, 2017.
[16] Heng Fan and Haibin Ling. Siamese cascaded region proposal networks for real-time visual tracking. In CVPR, 2019.
[17] Christoph Feichtenhofer, A. Pinz, and Andrew Zisserman. Detect to track and track to detect. In ICCV, 2017.
[18] Tuo Feng, Licheng Jiao, Hao Zhu, and Long Sun. A novel object re-track framework point clouds. In ACM MM, 2020.
[19] Runzhou Ge, Zhuangzhuang Ding, Hu, Yu Wang, Sijia Chen, L, Huang. and Y. AFDet: Anchor free one stage 3D object detection. arXiv preprint arXiv: 2006. 12671, 2020.
[20] Andreas Geiger, Philip Lenz, and R. Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In CVPR, 2012.
[21] Silvio Giancola, Jesus Zarzar, and Bernard Ghanem, Leveraging shape completion for Siamese tracking. In CVPR, 2019.
[22] Neil Gordon, B Ristic, and S Arulampalam. Beyond the Kalman filter: Partiele filters for tracking applications. Artech House, London, 830(5).1-4, 2004.
[23] Anfeng He, Chong Luo, Xinmei Tian, and Wenjun Zeng. twofold Siamese network for real-time object tracking. In CVPR, 2018
[24] David Held, Sehastian Thrun, and Silvio Savarese. Learning to track at 100 fps with deep regression networks. ECCV, 2016.
[25] Joao F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. Exploiting the circulant structure of with kernels. ECCV, 2012.
[26] João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. High-speed tracking with kernelized correlation filters. IEEE Transactions on Puttern Analysis and Machine Intelligence, 37())583--5%, 2014.
[27] Le Hui, Jia Yuan, Mingmei Cheng, Jin Xie, Xiaoya Zhang, and Jian Yang. Superpoint network for point cloud oversegmentation. ICCV, 2021.
[28] Ugur Kart, J. Kämäräinen, and Jiri Matas, How to make an rgbd tracker? In ECCV Workshops, 2018
[29] Ugur Kart, A. Lukezic. M. Kristan, J. Kämäräinen, and Jiri Matas. Object tracking by reconstruction with view-specific discriminative correlation filters. In CVPR, 2019.
11

---

[30] Aleksandr Kim, Aljoša OSep, and Laura Leal-Taixé. EagerMOT: 3D multi-object tracking via sensor fusion. arXiv preprint urXiv:2104. 14682, 2021.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980.
[32] M. Kristan, Jiri Matas, A. Leonardis, Tomás Vojár, R. Pflugfelder, G. Fernández, G. Nehehay, F, Porikli, and Luka Cehovin. A novel performance evaluation methodology for single-target trackers. IEEE Transactions Pattern Analysis and Machine Intelligence, 38:2137 2155, 2016.
[33] Matej Kristan, Jin Matas, Ales Leonardis, Michael Felsberg, Luka Cchovin, Gustavo Fernandez, Tomas Vojir, Gustav Hager, Georg Nebehay, and Roman Pflugfelder. The visual object tracking vot2015 challenge results. In ICCV workshops, 2015.
[34] Matej Kristan, Jiri Matas, Aleš Leonardis, Tomás Vojīř, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay. Fatih Porikli, and Luka Cehovin. A novel performance evaluation methodology for singee-target trackers. IEEE Trunsactions on Puttern Analysis and Machine Intelligence. 38( 38(11))2137-2155, 2016.
[35] Loic Landrieu and Mohamed Boussaha. Point cloud oversegmentation with graph-structured deep metric learning. In CVPR, 2019.
[36] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. CVPR, 2018.
[37] K. Lebeda, S. Hadfield, and R. Bowden. 2D not 2D: Bridging the gap between tracking and structure from motion. In ACCV, 2014.
[38] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. PiamRPN++: Evolution of Siamese visual tracking with very deep networks. In CVPR, 2019.
[39] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with Siamese region proposal network. In CVPR, 2018
[40] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen, PointCNN: Convolution on XathcolX X -rrnnsformed points. In NeurlPS, 2018.
[41] Tsung-Yo Yi Lin, Priyal Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 42:318 327, 2020.
[42] Y. Liu, X. Jing, Jianhui Nie, Hao Gao, Jun Liu, and Guo-Ping Jiang. Context-aware three-dimensional mean-shift with occlusion handling for robust object tracking in rgb-d videos. IEEE Transactions on Multimedia, 21:664-677, 2019.
[43] Ye Liu, Xiao Yuan Jing, Jianhui Nic, Hao Gao, Jun Liu, and Guo-Ping Jiang. Context-aware three- dimensional mcan-shift with occlusion handling for robust object tracking in RGB-D videos. IEEE Transactions on Multimedia, 21(3):664-11 2018.
[44] Zhijian Liu. Haotian Tang. Yujun Lin, and Song Han. Point-Voss Voxel can for efficient 3D deep learning. In NewrIPS, 2019.
[45] Matthias Luber, Luciano Spinello, and Kai O Arras. People tracking in RGB-D CGB-Doo data with on-line boosted target models. In IROS, 2011.
[46] Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3D detection, tracking and motion forecasting with a single convolutional net. In CVPR, 2018.
[47] Abhishck Patil, Srikanth Malla, Haiming Gang, and Yi-Ting Chen. The H3D dataset for full-surround 3D multi-object detection and tracking in crowded urban scenes. In ICRA, 2019.
[48] Charles Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object detection in point clouds. In ICCV, 2019.
[49] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In CVPR, 2017.
[50] Charles R Qi. Hao Su. Matthias Niebner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification 3D data. In CVPR, 2016.
[51] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical feature learning on point sets in metric space. In NeurlPS, 2017.
12

---

[52] Haozhe Qi. C. Feng, Zhiguo Cao, F. Zhao, and Yang Xiao. P2B: Point-to-box network for 3D object tracking in point clouds. In CVPR, 2020.
[53]
Martin Rünz and L. Agapito. Co-fusion: Real-time time segmentation. tracking and fusion of multiple objects. In ICRA, 2017.
[54] Samuel Scheidegger, Joachim Benjaminsson, Emil Rosenberg, Amrit Krishnan, and Karl Granström. Mono-camera 3D multi-object tracking using deep learning detections and PMBM filterin.. In /V, 2018.
[55] Abhijeet Shenoi, Mihir Patel, Jun Young Gwak, Patrick Goebel, Amir Sadeghian, Hamid Rezatofighi. Roberto Martin-Martín, and Silvio Savarese. JRMOT: A real-time 3D multi-object tracker and a new large-scale dataset. In IROS, 2020.
[56] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: Point-voxel feature set abstraction for 3D object detection. In CVPR, 2020.
[57] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3D object proposal generation and detection from point cloud. In CVPR, 2019.
[58] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2020.
[59] Shuran Song and Jianxiong Xiao. Tracking revisited using RGBD camera: Unified benchmark and baselines. In ICCV, 2013.
[60] Luciano Spinello, Kai Arras, Rudolph Trichel, and Roland Siegwart. layered approach to people detection in 3D range data. In AAAI, 2010.
[61] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Milles Multi-view convolutional neural networks for 3D shape recognition. In ICCV, 2015.
[62] Hao Su, Fan Wang, Eric Yi, and Leonidas Guibas. 3D-assisted feature synthesis for novel views of an object. In ICCV, 2015.
[63] Hugues Thomas, Charles R Qi, Deann-mmanuull Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. KPConv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
[64] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. End-to-end representation learning for correlation filter based tracking. CVPR, 2017.
[65] P. Voigtlaender, Jonathon Luiten, P. Torr, and B. Leibe. Siam R-CNN: Visual tracking by re-detection. In CVPR, 2020.
[66] Lei Wang. Yuchun Huang, Yaolin Hou, Shemman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In CVPR, 2019.
[67] Qiang Wang. Zhu Teng, Junliang Xing, Jin Gao, Weiming Hu, and Stephen Maybank. Learning attentions: residual attentional Siamese network for high performance online visual tracking. In CVPR, 2018.
[68] Qiang Wang. Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In CVPR, 2019.
[69] Sukai Wang. Yuxiang Sun, Chengju Liu, and Ming Liu. PointTrackNet: An end-to-end network for 3D object detection and tracking from point clouds. IEEE Robotics and Automation Letiers, 5(2):3206-322 2020.
[70] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning point clouds. arXiv preprint /001::18000033300 2018.
[71] Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 3D multi-object tracking: A baseline and new evaluation metrics. In IROS, 2020.
[72] T. Whelan, Renato Moreno, B. Glocker, A. Davison, and Stefan Leutenegger. Flasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research, 15:1697-1716, 2016.
[73] Hai Wu, Wenkai Han, Chenglu Wen, Xin Li, and Cheng Wang. 3D multi-object tracking in point clouds based on prediction confidence-guee guided data association. IEEE Transactions on Intelligent Transportation Systems, 2021.
13

---

[74] Yi Wu, Jongwoo Lim, and Ming- Ming-Hlsunn Yang. Online object tracking: A benchmark. In CVPR, 2013.
[75] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. SiamFC++: Towards robust and accurate visual tracking with target estimation guidelines. In AAAI, 2020.
[76] Bin Yan, Xinyu Zhang, Dong Wang, Huchuan Lu, and Xiaoyun Yang. Alpha-refine: Boosting tracking performance by precise bounding box estimation. In CVPR, 2021.
[77] Ze Yang and Liwei Wang. Learning relationships for multi-view 3D object recognition. In ICCV, 2019.
[78] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3D object detection and tracking. In CVPR, 2021.
[79] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. PU-Net: Point cloud upsampling network. In CVPR, 2018.
[80] Yuechen Yu, Yilei Xiong, Weilin Huang, and M. Scott. Deformable Siamese attention networks for visual object tracking. In CVPR, 2020.
[81] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, and Chen Change Loy. Robust multi-modality multi-object tracking. In CVPR, 2019.
[82] Zhipeng Zhang, Houwen Peng. Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor- free tracking. arXiv preprint 2006. 10721, 2020.
[83] Chaoda Zheng, Xu Yan, Jiantao Gao, Weibing Zhao, Wei Zhang, Zhen Li, and Shuguang Cui. Box-aware feature enhancement for single object tracking on point clouds. In ICCV, 2021.
[84] Wenzhang Zhou, Longyin Wen, L. Zhang, Dawei Du T. Luo, and Y. Wu. SiamMan: Siamese motion-aware network for visual tracking. arXiv preprint arXiv: 1912.0551, 2019.
[85] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-awan Siamese networks for visual object tracking. In ECCV, 2018.
14


Reviews:
[{"review_id": "yZ-yVZ6S2hE", "reviewer": "Reviewer_QKmE", "summary": "This paper tackles the problem of single-object tracking from point clouds. Specifically, it adopts a tracking by detection framework to find the target in the search area, implemented with a siamese network that extracts both the template and search area features. To improve feature discriminativeness, the template feature is embeded to the search area feature, trained with a regularization shape completion loss. In terms of target localization, in contrast to previous proposal based method, this paper adopts a dense detection approach that directly localizes the target from dense BEV voxel locations. Experiments on KITTI dataset with custom split shows significant improvements over previous state-of-the-art methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Following aspects of the submission can be improved.\n\n1. Needs a clear comparison with previous methods.\nThe method proposed in the submission is largely based on two previous methods, P2B [12] and SC3D [13]. In particular, the overall two-step framework follows P2B, where the first step embeds the template feature into the search area feature, implemented by a Siamese network; and the second step localizes the target given the enhanced search area feature. The main difference lies in the second step, where P2B uses a proposal based method, and this paper adopts a dense detection method, which is shown by experiments to perform much better on objects with sparse points. A suggestion here would be to add qualitative results in addition to the Table 5 in the submission to show examples of these improvements.\nAnother claimed novelty of shape-awre feature learning also largely follows the scene completion regularization loss proposed in SC3D, with the difference being the specific input feature (Line 187-190). A suggestion here would be to add a specific ablation study for the changes made on top of the regularization loss from SC3D, so as to justify these modifications and thereof the technical novelty.\nOverall, I would suggest authors provide a clear comparison with previous methods that are technically related, and clarify what's inherited from them, and what's changed to better solve an existing problem.\n\n2. Needs better presentation of the method section.\nIt's hard to follow the method section as tons of technical details are thrown to the readers without providing enough context and motivation of why the architecture is designed so. I would strongly suggest the authors to re-write the method section in the revised submission.\n\n3. More results in the experiments.\nThere can be more results in the experiments to help readers understand how the model is working. For example, the ablation study with SC3D as mentioned in point 1, qualitative comparison with both SC3D and P2B in all four categories, especially the cyclist because the proposed method performs much worse in this category compared with SC3D. Also, in Table 3 the speed comparison, the official speed of P2B is 40FPS on a NVIDIA 1080Ti GPU, however, Table 3 reports 17FPS on a NVIDIA TITAN RTX GPU, any reason why this huge gap?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I didn't see specific discussions about the limitations and potential negative societal impact in the submission, please add these in the revised submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mxdJ9VPAmfa", "reviewer": "Reviewer_kZCW", "summary": "This paper proposes a point cloud-based 3D single-object tracking method. To improve tracking performance in sparse point cloud scenarios, the proposed method incorporates the shape completion technique to enhance feature learning. Results are evaluated on the KITTI dataset demonstrating that the proposed method outperforms baselines", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper Strengths\n\n1. The engineering efforts on designing local and global feature embedding for template feature extraction are intuitive, which also significantly improve performance as shown in Table 4 ablation experiments\n\n2. Qualitative analysis on results for sparse point cloud intuitively demonstrating the usefulness of the proposed method\n\nPaper Weaknesses\n\n1. It is still vague to me if the main contributions of this paper are very useful through current experiments:\n\na) Regarding shape completion, the main difference claimed in this paper compared to prior work is that the proposed method performs shape completion in the search area while [13] performs shape completion for the template. Although the ablation experiments in Table 4 shows that without shape information by completion, the performance drops, it also needs to compare with shape completion on the template as [13]. As a result, I am not sure if the proposed idea is better or worse than [13]. Also, the absolute performance comparison in Table 1 does not help answer this question because presumably, the code base is different so there are lots of other factors affecting the overall performance. It would be easier to compare two ideas (this paper and [13]) by adding shape completion for the template in the proposed method, and/or change the shape completion to the search area or candidate shapes for [13]’s codebase. Also, intuitively it is unclear to me how performing shape completion for template or search area will make differences. Or maybe we should do both?\n\nb) Regarding the voxel-to-BEV target localization, the proposed method essentially just converts the 3D voxel features into 2D by performing max pool on the z-axis. It is unclear to me, how will the feature max pool on the z-axis help the later network (z-axis head) to better localize the z-axis center? In general, people will think that max-pooling is throwing away information so intuitively it should make the z-axis center estimation harder using the maxpooled BEV features. This contradicts the paper’s claim (line 68) that after feature pool in the z-axis, the regression on both 2D center and z-axis center become more effective and accurate. Besides the above analysis, there is also no detailed analysis on this voxel-to-BEV ablation. For example, in addition to the final performance analysis, can we add an experiment showing that after adding or not adding vowel-to-BEV, will the performance on center estimation (both the 2D center and z-axis center) change significantly? \n\n2. The network design is not justified. According to the paper (e.g., line 126) PointNet++ is extensively used in the method. Why not use more advanced point cloud processing methods since PointNet++ is 4 years old (NeurIPS 2017). \n\n3. Also, it seems not reasonable to me that the evaluation (testing) is only performed on two sequences of KITTI? At this moment, even the entire KITTI tracking dataset is too small and cannot be used as a robust testbed for evaluation, let alone only two sequences of it. I understand that this paper is following prior work on this evaluation protocol, but I feel this protocol is not fully justified, in the cases that there are many better testbeds existing. For example, one can use nuScenes for more robust evaluation which has 150 sequences for validation and 150 sequences for testing with more objects. Under the current evaluation protocol, I am not confident any number or analysis we obtain from the only two KITTI sequences is robustly meaningful. Results on Cars might have some implications but results on Van and Cyclist should have a high variation because there are simply too few examples for those objects in two KITTI sequences. I would suggest that authors add one more dataset for evaluation and comparison with other methods, either nuScenes or Waymo for example, unless there is a strong reason that we can only use the two KITTI sequences for evaluation\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation is addressed in the paper and negative societal impact is not discussed which I think is okay", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "jsYdRF9In1Z", "reviewer": "Reviewer_KWCL", "summary": "This paper suggests a framework named Siamese voxel-to-Bird Eye View (BEV) tracker, which aims to localise and follow each object of interest in sparse point clouds. The framework first feed the template and search area into the Siamese network to extract point features,  and then by adaptively learning the correlation between the template and search area, it embeds the template’s feature into the potential object in the search area . It also learns to encode shape-aware feature learning to characterize the shape information by generating a dense point cloud from a sparse point cloud. The suggested approach decomposes the 3D center into a 2D center and a z-axis center, and regress them in a dense bird’s eye view (BEV). The framework is claimed to achieve new state-of-the-art results on the KITTI tracking dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1- Clarity: The paper is well written, and the idea is clear. The technical section can be followed easily. However, the application focus of the proposed technique is not very clear to me. The main claim in the paper is that it is a framework for tracking 3D object. is it single object or multiple object tracking? I don't see any component of tracking here such as track initiation, termination and occlusion handling. The framework seems to be a 3D detection method to me rather than any tracking frameworks. Many details clarifying how it is extended for tracking single/multiple objects are not explained.   Considering the dataset used for the evaluation, eg KITTI, I am not convinced that how it is used for tracking single object only and how it is compared with the other methods in the KITTI leaderboards.\n\n2- Many missing related works: The related works and the comparing frameworks seem to be very narrow and cherry picked. There exist many related works in 2D MOT community (e.g. MOTChllenge or KITTI) are missed here. Similarly for 3D MOT, there exist many state-of-the work methods in KITTI, NuScene, Waymo and JRDB datastes and the proposed framework could be compared with. To name just few:  \nX. Weng, J. Wang, D. Held and K. Kitani. 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics. IROS 2020 (KITTI, JRDB, NuScene ).\nA. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martin-Martin and S. Savarese: JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset. IROS 2020 (KITTI, JRDB ).\nA. Kim, A. Ošep, L. Leal-Taixé. EagerMOT: 3D Multi-Object Tracking via Sensor Fusion. ICRA 2021(KITTI,  NuScene ).\n\n\n3 -Contribution: the paper does not offer any new theoretical contribution. It is rather a pure application paper using built upon the existing  machine learning tools with some tweaks and minor extensions. While the application papers are also well acknowledged by the NeurIPS research community, but such papers require comprehensive experimental evaluation on different benchmark datasets and comapring with many state-of-the-art frameworks to validate the efficacy of their proposed system. Without this, I would be hesitant to vote for the acceptance of such a paper in this venue. \n\n4- Experiments and datasets: The experiments are tested on KITTI dataset  only using a new metric (not those used by this benchmark for MOT). Why the approach was not evaluated on KITTI test set and using KITTI tracking metrics such as MOTA or HOTA? I it is not clear to me how the framework can deal with non-rigid object like human (how template is defined) and if it can track human as non-rigid object why it is not tested on 3D human tracking datasets such as JRDB. What about other 3D tracking self-driving datasets such as NuScene or Waymo?  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors didn't address limitations and societal impact. \nTracking 3D objects in point clouds has high potential impact in important applications, such as transportation  and self-driving vehicles. However, tracking technology can be generally deployed in human monitoring and surveillance as well which raise ethical and privacy issues. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "XDbZYoLNFzh", "reviewer": "Reviewer_yABo", "summary": "this paper proposed a Siamese voxel-to BEV tracker, including global and local template matching, shape-completion network and a voxel-to-BEV target localization network.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The results  look promising.\n\nMy major concern if novelty is limited.  The proposed method is an ensemble of well-known components: template matching (eg. SiamRPN[1]), shape-completion and bev localization (e.g. LaserNet[2]).\n\nAblation studies of global and local template features are desired.\n\n[1]Li, Bo, et al. \"High performance visual tracking with siamese region proposal network.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n[2]Meyer, Gregory P., et al. \"Lasernet: An efficient probabilistic 3d object detector for autonomous driving.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QaoZxeB4YZG", "reviewer": "Reviewer_1Mqw", "summary": "This paper proposes a novel Siamese voxel-to-BEV tracker for object tracking in sparse 3D point clouds. The main contributions are two folds: It introduces a shape-aware feature learning network to enhance the discrimination of targets in the search area; It develops a voxel-to-BEV target localization network to better localize 3D objects. The proposed method achieves the state-of-the-art performance compared with previous arts and ablation studies demonstrate the efficacy of various designs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission is addressing an important problem, namely object tracking in 3D point clouds. The submission is clearly written, making it easy to follow. The content is also well organized with enough technical details provided. Generally speaking, the submission is technical sound. The motivation behind various designs is clear and the experimental results partially support some of the claims.\n\nHowever, there are also claims not fully justified experimentally. For example, the submission claims it is better at handling sparse point clouds compared with previous methods including SC3D and P2B. If we compare Table 1 and Table 2, when switching from all types of objects to sparse objects, only the proposed method suffers from a performance drop while all the other two baseline methods actually get better performance. Therefore, it is not clear whether the claimed shape aware feature really makes the proposed method more suitable for sparse point clouds. Another example is that there's no ablation studies justifying the necessity of using both global and local features in the template feature embedding module and in the shape-aware feature learning module.\n\nThe method is quite new and its difference from previous works are also clearly discussed. The related work is adequate as far as I can tell. The overall experimental results are quite impressive, which outperforms previous methods by a large margin.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As I mentioned previously, some of the claims are not fully justified (better at handling sparse points, necessary to use both local and global features in the template feature embedding module and in the shape-aware feature learning module). Also the ablation study only considers the car category, which is not really representative enough. All four categories should be considered to really making the claimed contributions convincing enough. I hope authors could address this issue during rebuttal. Some claims sound awkward to me, such as line 378-380 (stable does not mean accurate.) The paper should point out that this proposed method is not suitable for indoor scenes where a BEV map might not be sufficient for object proposal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "P1ER53b79TU", "reviewer": "Reviewer_CfXd", "summary": "This paper introduced a siamese voxel2BEV object tracker for autonomous driving applications. The proposed network architecture is robust to sparse point cloud inputs, and can improve tracking performance significantly. Experimental results on KITTI benchmarks suggest that the proposed approach is highly effective.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper tackles a challenging task in autonomous driving, and the overall design of the algorithm is a sensible solution for this task.\n- Experimental results on KITTI benchmarks are great. The method surpasses existing solutions by a fairly big margin.\n- The presentation of the paper is relatively clear, and the visualizations in the paper are helpful for readers to grasp the intuitions of the design. \n- Regardless, I hope the authors release the code as they promised in the supplementary material, especially since this is an application paper, and would be useful for future research projects.\n\nWeakness:\n- The worse performance on the cyclist category seems to suggest that there are limitations in the approach. The authors argue that this is due to the lack of training data.  Suppose there are not sufficient data in other categories, would the trend of the result still remain the same?\n- Since this is an object tracking paper, it'll be helpful if authors can include videos that demonstrate the tracking outputs. Visualizations in the supplementary material are helpful but I think videos are more intuitive.\n- Minor: is the color scheme in Fig. 3 mean anything? When I first saw this diagram, I tried mapping similar colors to certain sections but eventually failed.\n- Minor: The paper is only tested on the KITTI dataset, and there are certain places in the whole pipeline that requires parameter tunings (such as coefficients for loss functions). What is the general guidance for parameter tuning when presented with new datasets? How do authors envision the proposed algorithm generalizes to new datasets or real autonomous driving scenarios?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have a broader impact question related to the overall goal of the project: I understand the scope of the project is 3D object tracking on LiDAR inputs. However for autonomous driving applications where safety is very important, why not use additional RGB information if necessary? My guess is that the current algorithm can be extended to deal with RGB+LiDAR inputs, so it'll be interesting if authors can provide more discussions about it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8El_C2FdoI", "reviewer": "Reviewer_m71r", "summary": "This paper proposes a 3D object tracker that localizes the target object given its template and a small search space. Their proposed model largely consists of the following two modules: Siamese Shape-Aware Feature Learning Network and Voxel-to-BEV target localization. The feature learning network takes the template point clouds and the search space point clouds as input and generates a feature embedding for the search space, focusing on its correlation to the template point clouds and its ability to generate dense point clouds. The search space feature is then fed into Voxel-to-BEV target localization networks which flattens the 3D point clouds into BEV 2D image where the target localization happens. The proposed method outperforms the preceding works by a noticeable margin.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, while this paper outperforms the preceding works in terms of performance and computational cost, I have some concerns about the writing quality, novelty, and ablation study.\n\nOn the positive side, the proposed method clearly outperforms the preceding state-of-the-art methods. This paper has a proper baseline with improved performance, partial ablation study to support the usefulness of the proposed components, and thorough computational cost analysis for better real-world usefulness. I find it interesting that a point cloud generation task used only during the training phase helps the expressiveness of the point cloud features used for localization.\n\nHowever, I have a few concerns about this paper:\n\nFirst, the quality of the writing is poor. There are too many obvious grammatical errors and unpolished sentences. I believe this paper must be thoroughly proofread.\n\nSecond, the proposed Template Feature Embedding Networks is a huge complicated module with heavy engineering of features without detailed ablation study in itself. I do see a motivation behind the details of the network design however they are not properly backed by detailed ablation compared to its complexity. Especially considering the venue of the conference, I find it difficult to appreciate heavy engineering without proper theoretical study. For example, ablation of global / local features and different distance functions for correlation and similarity map may be useful.\n\nThird, the Voxel-to-BEV target localization network seems to be heavily inspired by numerous preceding works in 3D object detection and I find it difficult to agree that this is a novel component. This paper must cite papers such as PointPillar and SECOND, which are commonly used 3D object detection encoder that look exactly the same as the proposed module.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see a thorough analysis of the limitation or societal impact of this work. There is a limited study of failure cases in the supplementary material however as a tracking problem, I believe this approach has numerous limitations such as re-id and tracklet initialization. A couple sentences of a societal impact of this work seems to be expected by the conference as well.", "review_text": "", "extra_scores": {"ethical_concerns": "N / A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, one of the reviewers expressed concerns about the paper (lack of clarity and limited novelty) and ranked the paper below acceptance. Another area of concern was related to a number of claims that the authors made and were not fully justified experimentally. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper that deserves acceptance.   Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal, including addressing the over statements/claims as discussed above) in the final camera-ready version.
Decision: Accept (Poster)