Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of single-object tracking from point clouds. Specifically, it adopts a tracking by detection framework to find the target in the search area, implemented with a siamese network that extracts both the template and search area features. To improve feature discriminativeness, the template feature is embeded to the search area feature, trained with a regularization shape completion loss. In terms of target localization, in contrast to previous proposal based method, this paper adopts a dense detection approach that directly localizes the target from dense BEV voxel locations. Experiments on KITTI dataset with custom split shows significant improvements over previous state-of-the-art methods.
main_review: Following aspects of the submission can be improved.

1. Needs a clear comparison with previous methods.
The method proposed in the submission is largely based on two previous methods, P2B [12] and SC3D [13]. In particular, the overall two-step framework follows P2B, where the first step embeds the template feature into the search area feature, implemented by a Siamese network; and the second step localizes the target given the enhanced search area feature. The main difference lies in the second step, where P2B uses a proposal based method, and this paper adopts a dense detection method, which is shown by experiments to perform much better on objects with sparse points. A suggestion here would be to add qualitative results in addition to the Table 5 in the submission to show examples of these improvements.
Another claimed novelty of shape-awre feature learning also largely follows the scene completion regularization loss proposed in SC3D, with the difference being the specific input feature (Line 187-190). A suggestion here would be to add a specific ablation study for the changes made on top of the regularization loss from SC3D, so as to justify these modifications and thereof the technical novelty.
Overall, I would suggest authors provide a clear comparison with previous methods that are technically related, and clarify what's inherited from them, and what's changed to better solve an existing problem.

2. Needs better presentation of the method section.
It's hard to follow the method section as tons of technical details are thrown to the readers without providing enough context and motivation of why the architecture is designed so. I would strongly suggest the authors to re-write the method section in the revised submission.

3. More results in the experiments.
There can be more results in the experiments to help readers understand how the model is working. For example, the ablation study with SC3D as mentioned in point 1, qualitative comparison with both SC3D and P2B in all four categories, especially the cyclist because the proposed method performs much worse in this category compared with SC3D. Also, in Table 3 the speed comparison, the official speed of P2B is 40FPS on a NVIDIA 1080Ti GPU, however, Table 3 reports 17FPS on a NVIDIA TITAN RTX GPU, any reason why this huge gap?
limitations_and_societal_impact: I didn't see specific discussions about the limitations and potential negative societal impact in the submission, please add these in the revised submission.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel Siamese voxel-to-BEV tracker for object tracking in sparse 3D point clouds. The main contributions are two folds: It introduces a shape-aware feature learning network to enhance the discrimination of targets in the search area; It develops a voxel-to-BEV target localization network to better localize 3D objects. The proposed method achieves the state-of-the-art performance compared with previous arts and ablation studies demonstrate the efficacy of various designs.
main_review: The submission is addressing an important problem, namely object tracking in 3D point clouds. The submission is clearly written, making it easy to follow. The content is also well organized with enough technical details provided. Generally speaking, the submission is technical sound. The motivation behind various designs is clear and the experimental results partially support some of the claims.

However, there are also claims not fully justified experimentally. For example, the submission claims it is better at handling sparse point clouds compared with previous methods including SC3D and P2B. If we compare Table 1 and Table 2, when switching from all types of objects to sparse objects, only the proposed method suffers from a performance drop while all the other two baseline methods actually get better performance. Therefore, it is not clear whether the claimed shape aware feature really makes the proposed method more suitable for sparse point clouds. Another example is that there's no ablation studies justifying the necessity of using both global and local features in the template feature embedding module and in the shape-aware feature learning module.

The method is quite new and its difference from previous works are also clearly discussed. The related work is adequate as far as I can tell. The overall experimental results are quite impressive, which outperforms previous methods by a large margin.
limitations_and_societal_impact: As I mentioned previously, some of the claims are not fully justified (better at handling sparse points, necessary to use both local and global features in the template feature embedding module and in the shape-aware feature learning module). Also the ablation study only considers the car category, which is not really representative enough. All four categories should be considered to really making the claimed contributions convincing enough. I hope authors could address this issue during rebuttal. Some claims sound awkward to me, such as line 378-380 (stable does not mean accurate.) The paper should point out that this proposed method is not suitable for indoor scenes where a BEV map might not be sufficient for object proposal.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper suggests a framework named Siamese voxel-to-Bird Eye View (BEV) tracker, which aims to localise and follow each object of interest in sparse point clouds. The framework first feed the template and search area into the Siamese network to extract point features,  and then by adaptively learning the correlation between the template and search area, it embeds the template’s feature into the potential object in the search area . It also learns to encode shape-aware feature learning to characterize the shape information by generating a dense point cloud from a sparse point cloud. The suggested approach decomposes the 3D center into a 2D center and a z-axis center, and regress them in a dense bird’s eye view (BEV). The framework is claimed to achieve new state-of-the-art results on the KITTI tracking dataset.
main_review: 1- Clarity: The paper is well written, and the idea is clear. The technical section can be followed easily. However, the application focus of the proposed technique is not very clear to me. The main claim in the paper is that it is a framework for tracking 3D object. is it single object or multiple object tracking? I don't see any component of tracking here such as track initiation, termination and occlusion handling. The framework seems to be a 3D detection method to me rather than any tracking frameworks. Many details clarifying how it is extended for tracking single/multiple objects are not explained.   Considering the dataset used for the evaluation, eg KITTI, I am not convinced that how it is used for tracking single object only and how it is compared with the other methods in the KITTI leaderboards.

2- Many missing related works: The related works and the comparing frameworks seem to be very narrow and cherry picked. There exist many related works in 2D MOT community (e.g. MOTChllenge or KITTI) are missed here. Similarly for 3D MOT, there exist many state-of-the work methods in KITTI, NuScene, Waymo and JRDB datastes and the proposed framework could be compared with. To name just few:  
X. Weng, J. Wang, D. Held and K. Kitani. 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics. IROS 2020 (KITTI, JRDB, NuScene ).
A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martin-Martin and S. Savarese: JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset. IROS 2020 (KITTI, JRDB ).
A. Kim, A. Ošep, L. Leal-Taixé. EagerMOT: 3D Multi-Object Tracking via Sensor Fusion. ICRA 2021(KITTI,  NuScene ).


3 -Contribution: the paper does not offer any new theoretical contribution. It is rather a pure application paper using built upon the existing  machine learning tools with some tweaks and minor extensions. While the application papers are also well acknowledged by the NeurIPS research community, but such papers require comprehensive experimental evaluation on different benchmark datasets and comapring with many state-of-the-art frameworks to validate the efficacy of their proposed system. Without this, I would be hesitant to vote for the acceptance of such a paper in this venue. 

4- Experiments and datasets: The experiments are tested on KITTI dataset  only using a new metric (not those used by this benchmark for MOT). Why the approach was not evaluated on KITTI test set and using KITTI tracking metrics such as MOTA or HOTA? I it is not clear to me how the framework can deal with non-rigid object like human (how template is defined) and if it can track human as non-rigid object why it is not tested on 3D human tracking datasets such as JRDB. What about other 3D tracking self-driving datasets such as NuScene or Waymo?  


limitations_and_societal_impact: No, the authors didn't address limitations and societal impact. 
Tracking 3D objects in point clouds has high potential impact in important applications, such as transportation  and self-driving vehicles. However, tracking technology can be generally deployed in human monitoring and surveillance as well which raise ethical and privacy issues.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a 3D object tracker that localizes the target object given its template and a small search space. Their proposed model largely consists of the following two modules: Siamese Shape-Aware Feature Learning Network and Voxel-to-BEV target localization. The feature learning network takes the template point clouds and the search space point clouds as input and generates a feature embedding for the search space, focusing on its correlation to the template point clouds and its ability to generate dense point clouds. The search space feature is then fed into Voxel-to-BEV target localization networks which flattens the 3D point clouds into BEV 2D image where the target localization happens. The proposed method outperforms the preceding works by a noticeable margin.
main_review: Overall, while this paper outperforms the preceding works in terms of performance and computational cost, I have some concerns about the writing quality, novelty, and ablation study.

On the positive side, the proposed method clearly outperforms the preceding state-of-the-art methods. This paper has a proper baseline with improved performance, partial ablation study to support the usefulness of the proposed components, and thorough computational cost analysis for better real-world usefulness. I find it interesting that a point cloud generation task used only during the training phase helps the expressiveness of the point cloud features used for localization.

However, I have a few concerns about this paper:

First, the quality of the writing is poor. There are too many obvious grammatical errors and unpolished sentences. I believe this paper must be thoroughly proofread.

Second, the proposed Template Feature Embedding Networks is a huge complicated module with heavy engineering of features without detailed ablation study in itself. I do see a motivation behind the details of the network design however they are not properly backed by detailed ablation compared to its complexity. Especially considering the venue of the conference, I find it difficult to appreciate heavy engineering without proper theoretical study. For example, ablation of global / local features and different distance functions for correlation and similarity map may be useful.

Third, the Voxel-to-BEV target localization network seems to be heavily inspired by numerous preceding works in 3D object detection and I find it difficult to agree that this is a novel component. This paper must cite papers such as PointPillar and SECOND, which are commonly used 3D object detection encoder that look exactly the same as the proposed module.
limitations_and_societal_impact: I do not see a thorough analysis of the limitation or societal impact of this work. There is a limited study of failure cases in the supplementary material however as a tracking problem, I believe this approach has numerous limitations such as re-id and tracklet initialization. A couple sentences of a societal impact of this work seems to be expected by the conference as well.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduced a siamese voxel2BEV object tracker for autonomous driving applications. The proposed network architecture is robust to sparse point cloud inputs, and can improve tracking performance significantly. Experimental results on KITTI benchmarks suggest that the proposed approach is highly effective.
main_review: Strength:
- The paper tackles a challenging task in autonomous driving, and the overall design of the algorithm is a sensible solution for this task.
- Experimental results on KITTI benchmarks are great. The method surpasses existing solutions by a fairly big margin.
- The presentation of the paper is relatively clear, and the visualizations in the paper are helpful for readers to grasp the intuitions of the design. 
- Regardless, I hope the authors release the code as they promised in the supplementary material, especially since this is an application paper, and would be useful for future research projects.

Weakness:
- The worse performance on the cyclist category seems to suggest that there are limitations in the approach. The authors argue that this is due to the lack of training data.  Suppose there are not sufficient data in other categories, would the trend of the result still remain the same?
- Since this is an object tracking paper, it'll be helpful if authors can include videos that demonstrate the tracking outputs. Visualizations in the supplementary material are helpful but I think videos are more intuitive.
- Minor: is the color scheme in Fig. 3 mean anything? When I first saw this diagram, I tried mapping similar colors to certain sections but eventually failed.
- Minor: The paper is only tested on the KITTI dataset, and there are certain places in the whole pipeline that requires parameter tunings (such as coefficients for loss functions). What is the general guidance for parameter tuning when presented with new datasets? How do authors envision the proposed algorithm generalizes to new datasets or real autonomous driving scenarios?
limitations_and_societal_impact: I have a broader impact question related to the overall goal of the project: I understand the scope of the project is 3D object tracking on LiDAR inputs. However for autonomous driving applications where safety is very important, why not use additional RGB information if necessary? My guess is that the current algorithm can be extended to deal with RGB+LiDAR inputs, so it'll be interesting if authors can provide more discussions about it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a point cloud-based 3D single-object tracking method. To improve tracking performance in sparse point cloud scenarios, the proposed method incorporates the shape completion technique to enhance feature learning. Results are evaluated on the KITTI dataset demonstrating that the proposed method outperforms baselines
main_review: Paper Strengths

1. The engineering efforts on designing local and global feature embedding for template feature extraction are intuitive, which also significantly improve performance as shown in Table 4 ablation experiments

2. Qualitative analysis on results for sparse point cloud intuitively demonstrating the usefulness of the proposed method

Paper Weaknesses

1. It is still vague to me if the main contributions of this paper are very useful through current experiments:

a) Regarding shape completion, the main difference claimed in this paper compared to prior work is that the proposed method performs shape completion in the search area while [13] performs shape completion for the template. Although the ablation experiments in Table 4 shows that without shape information by completion, the performance drops, it also needs to compare with shape completion on the template as [13]. As a result, I am not sure if the proposed idea is better or worse than [13]. Also, the absolute performance comparison in Table 1 does not help answer this question because presumably, the code base is different so there are lots of other factors affecting the overall performance. It would be easier to compare two ideas (this paper and [13]) by adding shape completion for the template in the proposed method, and/or change the shape completion to the search area or candidate shapes for [13]’s codebase. Also, intuitively it is unclear to me how performing shape completion for template or search area will make differences. Or maybe we should do both?

b) Regarding the voxel-to-BEV target localization, the proposed method essentially just converts the 3D voxel features into 2D by performing max pool on the z-axis. It is unclear to me, how will the feature max pool on the z-axis help the later network (z-axis head) to better localize the z-axis center? In general, people will think that max-pooling is throwing away information so intuitively it should make the z-axis center estimation harder using the maxpooled BEV features. This contradicts the paper’s claim (line 68) that after feature pool in the z-axis, the regression on both 2D center and z-axis center become more effective and accurate. Besides the above analysis, there is also no detailed analysis on this voxel-to-BEV ablation. For example, in addition to the final performance analysis, can we add an experiment showing that after adding or not adding vowel-to-BEV, will the performance on center estimation (both the 2D center and z-axis center) change significantly? 

2. The network design is not justified. According to the paper (e.g., line 126) PointNet++ is extensively used in the method. Why not use more advanced point cloud processing methods since PointNet++ is 4 years old (NeurIPS 2017). 

3. Also, it seems not reasonable to me that the evaluation (testing) is only performed on two sequences of KITTI? At this moment, even the entire KITTI tracking dataset is too small and cannot be used as a robust testbed for evaluation, let alone only two sequences of it. I understand that this paper is following prior work on this evaluation protocol, but I feel this protocol is not fully justified, in the cases that there are many better testbeds existing. For example, one can use nuScenes for more robust evaluation which has 150 sequences for validation and 150 sequences for testing with more objects. Under the current evaluation protocol, I am not confident any number or analysis we obtain from the only two KITTI sequences is robustly meaningful. Results on Cars might have some implications but results on Van and Cyclist should have a high variation because there are simply too few examples for those objects in two KITTI sequences. I would suggest that authors add one more dataset for evaluation and comparison with other methods, either nuScenes or Waymo for example, unless there is a strong reason that we can only use the two KITTI sequences for evaluation

limitations_and_societal_impact: Limitation is addressed in the paper and negative societal impact is not discussed which I think is okay
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: this paper proposed a Siamese voxel-to BEV tracker, including global and local template matching, shape-completion network and a voxel-to-BEV target localization network.
main_review: The results  look promising.

My major concern if novelty is limited.  The proposed method is an ensemble of well-known components: template matching (eg. SiamRPN[1]), shape-completion and bev localization (e.g. LaserNet[2]).

Ablation studies of global and local template features are desired.

[1]Li, Bo, et al. "High performance visual tracking with siamese region proposal network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[2]Meyer, Gregory P., et al. "Lasernet: An efficient probabilistic 3d object detector for autonomous driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
limitations_and_societal_impact: yes
review_title: null

Meta-review: Initially, one of the reviewers expressed concerns about the paper (lack of clarity and limited novelty) and ranked the paper below acceptance. Another area of concern was related to a number of claims that the authors made and were not fully justified experimentally. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper that deserves acceptance.   Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal, including addressing the over statements/claims as discussed above) in the final camera-ready version.
Decision: Accept (Poster)