Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes MV-3DT, a multi-camera 3D object tracking pipeline. The major novelty of MV-3DT is how it handles detections results from multiple cameras, where unlike existing works, it aggerates all cameras' detections together and performs temporal track matching across all cameras. MV-3DT performances on large-scale real-world datasets are comparable with base models and is STOA on NuScenes. Also, the paper includes limited ablation study to demonstrate the performance contribution of their proposed multi-camera association pipeline. 
strengths_and_weaknesses: Strengths:
1. This paper demonstrates MV-3DT performances across multiple large scale datasets and compare it with many base models. 
2.  The idea of using two LSTM to handle loss-tracked objects and tracking objects separately is an interesting approach for disappearing and reappearing objects.
Weakness
1. For 3D estimation network, the actual model architecture detail as well as the training loss for its three heads are not mentioned in the paper. 
2. For Cross-camera association, the merge state detail is missing, the paper doesn't discuss how tracks or detections are aggregated. For example for the later detect -> merge -> track pipeline, how are detections of the same object appears in two different cameras be aggregated?
3. For section 3.2, the paper claims "The intuition is that due to object truncation, the 3D localization at image borders is usually brittle", and uses this as support for the idea of cross-camera trajectories. However, my argument is that merging multi-cameras images into a single panoramic image could handle the image border problem even better.  As a follow-up question about this paper's comparison result is that, there is no comparison with methods using bird eye view based detection methods, and I think that is an important part to justify the proposed model is better.
4. The novelty of this paper is weak, merge detection before track doesn't seem to be enough to justify the novelty 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes MV-3DT, a multi-camera 3D object tracking pipeline. The major novelty of MV-3DT is how it handles detections results from multiple cameras, where unlike existing works, it aggerates all cameras' detections together and performs temporal track matching across all cameras. MV-3DT performances on large-scale real-world datasets are comparable with base models and is STOA on NuScenes. 
The novelty of this paper is weak, merge detection before track doesn't seem to be enough to justify the novelty. Some model details and pipeline processing methods are not clearly stated in the paper. Also, the paper includes limited ablation study to demonstrate the performance contribution of their proposed multi-camera association pipeline, yet the claim of "reduces identity switches significantly" is not supported by ablation study. 
issues: 1. For 3D estimation network, the actual model architecture detail as well as the training loss for its three heads are not mentioned in the paper. 
2. For Cross-camera association, the merge state detail is missing, the paper doesn't discuss how tracks or detections are aggregated. For example for the later detect -> merge -> track pipeline, how is detections of the same object appears in two difference cameras be aggregated?
3. For section 3.2, the paper claims "The intuition is that due to object truncation, the 3D localization at image borders is usually brittle", and uses this as support for the idea of cross-camera trajectories. However, my argument is that merging multi-cameras images into a single panoramic image could handle the image border problem even better.  As a follow-up question about this paper's comparison result is that, there is no comparison with methods using bird eye view based detection methods, and I think that is an important part to justify the proposed model is better.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Accurate and consistent object tracking across multiple cameras is a critical necessity for autonomous vehicles to safely navigate in the real world. The authors introduce a cross-camera 3d fusion method, called MV-3DT, that fuses 3D object detection results from multiple cameras before object identity association while tracking objects in a multi-camera setup. In contrast, existing camera-based 3D object tracking methods first detect objects using a single-camera setup and later post-hoc fuse detections to track objects in a multi-camera setup. Experiments conducted by the authors on large-scale driving datasets namely NuScenes 3D tracking benchmark and Waymo open benchmark empirically demonstrate that by fusing 3D detection results before object association, MV-3DT results in better average multi-object tracking accuracy compared to the existing methods with a significant reduction in object identity switches temporally and across cameras.
strengths_and_weaknesses: Strengths:
- The authors present an intuitive and effective idea of how information across multiple cameras should be combined to have accurate and consistent object tracking.
- The manuscript is well written and captures the main ideas of the work comprehensively. Showing tracked objects across cameras in figure 1 is quite informative.
- Videos showcasing object tracking accuracy across multiple cameras were quite insightful in appreciating the effectiveness of the work.

Weaknesses:
- Bird-eye view figures, especially figure 2, can be improved to make it easier for the reader to follow tracked objects across time steps. Discussed further in the “Issues” section.
- Further quantitative results on the Waymo open benchmark should also be included.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The ideas introduced in the work are novel and exciting. The manuscript is well-written. It presents the core ideas comprehensively, warranting an acceptance of the work at the conference. However, the authors must address a few issues before I would be confident in recommending the work for publication. If the authors address the raised concerns in their rebuttal satisfactorily, I would be happy to recommend the manuscript for publication.

issues: - The symbol $I^m_t$ has been used twice (in lines 89 and 98) to denote two different entities, first given input images from a set of cameras and second, a single input image to the 3D estimation network. Different symbols should be used to denote different entities.
- The affinity matrix A is computed as a weighted sum of the appearance affinity and a hybrid affinity of motion and location correlation. Please clarify why a hybrid affinity of motion and location correlation was considered instead of adding them as individual affinities. 
- In Table 1, the baseline KF3D leads to fewer identity switches compared to the proposed VeloLSTM method (see the last couple of rows). Similarly, in Table 2, MV-3DT without the new training scheme for the 3D estimation network reports fewer object identity switches compared to when the new training scheme is used. Please discuss potential reasons for these in the text and how to mitigate them.
- In Figure 2, it is difficult to follow what the major implications of the plots are. For example, it is not clear what the green, yellow and red circles represent. If red circles represent smooth and precise object tracking, then why are there no red circles around the blue and green objects in the first row plots?
- Briefly describe what are the major implications of the data presented in the tables in the caption. This helps the reader to quickly read and appreciate the experimental results.

Minor comments:
- In line 80, please say that “... Zhang et al. [60] propose MUTR3D, a method ….” as it is an important baseline and mentioning its name here helps the reader to follow the discussion more clearly.
- Rephrase statement “... i.e. 2.6% vs. 1.6% vs. 3.4% difference 223 of using VeloLSTM versus using no motion model.” in lines 223-224 for better readability.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a new way targeting at 3D object tracking in a multi-camera setup. Compared to previous post-hoc fusion approaches, this paper presents presents two main contributions:

1. cross-camera ssociation;
2. cross-camera motion modelling.

The idea of fusing 3D detections from individual camera inputs before data association and then predicting and tracking objects is throughly verified in two different benchmark datasets, showing the tracking accuracy improvement brought by the contributions in the paper

strengths_and_weaknesses: Strengths:
The idea of fusing the detections before performing tracking and predicting object trajectories is good and has been throughly verified in the experiment section.  

Weakness:
This paper presents a quite large system, with many components and hyparameters. To understand the core contribution of this paper better, it will be better to address the issues that I have raised below. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper shows an advancement in the multi-camera 3D MOT field. The idea is not difficult but seems to work well. There are a few points that could be further explained better.
issues: First of all, I would like to admit that I do not have a lot of experience in the field of 3D MOT. From my perspective, the idea of fusing information across camera views before tracking is a good idea. However, after reading this paper, I have a few questions in my head.
1. The whole paper presents a large system, with different components using different network components. For better reproduction of this paper, it will be better to explain the architecture of each component, for example, in the supplementary materials. 
2. There are some unexplained hyparameters within each model, such as the number of object velocities (L119), embedding dimension (L93), deep weight $w_deep$ (L115), and track management window $L$ (L134).Those parameters probably affect the performance of the system. It is not explained in the main paper or the supplementary document what values are used for these parameters in the experiments and how they are tuned. 
3. Why does Eq (4) choose to directly multiply the motion and velocity correnlation matrix? 
4. When referring to the main contribution in section 3.1 and 3.2, this paper frequently refers to [6], leaving an impression that [6] serves as a baseline of its system. However, system-wise, it is not clearly explained which components are inspired or borrowed from [6].
5. As a submission to a robotics conference, I would expect to see some analysis (ideally a break-down of the components) of the computation time of the proposed method. For example, compared to the Kalman filter, how much of a computation time trade off does it make to gain the accuracy improvement? 
6. The experiments were conducted on two benchmarks. In the implementation details, it seems that different hyperparameters are needed to tune for different datasets. From my perspective, it is fine, but an explanation may be required. I assume the same training setting does not work well. Is it because of the sampling rate and the FOV difference or something else? And does it mean that the trained model may not generalise well to different sensor setups?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper “MV-3DT: Multi-View 3D Object Tracking via Cross-Camera 3D Fusion” is proposing a pipeline for tracking objects across multiple cameras and time for autonomous driving applications. The pipeline is relying on a monocular 3D object detection network that predicts bounding boxes for each image in a multi camera system. Afterwards, the object detections are associated across all cameras and merged using 3D non-maximum suppression. The merged 3D objects are tracked over time using a learned LSTM-based motion model. Furthermore, a track management is proposed, where each object track can have the following states: active, inactive, dead. This way objects can be tracked even if they are not detected over a longer time frame.
The proposed pipeline is evaluated against recent single-camera (QD-3DT) and multi-camera (MURT3D) on the NuScenes and Waymo Open dataset. On both datasets the proposed method is outperforming the baselines.
strengths_and_weaknesses: Strengths:
* The paper is very well structured and easy to read. Therefore, one can easily get the idea of the proposed method and how it differentiates from the existing state of the art
* All design choice are well explain and a validated in the presented ablation studies
* Implementation details are given which can be helpful to reproduce the results presented in the paper
* Extensive experiments are perform to evaluate the proposed pipeline against existing state-of-the-art approaches

Weaknesses:
* The presented paper shows only minor weaknesses, which are as follows:
It would be beneficial to also report runtimes in the evaluations section. E.g. it is shown that the LSTM-based motion model is outperforming a classical EKF. However, it would be beneficial to understand how both model differ in runtime.
* To improve the presentation of the paper it would be beneficial to move some of the qualitative results from the supplementary to the main paper.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is nicely written and easy to follow. Furthermore, it is proposing a novel approach and clearly explaining the improvements to the current state-of-the-art. The performance is evaluated based on extensive experiments.

issues: * Add runtime measurements for the pipeline
* Move qualitative results from supplementary to main paper
review_title: null

Meta-review: metareview: The paper proposes a cross camera three dimensional fusion method for object tracking. The paper is of good technical quality. It is well presented and easy to follow. The reviewer concerns were adequately addressed and it is recommended to include them in the final submission of the manuscript. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)