Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores the long-term occlusion problem in multi-object tracking, and proposes to using trajectory forecasting methods to compensate the tracking losts. The forecasting module is conducted in bird-eye-view. The method can advance state-of-the-art trackers on the MOT Challenge dataset. 
strengths_and_weaknesses: Overall, I like the idea in this paper. I agree that most current multi-object trackers do not tackle long-term occlusions. The solution in this paper is sound and makes sense to me. However, I feel that the work of this paper hasn’t been finished yet. It doesn’t have supportive experiments. I hope the authors can improve the draft with the following comments. 

- Strengths
1. The paper is well-written and easy to follow. 
2. The paper tackled an existing problem in multi-object tracking and the solution is reasonable. 

- Weaknesses and Suggestions 
1. It makes sense that long-term occlusion rarely occurs in MOT Challenge datasets. However, the paper should have supportive experimental results. Current results, boost tracker by ~0.1 HOTA cannot prove the effectiveness of the method. I suggest the authors try to test the oracle of forecasting matching and show the advanced percentage of the method in this part of errors. 
2. How about doing experiment in autonomous driving datasets? The dataset has camera parameters. 

questions: See weaknesses. 
limitations: The authors discussed the limitations in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates the exploitability of trajectory forecasting in multi target tracking. First, unmatched trajectories and unmatched detections are projected in bird-eye view (BEV), trajectories are then extended according to some model and tested for association. Projection in BEV also introduces visibility constraints that further reduces the search space of true matchings. The author validate their projection method against the true 3D position of pedestrians (in MOTSynth), then study the applicability and benefits of different trajectory forecasting models. They show that the proper combination can improve the HOTA score and reduce ID switches of 7 SOTA methods on MOT17 validation set. In the supplementary, the authors detail how their projection method can be extended to moving sequences.
strengths_and_weaknesses: - (+) paper is well written and nicely structured, the reading is fluent; figures and tables are also helpful in the understanding of the key steps
- (+) the analysis of the contributions is carried out with an appropriate level of rigour
- (+) results are in favor of the proposed approach 

- (-) HOTA is a complex and composed measure that tries to unify/project many aspects of tracking to a single scalar, so the authors have to compensate by also using AssA, AssPR as well as ID switches. Why did the authors thought these metrics to be more appropriate than ID recall [A] which measures the ability of a tracker to associate the same identity to a trajectory despite occlusions and interruptions? Isn't this exactly what the authors are trying to improve? Is the "ID Recall" in Fig.1 the one from [A]?

[A] Performance measures and a data set for multi-target, multi-camera tracking

- (-) From what I can see from Tab.1 most of the gain comes from a linear prediction method in 3D space. Other than that there is only a 0.5% HOTA left to gain from other trajectory forecasting methods. In this perspective I think the paper is overstating the importance of trajectory forecasting methods in multi target tracking and not helping the reader draw the correct conclusions. 3D tracking has been around since MOTChallenge 15 and kalman filtering even earlier and this seems to be the thing that explains 90% of the performance improvement (54.11-50.71)/(54.52-50.71).

- (-) For the same reason as above, I would have liked to see a comparison on 3D MOT 2015

- (-) In Fig.4 B we see a small difference between H and pixels L2, and a large difference between pixels L2 and pixels. The only way the reviewer can explain the difference between pixels L2 and pixels is through the use of a threshold in different domain, which seems to indicate that the threshold in BEV is less tight than the threshold in pixel space. As a matter of fact 2m seems like a very large margin to associate a trajectory and a detection, how did the authors choose this threshold? I think to remember that for both MOTA and IDF1 the threshold for 3D tracking was 1m instead of 2m (can be checked in the MOTChallenge evaluation kit).

- (-) lines 127-129 "simply applying trajectory prediction to MOT is not trivially possible" are misleading, the community has been doing this for years and it's also reported in Tab.1 Kalman Filter (pixel) with decent results.



questions: - Can the authors add the ID recall from [A] to the ablation studies in Tab.1 (the MOTChallenge evaluation kit does provide the means to compute it as it is required to compute the final ID F1) or, in alternative, justify why they don't find the metric appropriate and instead use AssA, AssPR and IDS.
- Can the authors help the reader putting in perspective the impact of the different contributions, i.e. working in bird eye view vs trajectory forecasting?
- Can the authors motivate the choice of 2m as a threshold for BEV matching and try to justify the difference between pixels and pixels 2D in Fig.4b?
- Can the authors help the reader understanding the end of line 322 "tracking results suggest otherwise"? How do the authors explain this unexpected result?
limitations: nothing else to add
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates the problem of long-term multi-object tracking. This is a relatively under-explored problem that worths investigation since most existing tracking methods focus on short-term tracklets (usually shorter than 2s). The main contribution of this paper is that they project the scene into bird-eye views (BEV) using homography transformation, and then apply trajectory forecasting and tracking jointly in the BEV space. The original version described in the main text supports only videos captured from fixed views, and an extending version described in the supplementary material steps further to support videos captured from moving cameras. Results on standard test sets suggest the proposed method improves tracking accuracy when combined existing trackers, and refreshes state-of-the-art when combined with the best performant tracker ByteTrack.
strengths_and_weaknesses: Originality:

Good. Existing MOT datasets and evluation metics focus more on short-time tracking accuracy. While many methods already perform fairly well on these metrics, they always fail when targets lost for relatively long time.  This paper introduces trajectory forecasting in the BEV space to handle long-term lost tracklets and shows promising results. To my knowledge, this is not well-explored in previous literature so the originality is good.

Quality:

Good. The entire pipeline is simple, easy to understand, and seems work well for long-term tracklets.  
My biggest concern is about the forecasting module. From Table 1 it can be seen that applying Kalman Filter in the BEV space already performs well in terms of both prediction and tracking, and using advanced learning based methods seems do not bring too much gain. This is somehow below my expectation since Kalman Filter is such a simple linear model. Perhaps the problem lies in that most motion in the considered dataset is linear, and in this case, comparing different forecasting modules in a dataset with more complicated (non-linear) motion partterns may be helpful to validate the effectiveness of GAN. 

Clarity:

Good but can be improved. In general, the presentation is clear and easy to follow. However, it would be better if key experimental conclusions are highlighted more clearly, especially for Table 1. 

Significance:

Good. The long-term tracking problem is absolutely an imporant problem that is not well-explored due to limitation of existing datasets and metrics. This paper sheds new light on a promising direction towards solving this problem:  combine forecasting and tracking in BEV space.

questions: 1. As mentioned in the paper, forecasting multiple possible trajectories for lost tracklets bring more computation cost when perform association. It would be good to see how this affect the tracking accuracy/efficiency and how to do the trade-off.

2.  When dealing with moving cameras, homography is estimated for each single frame. This seems contradict conclusions in Figure 4(a), where it is shown that per-frame depth estimation is not stable so that the esitmated homography is largely affected. Is it possible to stablize the  depth estimation along the temporal dimension? 

Typo:
L#309 0.43% -> 43.0%
limitations: The authors carefully discussed the limitation and and potential negative societal impact of their work in the main text.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper discovers that trajectory predictions for moving agents will significantly reduce this search space and explores the trajectory prediction can improve long-term tracking robustness of MOT. Furthermore, they show that their proposed method reasons MOT in the bird-eye space and generates a small yet diverse set of forecasts while accounting for their localization uncertainty. Therefore, they manage to advance state-of-the-art trackers on the public benchmarks.


strengths_and_weaknesses: ### Strengths
- This idea is interesting and it will benefit the development of MOT community. Considering occlusions are one of the main challenges in tracking, this work proposes a novel way to overcome this concern. 
- This paper identifies several hurdles in the integration of trajectory prediction and MOT. Besides, there are several interesting and inspiring conclusions. 
- The presentation of this paper is fine and the organization is clear.
- They achieve the state-of-the-art performance on public benchmarks. Maybe, it may inspire more following MOT methods on this path. 

### Weaknesses
- The framework seems too complex. It includes five independent sub-models, e.g., depth estimation network and segmentation network. All of these sub-models require separate training, which may degrades the robustness of the framework.
- Before the deep learning era, there are a few works that already attempted to incorporate crowd motion prediction or crowd motion models, e.g., social force, to MOT, some of which are listed below. It'd be better to refer to these works.
> G Antonini，SV Martinez，M Bierlaire，JP Thiran: _Behavioral Priors for Detection and Tracking of Pedestrians in Video
Sequences_ IJCV 2006  
> Stefano Pellegrini, Andreas Ess, Luc Van Gool: _You'll Never Walk Alone: Modeling Social Behavior for Multi-target Tracking_ ICCV 2009  
> Kota Yamaguchi, Alexander C. Berg, Luis E. Ortiz, Tamara L. Berg: _Who are you with and Where are you going?_ CVPR 2011  
> Wenxi Liu, Antoni B. Chan, Rynson W. H. Lau, Dinesh Manocha: _Leveraging Long-Term Predictions and Online Learning in Agent-Based Multiple Person Tracking_ TCSVT 2015  





questions: - The proposed model estimates the depth and segmentation masks of the video frame, and thus project to BEV space. The quality of BEV is the basis of trajectory prediction. This paper indeed investigated how good the view projection (in Fig. 4) in synthetic data. I wonder if is it possible to conduct experiments to assess the impact of different view projection methods on MOT? 
- The pointers to tables are missing, which makes reading a bit difficult.
- MOTA seems to be a more popular metric in MOT challenge. In previous MOT works (e.g., Bytetrack), they provide MOTA metrics. For better reference, it would be better to provide MOTA as well.
- The implementation details on how to cope with the moving camera are not clearly provided. 
limitations: - The major limitation of this work on the uncertainty of homography transformation has been discussed. Their proposed method partially account for these uncertainties via simple strategies. Considering this is a novel exploration along this direction, I think the limitations have been sufficiently addressed. 

- The potential negative societal impact of their work was not dicussed.
review_title: null

Meta-review: metareview: The paper initially had mixed reviews 4567. The main concerns of the reviewers were:
1. can better show the improvement on long-term occlusions (cbmW)
2. lack of results on autonomous driving datasets w/ camera parameters. (cbmW)
3. Questions about the evaluation metrics used (yuJE, Tgjz)
4. In Tab 1, most of the HOTA gain comes from linear prediction in 3D space, i.e., Kalman filters.  (yuJE)
5. comparison on 3D MOT 2015 (yuJE)
6. missing ablation study on association threshold (yuJE)
7. what is the tracking / efficiency tradeoff for forecasting (XrjC)
8. how to deal with moving cameras (XrjC, Tgjz)
9. complex pipeline requires training separate sub-models (Tgjz)
10. ablation study on the different view projection methods (Tgjz)

The authors wrote a response to address these concerns. The reviewers were largely satisfied with the response. Reviewer yuJE still had a concern about the message of the paper (Point 4; Reviewer's point [A.1]), and responded:
> The authors replied by assessing that working in BED is already trajectory forecasting. I do not agree with that, that is just 3D or metric tracking. And metric tracking + kalman filter, which explain 90% of the contribution of the paper, should not be advertised as novelty, nor as trajectory forecasting. This view that I am suggesting here, clearly help the reader in understanding that trajectory forecasting is really of little help in MTT (~0.5% HOTA), which is the opposite of what the paper is claiming.

> As I see it, the paper has merits, e.g. ways to go from image to BED in static as well as in moving sequences, but that is not the story told by this paper (the most interesting part being in the supplementary material).

Nonetheless, the final ratings were positive (5667), and the reviewers appreciated the problem solution to handle long-term occlusions, and brings a promising direction for future research.  The AC agrees and recommends accept. The authors should revise the paper according to the reviewers' comments and the discussion. 

confidence: Certain
Decision: Accept