Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors tackle the problem of human trajectory forecasting with the help of a novel architecture. The architecture has two components, the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement, respectively. The authors construct a new challenging dataset, PAV for long-term prediction. The article demonstrates the superiority of the proposed model on ET/UCY and the new PAV dataset over state-of-the-art baseline methods. A new set of metrics, curve smoothing ADE/FDE have also been introduced here.
strengths_and_weaknesses: Strengths:
- The work is of moderate originality, with a fair presentation and writing quality. The proposed ideas of group memory and cross-modality fusion are interesting.
- The proposed models seems to be able to improve the prediction performance concerning the reported state-of-the-art results. Nevertheless, there is some lack of clarification in the experiments and evaluation part, especially about the reported SOTA results and how they were obtained.

Weaknesses:
- It is not clear the exact contribution of the PAV dataset; what is the criteria for the chosen subset of the MOT15 videos? What is the reason for using this dataset instead of existing and used datasets, such as NuScenes, Oxford RobotCar, and Cityscapes?
- It makes sense to enhance the loss function with curve smoothing during training; however, is it fair to use curve smoothing only on the ground truth during evaluation, especially for the state-of-the-art works that did not apply CS-loss? Actually, it is not mentioned if the SOTA works were retrained with CS-loss or not, and maybe it would be more convincing to include an evaluation with the default metrics as well as the new metrics.
- The reported performance of some SOTA works on the ETH/UCY datasets is much worse than the results of their original papers. Why are the original results not reported in the comparison?
- It is not mentioned anywhere in the paper what are the typical values of the trajectory threshold, beta and distance threshold, theta, used as hyper-parameters in the construction of group trajectory bank? It is also not said anywhere if the hyper-parameter values are specific to a dataset? 
- A comparative study could be performed with different types of clustering methods, used for constructing the group trajectory bank.

Minor comments:
- One minor mistake in row 108: t should start from -T_{pas+1}, instead of -T_{pas}.
questions: As also previously mentioned:
- What are the typical values of the trajectory threshold, beta and distance threshold, theta, used as hyper-parameters in the construction of group trajectory bank? Are the hyper-parameter values specific to a dataset?
- What is the exact contribution of the PAV dataset? What are the criteria for the chosen subset of the MOT15 videos? What is the reason for introducing this dataset instead of experimenting on the existing and used datasets, such as Stanford Drone Dataset (SDD), Intersection Drone Dataset (inDD)?
- The reported performance of some SOTA works on the ETH/UCY datasets is much worse than the results of their original papers. Why are the original results not reported in the comparison?
limitations: The authors did not discuss any limitations or potential negative social impact of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use the "scene history" for the human trajectory prediction. The "scene history" consists of two parts: historical group trajectories and individual-surroundings interaction. This paper proposes a SHENet to jointly use these two "scene history" clues.

Extensive experiments are conducted on ETH, UCY, and a new, challenging benchmark dataset PAV.

strengths_and_weaknesses: [Strength1] The proposed method which uses the "scene history" (especially historical group trajectories) is well-motivated. 

[Strength2] The basic idea makes sense. Historical group trajectories and individual-surroundings interaction are helpful to understand human behavior. 

[Strength3] This paper provides a new challenging dataset.

[Strength4] The proposed method achieve good performance.

[Weakness1] It is suggested to add the experiments on the SDD dataset, which is also an important benchmark for human trajectory prediction.

[Weakness2] For most real-world applications, we can not obtain the scene in which the model will be used. In the proposed method, the bank is fixed for inference after training. How to guarantee the learned bank with training scene can be used for the new environment. 

[Weakness3] The reproduction is difficult since no source code is attached in the supp.
questions: See [Weakness1] and [Weakness2].
limitations: No clear limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focus on the task of human trajectory prediction. The authors stress the importance of scene history, including historical group trajectories and individual-surroundings interaction, and design a clustering & update procedure to maintain a group trajectory bank, which is used to generate trajectory prediction proposals. A cross modal transformer fuse the feature of observed trajectory and scene information, which is then used to predict the offset from the trajectory proposals. The authors conduct intensive experiments on ETH, UCY, and PAV datasets.
strengths_and_weaknesses: Strengths:
1. The paper is well-organized and easy to follow. The authors state their motivation quite clear.
2. The novelty and originality are good. The group trajectory bank module is innovating.
3. The ablative study is comprehensive and convincing. The results shows that the main module - group trajectory bank, improve the performance by a non-trivial margin.

Weaknesses:
1. In the experiment section of ETH/UCY dataset, the authors didn't states the setting clear: what is the sampling protocol when calculating ADE/FDE? Standard protocol is best-of-20. Also, as the architecture itself is not stochastic, it'll be better if deterministic prediction ADE/FDE is added.
2. The performance of previous work in the table is different from the results from the original paper. For example, Y-net achieve 0.18/0.27 according to the original paper, but in table 2 it becomes 0.24/0.39. Is it due to different experiment setting or different implementation?
questions: None
limitations: About the ethical impact: acquiring human trajectory dataset may violate privacy under some circumstances. The authors may consider to add the potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, a method named Scene History Excavating Network (SHENet) is proposed for pedestrian trajectory prediction. The history information is divided into two categories: historical group trajectories and individual-surroundings interaction. Specifically, a group bank module and an interaction module are designed to explore these two information. In addition, the curve smoothing strategy is utilized to alleviate the individual bias. The proposed method can achieve SOTA performance on ETH/UCY as well as a new PAV dataset.
strengths_and_weaknesses: Strengths:
- Clear-written and well-organized.
- Promising performance.
- A new complex trajectory dataset.


Weaknesses: (See Questions for detailed comments)
- Lack of novelty.
- Missing technique details.
- Insufficient experiments.
questions: Concerns:
1. Memory Network is well used in many relevant tasks, i.e., trajectory prediction [1-3], question-answering, anomaly detection, etc, a subsection of related work should be included. Only one paper [4] is mentioned to tell the differences. The approach of trajectory clustering (group idea) is also not new [5-8] here. The similarity function (Eq. (1)) is the same as [4], it should be noted that it is adopted from other papers. In addition, the Transformer structures, including cross-attention and self-attention are also widely used in the trajectory prediction task. Thus, in my opinion, the novelty is not significant.

2. Some key details are missing.  
(a). Based on my knowledge, the original videos of ETH/UCY are not aligned with the pedestrian coordinates, so how to align the scene with the trajectory? Or is it directly operated in the feature-level for feature fusion?  
(b). In Line 4 of Algorithm 1, the K is initialized as 32 (Line 211), what's the reason/insight of setting this variable equals to 32. It's a really important hyper-parameter since it represents the initial cluster number. I couldn't see any analysis or experiments about this variable.  
(c). In Line 155 of Sec. 3.3, the Swin Transformer is applied for extracting scene semantic features, but the number of semantic classes is 150. I assume it is pre-trained here, but why use150 in the trajectory prediction task? I don't think it is reasonable to directly use150 semantic classes here. Details and explanations are missing here.  
(d). As for the results of ETH/UCY, some of these baselines are stochastic, the results are usually the best of 20. I couldn't see any stochastic properties in this proposed method, so are the results deterministic or stochastic?

3. Other questions.  
(a). The bank is fixed while inference, how to deal with the situation when there is a novel/unseen trajectory (pattern) when testing?  
(b). In Eq. (1), the Cosine Similarity is used there, the range of coordinates is not mentioned, usually we normalized the coordinates of ETH/UCY into $[-1, 1]$, which makes the range of $s_{i}$ is $[-1, 1]$. When $s_{i}=-1$, these two trajectories are totally opposite, how to define this kind of similarity? Is it similar or different? I think more explanations should be included here.  
(c). What's the meaning/insights of introducing curve smoothing (CS)? If using CS loss could have better performance, does it mean the trajectories in this dataset are more "stable"? Will using CS loss affect the generalization ability of the model itself, only making smooth predictions? Is that prior knowledge? 

4. Insufficient experiments.  
(a).  No experiment to support the necessity of using CS loss.  
(b).  When memory mechanism is used, I think the analysis of complexity/cost is necessary.  
(c). The study of Size($Z_{bank}$) is missing, how to initialize the memory bank, the changing process of this memory bank is necessary to support the whole method.


Minors:  
1. Line 2, Line 6 and Line 14 of Algorithm 1, $/*$ may need to adjust.
2. Line 7 of Algorithm 1, I believe Size($Z_{bank}$) is K.
3. In References, the names of conferences or journals, i.e. [1] IEEE or IEEE/CVF, are not consistent.

[1] Remember Intentions: Retrospective-Memory-based Trajectory Prediction. CVPR 2022  
[2] Graph-Based Spatial Transformer With Memory Replay for Multi-Future Pedestrian Trajectory Prediction. CVPR 2022  
[3] Multiple Trajectory Prediction of Moving Agents with Memory Augmented Networks. IEEE TPAMI 2020  
[4] Mantra: Memory Augmented Networks for Multiple Trajectory Prediction. CVPR 2020    
[5] Recursive Social Behavior Graph for Trajectory Prediction. CVPR 2020  
[6] Group lstm: Group Trajectory Prediction in Crowded Scenarios. ECCV Workshop 2018  
[7] Three Steps to Multimodal Trajectory Prediction: Modality Clustering,Classification and Synthesis. ICCV2021  
[8] PoPPL: Pedestrian Trajectory Prediction by LSTM With Automatic Route Class Clustering. IEEE TNNLS 2020  
limitations: No limitations described in the main body paper as well as the supplementary material.
review_title: null

Meta-review: metareview: Initially, the paper received mixed reviews (3456).  The major concerns raised by the reviews were:

1. What is the contribution of the PAV dataset? (XkVC)
2. There should be experiments on existing datasets, e.g. SDD or inDD. (XkVC, tAHR)
3. Is it fair to use curve smoothing on the GT during evaluation? To be fair, other works should be trained with CS loss too. What is the reason for CS, does it affect generalization of the model? No ablation for CS loss (XkVC, ZK4Z)
4. reported results of some SOTA works on ETH/UCY are worse than the published results. Why not use the original reported results? (XkVC, bcbq)
5. what are the settings of the hyperparameters? Is it dataset specific? (XkVC)
6. ablation study on the clustering methods used for constructing the group trajectory bank. (XkVC)
7. The bank is fixed after training -- how to guarantee the learned bank can be used in a new environment, new scene (tAHR, ZK4Z)
8. show results on ETH/UCY w/o using video data (tAHR)
9. can the proposed method work well from bird-eye view or when assumed information is missing? (tAHR)
10. evaluation metric used on ETH/UCY is not clear. stochastic vs deterministic? (bcbq, ZK4Z)
11. novelty is not significant (ZK4Z)
12. Missing details: scene-trajectory alignment, how to set K, how to set no. of semantic classes, range of 13. coordinates for cosine similarity, (ZK4Z)
14. No ablation on memory vs complexity (ZK4Z)
15. No ablation on the memory bank size and initialization (ZK4Z)

The authors wrote a response to address these concerns. All reviewers were satisfied with the response.  Overall, the reviewers found the paper interesting, although the approach is somewhat incremental and more experiments could be added (e.g., on SDD and inDD, as well as ablation studies).  Nonetheless, the final ratings increased to 5666.  The AC agrees with the reviewers and thus recommends accept. The authors should further revise the paper according to the reviews and discussion. 
confidence: Certain
Decision: Accept