Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes an interesting dataset that focuses on the new task of trajectory retrieval, named the Argoverse Trajectory Retrieval Benchmark. The dataset proposed contains 2795 trajectories of 13 different driving intents that are annotated by human annotators as well as some domain experts. It is a rather challenging task and the experiments show a relatively large performance gap between current methods and the upper bound. 
strengths: The paper is clearly written with sufficient initial experiments demonstrating a promising and challenging research area. The dataset, as well as the proposed task, opens a new area that allows future AV research to model driving behaviors and to analyze the ambiguous driving intent.
weaknesses: The scale of the dataset seems to be a concern in my opinion. With a total number of 2795 trajectories, I have some doubts that the current status of the dataset would be sufficient to train large-scale models. I hope to see a larger number of trajectories in future iterations.

Another concern is that it seems that there are many types of intents that are related to "Deceleration", at least in the training set. I am concerned that this introduces some intrinsic bias in the dataset. Was there any reason why most of the intents are related to "Deceleration"? Do you plan to mitigate this potential bias?
correctness: Yes the dataset is constructed in a sound way, but there are some concerns as stated in the "Weakness" section. The evaluation criteria and experiments seem reasonable.
clarity: Yes the paper is clearly written.
relation to prior work: Yes this is stated in the related work.
additional feedback: N/A
review_title: The Argoverse Trajectory Retrieval Benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this work, the authors proposed the Argoverse Trajectory Retrieval Benchmark, a retrieval benchmark that is built upon the Argoverse Motion Forecasting dataset. The goal of this benchmark is to advocate works in finding similar trajectories from a dataset given some query trajectory. The authors split 2,795 trajectories into query and retrieval sets and annotated 13 types of intents with 5 only present in the test set. Baseline experiments on the retrieval task are also shown using nearest-neighbor retrieval with different embedding functions. The authors also showed fine-tuning using contrastive learning and a "relevance feedback" framework that can provide the ground-truth relevance feedback to a small set of initially retrieved trajectories.
strengths: 1. The proposed work is the first benchmark of its kind, which contains trajectories annotated with clearly defined intents.
2. The proposed work can facilitate research in trajectory retrieval and intent prediction and can be applied to the field of autonomous driving and beyond.
3. The proposed work can promote explainability in the maneuvering of autonomous vehicles and help to query interesting scenes in large-scale autonomous driving systems.
4. The proposed work builds upon an existing trajectory dataset, which enables multitask learning and unsupervised learning that can leverage the unlabeled data.
weaknesses: 1. Although the proposed dataset is the first of its kind, the number of scenarios is relatively low compared with current deep learning datasets.
2. Experiments can be more sufficient. For example, the role of triplet loss in Section 5.2 is not clear because the results are inconsistent. The authors should include the proposed joint training strategy using triplet loss.
3. The filtering rationale discussed in line 175 is not mentioned in the paper, plus not justified. Also, it seems that <10% of trajectories are selected for labeling. Would it be helpful to release some "distractors" with no clear intent found?
4. The selection of the types of intent is not justified. It would be helpful for the authors to explain how they arrived at the set of 13 intents proposed in the paper.
correctness: The benchmark is designed in a fairly straightforward way, and the setup is clearly defined. The benchmark is developed based on the existing Argoverse dataset. The baseline experiments in the dataset include multiple learned and non-parametrized embedding functions. However, there are certain areas that are not clear:
1. The rationale of hand-selection in the dataset (such as selecting the query set) is not explained clearly. Will the hand-selection part introduce undesired bias? In addition, the hand-selection strategy might not be optimal for scalability.
2. The paper mentions that each trajectory is 5 seconds long, but the authors have not mentioned how these trajectories are trimmed, or if the trimming was reused from the Argoverse Motion Forecasting dataset. If the former is the case, what's the rationale for the trimming? If the latter is the case, is the trimming strategy suitable for both of the two benchmarks?
clarity: In general, the paper is well-presented.
1. Equation (1) is confusing; NDCG should be explained more clearly.
Misc:
* Line 178: "then THEN decelerate".
* The best numbers in Table 2 can be bolded to make the table easier to read.
relation to prior work: The related work is well-reviewed and presented, and the authors compared the proposed work to prior ones. However, is there existing dataset that provide any form of annotation for intent? If so they should better be included in the paper.
additional feedback: 1. For relevance feedback, only m=5 is shown. It might be helpful to show the impact of different numbers/rounds of relevance feedback on the performance. The application of relevance feedback should also be discussed more clearly to motivate this setting.
2. It would be interesting to extend the work to leverage the unlabeled data by unsupervised learning.
3. It would also be interesting to use a trained model to query the unlabeled dataset in Argoverse and showcase the potential of this work for automatic labeling of intents.
4. Since sometimes intents might coincide, it would be interesting if the authors can visualize the correlation between intents.
review_title: The Argoverse Trajectory Retrieval Benchmark

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper develops a benchmark dataset for trajectory retrieval task in driving scenario.  It introduces a new dataset where a trajectory annotated with intent that is to be matched and ranked against other trajectories based on a similarity metric. To set the baseline, the paper evaluates the dataset against four embedding functions on top of a feedback based method.

strengths: It is claimed to be the first work on standardized benchmark for trajectory retrieval under driving scenario.

The authors provide a number of baseline evaluation results based on diverse set of features/methods.
weaknesses: It is not clear whether an additional explicit intent label is required to capture interesting information arising from a social interaction. For one, one of the experiments, FEAT, which only uses domain specific hand crafted features without an intent, seems to do comparably to a more elaborate one, WIMP. In fact, there is no strong evidence in paper text that intent was used in any of the experiments in section 5.1.

Second, if intent were used meaningfully to contribute to improvements ranking, there should be an ablation study that corroborates its utility.  

Third, I argue that the social interaction is already encoded in the trajectory data without an explicit label. Since the trajectory data contains data of k-agents, a model with sufficient capacity should be able to learn how an ego agent behaves with respect to other k-agents. Not only the labels lack capacity to depict a more complex scenario (imagine, within 5 seconds, a car behind an ego agent overtakes and appears in front in the end; would the label, ‘Decelerate for moving lead vehicle’, mean a leading vehicle in the beginning or in the end?), but also fails to capture relationships with respect to j <= k agents (e.g., a car behind the ego is an ambulance car and an ego slows down for the car behind while keeping its relative speed and position to the leading vehicle). Instead of enumerating intent labels, it may be better for queries to indicate an ego and j-agent trajectories.

Lastly, the size of the dataset is not large. From the content in the paper alone, I cannot be confident that a model can learn social interactions especially when one is significantly different than known ones (i.e., when an intent cannot be characterized as a linear combination of known intents).
correctness: The submission is missing a number of items in the checklist including report on error bars, details on dataset and all supplementary materials that should accompany the paper. As it currently stands, the submission is incomplete.
clarity: The paper is fairly well written and not difficult to follow.
relation to prior work: I lack knowledge in the autonomous vehicle domain that I cannot confidently cover all prior contributions. However, amongst the prior work presented by paper, it was not clearly indicated how this work differs from work of Segal et al.
additional feedback: See above.
review_title: Initial review of the Argoverse Benchmark

Meta-review: Through discussion with the other AC and PC, we conclude that the paper is still lacking in experiments and justification for the limited size and variety of the proposed dataset. The overall rating of the paper is leaning towards rejection and we don't see a strong argument to override the reviews.
Decision: Reject