Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors contribute a substantial dataset and baseline models for addressing map change detection from egocentric and top-down perspectives and using various sources of information.

strengths: The need for the map change detection dataset is strongly motivated.

The authors show that training on synthetic data can lead to decent generalization onto real test data, but with lower accuracy than on synthetic validation data.

The authors' findings include (1) modality fusion is better done early than later, (2) combining RGB, semantics, and map information is most beneficial for generalizing to test data, and (3) comparing maps is better done from an egocentric perspective than from a top-down perspective.
weaknesses: I believe the formulation of the mean accuracy metric should be moved to the main text.

Aside from that, I found that the main text and appendix addressed all of the questions I had while reading.
correctness: The dataset was annotated carefully through 3 reviewing panels.
clarity: Yes, the exposition is very clear.
relation to prior work: The authors thoroughly establish the differences between their work and prior work.
additional feedback: N/A.
review_title: TbV Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work proposed a nice real-world dataset for HD map change detection, and also studied the cross-modality fusion for the detection performance. Overall, this problem is interesting and the dataset is non-trivial. The paper is well-written and well-organized. The main concerns lie in the significance of the problem and the practical value of the framework. 
strengths: 1. The motivation is clear and the problem is interesting.
2. The dataset collection is non-trivial.
3. The experiments are comprehensive and convicing.
4. The code is open source for the community.
weaknesses: The two concerns are: 
1. Does the task of HD map change detection really play a crucial role in self-driving? What kind of consequences will the HD map change lead to? How the perception and planning modules will be influenced with only a subtle change in HD map, quantitatively or qualitatively? More discussions are required to evaluate the value of studying such a task, since there are many other solutions to promote the robustness of autonomous driving, e.g., online HD map creation. The authors should provide more evidence to show that the task is worth studying.

2. Can the proposed framework be used in practice? The proposed method seems to be quite expensive, yet it should be implemented in real-time for the following-up perception or planning modules which incorporate HD maps. This will further increase the computation burden for the autonomous vehicles. Considering of the low probability of HD map change in reality, should we add this computationally-expensive module to deal with a rare event? Are there any other lightweight and effective solutions?
correctness: Good.
clarity: Yes. The paper is well written and well organized.
relation to prior work: Yes. The study of HD map change is scarce and the authors have reviewed existing studies.
additional feedback: See weaknesses.
***
The authors addressed my concerns, and they also improved the paper with more discussions. So I would like to raise my score from 5 to 7.
review_title: A useful dataset for HD map change detection in autonomous driving.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a benchmark dataset for permanent HD map change detection. The goal is to decide whether the layout of the road (cross walk and lane) changes from the previously captured HD map. Various baseline models are tested on the benchmark and several research problems are identified.
strengths: Due to activities in the real world, the layout of the road may be different from the recorded HD map. As a result, map change detection is a critical task. The benchmark proposed in this paper focuses on the recognizing permanent changes. Instead, those temporary changes are left for the on-board object detection and recognition systems. The proposed benchmark provides synthetic training data that (almost) come for free without manual labeling and real-world test data.
weaknesses: 1. There is significant performance drop between the results on synthetic validation data and real-world testing data. Apparently, an immediate research direction is to apply domain adaptation models to this benchmark to bridge the map. But probably results of a simple unsupervised domain adaptation method can be reported in the paper. For example, during training, a binary classifier is trained to tell whether the HD map is real or synthetic.

2. Is it possible to consider the physical constraints in the real-world? For example, if there are no sidewalks on a road, it probably does not make sense to add a crosswalk in the HD map. Of course, it may require some manual effort, but doing so may reduce the domain gap between real-world and synthetic data.

3. Although in the paper, it claims to be focusing on the map change "detection", it is actually a "classification" task. I personally think there is a chance to provide bounding boxes (or other formats) to denote what changes are. But I have to admit I am not an expert in autonomous driving. Not sure if the (real) detection task is an overkill.
correctness: See my comments about detection vs classification.
clarity: The paper is well written.
relation to prior work: yes.
additional feedback: No.
review_title: Better base models and HD map change "detection" (not classification) task

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Detecting high-definition (HD) map change is a clear requirement for L4 autonomy due to the reason that confident but incorrect assumptions about the environment will be made with an incorrect hd map. To address this issue, this paper first introduces a hd map change detection dataset called TbV. Based on this dataset, learning-based methods are presented and detailed experiment results and ablation studies are provided.
strengths: 1. First dataset to address hd map change detection problem, which is crucial for self-driving system when adopting hd map as a prior.
2. Dataset statistics and annotation pipeline is provided.
3. Experiment results of learning-based methods and detailed ablation studies are presented.
4. Code is available for reproducing the baseline result and dataset can be easily downloaded.
weaknesses: 1. Is there any legal concern about realeasing high-definition map?
2. If government departments and car company cooperate closely and update every road change simultaneously, is there still any significance about this task?
3. I don't really know about this topic, but wonder how many car companies use hd maps as hard priors?
4. As you said in abstract, "Because real map changes are infrequent and vector maps are easy to synthetically manipulate,we lean on simulated data to train our model.". Does this indicate we can use any dataset with hd map to formulate this hd map detection task?

correctness: The content appears to be correct.
clarity: The paper is clearly written, with detailed descriptions of the datasets, annotation, and benchmark results.
relation to prior work: Discussion with related works is sufficient and clear.
additional feedback: NaN
review_title: hd map change detection

Meta-review: 3 out of 4 reviewers recommend accepting the paper and the concerns of the 4th reviewer are clearly discussed in the authors' feedback. Overall, HD maps have been heavily used in production systems for autonomous driving. The authors leverage the industrial resources and open the venue for researchers to study the HD map maintenance problem. The contribution can be very important for the machine learning research.
Decision: Accept