ESDMotion: End-to-end Motion Prediction Only with SD Maps

Authors: Zhiyuan Zhang, Xiaosong Jia, Qifeng Li, Guanyu Chen, Junchi Yan
Primary area: applications to robotics, autonomy, planning
Keywords: motion prediction

Abstract:
Motion prediction is a crucial task in autonomous driving. Existing motion prediction models rely on high-definition (HD) maps to provide environmental context for agents. However, offline HD maps require extensive manual annotation, making them costly and unscalable. Online mapping-based methods still require HD map annotation to train the online mapping module, which is costly as well and may also suffer from the issue of out-of-distribution map elements.
In this work, we explore conducting motion prediction with standard-definition (SD) maps as substitution, which are more readily available and offer broader coverage. One crucial challenge is that SD maps have low resolution and poor alignment accuracy. Directly replacing HD maps with SD maps leads to a significant drop in performance. 
We introduce end-to-end learning and specially tailored modules for SD maps to solve the problems. Specifically, we propose ESDMotion, the first end-to-end motion prediction framework that uses SD maps without any HD map supervision. We integrate BEV features obtained from raw sensor data into existing motion prediction models, with tailored designs for anchor-based and anchor-free models respectively. We find that the coarse and misaligned SD maps bring challenges to feature fusion of anchor-free model and on anchor generation of anchor-based model. Thus, we design two novel modules named Enhanced Road Observation and Pseudo Lane Expansion to address these issues. Benefiting from the end-to-end structure and new modules, ESDMotion outperforms the state-of-the-art online mapping-based motion prediction methods by 13.4\% in motion prediction performance and narrows the performance gap between HD and SD maps by 73\%. We will open source our code and checkpoints.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730280217475

thread_type:
official_review

<thread_object>
creator: reviewer_9kqe

<content>
summary:
The paper presents a method to integrate Standard Definition (SD)-maps information in trajectory prediction models. The model jointly integrates sensor-based features (BEV maps) with SD-maps features with a deformable attention. Additionally, to account from the poor spatial localization of SD-maps, and missing fine-grained details, the method suggests a simple “pseudo-lane expansion” where artificial lanes are created parallelly to the main lane of the SD map. Each of these lanes serves in the deformable attention (“Enhanced Road Observation”) and provides goal candidates in anchor-based methods. Experiments are conducted on nuScenes.

strengths:
* Introducing SD-maps, that are readily available with cheap and crowd-sourced maintenance, in trajectory prediction models is a good idea. While some previous works exist to use SD maps as priors to build HD maps, the impact on downstream application (here trajectory prediction) is an underexplored and important topic.

* The proposed modules are motivated, and their instantiation is sound.

* The method is designed for both anchor-based and anchor-free methods.

* The paper is well illustrated.

* The discussion on the contribution differences between SD / HD maps in anchor-based/anchor-free models was appreciated.

weaknesses:
`[Model presentation]` The presentation of the method (section 3) could be significantly improved. In the current form, the model presentation interleaves the general trajectory prediction framework (from previous works), contributions of the work (enhanced road observation, pseudo lane expansion), high-level insights, and technical details (e.g. instantiation of the encoders, etc…). The presentation of the instantiation of the method for either anchor-based or anchor-free models can be also improved. For instance, the title of Section 3.2 targets anchor-free models but “the anchor-based model DenseTNT” is discussed at the end.

`[Pseudo-lane expansion]` DenseTNT already “expands” the lanes by densely sampling candidate goals around the lanes. It looks like the proposed pseudo-lane expansion simply amounts to increasing the size of the sampling kernel in denseTNT. Can the authors comment on this?

`[Prior works]`
There are prior works that try to infer HD online maps from SD-maps and sensor-data [1,2,3]
The reviewer suggests discussing these approaches in the related work section, as it offers another way to integrate SD maps.

* [1] Mind the map! Accounting for existing maps when estimating online HDMaps from sensors. Sun et al. 2023
* [2] Driving with Prior Maps: Unified Vector Prior Encoding for Autonomous Vehicle Mapping, Zeng et al. 2024
* [3] Local map Construction Methods with SD map: A Novel Survey, Li et al. 2024

`[Fair baselines]` The comparison between ESDMotion and “Unc”/”BEVPred” is unfair as ESDMotion uses SD maps which the latter ones do not (lines 369-374). Related to the previous remark, it would be valuable to compare ESDMotion with “Unc” + “online HD mapping based on SD maps” and “BEVPred” + “online HD mapping based on SD maps”.

`[Impact of the Enhanced Road Observation]` The improvement brought by the Enhanced Road Observation module is somewhat limited (Table 4).

`[Map updates]` The review agrees that HD maps must be often updated (l.43). However, I believe the use of SD maps does not solve this problem as SD maps should be updated as well.

`[Multimodal future prediction]` How many modes are generated by the models? (l.362,363).

`[Title]` The use of the word “only” in the title is questionable. There are some works doing end-to-end motion prediction without any maps (SD / HD).

`[Writing clarity]` The many paper typos, syntax and grammar issues hurt the readability of the paper. A thorough proof-read is strongly recommended. A non-exhaustive list of typos are shown below.
* “ground truth” → Hyphen is needed when used as an adjective
* Multiple spaces “ “ are missing, e.g., before citations, parenthesis, after “.” etc…
* Line 220. Is it Sec 4.3?
* Line 334. The subject is missing. “We”?
* Line 413 “tab1”
* Line 296 “an”
* Please refer to Figure 1 and Figure 6 in the text where appropriate.
* Figure 3 is discussed after Figure 4.
* Some notations are introduced but never used (e.g., l.293,335) 
* Opening double quote signs are off
* Illustrations are pixelated, pdf figures are generally preferred as they also allow for text selection and search.
* Tab 3 and 4 should be at the top of the page.
* In the reviewer’s opinion, the usage of bold sentences is too excessive.

questions:
Beyond the need for improved paper presentation and proofreading, answers to the points on `[Pseudo-lane expansion]` and `[Fair baselines]` may lead to a reconsideration of the reviewer’s recommendation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536202243

thread_type:
official_review

<thread_object>
creator: reviewer_tmwe

<content>
summary:
To address the dependency of trajectory prediction models on high-cost offline high-precision maps or online mapping modules, this paper proposes ESDMotion. ESDMotion uses surround-view images and globally covered SD maps as environmental input for trajectory prediction, and experiments were conducted on nuScenes. The method proposed in this paper to use SDMap is interesting. However, I have concerns about the contributions of this paper and the fairness of some experiments.

strengths:
1.Direct and clear motivation with coherent writing.  
2.The approaches of better utilizing SDMap (Pseudo Lane Expansion and Enhanced Road Observation) are interesting.

weaknesses:
1.The paper is misleading (including the title and contribution 1) readers. It does not only rely on SD Map for trajectory prediction but rather combines surround-view images. The surround-view images provide crucial local environmental information, which models like HiVT and DenseTNT do not utilize.  
2.The emphasis on the end-to-end architecture of ESDMotion as a contribution is questionable. With BEVPred and ViP3D's work[1-2] and open-source code as a foundation, training a model end-to-end from images to BEV features to trajectory prediction presents no real challenges or contributions. The contribution of this paper seems lacking. Although the code[3] provided by BEVPred replaces mapping module features with HD maps as input for trajectory prediction, it can easily be modified to enable end-to-end training of the BEV encoder and trajectory prediction model.  
3.From Table 2, it can be seen that for HiVT, the minADE and minFDE using HDMap and SDMap are quite similar. While ESDMotion claims to improve minADE and minFDE through an end-to-end architecture or BEV features, as I mentioned in point 2, I believe this end-to-end architecture should not be counted as a contribution of this paper. This improvement can also be observed in the comparison between BEVPred and offline HDMap in Table 1.


References:  
[1]. Gu X, Song G, Gilitschenski I, et al. Accelerating online mapping and behavior prediction via direct bev feature attention[J]. arXiv preprint arXiv:2407.06683, 2024.  
[2]. Gu J, Hu C, Zhang T, et al. Vip3d: End-to-end visual trajectory prediction via 3d agent queries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5496-5506.  
[3]. https://github.com/alfredgu001324/MapBEVPrediction

questions:
1.Although SD maps are globally covered, there are small areas like parking lots (which indeed exist in nuScenes) that are not covered. How does DenseTNT handle such situations?  
2.Why is ESDMotion unable to outperform BEVPred on DenseTNT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716587560

thread_type:
official_review

<thread_object>
creator: reviewer_arah

<content>
summary:
The paper presents ESDMotion, an end-to-end motion prediction framework that uses only standard-definition (SD) maps for autonomous driving motion prediction to address the limitations of high-definition maps. It integrates BEV features from raw sensor data into existing motion prediction models and presents Enhanced Road Observation and Pseudo Lane Expansion to handle the challenges brought by SD maps. Experiments on the nuScenes dataset show competitive performance compared to online mapping-based methods and mitigate the performance gap between HD and SD maps.

strengths:
1. This paper proposes an end-to-end motion prediction framework that relies solely on SD maps, addressing the scalability and cost issues associated with HD maps.
2. This paper introduces Enhanced Road Observation and Pseudo Lane Expansion to enhance feature fusion and anchor generation for SD maps, addressing the limitations of using SD maps in motion prediction.
3. The proposed ESDMotion obtains comparable or even better performance compared to methods based on HD maps or GT maps.

weaknesses:
1. Although the proposed approach ESDMotion obtains improvements in Tab.1&2, the baseline methods (DenseTNT and HiVT) are too old. It will be better to adopt the latest methods.
2. The proposed ESDMotion aims for end-to-end motion prediction however it still relies on SD maps as extra inputs, which is different from previous end-to-end works[1,2].
3. The use of SDMap feature fusion as a way to enhance BEV features is already common; the authors focus on fusing BEV features and SDMap prior features around the agent, which is innovative but lacks sufficient novelty.
4. The nuScenes dataset mostly contains simple road structure scenarios (PARA-Drive, BEVPlanner), in some complex scenarios, will the ESDMotion scheme proposed by the authors perform more excellently, for example, compared with PARA-Drive?


References\
[1] Jiang et al. Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction. arXiv 2022.\
[2] Gu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730820116302

thread_type:
official_review

<thread_object>
creator: reviewer_ngwg

<content>
summary:
This paper proposes an end-to-end trajectory prediction approach based on SD-map, termed ESDMotion, and validates its effectiveness across two common trajectory prediction paradigms: anchor-based and anchor-free schemes. The core contributions include an Enhanced Road Observation strategy to facilitate interaction between agent and SD-map features, as well as a Pseudo Lane Expansion approach to address the insufficient sampling of goal points in anchor-based methods by broadening the sampling range. Experiments on the nuScenes dataset shows the performance improvements in trajectory prediction using SD-map, narrowing the gap with the strategies that rely on HD maps.

strengths:
1.This paper is the first to propose and validate the use of SD-Map for end-to-end trajectory prediction tasks. It achieves performance that is comparable to, or even surpasses, approaches utilizing online predicted HD-map.
2.The proposed strategy effectively reduces the trajectory prediction performance gap between using SD-map and HD-map.

weaknesses:
1.The Pseudo Lane Expansion strategy proposed is a relatively coarse goal point sampling approach, as the generated pseudo lanes may not align with the actual road structure, potentially resulting in out-of-bound instances and other inaccuracies.
2.In the experimental section, validation is lacking for the performance of replacing SD-map with online HD-map predictions from models such as MapTRV2, combined with the proposed strategies in this study. Including this experiment would greatly enhance the validity of the paper's findings.

Limitations

Although HD-maps are not required, this study still relies on offline-generated SD-Map, which limits its applicability. Additionally, for anchor-based trajectory prediction approaches, the proposed strategy does not demonstrate a performance advantage over methods that utilize online-predicted maps, highlighting a limitation of this approach.

questions:
What will the trajectory prediction performance be like if the online predicted SD-map is used as input instead of the offline SD-map?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732200706495

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear AC and reviewers, 

We express our gratitude to all reviewers for their valuable time and insightful comments. 
Following your kind advice, we add more experiments, results, and explanations. Please check [our updated paper](https://openreview.net/pdf?id=sEJYPiVEt4) where the changes are marked in blue.

There are two common concerns raised and we clarify below.

> **Q1. End-to-end learning for motion prediction is not new. What is the contribution of this work?**

- The major focus of this work is not about end-to-end learning. In this work, we aim to address a previously unexplored problem: **using SD maps as the sole map information for motion prediction to reduce the cost of relying on HD maps**.
- We find out two issues (low resolution and poor alignment) caused by using SD maps for motion prediction. To overcome these issues, we design an end-to-end framework and tailored modules to better utilize SD maps. **The end-to-end learning is the high-level methodology of our solutions while our contributions lie in the SD map specific designs**.
-  As a result, our method demonstrates that **even with the low-cost SD maps, it is possible to achieve motion prediction performance comparable to that using HD maps**.
- As pointed out by multiple reviewers, we agree that current titles and writing may cause misunderstanding regarding the contributions. Thus, following reviewers' kind advice, we change our title to **ESDMotion: SD Map Oriented Motion Prediction Enhanced by End-to-end Learning** to emphasize our focus of SD map for motion. Accordingly, we revise the structure and wording in [our updated paper](https://openreview.net/pdf?id=sEJYPiVEt4) to emphasize the SD map oriented motivations. 

> **Q2. Why do anchor-based methods fail to achieve the same level of improvement as anchor free method?**

The reason is that: **Anchor-based methods like DenseTNT use lanes to generate goal candidates**. The sparesity and misalignment of lanes in SD map would lead to missing of nearby candidates and thus bad performance.

Inspired by reviewers' advice, we improve DenseTNT under SD map scenario in two ways:
1. There are small areas like parking lots that are not covered by maps. For these areas, we define a set of pseudo lanes at specific angles capturing the potential driving directions to generate goal candidates, as shown in this [figure](https://postimg.cc/hhgXPL3b). This updated version is named DenseTNT+ and ESDMotion+. As shown in Table below, such modifications explicitly improve the performance of DenseTNT.

| Models    | Maps      | minADE $\downarrow$ | minFDE $\downarrow$ | MR $\downarrow$ |
| ----------| ---       | ------------------- | ------------------- | --------------- |
| DenseTNT  | SD map | 1.2117              | 1.9849              | 0.2776          |
| DenseTNT+ | SD map | 1.1746              | 1.9421              | 0.2630          |
| ESDMotion | SD map | 0.7941              | 1.3863              | 0.1627          |
| ESDMotion+| SD map | 0.7783              | 1.3784              | 0.1616          |

2. In the origin version of Pseudo Lane Expansion, we expand a fixed number of parallel lines at uniform distances, which introduces too many candidates and thus hurts performance. To reduce those unnecessary candidates, **we propose to dynamically adjust the parameters of Pseudo Lane Expansion based on the position and density of SD map.** Specifically, for misaligned SD map as shown in [figure](https://postimg.cc/qNZJsnLY) (upper part), we generate more pseudo lanes on the side closer to the vehicle and reduce the number on the opposite side. To deal with different density of SD lanes as shown in [figure](https://postimg.cc/qNZJsnLY) (lower part)/Figure 5 in our paper, we increase the number of pseudo lanes to ensure better coverage for spare ones while decrease the number for dense ones. This further updated version is named ESDMotion++. As shown below, the performance is further enhanced.
 
| Models      | Maps   | minADE $\downarrow$ | minFDE $\downarrow$ | MR $\downarrow$ |
| ----------- | ------ | ------------------- | ------------------- | --------------- |
| ESDMotion   | SD map | 0.7941              | 1.3863              | 0.1627          |
| ESDMotion+  | SD map | 0.7783              | 1.3784              | 0.1616          |
| ESDMotion++ | SD map | 0.7597              | 1.3105              | 0.1523          |

We add these discussions, methods, and experiments into [our updated paper](https://openreview.net/pdf?id=sEJYPiVEt4). We sincerely thank the reviewers for their constructive comments.

We hope our responses could solve your concerns. If you have any further concerns or questions to discuss, we are more than willing to address them. Looking forward to your further suggestions to polish the paper!

Thanks,
Authors of Submission 2920
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a SD map conditioned motion prediction framework, which solely relies on SD maps and onboard signals. This paper reveals the gap between the models trained with SD maps and those with HD maps.  After reviewing the extensive discussions among the reviewers and authors, there is still a mix of borderline evaluations. Below, I summarize the key considerations.

Strengths: 

1. This paper presents a novel setting that could benefit future autonomous driving pipelines. 
2. The proposed methods are general, working with both anchor-based and anchor-free methods. 
3. The results with SD maps are comparable to those with HD maps under several settings.
4. Thorough response and extensive additional experiments are provided in the rebuttal period. 

Weaknesses:

1. The proposed method is not technically novel and similar ideas have been studied in prior arts.
2. Reviewers suggested experiments to combine SD maps and online predicted maps. Although authors provided additional experiments, however, reviewers suggested potential unfairness if authors did not include the aforementioned experiments. 
3. Paper presentation quality has to been improved. 

After reading these divided reviews and discussing with reviewers, AC decided to reject this paper because: 

1. The paper's technical novelty is limited given the prior work suggested by reviewers. Also the empirical results do not justify the usage of SD maps due to the remaining gap. Both aspects do not meet the standard of ICLR. 
2. Writing needs to be greatly improved to make the cut of ICLR.