Learning to Drive Anywhere
Authors: Ruizhao Zhu
Peng Huang
Eshed Ohn-Bar
Venkatesh Saligrama
Venue: CoRL 2023 Poster (2023)
Keywords: Global-scale Autonomous Driving
Imitation Learning
Transformer

Abstract: Human drivers can seamlessly adapt their driving decisions across geographical locations with diverse conditions and rules of the road, e.g., left vs. right-hand traffic. In contrast, existing models for autonomous driving have been thus far only deployed within restricted operational domains, \ie, without accounting for varying driving behaviors across locations or model scalability. In this work, we propose GeCo, a single geographically-aware conditional imitation learning (CIL) model that can efficiently learn from heterogeneous and globally distributed data with dynamic environmental, traffic, and social characteristics. Our key insight is to introduce a high-capacity, geo-location-based channel attention mechanism that effectively adapts to local nuances while also flexibly modeling similarities among regions in a data-driven manner. By optimizing a contrastive imitation objective, our proposed approach can efficiently scale across the inherently imbalanced data distributions and location-dependent events. We demonstrate the benefits of our GeCo agent across multiple datasets, cities, and scalable deployment paradigms, \ie, centralized, semi-supervised, and distributed agent training. Specifically, GeCo outperforms CIL baselines by over 14% in open-loop evaluation and 30% in closed-loop testing on CARLA.

Learning to Drive Anywhere
Ruizhao Zhu Peng Huang Boston University
Eshed Ohn-Bar
Venkatesh Saligrama
Abstract: Human drivers can seamlessly adapt their driving decisions across ge- ographical locations with diverse conditions and rules of the road, e.g., left vs. right-hand traffic. In contrast, existing models for autonomous driving have been thus far only deployed within restricted operational domains, i.c., without ac- counting for varying driving behaviors across locations or model scalability. In this work, we propose AnyD, single geographically-awar conditional imita- tion learning (CIL) model that can efficiently learn from heterogeneous and glob- ally distributed data with dynamic environmental, traffic, and social characteris- tics. Our key insight to introduce a high-capacity cro-location-basee channel attention mechanism that effectively adapts to local nuances while also flexibly modeling similarities among regions in data-driven manner. By optimizing contrastive imitation objective, our proposed approach can efficiently scale across the inherently imbalanced data distributions and location-dependens events. We demonstrate the benefits of our AnyD agent across multiple datasets, cities, and scalable deployment paradigms, i.e., centralized, supe-super, and distributed agent training. Specifically, AnyD outperforms CIL baselines by over 14% open-loop evaluation and 30% in closed-loop testing on CARLA.
Keywords: Global-scale Autonomous Driving, Imitation Learning, Transformer
1 Introduction
Driving at scale involves complex and nuanced decision- decision-making process across diverse social and environmental conditions. For instance, driving model for turning at intersections in San Francisco, CA may be able to generalize relatively well when deployed in Washington, DC, over 2400 miles away. However, when deployed just north of in New York City, the model would discover that right turns on red, which have been banned the city [1], could result in social disruption and potentially unsafe conditions at intersections. When deployed at intersections in Pittsburgh, PA, the model may begin accelerating
green light only be confounded by the frequent occurrence of the Pittsburgh left [2], resulting in freeqeet and uncomfortable braking. These examples illustrate how the lack of modeling of location- based traffic behavior and social norms can lead to potentially safety-critical consequences. Beyond city-level variability, failing to accurately account for state or country-level differences in traffic regulations and social norms can also have dire consequences, e.g., from the directionality of travel [3] to varying maximal speed limitations [4] or yielding expectations [5]. How can we design and learn models that flexibly accommodate the heterogeneous data encountered across challenging and diverse geographical, environmental, and social conditions?
Despite recent advances in decision-making models for autonomous driving, models are often trained and evaluated within limited operational domains, i.c., a handful of geographical regions and social conditions (c.g.. Waymo's service in Pheonix, AZ, and San Francisco, CA [6]). Autonomous driving benchmarks are often collected in handful of cities and routes [7-10]. Existing frameworks for learning drive (e.g.. [11-15)) train single policy without considering geo-location or policy adaptation. While such methods may be used to train and adapt different regional models, there often many similarities which can be shared among the different locations and benefit potentially small and rare datasets with imbalanced distributions. In this work, we propose an approach for
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

learning to modulate predictions across settings and locations, i.e., even in seemingly similar visual settings, within a single driving agent.
While prior works have leveraged transfer learning [16-23], e.g., through access to unlabeled data of target domain, or introduced internal layers that learn to adapt model output across various do- mains [24-28, 16, 29-31], these methods have exclusively focused on low-level object classification and detection tasks, and have not explicitly accounted for geographical priors or reasoning. In con- trast, we study end-to-end models for learning safe perception and recision-making in intricate 3D navigation scenarios. In this case, to avoid a potential accident, perception and action characteristics must both be carefully tuned to consider geographical location when reasoning over traffic maneu- vers and predicting social behavior. Moreover, the training process of our sensorimotor models may require order-of-mann urder-of-magnittuee higher sample complexity, i.e., due to the higher rarity of policy-level events and intricate maneuvers [32]. Thus, geo-aware model capacity should be explored jointly with approaches for efficient adaptation and parameter sharing, as we do in this work.
Contributions: We make three key contributions towards autonomous systems at scale: 1) We revisit current end-to-end driving models identify limitations in learning from heterogeneous and distributed data sources. In particular, we build on recent advances in waanformer-basee mod- els [33, 34] for learning high-capacity, geo-aware imitation learning agents that can adapt across geographical locations while sharing parameters and computation within a single network. 2) To fa- cilitate efficient training across inherently imbalanced data distributions and maneuvers, we further generalize conditional imitation learning by designing a supervised contrastive loss over condi- tional commands and locations. 3) We combine three public autonomous driving datasets collected by different companies and platforms across 11 locations to extensively evaluate the impact of the proposed scalable learning framework. To understand generalization across diverse use-cases and model training regimens, comprehensively analyze the benefits of our framework for various scalable deployment scenarios, including centralized (i.e., within a single company or server with shared raw data logs), distributed (i.c., with scalable federated computation), and semi-supervee (i.e., with unlabeled data) training.
2 Related Work
Learning to Drive from Demonstrations: Despite impressive recent advances learning to drive, approaches often leverage simple navigation tasks, i.c., lane following, intersection turning, and basic collision avoidance (e.g., with CIL [12, 35, 15, 36-39, 13]), or short real-world routes in handful of locations (e.g., [40, 11, 41, 42, 14, 43, 10, 44.46, 7J). We note that GPS localization in prior approaches may only be used to determine a next high-level command at an intersection [35], and not to learn regionally or socially appropriate decisions. Yet, training models among locations without such geo-awareness results in an ill ill-posed problem with ambiguous samples. Thus, our work can be seen as a natural generalization goal-conditional imitation learning frameworks [35, 47, 13] incorporate geographical information for learning high-capacity and controllable model.
Domain Adaptation: Model adaptation, i.e., from a source to a target domain with unlabeled data, has mostly focused on segmentation and detection tasks [16-22]. However, the robustness and reliability of current domain adaptation techniques at large scale have been repetitively ques- tioned (48-50). Moreover, the aforementioned techniques have not been previously studied within the more complex end-to-end training paradigm for decision-making models. Particularly relevant to our study are approaches that learn universal object detection models [24, 26] via self-attention and weighing feature channels based on the output of multiple parallel layers (i.e., adapters [51]). In contrast, our proposed cross pooss-tttenion-bare based network architecture can more effectively fuse geographically-derived and visual features while also outperforming adapter-based methods [24].
Benefits of Contrastive Learning: Researchers have been increasingly exploring the benefits of contrastive learning frameworks for learning generalized representations, even under imbalanced

---

Command
Speed
Planner
Luc
Geo-Aware Features
Image
Image Encoder
Features
Aordee
F
Fe
Multi-Head Module
E
K
Cross Attention
Cyss
Region
Embedking Matrix
Regin
Nert
Figure 1: Model Overview. Our model maps image, region, speed, and conditional command obser- vations future decisions, parameterized as waypoints the map view. To efficiently learn high- capacity model, we leverage a multi-head cross attention module which fuses and adapts internal representations in a geo-aware manner. Our imitation objective, defined over human-demonstratte waypoints (outlined in green), the other command branches (outlined in red), and the predicted weights by the multi-head module, regularizes model optimization under diverse data distributions.
long-tail settings [52-56]. Our main use-case inherently involves learning diverse and im- balanced underlying data distributions. For instance, Tesla may suddenly trigger warning in unsupported region, in which case small amounts of demonstration
challenging scenario data from the driver may be collected and available for training, Moreover, although diverse and rare traffic scenarios can occur within any local city region or country, the underlying distribution of such events can significantly shift among locations. Recently, Mandi et al. demonstrated the benefits of unsupervised contrustive learning for improved imitation learning within simple robotic use-cases [57]. Instead, we demonstrate the benefits of supervised contrastive learning techniques (e.g..(5)) by designing novel loss function for conditional imitation learning frameworks scale.
Distributed Learning to Drive: We comprehensively analyze our proposed approach across train- ing paradigms suitable for scalable deployment in order to ensure the generalization of our findings. In particular, Federated Learning (FL. provides a natural framework for implementing AnyD in the real-world. The goal of FL drive is train global model leveraging distributed data and mod- els from different agents [58], i.e., where agents may avoid sharing driving logs and data due to various privacy and efficiency considerations. However, dealing with data heterogeneity among agents [59-62] remains a challenge. We demonstrate our novel geo-conditional mechanism to com- plement current federated learning algorithms. Somewhat surprisingly. our FL model variants result in outperformance compared to the searualized-ttainned counterparts due to the effective regional bias handling. We note that this without having share potentially sensitive geographical information, our embedding matrix (defined in Sec. is kept local and private in implementation.
Method
We propose a geo-conditional agent (AnyD) which generalizes existing conditional imitation learn- ing methods [35, 63] through two key aspects. First, we propose a novel network structure that leverages multi-head transformer module for geo-aware adaptation of visual features across re- gions (Sec. 3.2). Second, we design contrastive learning objective which regularizes training and addresses imbalances across locations and capture settings (See. 3.3). An overview of our approach is depicted in Fig. 1.
3

---

3.1 Problem Definition
Our objective is to learn goal-directed agent that can effectively reason varying traffic rules and social norms in complex and dynamic real-world settings. We leverage offline approaches re- lying on learning from driver demonstrations [64, 65, 40, 66, 67] as they can safely learn to map sensor observations to actions, i.c,, as opposed to interactive methods [68, 69. 36, 70, 71J. As an example use-case, consider deployed Tesla or Waymo fleet encountering challenging settings be- yond its current constrained and geo-fenced deployment [72, 73]. Here, human can take-over and demonstrate desired driving behavior which can subsequently be uploaded to shared cloud server (i.e., centralized training) updated to improve the model locally (i.e., federated training, we sider both cases in Sec. 3.4). However, current end-to-end agents that leamn to drive in data-driven manner, e.g., based on CIL [35, 12, 74, 13], do not differentiate among regional norms.
Ceo-Conditional Imitation Learning: We assume a dataset of demonstrations D {(x. y) Le., measurements of x =
X, where I E R x is an image of the current environment, 1 R is the speed, E N is a navigation command [35, 12], {0, 1) is region index encoded as a one-hot vector over total G' regions, and corresponding action labels y E y based on human drivers. Consistently with prior work [75, 37, 76]. we predict a waypoint-based label in the bird's 's eye view over the next five planned locations (2.5 seconds), such that y = {wi]e=1 and w, R22 The high-level waypoint output the bird's eye view can also help standardize policy decisions across globally distributed platforms with heterogeneous sensor configurations [41]. In our work, experiment with various definitions for g (manually defined city labels and unsupervised neighborhood-Iee level labels, these do not require precise GPS localization). Moreover, note AnyD does not rely on image-level perception labels or high-definition map information. As such, it can be trained based on cheaply collected GPS-based waypoint labels and benefit from rapid advancements in positioning technology (analysis of localization noise when training Any models can be found in the supplementary). We train a geographically-wven awwae policy funetion X + y using supervised learning [12, 13]. Learning the policy ı in geo-conditional imitation learning requires carefully fusing image and geo-location information, i.e., opposed to just basic concatenation. Next, introduce our network architecture which efficiently generalizes the branch-based architectures of ClL-based approaches [35].
u. U,,
3.2
Geo-Conditional Transformer Module
To learn a scalable policy function, i.e., across cities. countries, and platforms, we design a single network which adapts its decisions based on an efficient multi-head module. The mechanism is mo- tivated by transformer [34, 33, 771, with three main aspects. First, the queries are only conditioned on the part of the input, i.e., the region-based features. Second, we not use spatial attention as in ViT-type architectures [33], but instead learn a low-dimensional channel weight vector which can be trained efficiently [77]. Third, while multi-her head mechanisms have been used by prior methods [33], propose to jointly predict scalar weight for each head prior to the summation of the heads. This formulation is analogous to a mixture or adapter-based model [78, 28]. Our domain attention mech- anism is implemented via a region token that enables the model to specialize the heads to specific domains or tasks and subsequently combine the heads based on the current appropriate region and decision. For instance, we identify the emergence of traffic rules, such as left vs. right-hand driving when inspecting the output of the learned heads in Sec.
Our model first extracts image features from the input image I. Subsequently, multi-head trans- former module computes channel weights for the features based on the current region definition g. Finally, the planner utilizes the re- re-weighted geo-aware features F and speed information v to generate waypoints for different commands. The command input c then selects the required way- points ý for execution. The multi-head transformer module takes as input visual features extracted using a ResNet-34 encoder & [79], F (assuming column vector g), extracted from trainable embedding matrix E E R [80]. We use C 512 such that the output of the multi-head module is 512- 12-dimensional vector for weight-
<(I) E 1Xx13xCC and a regional embedding e g

---

ing each channel in F and computing the geo-aware features Fg using the weighted and summed H output heads (Eqn. 3). The visual features are pooled (to accommodate the channel-wise atten- tion), processed through a Fully Connected (FC) layer, and concatenated with a region token a, [(C+1)xd where d 128 sets number hidden units. a will be updated and used weigh the multiple heads the output of the module, as shown in Eqn. 3. Similarly, the region embedding 7g (a, FC(c)] E R(C+1)xd The computation steps for the geo-aware transformer can then be summarized as:
21 = ax, FC (F)]
+ Attention(LN((z)) LN(z))
(1)
2 = z + MLP(LL((2)
(2)
F g Σ
@0 F
where LN denotes Layer Normalization, @ denotes channel-wist multiplication, and â is the up- dated region token values. 2 in the second step is pooled before addition to make the shape consis- tent. We follow VIT [33] to compute attention as
QK T
Attention 21, 2gg = softmax
V
where Q 2g W K 21 WK V 21 WV and Wº Rdddd WK Rxdd and W Rdxd learned matrices. Unlike ViT, we do not merge the multiple heads by concatenating such that there are H outputs, each C+1-dimensional, i.e., å E #x+(++1), Here, the weights for the h-th head are stored at the first index of the head output vector, i.e., OG = 2 The adapted geo-aware features are then given to a command-conditional branch as shown in Fig. for predicting the final waypoints. To optimize the network, we leverage contrastive loss function over maneuvers and regional decisions, as discussed next.
3.3 Contrastive Imitation Learning
Loss Function: Standard supervised learning approaches for imitation learning leverage an L₃ loss, iee, behavior cloning, between predicted and demonstrated ground-truth waypoints [71, 12]. How- ever, our case of highly heterogencous and imbalanced data over maneuvers and regions, this loss can result poor performance and overfitting to local biases [81, 55J. In particular, the distribu- tion of both conditional commands and regions g can be highly skewed, with certain critical events (e.g., turns) occurring at much lower frequency. While employ branched architecture [35, 13), subset of the branches may be trained over a fraction of the total samples, i.e., with most updating the 'forward' branch. We hypothesize that such imbalances can introduce noisy predictions from poorly-trainge branches. When adding the additional complexity of learning region-conditiona poli- cies, issues in data imbalance and heterogeneity compound. To tackle this practical safety-critica issue, Chen et al. [65] employed a privileged teacher (i.e., learned from complete ground truth obser- vations of the 3D surroundings instead of raw images) that can be used for additional sampling and data augmentation. However, training such a privileged expert requires extensive annotation of real- world data, which is not scalable. Instead, propose introduce command and region- contraasvee objectives simple and effective strategy for improving model optimization and providing more supervision when handling imbalanced data. In our analysis, we demonstrate the utility of this ap- proach for both vanilla CIL and the proposed geo-CIL. As far as we are aware, we are the first to empirically analyze such benefits for imitation-
driving agents at scale.
We propose incorporate additional terms in addition to the main behavior cloning loss, LBC- The total loss can be computed
C = L + XcLemd-et AgGa-ot
(5)
where >e, Ag are hyperparameters, Lemd-et and L contrastive losses. Next, we define each of the proposed loss terms.
5

---

The branched conditional command architecture of CIL updates each branch based on a subset of the data samples each batch B. As some commands are highly underrepresented in natural driving data, we propose a command contrastive loss that leverages predictions for other commands for the same sample as negative examples,
Command Contrastive Loss:
1
exp(d(y; vt)/T) Ee exp((d()).
Lond
log SBB
B
where the loss is computed over a single positive example with prediction outputted by the ground truth command branch C,, and the other predictions (i.e., hypothetical commands) as negatives. d is similarity function (we use negative Lg distance) and r IR' + is scalar temperature parameter. The command contrastive loss is natural extension supervised contrastive learning [55] to case of conditional imitation learning, with two key differences. First, do not apply it to the feature space (as commonly done) but instead to the ouipul space of the network, i.e., in order to better model differences among maneuvers. Second, the command contrastive loss is not computed over all other samples in the same batch with different commands as in standard supervised contrastive loss [55]. While this practice may work for simple classification tasks, our case such other samples tend to also involve different driving situations. We found leveraging other samples in this manner when training an imitation model degrade policy performance, most likely due to the added learning complexity and ambiguity. A similar reasoning can be applied improve optimization of the geo- conditioned transformer module, as discussed next.
While driving behavior across cities and regions can often similar, in practice, the cities in our employed datasets (detailed in Sec. 4) all have unique local characteristics effectively modeled. Thus, also propose to incorporate a region (i.e., city)-based contrastive loss. We apply the loss within the transformer module over different output head weights à E R Here, we follow standard contrastive loss implementation [55] and select the i-th sample as an anchor. During training, for the i-th sample in a batch, positive samples P(i) are defined within the same city, while negative samples (ii) from differing cities,
Cro-Contrastive Loss:
log Eper() exp(d(á,; ap)/T) EnEA(e) exp( (xp(d(a;; a///T)
Cg-et
B
P())
where A(i) = Ui)) and d is similarity function (we use negative L2 distance).
3.4 Scalable Training Settings
We comprehensively analyze the training of our model using three different scalable deployment set- tings. First, in Centralized Learning (CL), agents are able share all sensor data and geographical information with centralized server. Consequently, the server conducts supervised learning of our AnyD model over all of the raw data. To further analyze model scalability, we implement a Semi- Supervised Learning (SSL) model, which can leverage ample unlabeled data that may be available across locations (we follow [13]). Finally, we study the applicability our findings within federated learning approaches, as sharing raw sensor data can be inefficient or even potentially undesirable. For instance, our AnyD agent may be distributed over numerous heterogeneous data sources with various constraints, i.e., local regulations, legal authorities, and privacy requirements preferences. Thus, we also analyze a Federated Learning (FI) agent which does not require sharing raw and ge- ographical information with a server. To fully understand the role of our proposed network structure within such paradigms, we optimize AnyD using two federated learning algorithms, FedAvg[58] and FedDyn[59], We note that the geographical embedding matrix which contains cite-level in- formation remains locally updated on each agent (i.e., akin to form of local model personalization). In this manner, E reduces a row vector as the embedding vector for the specific region (city in our implementation). We note that this results in the removal of the geo-contrastive loss term Lg in Eqn. 7. The supplementary contains additional details regarding our implementation.

---

Experiments
In this section, we first introduce our combined multi-city benchmark extracted from multiple pub- liely available datasets. Specifically, we present ablation studies for various model design choices and loss terms. To understand the benefits of the proposed framework on various training schemes, we also report analysis with three different training paradigms. This ensures our model and findings relevant across real-world use-cases, e.g., with efficient distributed settings large-scale.
4.1 Datasets and Metrics
To learn global scale driving policy, we conduct training on data from three different datasets in- cluding Argoverse 2 (AV2) [9], nuScenes (nS) [10] and Waymo (Waymo) [7]. While these datasets do not have any official waypoints prediction benchmark, we extract these from the provided raw data logs. Specifically for each frame, we processing the raw data to get the future 2.5s as ground truth waypoints, current velocity, front-view RGB image, navigational commands and a city-level information. The data spans 11 cities. We split the data into training, validation and testing data. Our split utilizes 190k, 20k, and 35k training samples for AV2, nS and Waymo datasets respec- tively. We follow standard evaluation using Average L Displacement Error (ADE) and Final L2 Displacement Error (FDE) over future waypoints in the BEV space. We also evaluate closed-loop policy performance using CARLA [82]. As CARLA benchmarks do not generally involve regional modeling, we introduce left-hand driving and town-varying behavior of agents. Our supplementary provides additional details and experiments.
4.2 Results
Model and Loss Ablation: We study the underlying architecture of the model in Ta- ble 1. We find that replacing interme- diate image-level heatmaps (used in sev- eral CIL-based baselines [65, 13J) with fully-connected layers provides improved reasoning for our diverse perspective set- tings (reducing ADE from 1.24 to 1.16). We also demonstrate transformer framework with three heads to outperform other supervision choices, e.g., embedding concatenation and super- vision as an auxiliary prediction task as in Ayush et al. [83] (1.09 vs. 1.15 ADE). AnyD also outperforms another attention-based method e.g., Hybrid ViT [33] on concatenated image-city features (1.09 vs. 1.20 ADE), which validates its efficiency on adaptation. Moreover, the proposed geo-conditional module can be used to increase the modeling capacity of the agent, and thus can scale beyond simple task supervision. We also find holistic effect among the proposed loss terms, with a combination leading to the best results (1.93 vs 2.08 FDE for the vanilla behavior cloning loss). Fig. depicts a case where AnyD handles diverse traffic regulations and social norms such as turning right (wider turn) in Singapore and yielding a 'Pittsburgh left.'
Table I: Ablative Studies on Model Architecture, Geo-Cond Cro-Cooniiiioaal Module and Loss. We start from CIL architectures [65, 13] and gradually add different components losses to get AnyD.
Ablation
Method
ADE
FDE
CIL [65]
1.32
2.55
CIL Architecture
BEV Planner [13]
1.24
2.45
Our Planner
1.16
2.17
Concatenation
1.14
2.12
aco-conditional
Task Supervision [83]
1.15
2.24
Geo-CIL Architecture
Hybrid ViT [33]
1.20
2.30
Universal Adapter [84]
1.10
2.11
Geo Transformer w/ Lnc
1.09
2.08
Lec, Lend-et
1.07
1.97
Loss Function
Lec. Ly-a
1.06
2.00
AnyD (C)
1.05
1.93
Training Paradigms: Table 2 reports the impact of various model training schemes on ADE per- formance (additional details, including FDE-based analysis, are in the supplementary). We observe consistent improvements across paradigms and cities even with severe data imbalance. Moreover, leveraging unlabeled YouTube data for each city results in further gains, specifically for cities with lesser data (MTV, PAO. SGP, and PHX). For instance, MTV improves from 1.40 to 1.23 ADE due to the unlabeled data, showing the importance of this mechanism for our use-case. Overall, the model is shown outperform the baseline of Zhang et al. [13], which does not leverage geo-location. Consistently with prior work, federated learning algorithms tend to under-perform their centralized
7

---

Table 2: Evaluating AnyD with Different Training Paradigms. AnyD efficiently integrates into various training paradigms (CL-Centralized Learning, "SL-Semi-Supervee Federated Learning.. ADE is computed across the 11 cities in dataset. Our planner is our proposed architecture for direct image-to-BEV prediction (without the geolocation information or introduced auxiliary loss terms, see supplementary for additional architecture details).
Learning, and FL-
Settings
Method
Avg
PIT
WDC
MIA
ATX
PAO
DTW
BOS
SGP
PHX
SFO
MTV
CIL [65]
1.32
1.14
1.40
1.47
1.23
1.49
1.01
0.89
1.09
1.65
1.67
1.48
CILRS [12]
1.27
1.18
1.28
1,43
1.16
1.52
1.11
0.84
1.02
1.39
1.60
1.40
BEV Planner [13]
1.24
1.18
1.01
1.34
1.23
1.55
1.03
0.90
1.07
1.38
1.58
1.39
CL
TCP [38]
1.22
1.09
1.23
1.41
1.14
1.47
0.99
0.87
1.01
1.39
1.49
1.40
Our Planner
1.16
1.24
1.12
1.12
1.38
1.39
1.02
0.92
1.10
1.08
0.89
1.41
AnyD
1.05
1.12
0.95
1.16
1.31
0.89
0.82
1.03
0.98
0.83
1.40
SelfD [13]
1.02
1.13
1.03
1.01
1.25
1.26
0.93
0.80
0.95
0.82
0.79
1.29
SSL
AnyD
0.97
1.06
0.93
0.92
1.19
1.24
0.89
0.76
0.94
0.84
0.75
1.23
FedAvg [58]
1.42
1.38
1.43
1.41
1.73
1.63
1.23
0.93
1.21
1.53
1.42
1.64
FedDyn [59]
1.19
1.23
1.15
1.21
1.59
1.51
1.11
0.797
0.95
0.97
0.99
1.30
AnyD (FedAvg)
1.20
1.23
1.06
1.04
1.62
1.61
1.00
0.81
1.07
1.33
1.01
1.41
AnyD (FedDyn)
0.98
1.08
0.91
0.91
1.50
1.54
0.90
0.68
0.82
0.62
0.70
1.12
Table 3:
Closed-Loop Evaluation in CARLA. We report closed-loop metrics of Success Rate (SR), Route Completion (RC), Infraction Score (IS) and Driving Score (DS) compared baseline planner which is not trained in a geo-aware manner.
Metries
ADE 4
FDE 4
SR1
RC f
1St
DS
BEV Planner [13]
0.58
0.97
0.29
0.50
0.61
0.36
AnyD
0.46
0.79
0.36
0.69
0.67
0.50
counterparts. Yet, due better handling of local biases, AnyD is shown to benefit federated learning. surpassing centralized training (0.98 vs. 1.05 ADE).
Closed-Loop Evaluation: The results of the closed-loop experiments CARLA are shown in Table 3. We find consistent improvements in success rate, route completion, and infractions (the supplementary video shows qualitative examples where the baseline model struggles with left-hand driving). We compute both open and closed-loop metrics by saving the expert actions for the test sequences. AnyD outperforms the baseline planner [13] on both open-loop metries (reducing ADE from 0.58 0.46) and closed-loop metrics (improving driving score by 38%, from 0.36 0.50). Nonetheless, overall success rates quite low for our benchmark, it contains significant behavior variability. This highlights the challenging nature of the region-aware decision-making task for current imitation learning models, either in the real-world or in simulation.
5 Conclusion
We envision large-scale navigation agents that can seamlessly operate in hetero- geneous and distributed locations. Our work introduces an efficient framework for training universal high-capacity naviga- tion agent across diverse locations. Us- ing our proposed agent, fleets of vehicles can increasingly grow their operation ca- pacity to novel conditions, i.e., by involy- ing humans and collecting both unlabeled or labeled demonstration data for policy training. Nonetheless, effectively incorpo- rating geo-awareness into driving models remains challenging and under-explored research problem.
10
Gt
Daedl
7.Meness
((Metes)
22 (Meeas))
Figure Qualitative Results. We plot predicted way- points in the BEV for comparison. AnyD exhibits robustness in region-specific cases, including turning right (wider turn) in Singapore, yielding 'Pittsburgh left' vehicle Pittsburgh.
8

---

Acknowledgments
This research was supported by a Red Hat Research Grant, Army Research Office Grant W91 INF21102246, National Science Foundation grants (CCF-2007350, CCF-1955981, and IIS- 2152077), and AFRL Contract no. FA8650-22-C-1039.
References
[I] Wikipedia. Turn on red. http:://ewwwiiiiiccii.oggvvvuiiu 2022.
[2] Wikipedia, Pittsburgh left. hutps://en.wikkppeiii.e http://wwwwwkkkooooooo////////ss 2019.
[3] ChartsBin. Worldwide driving orientation by country. http://chharrbiin.co////weeeddr 2009.
[4] VividMaps. What traffic rules are different in different countries. https://viridmupsscommiiii traffic-rules-ae corulle-are-dfffeee arr--diferret-iie iit-diffrent-counniie 2019.
[5] Wikipedia. Priority the right.
to the right, 2022.
[6] Waymo one. http::/wwaymmo.com/wwwwyyoooo 2022.
[7] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[8] X. Huang, X. Cheng, Q. Geng, Cao, D. Zhou, Wang, Lin, and R. Yang. The apolloscape dataset for autonomous driving. In CVPRW, 2018.
[9] B. Wilson, W. Qi, T. Agarwal, Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, et al. Argoverse 2.0: Next generation datasets for self-driving perception and forecasting. NeurlPS, 2021.
[10] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O, Beijbom. nuscenes: multimodal dataset for autonomous driving. CVPR, 2020.
[11] M. Bansal, A. Krizhevsky, and A. Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. In RSS, 2019.
[12] Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. ICCV, 2019.
[13] Zhang, R. Zhu, and E. Ohn-Bar. SelfD: Self-learning large-scale driving policies from the web. In CVPR, 2022.
[14] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. In arXiv, 2016.
[15] D. Chen, V. Koltun, and Krähenbühl. Learning to drive from world on rails. In ICCV, 2021.
[16] N. Dvomik, C. Schmid, and J. Mairal. Selecting relevant features from multi-domain repre- sentation for few-shot classification. In ECCV, 2020.
[17] Y. Wang, J. Peng, and Zhang. Uncertainty-aware pseudo label refinery for domain adaptive semantic segmentation. In ICCV, 2021.
[18] K. Saito, Y. Ushiku, and T. Harada. Asymmetric tri-training for unsupervised domain adapta- tion. In ICML, 2017.

---

[19] J. Yang, S. Shi, Z. Wang, H. Li, and X. Qi. ST3D: Self-training for unsupervised domain adaptation on 3d object detection. CVPR, 2021.
[20] Z. Lin, D. Ramanan, and A. Bansal. Streaming self-training via domain-agmoste unlabeled images. In arXiv, 2021.
[21] Y. Liu, W. Zhang, and J. Wang. Source-free domain adaptation for semantic segmentation. In CVPR, 2021.
[22] K. Wang, X. Gao, Y. Zhao, X. Li, D. Dou, and C.-Z. Xu. Pay attention features, transfer learn faster cnns. In ICLR, 2019.
[23] L. Lai, Z. Shangguan, J. Zhang, and E. Ohn-Bar. XVO: Generalized visual odometry via cross cross-modal modal self-training. In ICCV, 2021.
[24] X. Wang, Z. Cai, D. Gao, and N. Vasconcelos. Towards universal object detection by domain attention. In CVPR, 2019.
[25] I. Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, 2017.
[26] W.-H. Li, X. Liu, and H. Bilen. Cross-domain few-shot learning with task-specific adapters. In CVPR, 2022.
[27] S.-A. Rebuffi, H. Bilen, and Vedaldi. Learning multiple visual domains with residual adapters. In NeurlPS, 2017.
[28] W.-H. Li. X. Liu, and H. Bilen. Universal representation learning from multiple domains for few-shot classification. ICCV, 2021.
[29] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
[30] P. Bateni, R. Goyal, V. Masrani, F. Wood, and L. Sigal. Improved few-shot visual classification. In CVPR, 2020.
[31] Requeima, Gordon, Bronskill, S. Nowozin, and R. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurIPS, 2019.
[32] S. Shalev-Shwartz and A. Shashua. On the sample complexity of end-to-end training vs. se- mantic abstraction training. In arXiv, 2016.
[33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In ICLR, 2021.
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, E. Kaiser, and Polo- sukhin. Attention is all you need. In NeurlPS, 2017.
[35] F. Codevilla, M. Miilerr, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, 2018.
[36] E. Ohn-Bar, A. Prakash, A. Behl, K. Chitta, and A. Geiger. Learning situational driving. In CVPR, 2020.
[37] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In ECCV, 2022.
[38] P. Wu, X. Jia, L. Chen. J. Yan, H. Li, and Y. Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong bascline. In NeurIPS, 2022.
10

---

[39] A. Hu, G. Corrado, N. Griffiths, Z. Murez, C. Gurau, H. Yco, A. Kendall, R. Cipolla, and
J. Shotton. Model-based imitation learning for urban driving. NeurIPS, 2022.
[40] J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, et al. Urban driving with conditional imitation learning. In ICRA, 2020.
[41] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun. Driving policy transfer via modularity and abstraction. In CoRL, 2018.
[42] W. Zeng, Luo, Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[43] L. L. Li, B. Yang. M. Liang, W. Zeng, M. Ren, Segal, and R. Urtasun. End-to-end contextual perception and prediction with interaction transformer. In IROS, 2020.
[44] M.-F. Chang, J. Lambert, Sangkloy, Singh, Bak, Hartnett, Wang, Carr, S. Lucey, D. Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
[45] Houston, G. Zuidhof, L. Bergamini, Ye, L. Chen, A. Jain, S. Omari, V. Iglovikov, and Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In CoRL, 2021.
[46] X. Huang, Wang, X. Cheng, D. Zhou, Q. Geng, and R. Yang. The ApolloScape open dataset for autonomous driving and application. IEEE PAMI, 2020.
[47] Z. Huang, Z. Shangguan, J. Zhang, G. Bar, M. Boyd, and E. Ohn-Bar. ASSISTER: Assistive navigation via conditional instruction generation. In ECCV, 2022.
[48] V. Prabhu, R. R. Selvaraju, J. Hoffman, and N. Naik. Can domain adaptation make object recognition work for everyone? In CVPR, 2022.
[49] S. Sagawa, P. W. Koh, T. Lee, I. Gao, S. M. Xie, K. Shen, A. Kumar, W. Hu, M. Yasunaga, H. Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. In ICLR, 2021.
[50] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In NeurIPS, 2020.
[51] J. Hu, L.. Shen, and G. Sun. Squeere-and-excitaioon networks. In CVPR, 2018.
[52] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[53] Bai, J. Chen, J. Zhao, B. Wen, X. Jiang, and Kot. Feature distillation with guided adversarial contrastive learning. In arXiv, 2020.
[54] Li, P. Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. ICLR, 2021.
[55] Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, Isola, A. Maschinot, C. Liu, and D. Kr- ishnan. Supervised contrastive learning. NewrlPS, 2020.
[56] J. Cui, Z. Zhong, S. Liu, B. Yu, and J. Jia, Parametric contrastive learning. In ICCV, 2021.
[57] Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. In ICRA, 2022.
[58] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas. Communication--fficieett learning of deep networks from decentralized data. In AISTATS, 2017.
[59] D. A. E. Acar, Y. Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021.

---

[60] D. A. E. Acar, Y. Zhao, R. Zhu, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama. Debiasing model updates for improving personalized federated training. In ICML, 2021.
[61] Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization heterogeneous networks. In MLSys, 2020.
[62] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.
[63] Zhang, Z. Huang, and E. Ohn-Bar Bar. Coaching a teachable student. In CVPR, 2023.
[64] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. In Foundations and Trends@ in Robotics, 2018.
[65] D. Chen, B. Zhou, Koltun, and Krähenbühl. Learning by cheating. In CoRL, 2020.
[66] O. Scheel, L. Bergamini, M. Wołezyk, B. Osiński, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In CoRL, 2021.
[67] A. Behl, K. Chitta, A. Prakash, E. Ohn-Bar, and A. Geiger. Label-efffcient visual abstractions for autonomous driving. In IROS, 2020.
[68] X. Liang, T. Wang, L. Yang, and E. Xing. CIRL: Controllable imitative reinforcement learning for vision-based self-driving. In ECCV, 2018.
[69] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. ICCV, 2021.
[70] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
[71] A. Prakash, A. Behl, E. Ohn-Bar, K. Chitta, and A. Geiger. Exploring data aggregation in policy learning for vision-based urban autonomous driving. CVPR, 2020.
[72] WIRED. Why people keep rear-ending self-driving cars. http::/wwww.virrd...omooooyss/t www.rrssseeeeeeeeeeeeennnww-----
2018.
[73] electrek. We tested tesla full self-driving beta in the blue ridge mountains, and it was scary. -etee--bbbbbb-iiieeeeeee -iodi---ooottiissscue sccar// 2022.
[74] F. Codevilla, A. M. Lopez, V. Koltun, and A. Dosovitskiy. On offline evaluation of vision-based driving models. In ECCV, 2018.
[75] K. Mangalam, Girase, S. Agarwal, K.-H. Lee, Adeli, J. Malik, and A. Gaidon. is not the journey but the destination: Endpoint conditioned trajectory prediction. ECCV, 2020.
[76] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger. Transfuser: Imitation with transformer-basee sensor fusion for autonomous driving. In PAMI, 2022.
[77] Woo, J. Park, J.-Y. Lee, and I. S. Kweon. CBAM: Convolutional block attention module. In ECCV, 2018.
[78] X. Wang, F. Yu, L. Dunlap, Y.-.. Ma, R. Wang, A. Mirhoseini, T. Darrell, and J. E. Gonzalez. Deep mixture of experts via shallow embedding. In ICML, 2018.
[79] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[80] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv, 2013.
12

---

[81] B. Kang, Y. Li, S. Xie, Z. Yuan, and J. Feng. Exploring balanced feature spaces for represen- tation learning. In ICLR, 2020.
[82] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In CoRL, 2017.
[83] K. Ayush, B. Uzkent, Meng, Tanmay, M. Burke, Lobell, and S. Ermon. Geography- aware self-supervised learning. In CVPR, 2021.
[84] H. Bilen and A. Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. In arXiv, 2017.
[85] Carla autonomous driving leaderboard. https:/leaderttarrlccaaaarrr 2022.
[86] Bennett. OpenStreerMap. Packt Publishing Ltd, 2010.
[87] X. Huang, P. Wang, X. Cheng, D. Zhou, Q. Geng, and R. Yang. The apolloscape open dataset for autonomous driving and its application. In TPAMI, 2019.
13

---

Supplementary
We provide additional implementation details, including network architecture and training proto- col, as well as additional analysis, including ablative studies, results on CARLA, and additional qualitative examples.
6.1 Implementation Details
This section first provides details regarding our proposed network architecture and discuss differ- ences with baseline models (Sec. 6.1.1). Next, we provide details regarding the processing of the driving datasets to construct the multi-city benchmark used throughout the analysis (Sec. 6.1.2). Finally, we discuss evaluation settings (Sec. 6.1.3) and training protocol (Sec. 6.1.4).
6.1.1 Architecture and Baselines
We leverage an ImageNet-pretrained ResNet-34 [79] as our backbone Φb All images are resized to 400 x 225 prior to being inputted to the model. To leverage diverse camera viewpoints, we build on prior models in conditional imitation learning [65, 13] and train a direct image-to-BEV prediction model, i... without assuming fixed known BEV perspective transform. Moreover, we also find the removal intermediate image-level heatmaps [65, 13] (and directly regressing the BEV waypoints) proposed network architecture for image-to-
to improve model performance. Fig. 3 compares BEV planning to standard baseline architecture (e.g., [13, 65]). Prior image-based models may utilize deconvolutional layers to obtain an image-aligned heatmap and followed by a soft-argmax ('SA' Fig. 3) and 2D waypoint projection to the BEV space. The projection can be implemented either using homography (i.e., known extrinsic parameters [65, 38J) or with a learned projection layer [13]. contrast, we find it beneficial remove the intermediate image-level processing and directly predict BEV waypoints,
shown in Fig. 3(b). We replace the upsampling layers with 3 x convolutional layer which fuses the image and speed-based features prior to inputting to three fully-connected final prediction layers. By removing unnecessary processing steps and enabling more expressive image-to-BEr BEV mappings, the proposed planner architecture improves from 2.45 FDE to 2.17 FDE. This improvement comes with minimal gain in parameters (23.8M vs. 24. Incorporating the geo-conditional module with three adapters further improves trajectory prediction performance to 1 1,93 FDE with a 24.2M parameter model. We do not find it necessary to leverage more complex, e.g., GRU-based [38], prediction heads.
While related methods are often studied in simulation [38, 65], we provide several base- lines by following publicly available implementations. particular, we leverage the wate-of-the-ar monocular agent TCP [38]. To ensure meaningful comparison with single-frame models, we also remove the temporal refinement module. As TCP leverages a control prediction branch in addi- tion to the waypoint prediction branch, normalize the raw control signals among the different vehicle platforms across the multiple datasets to [0, 1]. When comparing with the semi-supervised learning scheme of SelfD [13], we leverage a 10- hour YouTube driving dataset with available city (i.c.. region) descriptions. In this manner, we are able to use AnyD to pscudo-label videos that were taken within the cities leveraged in our experiments. Subsequently, we mix the datasets to train a semi-supervised AnyD model, which is then evaluated over our multi-city benchmark.
Baselines:
6.1.2 Data Processing and Distributions
To obtain BEV waypoints for training, we standardize formats across three datasets, Argoverse (AV2) 19], nuScenes (nS) [10] and Waymo (Waymo) [7]. The datasets provide proteprocessed world coordinates for each frame obtained from GPS and other mounted sensors, e.g., LiDAR [10]. We leverage these reported ego-poses to generate a waypoint prediction benchmark. For each frame, use the global coordinate an intermediate get relative positions of the future 2.5s as ground truth waypoints. The conditional command (left, forward, right) is inferred in semi-automatic process. First, extract the preliminary command by thresholding the curvature of the trajectory.
14

---

Command
Speed
Image I
Image Encoder
Features
Deconvolution
0:00
SA
-000
:
(a) The baseline BEV Planner [13] relies alignment with the image input
through upsampling (i.c., to obtain waypoint heatmap), followed by soft-argmax (SA) [65].
Command
Speed
Image I
Imuge Encoder
Features
+000
+::00
(b) The proposed planner model simplifies the architecture such that each command branch directly predicts BEV waypoints without intermediate upsampling, 2D heatmap, or soft-argmax.
Command
Speed
Geo-Aware Features
Image I
Image Encoder
Features
F
Multi-Head Module
E
Region g
Embedding
(c) The complete AnyD architecture with geo-aware feature modulation.
Figure Comparing Network Architectures. As discussed in the main paper (Sec. 3.4 and Sec. 4.1). our proposed planner architecture abandons the intermediate image-aligned heatmaps and consequent soft- aoft-argmax employed by several prior works [65, 13] (top figure) and directly predicts waypoints (middle figure). The proposed architecture improves BEV waypoint prediction results by 12.9% FDE (2.45 vs. 2.17 FDE, shown Table of the main paper). Incorporating a geo- conditional module (bottom figure) further boosts performance by 4.1% (2.08 FDE) and 11.1% (1.93 FDE) without and with the proposed loss function, respectively.
However, this process cannot detect subtle maneuvers, such as lane changes, which are included in dataset. Consequently, manually verify and annotate the initial automatic command predic-
15

---

Data Distribution Across Cities
80000
60000
20000
MM
PIT
WDC
DTW
PHX
8FO
805
SGP
ATX
MTV
PAO
Figure 4: Imbalanced Training Data Distribution. Our training data is unevenly distributed across different cities, as often is the case in real-world data (y-axis is sample count).
tions. For our geo-conditional module, we not require accurate GPS information as we quantize each latitude and longitude into a city-level cluster. Each dataset provides front-view RGB image and speed (either from the raw CAN bus from the positioning information), which are inputted into model observations.
The data spans 11 cities: Pittsburgh (PIT), Washington, DC (WDC), Miami (MIA), Austin (ATX), Palo Alto (PAO) and Detroit (DTW), Boston (BOS), Singapore (SGP), Phoenix (PHX), San Fran- cisco (SFO) and Mountain View (MTV). The data distribution across different cities is shown in Fig. 4.
6.1.3 Evaluation Metrics and Settings
Open-Loop Evaluation: Average Displacement Error (ADE) and Final Displacement Error (FDE) are standard metrics for trajectory prediction [75]. We first compute the error within each city, and then average to obtain a balanced average metric. We also generate more fine-grained analysis by providing a breakdown over 1l semantic events in the dataset, as will be further discussed in Sec. 6.2. The extracted events include left turns, forward command, right turns, highway driving, heavy downtown traffic, red traffic lights, stop signs, uncontrolled intersections, pedestrian cross- ings, rain, and construction zones.
The open-loop evaluation measures the distance of waypoint predictions with real-world human drivers under complex maneuvers, including yielding, merging, and irregular in- tersections. To further validate our proposed approach, we sought to evaluate AnyD in closed-loop settings where continuous predictions are made in order navigate vehicle a destination along route. While closed-loop real-world evaluation is challenging due safety requirements, age the CARLA [82] simulator. Yet, standard CARLA evaluation does not generally involves social and regional behavior that is dynamic across towns. Subsequently, models do not currently incorpo- rate regional modeling, and GPS information is solely used to determine a command at intersections along a route. Hence, motivated by our 1 11-city real-world benchmark, we introduce a new bench- mark where different towns have different traffic behavior. Our benchmark is defined over Town 1, Town 2, and Town 10. We have modified Town for left-hand driving and added pedestrians and vehicles with more aggressive behaviors, e.g., with jaywalking. higher speeds, and closer proximity, to Town 10. We tune the autopilot's controller in order generate optimal behavior under the novel settings. When performing closed-loop evaluation in CARLA, we also compute Driving Score (DS) [85], which is a product between the route completion and a penalty based on infractions.
CARLA Evaluation:
lever-
6.1.4 Training Protocol
We study the role of our proposed network for scalable deployment use-cases using three training paradigms. To ensure standardized training across both centralized and federated training, we train the model using Stochastic Gradient Descent (SGD) [59]. centralized training (where all the raw observation data is shared in a single server), we use a batch size of and train for 7,500 iterations. We set the initial learning rate to le-1, learning rate decay as 0.997, and weight decay as 1c-3. The loss dyper-parametted are set A 1e-3, Xg 1c-4, A 1e-4. For semi-supervised training settings, model training is done in three stages. We first download a set of online videos
16

---

based on their tag which provides city-level information. We then train a supervised model using the same hyper-parameters as in centralized training above, and pseudo-label the unlabeled videos. We train three models from different initial seeds to compute a confidence score (i.e., variance) for each pseudo-label and filter low-confidence predictions. Subsequently, we train our model using the original training dataset combined with the large pseudo-labeled dataset. Here, we set the initial learning rate to le-3 and train the model for 500 iterations. Finally, for fair comparison between the centralized training and the federated training settings, we train our federated model for 1,500 synchronous communication rounds. We treat each city as its own 'node' or 'device," but do not share the private geo-embedding with the server (i.e., we aggregate all model parameters on the server using FedAvg [58] and FedDyn [59] excluding E). For each communication round, the model updated for five local iterations with SGD (in this manner, total iterations remain at 7,500). We further note that we remove the geo-contrastive loss term Ly-et cct in the federated learning settings (as this information is shared among the locations). We keep all other hyper-parameters fixed throughout the training settings.
6.2 Additional Ablation and Results
To supplement our findings in the main paper, we discuss four additional results. First, we provide supplementary analysis in terms of FDE (corresponding to Table 2 of the main paper with ADE), context and event-based performance evaluation (Sec. 6.2.1). Second, we perform additional abla- tion studies regarding the role of the number of heads in the multi-head module, impact of GPS noise over waypoints ground-trut in training, and clusters of the neighborhood- Third, we analyze an adaptation experiment to novel city in Sec. 6.2.3 and show additional qual- itative waypoint prediction results Sec. 6.2.4. Finally, we perform closed-loop evaluation results using the introduced CARLA benchmark.
models (Sec. 6.2.2).
6.2.1 Additional Analysis
For completeness, we report FDE results across training paradigms and models in Table 4. While FDE is more challenging as it emphasizes long-term prediction, we ob- serve similar trends among the models and cities compared to the complementary ADE-based anal- ysis the main paper. Specifically, we demonstrate AnyD to improve over our baseline even using this harsher metric, i.e., from 2.55 1.93 average FDE. Semi-Supervised Learning (SSL) provides further gains compared to the centralized AnyD for most cities (excluding ATX and DTW, which show slight under-penne pedrrr-erform...c).. When compared with Federated Learning (FL), the improvement is less pronounced compared to the gains observed with ADE and FL. While some cities are shown to significantly benefit FL over CL (e.g., BOS, SGP, PHX, SFO, MTV) others do not (e.g.. MIA and ATX). ATX is a small dataset with limited diversity and high speed variability. While evaluation becomes less reliable, a low-shot learning setting can also be studied to understand such challenges in the future.
Final Displacement Error:
Event-Driven Analysis: Table 5 shows a breakdown of driving performance over different events and maneuvers. Here, we see a significant benefit for the introduced regional awareness, higher modeling capacity, and more balanced contrastive objective. For instance, AnyD improves perfor- mance over the unevenly distributed commands ratio of 20 3 among left:forwardrrght in our dataset), for right command from 1.43 ADE with the planner and up 1.19 ADE with AnyD. Other conditions, such as highway, downtown, crossings, and rain conditions all show improvements as well. These results suggest that the situational adapters are able to accommodate various conditions both within and across geo-locations.
6.2.2 Ablation Studies
Number of Heads: We investigated the impact of increasing the number of heads in Table on the performance of the model in cities with different amounts of data. We observe that when cities
17

---

Table 4: Analyzing AnyD with Different Training Paradigms (FDE Version). We analyze the Final Displacement Error (FDE) counterpart of Table in the main paper (which shows Average Dis- placement Error, ADE). FDE only considers the final waypoint, while ADE considers all waypoints along the predicted route, thus providing complementary analysis. Although both metrics demon- strate similar performance trends, final waypoint prediction is more challenging task (hence, errors are higher). We analyze the three AnyD training paradigms (CI-Centrallized Learning, SSL--Semi- Supervised Learning, and FL-Federated Learning). The results show FDE across the 11 cities in our dataset. Our planner refers to the direct Maage-to-BEY prediction (without the geolocation informa- tion introduced auxiliary loss terms, see middle architecture in Fig. 3).
Settings
Methods
Avg
PIT
WDC
MIA
ATX
PAO
DTW
BOS
SGP
PHX
SFO
MTV
CIL [65]
2.55
2.20
2.71
2.85
2.47
3.05
2.02
1.69
2.14
3.03
3.07
2.86
CIRL [68]
2.48
2.39
2.55
2.82
2.37
3.13
2.21
1.61
2.03
2.59
2.88
2.72
BEV Planner [13]
2.45
2.30
2.08
2.66
2.64
3.25
2.04
1.75
2.51
2.87
2.73
CL
TCP [38]
2.37
2.17
2.27
2.77
2.28
3.02
1.95
1.69
1.99
2.52
2.71
2.71
Our Planner
2.17
2.36
2.16
2.15
2.69
2.85
1.98
1.72
2.07
1.97
1.73
2.69
AnyD
1.93
2.19
1.97
1.94
2.48
1.80
1.55
1.96
1.88
1.62
2.80
Sclfi) [13]
1.98
2.24
2.08
2.03
2.49
2.70
1.89
1.54
1.84
1.55
1.52
2.53
SSL
AnyD
1.89
2.12
1.93
1.89
2.57
2.76
1.83
1.45
1.83
1.65
1.50
2.45
FedAvg [58]
2.63
2.65
2.85
2.69
3.68
3.43
2.55
1.76
2.31
2.87
2.67
3.13
FedDyn [59]
2.14
2.48
2.34
2.44
3.42
3.10
2.35
1.53
1.86
1.86
1.85
2.55
FL
AnyD (FedAvg)
2.23
2.51
2.21
2.12
3.52
3.40
2.04
1.58
2.09
2.43
1.97
2.79
AnyD (FodDyn)
1.92
2.25
1.91
1.94
3.26
3.29
1.90
1.37
1.68
1.31
1.49
2.28
Table 5: Event-Driven Analysis. We perform separate evaluations over subsets of our total eval- uation benchmark based semantic events defined commands and driving conditions. The settings are, in order: left turns, forward command, right turns, highway driving, heavy downtown traffic, red traffic lights, stop signs, uncontrolled intersections, pedestrian crossings, rain, and con- struction zones. Anyl) is shown to benefit from improved robustness across conditions compared to the baseline, i.e., due the improved modeling capacity.
Method
Metric
Left
Fwd.
Right
Hwy.
DTown
Red
Stop
Unctrl.
Cross
Rain
Constr.
Our Planner
1.60
1.01
1.43
7.66
1.12
0.66
0.63
0.77
1.36
1.37
0.76
ADE
Full AnyD
1.48
0.92
1.19
1.37
1.04
0.60
0.66
0.72
0.80
0.80
0.56
Our Planner
2.93
1.95
2,68
12.80
2.36
1.41
1.62
1.28
2.20
2.24
0.83
FDE
Full Anyl)
2.75
1.85
2.30
2.40
2.20
1.05
1.62
1.22
1.44
1.44
1.30
Table 6: Number of Heads in the Multi-Head Module. We vary the number of heads in the transformer model and compute the resulting model's ADE. We observe drop in performance beyond three heads (selected throughout the experiments). We demonstrate this to be due to the cities with small (less than 10,000) data samples which may benefit from the increased modeling capacity.
of Heads (H)
Large- Data Cities
Small-Data Cities
All Cities
1.09
1.27
1.20
1.00
1,28
1.15
3
0.98
1.22
1.09
5
1.01
1.28
1.13
1,00
1.27
1.12
have a sufficient amount of data (defined as more than 10,000 frames), the error rate decreases and remains consistently low as the number of heads increases. This can be attributed to the increased model capacity, allowing for more effective feature extraction and improved learning of city-speciff patterns. However, when the training data is limited, increasing the number of heads can still result in worse performance some of the cities, potentially due to overfitting the small data sample. Thus, efficiently increasing model capacity in small-data domains remains challenge.
18

---

0.9
0.7
WDC
MIA
ATX
0.5
0.4
BOS
SGP
0.2
PHX
MTV
Head 1
Head
Head
Figure 5: Head Weight Distribution over Cities. While the heads and their weights are learned in data-driven manner, do find specialization of certain heads across regions. For instance, the unique tropical scenery Miami (MIA) gives rise to a distinctive pattern among head and head 3, well Singapore (SGP).
Staggper www.ppee--eeooeeeeo Easbooling Shous Exsholing
Figure 6: Attention Visualization in the Cco-Conditional Transformer. We visualize the atten- tion pattern of the selected head (head with the highest weight) under the same input observations but given different region embeddings. Given Boston embeddings, the weight is shown concentrate more the objects in the middle and right portion of the image. While the Singapore embeddings guide attention towards the left portion of the image, which attributable the left-hand driving scenario in Singapore.
Fig. studies the frequency of the head of the highest weight. Notably, results indicate that the distribution in Singapore, where left-hand driving is practiced, differs from that of other cities. Additionally, the unique tropical scenery of Miami gives rise to distinctive pattern as well.
Effects of Geo-Conditional Transformer: Fig. shows the visual effect of the attention pattemn of the selected head (head with the highest weight) under the same observations, given different region embeddings. Given Boston embeddings, the head concentrates more the objects in the middle and right portion of the image. While the Singapore embeddings guide attention towards the left portion of the image, which is attributable to the left-hand driving scenario in Singapore. We note that our supplementary contains additional ablations regarding number of heads in the multi-head attention module and the impact of dataset size on training such higher capacity models.
To understand the role of scalable real-world deployment and data collection, we analyze the impact of potential GPS error in Table 7. While our analyzed datasets carefully post-process the reported world coordinates, we envision AnyD deployed across more diverse and potentially noisy settings. We therefore report the performance degradation due to the addition of Im and 3m Gaussian noise over the training waypoints. Specifically, we find that even with added noise, Any obtains decent prediction performance, outperforming prior state-of-the-ar models that trained with clean waypoints, e.g., 2.39 FDE with 3m noise 2.45 FDE for the baseline [65, 68]).
Ground-Trune Truth Noise Analysis:
19

---

Table Impact of GPS Noise. Positioning accuracy (e.g.. for obtaining waypoints for training over new locations) is known to be variable. We analyze training with different degrees of Gaussian noise imposed over the ground-truth waypoints (a standard deviation, in meters). We introduce the noise to simulate the data collection in real-world conditions, i.e., in scenarios where the model may be trained over raw GPS measurements without extensive LiDAR-basee filtering performed in current autonomous driving datasets. Metrics averaged over cities.
Noise
ADE
FDE
Original
1.05
1.93
=
1.16
2.21
1.28
2.39
Table 8: Number of K-Means Clusters vs. Model Performance. We vary the number of clusters for K-means when obtaining additional finer-grained regions (i.e., neighborhood-leww clusters) for each city from publicly available GPS traces [86]. We note that this data may not always be available within all city or country regions. While the additional fine-grained information can further improve our model, we find performance to the plateau beyond three clusters for our data.
# of Clusters
PIT
WDC
MIA
ATX
PAO
DTW
1
1.15
1.27
1.60
1.13
1.65
0.96
3
1.16
1.09
1.11
1.10
1.42
0.95
10
1.21
1.09
1.22
1.39
0.98
K-6, Mam
X-), Wishington, DC
X-10, Misni
K-10, Washington. DC
Figure 7: Example Neighborhood Clustering by GPS Traces from OpenStreetMap. We utilize GPS trace data from OpenStreetMap (OSM) [86] to automatically divide cities into sub-regions based on traffic patterns. We leverage K-means clustering to analyze the ability of our model to handle finer-grained regions within cities. Clustering results of Miami and Washington, D.C. with respect to the number of clusters are shown (for 3 and 10 clusters). Despite the coarse clustering, meaningful clusters emerge, e.g., Miami's beach (blue) vs. downtown area (green) in the leftmost K figure.
Unsupervised Clustering of Regions: In Table 8, we explore the benefit of finer-grained regional clustering choices, i.e., within each city, when defining g. This experiment can uncover potential benefits from modeling intra-city settings. To achieve this, we utilize GPS trace data from Open- StreetMap (OSM)[86) and cluster cities into sub-regions based on traffic patterns. For example, MIA clustering results in semantic regions, e.g., downtown vs. beach areas, with large improve- ments in prediction performance for the finer-grained model (from 1.60 to 1.11 ADE). Similarly, WDC is clustered into downtown and highway regions, also benefiting performance (from 1.27 to 1.09 ADE). We note that while these examples suggest our model can further benefit model im- provement, such clustering data may not be available for many locations, and thus cannot always be assumed.
Table further explores the benefits of finer- grained clustering on AnyD model performance, i.e.. within city neighborhoods. To achieve this, employ GPS trace data from OpenStreetMap (OSM) [86] and divide cities into sub-regions based traffic patterns via K-means clustering.
20

---

Table Adaptation New City, We fine-tume on additional data from Guangzhou, China [87].
AnyD not only outperforms the baseline model on the new city but also maintains the performance on previously seen cities.
Cities
Seen Cities (Before - After)
GZ
BEV Planner [13]
1.24 - 1.33
0.87
AnylD
1.05 1.04
0.84
Fig. 7 shows example clustering results in MIA and WDC with respect to K = 3 and K = 10 clus- ters. For instance, the downtown areas of both two cities can be seen as clusters for both = 3 and K While somewhat coarse in its clustering, we find that this privileged region information (which may always be available) can introduce additional benefits when used as g during model training and evaluation.
6.2.3 Adaptation to a New City
In practice, geo-aware model may be required to learn to drive in previously unseen region with significant domain gap. To understand the model performance of AnyD under such learning settings, we extract an additional city (Guangzhou, China) from a different dataset, ApolloScape [87]. In this case, large domain gap occurs due to the differing social norms and traffic density in China. We mix the new city with the prior 11, and continue fine-tuning the model. Our results in Table indicate that AnyD can learn drive in the new city while also maintaining similar performance levels for the previously observed cities (i.e., without forgetting). In contrast, the baseline model, which does not incorporate the explicit geo-aware module, has higher error while also impacting performance on prior seen cities.
6.2.4 Qualitative Results
We show additional qualitative results in Fig. 8 and Fig. (on the next page). As depicted in the figures, the AnyD generally provides better performance on challenging tasks of navigating across different regions and events, including turns and merging (requires reasoning over traffic direction- ality) as well as speed limits. Fig. 10 depicts failure cases, which show challenging conditions involving rare rule violations by other vehicles.
6.3 Limitations
Despite the multiple publicly available datasets used in our experiments, the diversity in existing benchmarks is still limited, i.e., compared to the vast diversity of geo-locations and events that an agent may encounter in the real-world. Besides Singapore, which provides a challenging general- ization use-case data logs in current datasets are often captured over short drives and are biased towards the US. Thus, our framework requires further validation with larger-scale settings with in- creased diversity in the future. Here, while approach for learning unified model is motivated by human drivers that efficiently learn to adapt generalized skills across locations (including traf- fie direction), it can be potentially challenging to learn a single model across drastically differing locations. Finally, incorporating various explicit constraints and specifications (e.g., of local traffic rules) could also be studied in the future in order to enable efficient agent adaptation.
21

---

Pittsburgh
Washington
Miami
Austin
Palo Alto
Detroit
325
Figure 8: Qualitative Results across Cities Argoverse Dataset. We plot predicted waypoints in the BEV for Anyl), the ground truth trajectory, and the baseline planner model. AnyD is shown improve reasoning over regional speed and scenarios as well as general navigation and intricate maneuvers.
22

---

Boston
Singapore
Phoenix
San Francisco
175
159
Mountain View
Figure 9: Qualitative Results across Cities on nS and Waymo Dataset. We plot predicted way- points the BEV AnyD, the ground truth trajectory, and baseline planner model. AnyD shown to improve reasoning over regional speed and scenarios well as general navigation and intricate maneuvers.
Figure 10: Example Failure Cases. Challenging cases where AnyD fails produce safe driving behavior, often due to dense settings or rare behaviors.
23


Reviews:
[{"review_id": "pAHd0AHEuj", "reviewer": "Reviewer_bBuC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "--", "summary_of_recommendation": "--", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "--", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "nNruVNUH1dm", "reviewer": "Reviewer_tmmM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This method proposes an approach for conditional imitational learning for driving. In particular, the authors propose an architecture choice of a geo-location based channel attention mechanism and a contrastive objective, which allows them to learn a useful agent for driving in dynamic environments. The method outperforms prior methods in open-loop evaluation in the CARLA simulator.", "summary_of_recommendation": "I recommend a weak accept of the paper. I have highlighted some strengths and weaknesses for this work as well as highlighted some questions for consideration. Thank you!", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would like to hear back regarding the baseline analysis I suggested in the weaknesses section. \n\nAdditionally, I had some additional questions for consideration:\n1. Several regions around the world are devoid of accurate geographical information. How would your architecture be resilient to noise/inaccurate geographical information.\n2. Your paper mentions that the CARLA benchmarks do not generally involve regional modeling. Are there other benchmarks that have this information present within their benchmark? If so, how does your method fare in this setting? In particular, I am concerned if this information provided to the model is privileged and/or unrealistic. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Paper is well written and clear to understand\n- Utilize a combination of public autonomous driving datasets to learn a representation that scales across different deployment scenarios\n- Several architectural suggestions for a geo-aware driving agent.\n\nWeaknesses:\n- For table 2, some of the baselines compared against that are bolded seem to be close under statistical uncertainty so adding error bars would be quite helpful to quantify how well the proposed method does indeed outperform other baselines.\n- Have you tried comparing the visual representation learned with other pre-trained representations? For example, features from a pre-trained Masked Auto-Encoder may perform quite reasonably. Additionally, with the myriad of PeFT approaches (e.g LoRA), these would be compelling in terms of efficiency as well.\n\nOverall, I would recommend a weak accept for this paper.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "lv4V1pdyYBP", "reviewer": "Reviewer_pg55", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a modification to an end-to-end learning agent that allows it to learn geographically conditioned policies. The claim is that such a feature allows the agent to have an extended operational domain. The key insight underlying the approach is a cross-attentional mechanism that can efficiently switch between the different domains. In the experiments, the authors have trained their agent with three combined publicly available datasets. ", "summary_of_recommendation": "In summary, the paper would need to drive the point\n- Do these modifications improve closed-loop performance?\n- What is the benefit of training with an aggregated dataset and why?\n(If you can show experimentally that only this kind of regional channel attention allows to learn similarity why excluding differences then the paper would gain much more strength)", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Could you make more clear the benefit of data aggregation in the paper?\n- Could you add an experiment where the backbone is frozen?\n- Could you add some updated limitation section about open loop result not translating to closed-loop performance?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Contribution\n\nThe paper presents an interesting extension of Behavior cloning for autonomous driving. The main challenge in end-2-end autonomy is to learn robust policies. Robustness is difficult to achieve due to many problems involving input data distribution shifts. Here the authors address geographical shift by adding a specialized network branch based on cross-attention to change the policy behavior based on a learned geographical embedding.\n\n= Research Idea\n\n== Quality\n\nThe paper is very well executed. The contribution is well-situated and argued. Even though geographical shift is probably not the major issue with end-2-end learning, it tackles an aspect of distributional shift which itself is the main challenge associated with these approaches. \n\n== clarity\n\nThe paper is quite clear, though the focus is somewhat not ideally placed. For instance, contrastive learning ends up having no statistical significance in the experiments. This should have been made clear from the start. Otherwise, the reader builds an expectation that it is an essential part of the contribution when it is actually not.\n\n== originality \n\nThe paper is rather original in tackling the open-loop question in a very straightforward way. Aggregating data leading to 15 geographical locations leads to interesting results. However I would have expected a focus on: is the data aggregation being leaverage by the agent and how? Is it simply learning better vision features? This could have been studied by freezing the backbone and seeing if the effect of E2E learning are important. Motivating a monolithic approach compared to separated networks.\n\n== significance\n\nThe significance of the work I would rate as medium. It definitely shows how to account for regional differences when trained on massive worldwide datasets. However, it could have more impact by treating the questions of closed-loop behavior and the reverse question which is: Is the proposed approach the best way to use a worldwide dataset with E2E learning.\n\n= Strength and Weakness\n\nStrengths\n-\tVery well-written paper\n-\tClarity of argumentation\n-\tInteresting idea\n-\tBreadth baseline comparison\n\nWeakness\n-\tNo statistical closed-loop results\n-\tDoes not tackle the main question with sufficient clarity?\no\t“ Is this useful?” or could we train on each dataset separately and arbitrate.\n-\tToo much time spent on contrastive learning with no statistically significant results\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "YRdMNWPTyPt", "reviewer": "Reviewer_etZf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper propose to consider the regional factor of autonomous driving explicitly in the form of extra parameters and network. They build a open-loop real world benchmark and one closed-loop CARLA benchmark. They verify the effectiveness of their methods in both benchmarks.", "summary_of_recommendation": "In summary, it is an interesting work with solid experiments, which is why I give a weak accept. However, as mentioned in the weakness part, there are important concerns unsolved. My scores would change according to authors‘ response to these concerns.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.  My largest concern is about open source. IMHO, the problem and benchmarks proposed in the paper are at least as significant as the model. If the benchmark is not open sourced, I do not think these result tables alone could provide much value to the community.\n\n2. How does your AV2+NS+Waymo and train, val, test split work? How many samples in each set? Is there any overlap among cities? For training, do you just feed all data from diferent source into the model?  There is not enough details about the evaluation protocol.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n1. It is reasonable and interesting to consider the regional factor in the application.\n2. The two proposed benchmarks is interesting\n3. Experiments show the effectiveness of the proposed method.\n\nWeakness:\n1. No statement about open source and thus it is difficult to evaluate how it could really be beneficial for the community.  \n2. Possiblly due to limited space, the experiment section is unclear and uncomplete. For example:\n- In Sec. 4.1 Dataset, the authors mentioned AV2+NS+Waymo and CARLA. However, in the paper,  only results on the former one are reported. All results regarding CARLA are in the supplymental materials.\n- In Line298, Youtube videos are mentioned without any context. In Table 2, the setting about SSL is unclear but demonstrated.\n- In Table 1 Table 2, and Sec 4.2, it is unclear that the metric is for which dataset. (Though by the minADE, I gusess it is AV2+NS+Waymo).\n\nThus, I suggest a complete rewriting of the benchmark & experiment section.  The experiments should be divided by setting or datasets in a clearer manner instead of mixed together. For example, if there is no experiments about CARLA in the paper, you should use much less text to mention it.\n\nMissed reference:\nThere are some SSL works specifically for AD in the community are not mentioned.\n[1] Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining. ECCV 22\n[2] PPGeo: Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling. ICLR 23\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "IS6n0hgfuN", "reviewer": "Reviewer_zjyo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work presents an approach that allows the adaptation of conditional imitation learning toward the specifics of region-based driving norms. The method comprises two main contributions: i) a cross-attention-based transformer architecture that allows forming queries based on region embedding, and ii) two additional regularizing contrastive losses: one is imposed on navigational commands, and the second is based on the regional head weights. The authors demonstrate the effectiveness of their method on several publicly available datasets, such as nuScenes, Argoverse2, and Waymo, and perform extensive ablation and comparison analysis with a family of existing methods in centralized and federated learning settings.", "summary_of_recommendation": "The paper brings a few interesting (albeit not groundbreaking) ideas and is generally pleasant to read. \nI believe it has important practical implications, thus I am inclined toward accepting this work. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The terms “branched architecture” and “navigation commands” require at least a brief explanation at the spot of introduction (with examples if possible).\n- The authors should provide more intuition behind contrastive losses. It is not completely clear why they would help with the imbalanced data specifically. \n- Sections on Contrastive Losses require a careful introduction of the notation used (\\hat(y)^(c)_i and y_i.), together with how the corresponding inputs are computed: for example, does the framework execute separate forward passes for each command c? These are core contributions, hence a more detailed and easy-to-understand description could be helpful for the reader.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Praise:\n- The paper addresses an important problem in self-driving: the seamless adaptation of a single policy (or a predictor) to the specifics of driving in different geographic regions.\n- The idea of using contrastive losses is simple and has strong practical implications.\n- The authors perform an extensive evaluation of their method with a decent set of baselines. I find the integration of the method in the federated learning setting particularly appealing.\n\nConcerns:\n- Although my general impression of the information flow and the writing style of the work is positive, I believe it could still be improved. I will leave a few specific comments in the “questions” section.\n- The main metrics used for comparison do not reflect infractions and traffic rules violations. This is not a strong drawback, of course, since the work focuses on the imitation learning setting solely, and ADE/FDE are the most commonly used metrics in this setting. However, I would encourage the authors to think beyond these simple KPIs, since they often hide driving quality issues.\n- It would be interesting to see a more detailed analysis of the limitations, as well as more evaluation of the adaptability of the method. I.e. it is important to point out where practitioners should give up and just keep separate models instead of adapting a single architecture. How and where this boundary should be created, could these regional boundaries be created automatically? To be fair the authors do mention the desire to evaluate their method on a more global scale in the limitations section and the lack of data is, of course, the primary problem here.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Scores:  zjyo: Weak Accept,  (bBuC: Weak Reject), etZf:Weak Accept, pg55: Weak Accept, tmmM: Weak Accept  

Quality:  The paper focuses on a significant challenge in decision-making in self-driving cars, which involves the seamless adjustment of a single policy to account for the nuances of driving in diverse geographic locations.  This is equivalent to the adaptation to distributional shifts  induced by changes in geographic location. The paper is well executed and the contribution is well-situated and argued.

Clarity: The paper is well written and clear.  The reviewers have acknowledged the efforts made by the authors to improve the manuscript.

Originality: Good. The  paper shows that aggregating data from 11 geographical locations leads to interesting results.

Significance: All reviewers have expressed that the paper contains interesting ideas with solid  experiments. 

Pros:
- interesting extension of behavior cloning for geo-aware autonomous driving with solid experiments
- several architectural improvements for a geo-aware driving agent and  breadth baseline comparison
- use of a combination of three public autonomous driving datasets to learn a representation that scales across different deployment scenarios
-  code is available

Cons:
- geographical shifts may  not be the main challenge in autonomous driving E2E learning
- the question  of how data aggregation is  leveraged and whether this it is the best way to make use of a worldwide data set for E2E-learning is not entirely clear
Decision: Accept (Poster)