Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
Authors: Eli Bronstein
Sirish Srinivasan
Supratik Paul
Aman Sinha
Matthew O'Kelly
Payam Nikdel
Shimon Whiteson
Venue: CoRL 2022 Oral (2022)
Keywords: Imitation Learning
Curriculum Learning
Autonomous Driving

Abstract: ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments. In the context of autonomous driving, it is common to treat all available training data equally. However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set -- we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset. We present a method to predict the inherent difficulty of a driving situation given data collected from a fleet of autonomous vehicles deployed on public roads. We then demonstrate that this difficulty score can be used in a zero-shot transfer to generate curricula for an imitation-learning based planning agent. Compared to training on the entire unbiased training dataset, we show that prioritizing difficult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, all while using only 10% of the training data.

Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
Eli Bronstein
Sirish Srinivasan"
Supratik Paul Shimon Whiteson
Aman Sinha
Matthew O'Kelly
Payam Nikdel
Waymo, LLC
(ebronstein, sirisha, supratikpaul, thisisaman, mokelly, payamn, shimonw] @waymo.
Abstract: ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments. In the context of autonomous driving, it is common to treat all available training data equally. However, this approach produces agents that do not perform robustly safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set-we show that an agent trained using only a 10% subset of the data performs just well as an agent trained the entire dataset. We present a method to predict the inherent difficulty of a driving situation given data collected from fleet of autonomous vehicles deployed on public roads. We then demonstrate that this difficulty score can be used in zero-shot transfer to generate curricula for an imitation-learning based planning agent. Compared training on the entire unbiased training dataset, we show that prioritizing difficult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, all while using only 10% of the training data.
Keywords: Imitation Learning, Curriculum Learning, Autonomous Driving
1 Introduction
Autonomous vehicles (AV) typically rely on optimization-be bbased motion planning and control meth- ods. These techniques involve bespoke components specific the deployment region and AV hard- ware, and require copious hand-tuning to adapt to new environments. An alternative approach is to apply machine learning (ML) to the lifetimes of experience that AV fleets can collect within days or weeks. A paradigm shift to ML-based planning could automate the adaptation of behaviors to new areas, improve planning latency, and increase the impact of hardware acceleration.
For example, imitation learning (IL) can utilize the large tranches of expert demonstrations collected by the regular operations of AV fleets to produce policies that perform well in common scenarios, without the need specify a reward function. However, both the distribution from which experi- ences are sampled and the policy used to generate the demonstrations can critically affect the II. policy's 's performance [1]. The training data distribution is especially important when learning meth- ods are applied to problems characterized by long tail examples (c.f. [2. 3, 4, 5, 6)). In the case of autonomous driving, the vast majority of observed scenarios are simple enough to navigated without any negative safety outcomes. A visual inspection of random subset of our data suggests that half of it consists of scenarios with the AV as the only road user motion, while another quarter contains other moving road users, but not necessarily close enough to the AV to affect its behavior. As a result, IL policies may not be robust in safety-critical and long tail situations.
Reinforcement learning (RL) can be used to explicitly penalize poor behavior, but due to the complex nature of driving [7], it is difficult to design a reward function for AVs that aligns with human expectations. Even if reward signals were provided for safety-critical events or traffic law violations (e.g., collisions, running a red light), they would be extremely sparse since such events are quite rare. Furthermore, exploration to collect more long tail data challenging due to safety concerns [8].
Despite these issues, most learning-basee based robotics and AV applications use the naive strategy of cre- ating training datasets from all available demonstrations. In the context of AVs, since most driving situations are simple, this strategy is both inefficient and unlikely generate policy that robust
Denotes equal contribution.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Fleet Learning
Zero-shot Transfer
Venicle Platfomm
Depkoymant Region
Folkcy
D
Development Planner Daployed in Courter factual Simulations
BR
Type?
Region
Parrer
Counterfactual Difficulty Score
H
Region
Expert
B
Fieet Data Segmeets
Run Segment Embedding
Difficulty Model
Same Ran Sogmast?
Seggment Emarodiing
Vileoe Cartgecoinn
Siiiartt
Dificultt Soore
Trajectory
Expert
Readggaph Other Roads
Murti layer Perceptron
The fleet collects experiences with variety of policies in multiple operational design domains. The fleet data is sharded into run segments. Fleet data is used learn an embedding that maps a run segment to a vector space based on similarity. counterfactual simulations and human triage; the outcome of this process is a labeled set of difficulty An MLP is trained to regress from embeddings to the difficulty labels.
Figure 1:
D Run segments are selected for
scores. E
to difficult scenarios. common solution is to upsample challenging examples, either by increas- ing their sampling probability by a predetermined factor [9] or with curriculum learning [10], i.e., dynamically updating the sampling probability during training based on the agent's performance.
However, both of these approaches include significant hurdles. Upsampling requires that we know which examples are part of the long tail priori, as in standard classification problems where the class-label imbalance can inform a sampling strategy. In IL and RL., no such labels are available. As such, curriculum learning is more suitable since it uses the agent's 's current performance to identify hard examples. However, standard approaches to curriculum learning are specific to the agent being trained; they do not, for example, utilize data collected by deployed AVs running other planners, which can provide more general, policy-agnostic insights into the long tail of driving.
In this paper, we propose an approach (summarized in Figure 1) that addresses the challenges of upsampling and curriculum learning applied to an AV setting. Developing road-ready AV gener- ally involves both collecting real-world data with an expert. which can be combination of human drivers and thoroughly evaluated AV planners; and evaluating new development planners, which are regularly simulated on the data collected by the expert to identify potential failure modes, generating a large counterfactual dataset. Our method uses this readily available data to train a difficulty model that scores the inherent difficulty of a given scenario by predicting the probability of collisions and near-misses in simulation. This difficulty model provides several key benefits: 1) it is computation- ally less expensive to predict a driving situation's 's difficulty than to simulate it for a policy being trained; 2) the model learns the inherent, policy-agnostic difficulty of scenario because it is trained on multiple development planners different geographic regions; and 3) the model predicts a con- tinuous score that can be used to identify scenarios within arbitrary difficulty range, rather than obtaining a few counterfactual failures.
We show that a zero-shot transfer of this model can identify long-tail examples that are difficult for a new -b-seeed planning ageht-without any fine tuning. This allows us upsample difficult training examples without expensive evaluation of the agent during training. Though we train the planning agent using II. as a case study, our approach can be applied to any ML--based planning approach. The main contributions of this paper are:
1. We train a model to predict which driving scenarios are difficult for development planners and show that it can zero-shot transfer to the task of finding challenging scenarios on which to train an based planning agent. This generalization suggests that the model can predict the inherent difficulty of driving situation.
2

---

2. We show that training an ML-based planning agent on unbiased driving data leads to poor per- formance on difficult examples since easy driving scenarios dominate rarer, harder cases.
3. We show that using our difficulty model to upsample more challenging scenarios reduces colli- sions by 15% and increases route adherence by 14% on the unbiased test set. This suggests that there are significant diminishing returns in adding common scenarios the training dataset.
2 Related Work
The application of RL to the task of autonomous driving has received significant attention in recent years [1I]: proposed methods span the gamut of methodologies and the AV stack itself. RL has been used address a variety of problems including end-to-end motion planning. behavior gener- ation, reward design, and even behavior prediction. In this work, we focus on imitation learning techniques [12], which avoid direct specification of a reward function, and rely instead on expert demonstrations. As a result, they can capture subtle human preferences and demonstrate impressive performance on a variety of robotics tasks. However, despite many attempts [13, 14, 15]. IL and RL techniques still struggle with the long tail present in the driving task [4].
Like this work, Brys et al. [16] and Suay et al. [17] consider how to leverage potentially subopti- mal demonstrations to improve the efficiency and robustness of learning. Unlike these works, we use offline methods to learn a model of each scenario's 's difficulty and bias the distribution that IL performed on. This approach is similar baselines [18, 19] inspired by Peters and Schaal [9]; however, unlike these works, our setting does not provide a reward signal for the proposed demon- strations. Instead, we use offline, off- off-policy simulations to learn a foundation model with which we can efficiently approximate scenario's 's difficulty, which would have otherwise required expensive counterfactual simulations during training. Our approach sidesteps the inefficiencies of performing rollouts of the learnt policy on the entire dataset since inference using the difficulty model is com- putationally much cheaper than simulation. Similar techniques have also been proposed by Brown et al. [20]: however, they focus largely on situations with severely suboptimal demonstrations where the reward is specified. Similar problems have also been identified in offline RL [21]. Interestingly, Kumar et al. [22] identify the tight relationship between imitation learning and offline RL, noting the theoretical advantage of incorporating reward information in settings like autonomous driving which must avoid rare catastrophic failures. Our experiments provide empirical support for this insight.
Curriculum learning (CL) [10] is also closely related to this work. While not originally classi- fied as such, methods like automatic domain randomization, prioritized experience replay [23], and AlphaGo's 's self-play [24, 25] have led to superhuman game-playing agents and breakthroughs in sim2real transfer [23, 26, 27]. CL methods solve for surrogate objectives rather than directly opti- mizing the final performance of the learner. They control which transitions are sampled, the behavior of other agents in an environment, the generation of initial states, or even the reward function. CL. methods are also characterized by whether they used on- off-policy. For example, Uesato et al. [28] exploit low quality policies to obtain failures in an on-policy RL setting. Finding hard examples in the training data using this approach requires repeatedly generating rollouts for each expert trajectory in the dataset. Such an approach is computationally infeasible when operating at scale since the training datasets can have hundreds of thousands of real-world driving miles. Simi- lar approaches known as nega-negative mining have been used in supervised learning settings [29]; like Uesato et al. [28] they evaluate the difficulty of examples online.
Instead, we consider variants of CL that exploit off-policy data. As in the on-policy case, the key problem to determine which data is interesting. Off-policy compatible methods are also gen- erally purrogate-based. For example, they can select for diversity [30], moderate difficulty [27], surprise [23], or learning progress [31]. Our approach is most similar to Akkaya et al. [27], but in- stead of performing expensive agent evaluation during training, we use off- off-policy data both to train a foundation model [32], which encodes experiences, and to classify the difficulty of an interaction. We also utilize large-scale real-world data and demonstrate that simpler curricula are effective.
3 Background
Model-based Generative Adversarial Imitation Learning: Behavior cloning (BC) [33, 13) is naive imitation learning method that applies supervised learning to match the expert's conditional action distribution: arg max Ex [log Te (e(a]s)]. BC policies may suffer from covariate shift,
3

---

resulting in quadratic worst-case error with respect to the time horizon [34]. To address this issue, generative adversarial imitation learning (GAIL) [35] formulates IL as an adversarial game between the policy wo and the discriminator D The discriminator is trained to classify whether given trajectory was sampled from wo (labeled ()) or from the expert demonstration (labeled 1), and the policy is trained to generate trajectories that are indistinguishable from demonstrations:
arg max arg min Ee,0~ro olog (s, a)] + Es (lo((1 - Du(s.a))l.
0 3
GAIL minimizes the gap in the joint distributions of states and actions p(s, a) between the policy and the expert, resulting in linear error with respect to the time horizon [36]. However, GAIL relies on high variance policy gradient estimates because it uses an unknown dynamics model, making its objective function non-differentiable. In contrast, model-based GAIL (MGAIL) [37] uses differen- tiable dynamics in combination with the reparameterization trick [38] to reduce the variance of the policy gradient estimates.
4 Method
A key challenge in commercial AV development is to design an AV planner that can safely and efficiently navigate real-world settings while aligning with human expectations. At any given time, there may exist multiple development planners under evaluation. Iteratively improving an AV plan- ner typically involves the following three steps. 1) Data Collection: Real-world data is collected by fleet of vehicles in the operational area. 2) Data-Driven Simulation: A development planner tested in simulation by having control the data-collecting ego vehicle in run segment, or short snippet of recorded driving data. 3) Evaluation and Improvement: The development planner is evaluated on key metrics based on these simulations, with potential issues identified and addressed.
As mentioned in Section 1, we consider ML-based approach to developing an AV planner from the ground up. One option is to use imitation learning to train a planning agent. Given an initial dataset of logged expert driving, naive approach is to train the agent the entire dataset. However, this means that challenging long tail segments are used only a few times during training, yielding a planning agent that has difficulty negotiating similar situations [2. 6]. Thus, to improve our agent, we require a method to upsample these rare segments.
4.1 Difficulty Model
The key idea behind our method is to use the real-world run segments replayed in simulation with development planners to learn difficulty model that predicts the difficulty of logged segment, i.e., whether development planner is likely to have a poor safety outcome simulation. We train the difficulty model on simulations of multiple development planners in different geographic areas, so it can be seen as marginalizing over a diverse distribution of development planners. This makes the model more likely to be able to identify the inherent difficulty of a segment. In turn, this facilitates the zero-shot transfer from training on data from development planners to inferring difficulty for a substantially different planning agent. Intuitively, segments that development planners find difficult are likely to be difficult for the planning agent as well. Specifically, we use the difficulty model's scores inform upsampling strategy for training the planning agent.
The evaluation process for development planners typically involves large-scale simulations, with potentially problematic behaviors flagged for engineers to address. We train the difficulty model to predict collisions and near-misses attributable to the development planner, as opposed to other road users. This data is generated the normal course of the AV planner development cycle, so no new training data is needed.
Since we want to marginalize out the idiosyncrasies of individual development planners, we model simulation's 's safety outcome y E {0, 1} (1 if a collision or near-miss occurred, 0 otherwise) as function of the logged run segment alone. The input to our model is a learned segment em- bedding from a separately trained model. Given a logged run segment, we collect static features (e.g., road/lane layouts, crosswalks, stop signs), dynamic features (e.g., positions and orientations of other road users over time), and kinematic information about the data-collecting ego vehicle. We use these features to generate two top-down images of the segment: one of the ego vehicle's trajectory, and another of the static features and other road users' trajectories. We encode each im- age into dense d-dimensional embedding vector (as in (39]) using a CNN and contrastively train

---

a classifier (e.g., [40]) with cross-entropy loss to determine if two images are from the same run segment (see Figure Ic). Our difficulty model is an MLP that learns function f IF - [0, 1] map- ping the embedding to the simulated safety outcome y. We trained this model using cross-entropy loss on a dataset of 5.6k positive and 80k negative examples. The number of negative examples was downsampled by multiple orders of magnitude since the prevalence of simulated collisions and near- misses is extremely low. The model produces uncalibrated scores by design, as trying to calibrate to the extremely small unbiased prevalence rate of positive examples is numerically unstable.
4.2 Sampling Strategies
Given the long-tail nature of the difficulty scores (see Figure 2), natural to upsample difficult segments during training. A standard solution for upsampling in classification problems is to create separate datasets for each class, and then generate a batch by sampling a specified proportion from each dataset. Since this requires discretized classes, it cannot be applied to our real-valued difficulty scores, Moreover, due to the large training dataset size it is not scalable to upsample individual seg- ments: the entire dataset cannot fit in memory and random access to individual examples from disk is incompatible with distributed file sharding of data. Instead, we partition the dataset into ten equally sized buckets, each corresponding to decile of the data by difficulty scores, with up/downsampling achieved by assigning different sampling probabilities to each bucket. This enables us to efficiently generate batches on the fly (e.g., sampling weighted batch of k run segments requires minimal overhead over the h constant-time accesses to the head pointers of each bucket). This decile-based bucketing also ensures that our method is agnostic to the model scores, which are uncalibrated.
We consider two training variants: 1) fixed weighting scheme for each bucket, held constant throughout training, and 2) a schedule of weights for each bucket that changes as training progresses. Specifically, we use the following three sampling strategies. "Highest- 10%" trains the agent only on the highest scoring bucket (i.e., on the segments with the highest 10% difficulty scores). "Uniform- 10%" upsamples difficult segments by setting each bucket's 's sampling weight to the range of diffi- culty scores in that bucket (in the limit of infinite buckets, this approaches a uniform distribution over the difficulty scores). "Geometric-scheduue 10%% implements a geometric progression of weights with each bucket weighted equally at the beginning of training and weighted proportional to its av- erage diffieulty score the end of training (see Appendix 8.2 for further details). In Section 5.4 we compare the performance of these training variants against several baselines. Our variants are trained on only 10% sample of available data.
5 Experiments
To prevent information leakage between the difficulty model and the planning agent, the former is trained on dataset collected more than six months prior to the dataset for the latter. The training dataset for the planning agent consists of over 14k hours of driving logged by a fleet of vehicles. We split the data into 10 second run segments, resulting in over 5 million training segments. We also create two test sets, chronologically separate from the training set to prevent train-test leakage. The first unbiased test set is composed of 20k segments sampled uniformly from logged data. The second set consists of 10k segments with difficulty scores in the top one percentile of the training data's 's score distribution. The distributions of the difficulty model scores for the train set and unbiased test set both have long tails (see Figure 2) - scores above 0.85 account for only around 0.5% of the dataset.
As described in Section 4.2, we split the training dataset into 10 equal sized buckets based on the difficulty score deciles. 200 run segments are further split from each training bucket to obtain vali- dation buckets for model selection. To highlight the effect of our training schemes on performance on segments of varying difficulty, we use the same bucketing approach for the unbiased test set as for the training set. We report the full, unbiased test set results by aggregating over all buckets.
5.1 Baselines
We report three baselines for comparison, which differ in their training data: "Baseline-all"i is trained on the full dataset, "Baseline- 100% trained on a uniformly randomly sampled 10% of the full dataset, and "Bascline-lowest-0g5 10%6 is trained only the bucket with the lowest difficulty scores.
5

---

Deneity
Density
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.4
0.6
0.8
1.0
Model Score
Model Score
(a) Train Dataset
(b) Unbiased Test Dataset
Figure 2: Distribution of the difficulty model scores for the train and test datasets. The ten alternating shaded backgrounds indicate the thresholds the decile buckets.
5.2 Training Details
We the planning agent described in Bronstein al. [41], which employs stochastic continuous action policy conditioned on goal route and is trained using a combination of MGAIL and BC. See Appendix 8.7 for additional details. We train 10 random seeds of each agent variant and baseline for 200k steps. After the initial 100k training steps, we evaluate each agent on the validation set at intervals of 10k steps. We select the agent checkpoint with the lowest sum of collision and off-road driving rates, and evaluate it on the held-out test set. Since the learnt policy is stochastic, we report the average performance of 16 independent rollouts for each test run segment.
5.3 Metrics
We assess the planning agent's performance using the following binary metrics (1 if the event of interest occurred in the segment, 0 otherwise):
Route Failure: the agent deviates from the goal "road route" at the start of the segment, which includes all lanes in the road containing the goal lane-specific route.
Collision: the agent's bounding box intersects with another road user's bounding box.
Off-road: the agent's bounding box exits the drivable road area.
4. Route Progress ratio: ratio of the distance traveled along the route by the agent and the expert.
We also report the overall failure rate the union of the first three metrics; segment is considered failure any of the binary metrics is nonzero. When comparing the performance of different agents, prioritize this failure rate due to the safety-critical nature of driving, while also considering the route progress ratio to ensure the agents are making efficient forward progress.
5.4 Results
We present the performance of our training variants and baselines the full, unbiased test set in Table 1. Each variant's action policy is conditioned on the expert's 'ss initial goal route, which is held constant throughout the segment.
We observe no significant difference between the performance of Baseline- and Baseline-all, demonstrating that simply increasing the training dataset size does not necessarily lead better performance. Also, Baseline-lowest-100 has the worst performance for the collision and off-road metries. This suggests that the easiest segments are not representative of the entire test set distri- bution and do not contain enough useful information to learn from. However, Baseline-lowest- 100 achieves the lowest route failure rate. We believe this is because the least difficult training bucket primarily composed of segments in which it is simple to follow the route, such one-lane roads with no other road users and minimal interaction. This could cause the Baseline-lowest 10% agent overfit the route features and follow the route well the expense of safety.
All three of our upsampling variants achieve significantly lower collision rates, and comparable off-road and route failure rates to the baselines (with the exception of Baseline- 10% 's route failure rate). This key result demonstrates that segments with high predicted difficulty contain the

---

Table 1: Evaluation of agents and baselines on the full unbiased test set (mean standard error of each metric across 10 seeds). For all metrics except route progress, lower is better.
Route Failure
Collision
Off-road
Route Progress
Failure
Agent Variant Baseline-all Baseline- Baseline-lowest 10%
rate (%)
rate (%)
rate (%)
ratio (%)
rate (%)
1.38
1.46
0.73 1.07
81.21 1..39
3.3310.20
1.34
1.50..0.09
0.67 10066
81.12
3.28 10.13
1.14
4.15.0..1
0.98 10.10
00.1
1.91100.1
Highest-IOgh
1.33+00.0
1.23
0.74
1..95=1.33
3.10.10.0
Uniform- 10%
1.35
1.17=20.08
0.75=0.07
80.67
3.07:10117
Geometric-schedull-1155
1.19
1.25
0.74
80.48
2.92:1..11
majority of useful information needed for good aggregate performance. Geometric-shhclul--100 has the largest improvement over the baselines, with a significantly lower collision rate, comparable route failure and off-road rates, and a minimal decrease in the route progress ratio. This highlights the advantage of observing the whole spectrum of data at the start of training, and progressively increasing the proportion of difficult segments to emphasize more useful demonstrations.
To get more nuanced view of each variant's performance, compare the variants to Baseline- 10% for each of the test buckets. Figure 3 shows the performance for the lowest (0-10%), low/mid (30-40%), highest (90-100%) and long tail (99-100%) test buckets. See Figures 5 and 6 in the Appendix for metrics for all the test buckets.
Not only does each agent's 's collision rate correlate with the difficulty score, but so do the route failure and off-road rates, with the exception of Highest- 10%'s 's off-road rate. This shows that segments that were challenging for development planners are also likely to be challenging for our planning agent, which enables the zero-shot transfer of the difficulty model. It also demonstrates that although the difficulty model was only trained to predict collisions and near-misses, its predicted score describes broader notion of difficulty, measured by other key planning metrics.
On the highest and long tail buckets, Highest- 100% and Uniform- 100% achieve much lower colli- sion rates and overall failure rates than Geometric-schcdule- 10% and the baseline. This shows that upsampling difficult segments results in better overall performance
those segments, not just on metrics that are highly correlated with the difficulty label (i.e., collisions and near- near-misses). This is encouraging, since it suggests that the training labels for the difficulty model do not need to fully define expert driving behavior in order for the resulting planning agent to exhibit improved perfor- mance across multiple metrics. However, Highest- 10% and Uniform- 10% perform comparable to, or worse than the baseline on the lowest and low/mid buckets across all metrics, with especially poor performance on the route failure and off-road metries. Thus, extreme upsampling of difficult segments sacrifices performance at the other end of the spectrum, since the easiest segments become too rare in the training data. Geometric-schedull- 100% addresses this issue by upsampling difficult segments while maintaining sufficiently broad coverage over the difficulty distribution. While it does not achieve equally low collision rates the Highest-109; and Uniform-10% variants on the highest and long tail buckets, it outperforms the baseline on collisions and performs well on the lowest and low/mid buckets, yielding the best overall performance.
6 Limitations
While our difficulty model successfully identified challenging segments, it was only trained to pre- diet collisions and near-misses, which are just one indication of difficulty. There are other labels that would be helpful for a more comprehensive difficulty model, such as traffic law violations, route progress, and discomfort caused to both the ego vehicle's passengers and other road users. Moreover, the difficulty model could be improved by incorporating the severity of the negative safety outcome into the training labels. Furthermore, as noted in Section 1, a large proportion of the available data consists of situations with very few other road users in the scene. The difficulty model could be replaced with a heuristics- of doing is difficult to gauge a priori.
approach of pruning such scenarios, though the viability
In terms of evaluation metrics, we focused primarily on safety metrics, since these are of paramount importance for real-world deployment. However, we have not considered other facets of driving like comfort and reliability, which can also significantly affect the viability of ML-based planners.
7

---

highest-10%
12
uniform-100 1rrmm-10%%
baseline-10%
0-10
30-40
90-100
99-100
0-10
30-40
90-100
99-100
(a) Overall failure rate (%)
(b) Route failure rate (%)
10.0
2.4
1.2
0.0
0.0
0-10
30-40
90-100
99-100
0-10
30-40
90-100
99-100
(c) Collision rate (%)
(d) Off-road rate (%)
Figure 3: Metrics for the Baseline- 10%% Highest- 10%, Geometrii-schhcull--105 and Baseline- 10% variants on multiple decile test buckets according to the difficulty score. For each metric, each variant's performance shown for the lowest (0-10%), low/mid (30-40%), highest (90-100%), and long tail (99-100%) test buckets.
Finally, while we have demonstrated that our method upsampling long tail segments leads to better performance, we have done only for agent trained using MGAIL. Quantifying the performance gains with other learning methods remains a topic for future work.
7 Conclusion
We showed that the naive strategy of training unbiased driving dataset is suboptimal due the large fraction of data that does not provide additional useful experience. By utilizing readily available data collected while evaluating development planners in simulation, we trained model to identify difficult segments with poor safety outcomes. We then applied this model in a zero- shot manner develop training curricula that upsample difficult examples. Planning agents trained with these curricula outperform the naive strategy in aggregate and are more robust in challenging, long tail scenarios. However, overly aggressive upsampling produces policies that do not handle simpler situations well. We conclude that sampling strategies that prioritize difficult segments but also include easier ones are likely to achieve the best overall performance.
We have also showed that training on the full dataset does not yield any significant benefit over training on only 10% of the data sampled uniformly at random, demonstrating that simply adding more unbiased data the training set does not necessarily improve performance. This suggests that we can use our difficulty model to reduce the cost of AV system development in two areas: targeting active data collection when operating fleet of vehicles and selective retention of large- scale sensor logs. Namely, since the difficulty model can predict which driving scenarios are likely be challenging for new planning agents, we could identify geographic "hotspots" where these scenarios occur and use these locations to inform our data collection process. Furthermore, since biasing the planning agent's training dataset toward difficult segments leads better results with only fraction of the available data, we could use the difficulty model scores to reduce the amount of stored data without sacrificing downstream performance.

---

Acknowledgments
We thank Ben Sapp, Eugene le, Jonathan Bingham, and Ryan Polkowski for their helpful comments, and to Ury Zhilinsky for his support with experiments and infrastructure.
References
[1] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072119 2020.
[2] J. Frank, S. Mannor, and D. Precup. Reinforcement learning in the presence of rare events, In Proceedings of the 25th international conference Machine learning, pages 336-343, 2008.
[3] N. Kalra and S. M. Paddock. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? RAND Corporation, 2016.
[4] Shalev-Shwartz, Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving, arXiv preprint arXiv:1610.088
2016.
[5] S. Paul, K. Chatzilygeroudis, K. Ciosek, J.-B. Mouret, M. Osbore,, and S. Whiteson. Al- ternating optimisation and quadrature for robust control. In AAAI Conference on Artificial Intelligence, 2018.
[6] S. Paul, M. A. Osborne, and S. Whiteson. Fingerprint policy optimisation for robust reinforce- ment learning. International Conference on Machine Learning. 2019.
[7] De Freitas, A. Censi, B. W. Smith, L. Di Lillo, S. Anthony, and E. Frazzoli. From driverless dilemmas to more practical commonsense tests for automated vehicles. Proceedings of the national academy of sciences, 1 11((1)) 2021.
[8] S. Lange, Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learn- ing. pages 45-73. Springer, 2012.
[9] J. Peters and S. Schaal. Reinforcement learning by soward-weightn regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[10] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum learn- ing for deep rl: A short survey. arXiv preprint arXiv:2003.cm 0..6644404044 2020.
[11] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
[12] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50(2), apr 2017. doi:10.1145/305911 URL https://dolt org/10. 1145/3054912
[13] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. httplhttss//ppceeedeiee www.igs..........om ottpre/ttgs.aarrpp..c//////////o .df.
[14] M. Bojarski et al. End to end learning for self-driving cars. CoRR, 2016. URL http:// // arxiv. orgoorgg/ooo wwwooogg////////oeoooooooeeeesse wwtp////../......
[15] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems, pages 356554573, 2016.
[16] T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Nowé, Reinforcement learning from demonstration through shaping. In Twenty-fourth international joint conference on artificial intelligence, 2015.
[17] H. B. Suay, T. Brys, M. F. Taylor, and Chernova. Learning from demonstration for shaping through inverse reinforcement learning. In AAMAS, pages 429-437, 2016.

---

[18] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34, 2021.
[19] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 168-168. PMLR, 2022.
[20] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[21] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review and perspectives on open problems, 2020. URL http://///arriv.orrrb///////////
[22] A. Kumar, J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05612 2022.
[23] T. Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized experience replay. In ICLR (Poster), 2016.
[24] A. L. Samuel. Some studies in machine learning using the game of checkers. ri-recent progress. IBM Journal of research and development, 11(6):01-617, 1967.
[25] G. Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-211 1994.
[26] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[27] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGirew A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 2019.
[28] J. Uesato, A. Kumar, C. Szepesvari, T. Erez, A. Ruderman, K. Anderson, K. D. Dvijotham, N. Heess, and Kohli. Rigorous agent evaluation: An adversarial approach to uncover catas- trophic failures. In International Conference on Learning Representations, 2018.
[29] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016.
[30] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. Curriculum-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019.
[31] C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P.-Y. Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages 1331-1340. PMLR, 2019.
[32] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07225
2021.
[33] D. Michie, M. Bain, and J. Hayes- Cognitive models from subcognitive skills, IEE control engineering series, 44:71-99, 1990.
[34] S. Ross et al. A reduction of imitation learning and structured prediction to no-regret online learning. In AI Stats, 2011.
[35] J. Ho and S. Ermon. Generative adversarial imitation learning- In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 29, Curran Associates, Inc., 2016. hRt /ttp:////////opppeeeeeeeeee neurips.. cc
Papeeff-aaap..aeeeeeee pdf.
10

---

[36] G. Swamy, S. Choudhury, J. A. Bagnell, and Z. S. Wu. Of moments and matching: A game- theoretic framework for closing the imitation gap, 2021.
[37] N. Baram, O. Anschel, I. Caspi, and S. Mannor. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning, pages 390-399. PMLR, 2017.
[38] M. Xu et al. Variance reduction properties of the reparameterization trick. In A/ Stats, 2019.
[39] M. Chidambaram, Y. Yang. D. Cer, S. Yuan, Y.-H. Sung, B. Strope, and R. Kurzweil. Learning cross-lingual sentence representations via a multi-task dual-encoder model. arXiv preprint arXiv:1810.122 12836, 2018.
[40] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 7748-8763. PMLR, 2021.
[41] E. Bronstein, M. Palatucci, D. Notz, B. White, A. Kuefler, Lu, S. Paul, P. Nikdel, Mougin, H. Chen, J. Fu, A. Abrams, P. Shah, E. Racah, B. Frenkel, S. Whiteson, and D. Anguelov. Hier- archical model-based imitation learning for planning in autonomous driving. 2022 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 2022.
IEEE,
[42] H. Namkoong and J. C. Duchi. Stochastic gradient methods for distributionally robust opti- mization with f-divergences. Advances in neural information processing systems, 29, 2016.
[43] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers, 2021.
[44] J. Mercat et al. Multi-head attention for multi-modal joint vehicle motion forecasting. In ICRA, 2020.
[45] Lee et al. Set transformer: framework for aitention-basee permutation-inwee invariant neural networks. In ICML, 2019.
[46] A. Jaegle al. Perceiver: General perception with iterative attention. In ICML, 2021.
[47] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint 1807 2018.
[48] C. Zhang, R. Guo, W. Zeng, Y. Xiong, B. Dai, R. Hu, M. Ren, and R. Urtasun. Rethinking closed-loop training for autonomous driving. In S. Avidan, Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022, pages 264-282, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-199944-.7


Reviews:
[{"review_id": "NwETewqAc5L", "reviewer": "Reviewer_CdL1", "summary": "The authors argue as follows:\n\nNaively training on all driving data via imitation learning is suboptimal since some examples are harder than others. A way to solve this is via upsampling/upweighting more difficult examples, and the main question is how one would determine this difficulty metric in an efficient way. We can design a separate model which predicts difficulty as a function of run segments, where the label is provided via simulation and simulated / ground-truth metrics of risk. This difficulty model is then run over the entire dataset and predicts a difficulty score for each run segment. This then allows for training of imitation learning with upsampling of the more difficult scenarios and with a curriculum of training on easier->harder examples, which is shown to improve results compared to equal weighting of data.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The authors present a clear way to predict driving difficulty via counterfactual simulation and explain the method clearly\n+ The data sampling methods do demonstrate zero-shot transfer, which implies that using difficulty based on counterfactual simulation does improve imitation learning of AV planning, based on the provided metrics.\n+ Several strategies for sampling are evaluated, which include a curriculum schedule, and performance is visualized which demonstrate tradeoffs between performance on easy and hard data\n+ The ablations of the upsampling strategy and curriculum strategy against the baseline which indicate that the difficulty score is a useful method of upsampling against uniform weighting of the data\n+ The authors provide more detailed experiments in the appendix, including adaptive importance sampling, which help understand the limits of these upsampling methods\n+ The visualizations in the provided video are helpful\n\nWeaknesses\n- The baseline is the naive method, and there are no comparisons to any prior work. Albeit, relevant prior works may be hard to compare against; the authors mention closest work are on-policy\n- There could be some discussion of or attempts at the feasibility of using real data instead of simulated counterfactual examples for training the difficulty model. Or, as an ablation, could some difficulty heuristic be computed for each imitation learning run-segment instead of using the difficulty model?\n- One of the key aspects of this work is the efficiency advantage compared to on-policy difficulty computation methods. The tradeoffs/feasibility here could be better understood if all the dataset sizes were provided.\n- There could be some discussion about using some weighting term based on the difficulty score and uniform sampling, rather than sampling the dataset into buckets. The authors mention that the scores are uncalibrated, but some kind of bucketing of a weighting term may resolve this similarly to what is used in the paper. This may allow for more hyperparameter search compared to the current method of fixed buckets which presents infrastructure challenges.\n- How should we understand the tradeoffs between the metrics, e.g. is failure rate or route progress ratio more important? It seems that the two are somewhat inversely correlated. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work provides compelling motivations and results for using difficulty score based on synthetic off-policy data in imitation learning planning. It is clear how such a method could improve AV planning, and how it could improve training such methods at large dataset scale, where other kinds of difficulty metrics may not be as feasible. Therefore, it seems that this work has the potential for larger impact. There are still some limitations in the set of works or alternative design choices compared against and understanding of the datasets and metrics used.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See questions in weaknesses\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "LaNCtdvih9u", "reviewer": "Reviewer_fmkD", "summary": "This work presents a method for predicting situational difficulty, given data collected from autonomous vehicles deployed on public roads, then uses the predictor to generate curricula for a planning-based imitation learner.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* Strong motivation. The particular choice of underlying data distribution driving imitation learning critically affects performance, particularly in safety critical scenarios with long tail events. This work adds to the body of work that shows just how large an affect changing the data distribution can have on outcomes. In particular this highlights how the common recent approach of \"just scaling up data\" can have diminishing returns. Although this is demonstrated in the autonomous driving domain, I think these findings should be quite relevant to any other where easy examples dominate much rarer safety critical events. Furthermore, L53-L65 do a good job of motivating the particular choice of a machine-learned \"foundation difficulty model\" over the more expensive conventional approach of using planners to generate counterfactual failure datasets.\n* Clear method description. Good description of the full inputs and outputs to the foundation difficulty model and how it was trained, as well as how each part of the overall system fit together.\n* Careful experiments highlight a surprising and important point: simply increasing the training dataset size does not necessarily lead to better performance. Prioritizing difficult scenarios during imitation learning, using the presented method, allows a model to be trained on 10% of the full dataset, that behaves competitively with one trained on the full dataset. This highlights a weakness in the popular \"just scale up data\" approach. I think the magnitude of the improvement in sample efficiency via this prioritization scheme could be an important finding for others working on autonomous driving.\n* Good discussion of related work, highlighting how this off-policy curricula approach compares to on-policy versions.\nInteresting study (4.2) comparing simpler fixed weighting amongst difficulty buckets to more complex scheduled weighting.\n* Careful seeded experiment design which help assess statistical significance.\n\n## Weaknesses\n* No ablations on foundation model. While the benefits of employing a \"foundation difficulty model\" seem clear from the experiment, the reader is not left with too much insight on which elements of this foundation model are the most essential. For example, L159 the model is trained on \"multiple development planners in different geographic areas\". This is claimed as the key to generalization, but we do not know *how important* it is. It would be interesting to see for example, how much data is required to train a sufficient foundation difficulty model, whether multiple regions of data or data coming from multiple planners are necessary, etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work provides experiments highlighting the surprising diminishing returns \"just scaling up data\" can have in learning autonomous driving. It presents a new \"foundation difficulty model\" approach, trained on real offline driving data, which is used to drive a reweighting curriculum imitation learning.  Careful experiments show that this system learns controllers that are competitive with those receiving 10x more data. I think this is a concrete result that others doing large scale imitation learning will benefit from.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* It would be helpful to understand the essential components of the foundation difficulty model, through ablation experiments.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "BOl6DFCHiW", "reviewer": "Reviewer_4k9b", "summary": "This paper proposes a system to label the difficulty of collected driving cases in a policy-agnostic manner so that this system can be used to pick safety-critical training data for ML-based planning modules. By changing the training data distribution according to the labeled difficult score, the performance of the IL policy can be improved. It has a lower failure rate in safety-critical scenarios, compared to the agent trained on unbiased training data.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\nThe motivation for this work is important. Ways to tame the long-tail distribution should be developed for acquiring a learning-based controller. This work proposes a new approach to achieve this by giving a difficulty score.\n\n**Weakness**:\n1. It is well known that changing the training data distribution can improve performance.\n2. The description of the method is ambiguous and difficult to understand. Consider adding a figure to better illustrate it.\n3. In the experiment, other policy-agnostic baselines, which can also score the difficulty of driving scenarios should be included as baselines, such as RSS (Mobile Eye *x* Intel). \n\nPersonally, I do not believe that training ML-policy in replayed cases is a good approach for autonomous driving, especially for solving safety-critical cases where rich interaction is required. Since the replayed vehicles can not react to the ego vehicles, the policy can not be adapted to the real world. We should consider generating some realistic interactive cases from the collected data for learning-based policy training and bridging the sim-2-real gap.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method description is not accurate. Notations and figures can be used to improve the quality. Also, more evidence should be provided to point out why this scoring method is better than other rule-based ones, such as RSS. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Which dataset is used in this work? Is it an open dataset or an in-house one? ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "1HxAKNZkzwk", "reviewer": "Reviewer_Dcm5", "summary": "This work presents a method to assess the difficulty of a scenario in an autonomous driving data set. Instead of using the full data set for motion planning in an imitation learning setting, the proposed method only considers a subset of the data set which is assessed as difficult to generate curricula. These curricula are used to train an imitation learning agent for motion planning. Experiments show that the proposed method achieves improved performance in various criteria such as reduced collisions in the autonomous driving setup. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides thorough evaluations of the sampling techniques and provides a nice related work part.\nHowever, I find it quite hard to follow which parts are optimized with only the text (see issues)\nAdditionally, a comparison to other baselines which use different techniques e.g. [16, 17] would make sense for me. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on my concerns and issues I recommend assessing a weak reject. However, as I am not an expert in this field, I am fine if the other reviewers think that the current version of the paper is worth being published.\n\n# Post Rebuttal decision:\nBased on the answers of the authors, I would like to raise my recommendation from weak reject to **weak accept**.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "- While reading the text I find it quite hard to follow which parts of the framework are optimized. I believe the reader would appreciate it if the objectives which are optimized are explicitly mentioned in the text in the sense that also the optimized models/distributions have an explicit mathematical description. By these definitions the text can also easily refer to the parts and explaining will probably be easier\n\n- While the evaluations of the paper are thorough and nice, I was wondering if a comparison to other similar works (e.g. [16, 17]) (as stated in the related work part) would make sense. Is there a specific reason why they were not conducted?\n\n- Does the sentence in lines 94-96 make sense? It is mentioned that a \"counterfactual simulation\" is used, but later in the same sentence it is mentioned that \"... would have otherwise required expensive counterfactual simulation\". Probably I missed something and got confused with this sentence. "}}]

Author rebuttal:

Decision: Accept (oral)