TITRATED: Learned Human Driving Behavior without Infractions via Amortized Inference
Authors: Vasileios Lioutas
Adam Scibior
Frank Wood
Venue: Accepted by TMLR ()

Abstract: Models of human driving behavior have long been used for prediction in autonomous vehicles, but recently have also started being used to create non-playable characters for driving simulations. While such models are in many respects realistic, they tend to suffer from unacceptably high rates of driving infractions, such as collisions or off-road driving, particularly when deployed in map locations with road geometries dissimilar to the training dataset. In this paper we present a novel method for fine-tuning a foundation model of human driving behavior to novel locations where human demonstrations are not available which reduces the incidence of such infractions. The method relies on inference in the foundation model to generate infraction-free trajectories as well as additional penalties applied when fine-tuning the amortized inference behavioral model. We demonstrate this "titration" technique using the ITRA foundation behavior model trained on the INTERACTION dataset when transferring to CARLA map locations. We demonstrate a 76-86% reduction in infraction rate and provide evidence that further gains are possible with more computation or better inference algorithms.

Published in Transactions on Machine Learning Research (08/2022)
TITRATED: Learned Human Driving Behavior without Infractions via Amortized Inference
Vasileios Lioutas
Inverted AI University of British Columbia
Adam Scibior Inverted AI
adam.sciborr@mmeerte..cc
Frank Wood
fronk, www.:oowowwwweeeecccco
Inverted Al
University of British Columbia
Montréal Institute for Learning Algorithms (Mila)
Reviewed on OpenReview: https: optpr/veeeeeee...ooo net/ forum9 M8D542smrO
Abstract
Models of human driving behavior have long been used for prediction in autonomous vehi- cles, but recently have also started being used create non-playable characters for driving simulations. While such models are in many respects realistic, they tend to suffer from umac- ceptably high rates of driving infractions, such collisions off-road driving, particularly when deployed in map locations with road geometries dissimilar to the training dataset. In this paper we present a novel method for fine-tuning foundation model human driving behavior to novel locations where human demonstrations are not available which reduces the incidence of such infractions. The method relics on inference the foundation model to generate infraction-fis frre trajectories as well additional penalties applied when tne-tuning the amortized inference behavioral model. We demonstrate this "titration" technique using the ITRA foundation behavior model trained the INTERACTION dataset when trans- ferring to CARLA map locations. We demonstrate 76-86% reduction in infraction rate and provide evidence that further gains are possible with more computation or better inference algorithms.
1 Introduction
Predicting the behavior of human-operated vehicles is a crucial capability for self-driving cars. Traditional
approaches are based on simple kinematic models (Cosgun et al.
2017 Ziegler et al. 2014 Lefevre et al.
2014 Haarnoja et al. 2016) or hand-coded Dociiiio-mmkingg systems systemns (Kesting et al. 2009
which may include
parameters learned from data (Bhattacharyya et al.
2021).
Recently data-driven approaches utilizing deep
lcarning have bocome increasingly popular (Lee et al.
2017
Tang & Salakhutdinov
2019
Djuric et al. 2020:
Mo el al. 2020 Rhinehart et al. 2019 Gupta et al.
2018
Sadeghian et al. 2019 Zhao et al., 2020 Hong
al. 2019 Zhao et al. 2019
Apart from their use in on-board prediction systems for planning in sclf-driving cars, models of human driving
behaviors are also used to create realistic non-playable characters (NPCs) for simulators in which self-driving
cars can be tested and trained (Suo et al.) 2021 Bergamini et al. 2021 both settings it problematic if the models too-frequently generate behaviors that result in serious infractions, such as colliding with other agents or going off-road often, as the resulting simulations would be sufficiently unrealistic adversely impact downstream task performance. The problem of excess infractions be to some extent. mitigated

---

Published in Transactions on Machine Learning Research (08/2022)
Observed Behavior
AV Simulator
OOD Locations
Pre-train
Foundation Model p(81:))
Inferred Distribution p($1r) A)
Inference
Plaurl Fulac) p(surlA True)
Amortized Inference
Synthetic Data
TITRATED Model
AV Simulator NPCs
θ
po(81:) Deploy
Figure I: Overview of our model "titration" method. Starting from a pretrained loundation behavior model p($1:)) on observed human behavior (gray shaded boxes are taken as given starting point), our aim is transfer its predictive power to out of distribution (OOD) map geometries, particularly infraction-free behavior models to drive coa.ppaeaallscharrateess (NPC)s in AV simulators. Our method is rooted in the use of inference to generate infraction-free trajectories in OOD locations from the conditional distribution pA) where A is shorthand for the event "no infractions occur." By sampling behavioral rollouts from this distribution we generate synthetic data which is then, turn, used to train a parametric "amortized" infraction-free behavior model (0(11:) As p(s1:T|A).
to nearly-
by introducing additional penalties for infractions at foundation behavior model training time, which are referred to as "common sense losses" in Suo et al. 2021
Like all machine learning models, the ones predicting human driving behavior suffer from degraded perfor- mance under domain shift, which in this case, in particular, occurs when the models are deployed in locations not covered in the training dataset. This includes increased infraction rates, which is a major obstacle when creating simulations with NPCs learned from data collected on different roads. In this paper we present a general algorithm for fine-tuning behavior prediction models to novel settings where human demonstrations are not available.
Our key observation is that models predicting distribution over possible future behaviors, rather than just single behavior, often predict distributions that contain good predictions, but also others that perform infractions. Such predictions can be improved by further conditioning on not performing such infractions. While in certain settings such conditioning can be performed online during simulation (Suo et al. 2021 doing so generally too computationally expensive.
We generate infraction-free trajectories target locations from the conditional distribution defined by the predictive model via probabilistic inference offline. Subsequently, these trajectories to train new
2

---

Published in Transactions on Machine Learning Research (08/2022)
predictive model, typically by fine-tuning the original one, which targets the infraction-fres conditional distribution. This effectively creates a synthetic dataset of behaviors that avoids infractions in the target locations, which can be used for learning behavior models through the usual means. This can be seen as a kind of highly structured data augmentation process. Alternatively, we can view this process as an instance of amortized inference, where the observations are known in advance, but the model is further conditioned, here on constraints. Figure graphically illustrates our approach.
In this paper, use ITRA (Scibior et al., 2021 trained on the INTERACTION dataset (Zhan et al. 2019) as our foundation model (Bommasani et al.) 2021 and target the creation of NPCs for CARLA (Dosovitskiy ct al.] 2017). We find that out-of-the-box ITRA generates trajectories that are in many ways realistic but sulfer from unacceptably high rates of collisions and off-road driving on CARLA maps motivating the development of the method presented in this paper. We call the model output by applying our methodology TITRATED, which stands for "Training ITRA to Emulate Desiderata.. The name is also loosely inspired by a form of "titration" we perform in Section 4.1 as we increase the number of rejection sampling attempts.
2 Related Work
Whether for producing onboard agent behavior prodictions for autonomous vehicles (AV) or for generating realistic non-playable character (NPC) behaviors for AV simulators, behavior prediction models are at the heart of solving the AV control problem. The imitation of human driving behavior has been used to learn control policies for autonomous vehicles (Bojarski et al., 2016 Hawke et al. 2020). More recently, emphasis was put on probabilistic prediction models that behave more human-like and realistic (Suo et al.] 2021 Bergamini et al. 2021 Scibior et al., 2021). All machine learning data-driven behavior models suffer from generalizability issues, especially when the available training data is limited in the diversity of road structures and interesting agent interactions. To improve the performance of behavior models, [Bnssal et al. (2019) proposed using collision and off-road losses during training to reduce infraction rates. These losses are computed in the pixel space from the generated birdview representation masking the corresponding disallowed areas (i.e. the other agent and off-road parts). They complement the standard imitation loss that used against human-recorded ground truth data, and thus, as training progresses, the model produces fewer and fewer infractions, leading the auxiliary losses to provide less information compared to the initial stages of training. On the other hand, Park et al. (2020) proposed to give as input to the probabilistic behavior model probability mea that explicitly highlights the areas that are considered non-drivable. Zhhu et al. (2021) suggested using an unlikelihood loss during training that will minimize the likelihood of undesired predictions possibly reducing the diversity of the predicted trajectories. Rhinehart et al. (2018) proposed R2P2 which is a single agent model that uses an autoregressive Gaussian at every step, that is conditionally dependent on all the previous steps. R2P2 emphasizes the use of a custom loss, which is a symmetrized cross-entropy loss between the model and data. This resembles a forward and reverse KL term. The latter is evaluated by rolling out under the model and evaluating an approximate data density, either KDE estimate custom model scoring drivable surfaces trained separately. In contrast, in this work, are interested in improving the performance of existing foundation probabilistic behavior models in OOD locations where acquiring human demonstration is difficult or even impossible. We amortize the inference process of generating infraction-free synthetic examples by finetuning with these examples and additionally applying collision and off-road losses by rolling out under the model during training.
3 Method
3.1
Foundation Model
Our starting point is ITRA Scibior et al. 2021 pre-trained on the INTERACTION dataset. This is the foundation model of driving behavior that we use in this work, but any other sufficiently general probabilistic behavior prediction model (e.g. (Zhao et al. 2019 Suo et al.] 2021)) trained on any other dataset with sufficient coverage could equally well have been used. The method we develop is model-agnostic.
For convenience we share notation with ITRA. Let s² - st...,, s denote a sequence of states st for agent i where i (1...., N.) and N₂ is the number of agents in the scene at time t. Fach agent state is
s(....,,.........
3

---

Published in Transactions on Machine Learning Research (08/2022)
ITRA
TITRATED
ITRA
TITRATED
ITRA
TITRATED
ITRA
TITRATED
BABAA
BERSO
Figure 2: Examples from finetuning ITRA to the novel setting of CARLA. On its own ITRA produces umacceptable levels of infractions on CARLA maps, both in terms of collisions (top row) and off-road- driving (bottom row). Our method, TITRATED, substantially reduces both forms of infractions. Colored lines indicate the path moving vehicles took to get where they are.
tuple s/= vi. vit. of) E R' where (r7, vt) is the geometric center of the agent, v is the agent's 's orientation, and ug is its speed. Each agent has a fixed rectangular shape with length I' and width w², which is assumod known and implicitly part of each agents' state.
In ITRA, the interactions of the agents with each other and the environment are mediated through birdview image, which for each agent i and time t is obtained via differentiable rendering Ij render(í, s, s!:"x). These images are centered on each ego vehicle and rotated lo match its orientation, but otherwise are similar to the images shown in Figure 2 albeit without the colored paths. Based on Z₃ and a hidden recurrent state, the agent i at time selects action a; = (;), consisting of acceleration and stecring, which is translated into subsequent state s/41 using a bicycle kinematic model (Polack et al. 2017). This combination birdview images and action space is selected facilitate domain transfer of ITRA predictions to novel maps.
Each agent in ITRA is modeled as conditional variational recurrent neural network (VRNN) (Chung al. 2015), with latent variables zf. The joint predictive distribution of ITRA factorizes
T N.
/
(((i:))
I
M4 8 d2
I:N.
(1)
where
p22t)= N(2
(2)
p( Ij 2f, hi 5/) = (kin af), oI)
(3)
of = (TT; #/ M(_1)
(4)
with to denoting the neural network and kin the kinematic bicycle model. The model is trained as usual, jointly with separate inference network, by maximizing the evidence lower bound (ELBO). We refer the interested reader to [Scibior et al. 2021 for more details.
3.2 Infractions
We start by formalizing the notion of infraction, notably collision and off-road invasion. We chose metrics that can indicate both the presence of such infractions and their severity. In Section 3.4 we use these metrics directly as losses in training.
For each agent i and time f, the current state of and the fixed dimensions (F,w") w²) define bounding box Cj R1x2, represented as four corners of the vehicle. The overlap between bounding boxes be quantified
4

---

Published in Transactions on Machine Learning Research (08/2022)
using the standard Interscction--vee- (IoU) metric
[, n B,
loU(B,, B,)
(5)
[B, Bjl
Using that, we define our collision metric as the sum of individual loUs across time
N Ce(Cir)) = Σ LoU(C, t
(6)
As this metric is generally expensive to compute, particularly in a differentiable fashion, we use an approxi- mate version of it described in Zhou et al. (2019). Contrary to other methods (Suo et al. 2021 Rempe et al. 2022) that employ a differentiable collision loss that approximates each vehicle as 5 circles and maximizes the distance between the closest centroids, our use of the technique of [hou al. (2019) is novel in that it is highly accurate differentiable collision loss that directly minimizes the overlapping area. between two collided vehicles.
For off-road infractions, we assume that we have access to a triangle mesh V defining the driveable area (road surfaces, etc.). Given a function & that computes the distance from a point to a triangle, we define a novel off-road infraction metric as the sum of distances the corners of all vehicles from the road mesh across time
T N₂ ΣE min ((g.
(oR(CI.T
eev
1= i
This metric can be computed efficiently in a disferentiable manner (Eberly) 1999 and well-established im- plementations exist [ When used in training, this metric is referred to as the point-to-surface loss (Smith) ct al. 2019). Note that it is zero only when all four corners of all vehicles are contained within the driveable area.
3.3 Inference
Let A be the event that no vehicle performed infraction within the specified time window
A:= 0) A (COR = 0)
(8)
where A means logical and. The probability of A under the real-world distribution of human driving be- haviors is remarkably close to 1, but in predictive models such ITRA it can be much lower, indicating an unacoeptable, unrealistic nunber of infractions. Our key insight is that we can obtain a better model by further conditioning on not performing infractions, demoted here as the conditional density p(sur) V). Note that in order to keep the notation relatively uncluttered we have throughout deliberately elided nota- tion that indicates the conditional dependence on the underlying map layout V. It should be kept in mind throughout that all of our objectives averaged over V.
Unfortunately, sampling from the conditional distribution p(51.T (A) can be computationally expensive. This is particularly problematic in the simulation context, where we would normally unroll the predictive model one step at a time to derive NPC behaviors, but conditioning on not performing infractions in the future requires also sampling subsequent states. For this reason, in Section 3.4 we learn a model po(s1:)) = p(Si.T|A) A), which can be sampled from sequentially in fixed time. First, we show how to use inference to generate examples of infraction-free trajectories in an off-line setting.
Given collection D of initial conditions on target maps, free trajectories by repeatedly sampling joint trajectories from the predictive model and rejecting them until no infractions are found. This procedure is simple but can be computationally expensive, since rejection sampling can take arbitrarily long to produce acceptable samples. In order to limit computational cost, we introduce a max mar_trials parameter, which indicates the maximum number of sampling attempts performed,
construct synthetic datasct D of infraction-
5

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm
Infraction-Free Dataset Generation
Input: Initial conditions dataset D
Driving behaviour model p(s1:))
Maximum number of trials max trials
Output: Infraction-Free dataset. D
D - 0
for (8o; V) D do
found false
for + to maa trials do
5:
Sample rollout $1.7 from p(s1:r(so)
6:
Convert states 81. to bounding boxes Cı.r if Le(Ci.)) = 0 and CoR(Ci.r, V) = then
7:
sound < true break
10: if found then
11:
De - Du ((sTr,
return D
12:
after which the (so. V) tuple is excluded from the datasct if an acceptable sample was not found. The details of this datasel generation process are given in Algorithm D Note that this process can be done "just-in- time", interleaved with training, which is what we actually do in our experiments. This has the advantage of not allowing the fine-tuned model to overfit to specific examples and requires, instead, it to fit the full conditional distribution.
3.4 Amortized Inference
Using the synthetic data consisting of infraction-free Irajectories from Section 3.3 we can train an amortized, "titrated", model with ELBO, following the standard training process of ITRA outlined in Section 3.1 This process minimizes the negative ELBO, defined as
T-1 Nr
LELBO
E
E (o(=(la;777)1))
10g DRr. (o))ll
$1.7 p1 (4)
Aml
E
log (o(12)))))
(9)
$1.2 AurmpleurAA
where q8 is separate inference network trained jointly with the model pa-
As discussed in Section 1 learned models of human driving behavior, such as ITRA, are prone to generate excessive infractions. We ameliorate this problem by introducing explicit infraction penalties in the form of
the collision metrics from Section 3.2 obtained when sampling from the amortized model pe, as additional loss terms
C - + AcCe + AOR
(10)
This is the final training objective for TITRATED, although some care needs to be taken when minimizing it, since LELBO involves sampling from the inference network, while Le and LoR involve sampling from the prior. We address this by performing two separate rollouts for each training example, shown in detail in Algorithm 2
4 Experimental Results
In our experiments, we, at a high level, learn human driving behaviors from the INTERACTION datasct
(Zhan et al., 2019) and use them to create NPCs in CARLA (Dosovitskiy al.] 2017 More specifically
take pre-trained ITRA model obtained exactly described in Scibior et al. 2001 and fine-tune it

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm 2 Amortized Inference with Infraction Penalties Training Process
Input: Infraction-Free datasct D (Alg]
Foundation model
Output: Amortized model po
Initialize Py
2: for ($1:7, V) E D do
3: $1:T-1 ~ po(s1:r) using proposal distribution qá
4: LELBO < compute using 81.7 and 81.7-1 (Eq 9
5:
$1T-1 ~ po((1:)) using prior distribution
Convert states 81:7-1 lo bounding boxes C1T--1
Lc compute using C1.T-1 (Eq 6)
LoR + compute using C1.T-1 and V (Eq
10:
11:
L <- - LELBO + Achc + AonCon (Eq 10
- dc
12:
13:
return pe
Table 1: Summary of the dataset collected using CARLA autopilot. These are the initial conditions used
adapt TITRATED to CARLA.
Avg. Num. Agents Per Trajoctory
Scene
Train
Val. Size
Town01 Ttwait_Sttaight
7338 / 492
8.6
Town01 3way
18632
930
19.8
Town02 Straight
36215 / 1916
35.5
Town02 3way
38206
2030
38.8
Town03 Roundabout
21830 / 1151
24.1
Town03 5way
22160 / 1076
24.1
Town03. 4way
12822
753
13.8
Town03 3way Unprottceee
14865
885
15.0
Town03 3way Protected
13045
688
14.5
Town03 CasStation
21127
1028
21.2
Town04 Merging
5268 / 292
7.3
Town04 3way Large
3931 / 193
5.3
Town04 3way Small
5076 / 394
6.2
Town04 4way__top Stop
7872 582
8.5
Town04 Parking
12794
701
13.8
Town06 Merge_ Morge__Single
Single
13154 / 664
16.5
Town06 4way Jarge
9110 / 362
9.7
Town06 Merge_Double Double
5028
240
7.5
Town07 3way
35601 / 1871
37.0
Town07 4way
30412
1586
31.8
Town10HD 4way
57826
3005
55.9
Town10HD 3way Protected
30456
30.0
Town10HD 3way Stop
50747
2566
50.0
with the method described in Section 3 to obtain TITRATED models for collection of selected locations in CARLA. Our goal is to maintain maximum similarity to ITRA predictions while reducing the incidence of driving infractions, consisting of collisions and off-road invasions, in these novel simulator contexts.
To create collection of initial conditions required by Algorithm in CARLA, we created a custom synthetic dataset using the built-in autopilot. Specifically, in each town we recorded 200 minutes of driving 10 frames per second, featuring 100 randomly selected and placed vehicles. We then selected 23 locations of interest

---

Published in Transactions on Machine Learning Research (08/2022)
Table 2: Rates of driving infractions various CARLA scenes, with (TITRATED) and without (TTRA) the fine-tuning proposed in this paper. TITRATED (All maps) refers single model trained on all 23 distinct locations.
Collision Rate x10
Off-road Rate x10
Scene
TITRATED
TITRATED
ITRA
TITRATED
Reduction
ITRA
TITRATED
Reduction
(AlI maps)
(AlI maps)
Towu01_ Strsih_St
26.0
2.0
5.6
78.46%
42.7
0
100%
Town01__wwe
19.4
4.8
4.3
77.84%
11.3
100%
Town02 Straight
69.64%
51.5
1.6
90.49%
Towa02 3way
10.0
3.3
3.1
69.00%
33.9
98.53%
Town03 Roundabout
5.0
1.7
1.2
76.00%
131.0
17.1
7.4
94.48%
Been08__way
4.6
80.43%
37.5
3.7
96.14%
Town03 4way
6.0
34.2
70.83%
Towa03 Sway Unprotected
6.3
2.7
2.6
58.73%
74.8
42.3
25.1
66.44%
2arn03_wway _ Protected
10.4
4.9
3.9
02.50%
84.9
41.5
27.4
67.73%
Town03 GasStation
6.6
0.6
0.3
95.45%
32.8
18.2
19.1
41.77%
Town01 Merging
0.2
100%
100%
Town04 Sway Large
8.0
2.3
1.4
82.5%
11.2
100%
Towa04 Ba1e044___a_____ee
3way__wwall
Small
16.6
6.4
3.4
79.522%
29.6
0.8
100%
Towa04 Awwyy_too
10.9
4.7
2.0
81.66%
32.4
11.6
0.09
89.72%
Town04 Parking
6.5
4.7
1.1
83.08%
44.7
5.3
0.1
99.78%
Town06 Merge Single
1.7
1.1
1.3
23.53%
109.6
27.6
20.0
81.75%
Town06 Away large
1.7
0.3
0
100%
7.3
1.5
3.3
$4.79%
Towa06 Merge Double
1.8
0.5
100%
2.3
100%
TownO7 3way
6.9
2.2
97.79%
Towa07 Owway
7.8
2.3
2.6
66.67%
60.6
3.1
0.1
99.83%
Towa10HD Away
72.22%
78.67%
Town10HD 3way Protected
4.5
80.00%
72.44%
TolnD 3way Stop
3.6
1.3
63.89%
62.5
17.2
12.8
79.52%
Average
7.8
2.4
1.8
76.11%
52.6
14.5
7.8
86.55%
acToss 7 towns and cropped the recordings to 100 meter radius from each of them, mimicking the structure of the INTERACTION dataset. We then sliced those datasets into second segments, consisting of 1 second of history and 3 seconds predictions, laking the second history from those segments as initial conditions. The dataset summarized Table We emphasize that the autopilot-gennerated trajectories are only used to specify initial conditions, the trajectories recorded in the subsequent 3 seconds never used in training.
In the evaluation we are only interested in the frequency of infractions and not their severity, Specifically, we define the
define
infraction rates using indicator functions on the metrics defined in Section 3.2 collision rate as
1 Collision Rate N KTT ΣΣΣ ce(
(11)
where N is the number of validation examples and K is the number of samples generated for each example n E N. Contrary to other methods (Suo et al. (021),, we apply zero collision tolerance since our method is able to detect collisions precisely instead of approximately. We define the off-road rate as
N
1
Off-road Rate
CoR(s (01)00
(12)
N KKT
where i refers to the index of the ego-vehicle.
In each of the CARLA locations, we produce a separale version of TITRATED by fine-tuning ITRA from the checkpoint trained on the INTERACTION dataset. In addition, we train a single TITRATED model for all CARLA locations. We use ITRA (Scibior et al.] (2021) as our bascline model pre-trained on the whole INTERACTION dataset (Zhan al. 2019). We follow the same syper-paramete setup as described in Scibior et al. (2021) and fine-tune mirror copy of the ITRA model for each unseen location from the
8

---

Published in Transactions on Machine Learning Research (08/2022)
Table 3: The effect of more effective inference on TITRATED infraction rates.
Max Trixls
10
100
100
20
100
Scstie
Callisian Rate x10-4
Off-rosd Rate x10-4
Rejected Exxmplex
Straighs Road
11.21
2.00
2.07
2.02
1.58
2.25
7.5%
2.6%
2.1%
1.7%
1.6%
Roundabout
3.45
1.72
1.72
1.43
63.32
47.15
37.58
35.19
34.60
10.1%
5.4%
4.8%
4.1%
3.7%
4-way Intersection
1.05
0.34
0.30
0
2.60
1.53
1.38
0.92
0.
2.8%
1 1.5%%
1.2%
1.0%
0.95%
Collision Rate
Off-road Rate
1.0
Normaoond Dreriaan
0.88
0.0
Sha Road
0.5
Striight Road
Hourdaatos 4-aay Internaction
Nomollzed
Nouncdbout <way Imarsection
0.4
0.4
0.0
4.0%
Percentage
Eximoles
Figure 3: Scatter plots of infraction rates achieved by TITRATED versus the number of items rejected from the synthetic dataset. Different points correspond to max trials values E {1, 10, 20, 20,50, 100} used in rejection sampling, increasing to the left. Straight lines are fit with least squares to extrapolate performance to setting with perfect inference algorithms where no items are rejected.
synthetic dataset using the proposed methodology. Specifically, we set the model hidden size da to 64, the latent representation size d_ to 2 and the number of recurrent layers to 2. We use differentiable rendering as described in Scibior et al. (2011) and a birdview image size 256 256. We use the ADAM optimizer (Kingma Ba 2015) with default values and 1 x 10-3 learning rate. The batch size set to trajectories using class-mates forcing (Scibior al. 2021 Tang & Salakhutdinov 2019) and trained on single NVIDIA RIX 2080 Ti GPU. We set the hyperparameters Àc and ÅOR to I x 10sh and 1 102 respectively and use maz trials of 10 and just-in-time sampling of infraction-free training examples.
Table 2 shows that TITRATED is able to significantly reduce the collision and off-road rates of ITRA. Specifically, we first generate an ITRA rollout for each initial condition in the corresponding CARLA dataset (see Table D and compute infraction rates. We do the same for TITRATED rollouts. We see that, averaged across all CARLA locations, individually trained TITRATED models are able to reduce collisions by 69% and off-road invasions by 72%. In some cases, TITRATED even achieves a full 100% reduction, entirely eliminating off-road driving in all validation examples. Furthermore, on all CARLA locations is producing 76% reduction in collisions and 86% in off-road infractions. Figure 3 and the related discussion in the following section suggest that TITRATED is likely to reduce infractions even further with increased quality of inference, albeit at potentially greater computational expense to construct infraction-free examples for all initial conditions.
single TITRATED model trained
4.1 Better Inference Leads to Lower Infraction Rates
The key component of TITRATED is the Bayesian inference algorithm for computing and sampling from the conditional distribution 44). In the main experiment, we used rejection sampling with max_frials = 10, resulting in a reasonable trade-off between computational cost and infraction metric improvements. However, this choice excludes many examples for which no accepted sample could be found in ten tries. Crucially, these are "difficult" examples in the sense that ITRA is likely to produce high infraction rates on them. This raises the following question: would finding accepted samples for all of these difficult examples drive TITRATED model infraction rates to zero?
9

---

Published in Transactions on Machine Learning Research (08/2022)
TownO1_Straight
Town03 Roundabout
Town06 21116644wwyyggg llagge
DR DR_ BU__DEEUMMM____ Merging_Mi
DR_ DR_USA USA Roundabout FT
DR_USA USA Intersection MA
Figure 4: Top row: three representative scenes from CARLA dataset; straight road, roundabout, and 4-way intersection. Bottom row: the structurally similar maps from the INTERACTION dataset that we used for the acceleration distribution comparison in Figure
To answer this question, we investigated how using increasingly more powerful inference algorithms could impact the performance of TITRATED, trying to extrapolate to the setting a perfect inference oracle that is able to produce samples with no infractions using rejection sampling with increasing mar max__rials to stand in for a sequence of increasingly powerful inference algorithms, although in practice could use different sequence of sampling techniques of growing sophist ication ication computational expense instead.
all cases. For simplicity, we conducted this study
We selected three representative locations from the CARLA dataset, covering a straight road, a roundabout, and a 4-way intersection, as depicted in Figure 4 For each of those locations, we ran the full TITRATED training procedure with varying values of mar trials E (1, 10, 20, 50, 100). As expected, both the number of rejected examples and the infraction rate decrcase monotonically with incrcasing max mux__trilss, as shown Table 3 This procedure of gradually increasing the number of rejection sampling atlempts loosely resembles titration, further motivating the naming of method.
To analyze the scaling behavior the inference algorithm improves, we plotted the infraction rate of TITRATED against the fraction of rejected examples in Figure 3 Somewhat surprisingly, this scaling tends to be close to linear, allowing us to extrapolate to a setting with perfect inference algorithm that finds acceptable samples for all training examples. In most, but all cases, the extrapolated line achieves zero infraction rate before achieving full dataset coverage, suggesting that the use of better inference algorithms could remove infractions entirely.
10

---

Published in Transactions on Machine Learning Research (08/2022)
Straightt Road
Roundabout
4-way Intersection
TITRATFI
IITRTTI ITR Autoplio Human
TITAATE
ITRA
ITAA
10
Autopfoo Human
Autoolot ot Human
10
Density
Density
2
-1.00 -0.75 -0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.50
-025
0.00
0.25
0.50
0.75
3.00
-3.00
-0.75 -0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Acceleration
Acceleration
Acceleration
Figure 5: Kernel density estimates of acceleration distributions for TITRATED, ITRA, CARLA autopilot, and real human data. Accelcration values are normalized using highest values present in human data.
4.2 Assessing Human-likeness of Driving Behavior
The primary motivation for this work was to create NPCs for CARLA exhibiting human-like driving be- havior. While this is largely subjective and no definitive metrics are agreed-upon by the community, in this experiment we sock to demonstrate that in some sense, TITRATED indced generates more human-like driving behavior than the CARLA autopilot.
When actual human trajectories for specific maps are available, prediction metrics such as average and final displacement error can be used to judge the realism of behavior models (Bergamini et al. 2021), but unfort unately those are not available in CARLA. For comparing different learning-based models, infraction rates can be used, but hand-coded models such as CARLA autopilot can easily achieve zero infraction rates while still not being very realistic.
The only alternative approach are aware of is to compare the distributions of various quantities associated with driving (Chao et al. 2020). In Figure 5 present the probability densities of vehicle acceleration. We chose three representative scenes from the CARLA dataset, the same ones as used in Section 4.1 and selected similar scenes from the INTERACTION dataset for comparison with humans in similar scenario. The scenes are depicted in Figure 4
In all three cases, CARLA autopilot has a highly peaked acceleration distribution around zero, while the distribution of human acceleration values is more spread, correspouding a higher diversity of controls applied by humans. We see that both ITRA and TITRATED are more realistic than the CARLA autopilot, in the sense of the distribution of accelerations being more spread. As might be expected, the additional conditioning in TITRATED sometimes leads to less realistic acceleration distributions, reducing the variance on 4-way Intersection and introducing excessive braking on Roundabout. On the other hand, on Straight Road the TITRATED distribution is slightly less peaked than ITRA, indicating a more realistic accoleration distribution ion after conditioning on not performing infractions.
5 Discussion
A limitation of TITRATED is that it can only be used to address infractions that we never want to see in predictions. However, humans commit those infractions in the real world at non-zero rate, and a realistic model should be able to replicate this rate of infractions. This is even more important for less catastrophic infractions, such as running a red light or failing to yicld. Our proposed methodology for amortized conditional inference is agnostic to the type and number of conditions. In principle, one can use more sophisticated conditioning to achieve such goals. A promising avenue for future work is accommodating soft constraints that we expect humans to satisfy usually, but not always, like obcying traffic rules or exccuting specific maneuvers and commands. Taking advantage of more powerful interference algorithms is essential for achieving this goal, thus we are currently investigating alternatives in this context.
11

---

Published in Transactions on Machine Learning Research (08/2022)
6 Conclusion
TITRATED provides way transfer learned models of human driving behaviors to new settings while avoiding unrealistic infractions typically associated with such domain transfer. The crucial ingredient of the process is generating samples from the model conditioned on not performing such infractions, which is chal- lenging, but better inference algorithms used for this purpose may result in large performance improvements. TITRATED is already used to provide NPCs for CARLA based on human driving trajectories recorded in the real world.
Acknowledgments
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, and the Intel Parallel Computing Centers program. Additional support was provided by UBC's Composites Research Network (CRN), and Data Science Institute (DSI). This research was enabled in part by technical support and computational resources provided by WestGrid (www.wesd westgrid.ca) Compute Canada (www.com www.ccomputecanada..aa)) and Advanced Research Computing at the University of British Columbia (arc.ubc.ca).
References
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst. Proceedings of Robotics: Science and Systems, FreiburgimBroisgann Germany, June 2019. doi: 10.15607/RSS 20.9.XV.031.
Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blažej Osiáski, Hugo Grimmet, and Peter Ondruska. SimNet: Learning Reactive Sclf-driving Simulations from Real-world Observations. In 2021 IEEE International Conference on Robolics and Aulomation (ICRA), pp. -. IEEE, 2021.
Raunak Bhattacharyya, Soycon Jung, Liam A. Kruse, Ransalu Senanayake, and Mykel J. Kochenderfor. A Hybrid Kule-Based and Data-Driven Approach to Driver Modeling Through Particle Filtering. IEEE Transactions on Intelligent Transportation Systems, pp. 1-14, 2021. doi: 10.1109/TTTS.. 201.3199...
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End end learning for self-driving cars, 2016. URL htttps://arxiii http://wiivvr.r.eeeeeeoo hwtpg//bs///1..... 07316
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the Opportunities and Risks of Foundation Models, 2021. URL https:://rrii: //arxiv/////.v.//aeaeaaear www...rgggaaa..... hbse//bs/2000 21181002222
Q. Chao, Z. Deng, Y. Xiao, D. He, Q. Miao, and X. Jin. Dietionary-based Fidelity Measure for Virtual Traffic. IEEE Transactions on Visualization and Computer Graphics, 2693):140- March 2020. ISSN 1941-0506. doi: (0.1109/VVCG.201 and Computer Graphics.
Conference Name: IEEE Transactions on Visualization
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A Recurrent Latent Variable Model for Sequential Data. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS'15, pp. 2980-2988, Cambridge, MA, USA, 2015. MIT Press.
Akansel Cosgun, Lichao Ma, Jimmy Chiu, Jiawei Huang, Mahmut Demir, Alexandre Miranda Anon, Thang Lian, Hasan Tafish, and Samir Al-Stouhi. Towards full automated drive in urban environments: demon- stration in gomentum station, california. 2017 IEEE Intelligent Vehicles Symposium (IV), Jun 2017. doi: 10. 1..1109/vv.2217..77
12

---

Published in Transactions on Machine Learning Research (08/2022)
Nemanja Djurie, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Cang-Chich Chou, Tsung- Han Lin, Nitin Singh, and Jeff Schneider. Uncertaint--wware Short-term Motion Prediction of Traffic Actors for Autonomous Driving. 2020 JEEE Winter Conference on Applications of Computer Vision (WACV), Mar 2020. doi: 10.1109/wacv455360 0tw.4455722220.909335
Alexcy Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1sl Annual Conference on Robol Learning, pp. 1 1-16, 2017.
D. Eberly. Distance Between Point and Triangle in 3D. In Geometric Tools, 1999.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks. 2018 IEEE/CVE Conference on Compuler Vision and Pattern Recognition,, Jun 2018. doi: 10.1109 2011109/002222018.00222 2018.00440.
Tuomas Haarnoja, Amurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop kf: Learning discriminative deterministic state estimators. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 4383 4391, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean hwakee, Nicolas Griffiths, Amar Shah, and Alex Kndall. Urban driving with conditional imitation learning. 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020. doi: 10. St0.r444945..220.19999
URL http: dwi. /ttp///1.1.10///440095555 2000:440004522000.....2 2020. 9197408
Jocy Hong, Benjamin Sapp, and James Philbin. Rules of the Road: Predicting Driving Behavior with Convolutional Model of Semantic Interactions. 2019 IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi:
09.065
Arne Kesting, Martin Treiber, and Dirk Helbing, Agents for traffic simulation. Computational Analysis, Syn- thesis, & Design Dynamic Systems, pp. 325-356, Jun 2009. ISSN 2154-9826. doi: 10.1201/97814200770222 chll.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http:///::/
6980
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chan- draker. DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents. 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1109//rp::2017.23.
S. Lefévre, D. Vasquez, and C. Laugier. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH Journal, 1:1-14, 2014.
Xiaoyu Mo, Yang Xing, and Chen Lv. ReCoG: A Deep Learning Framework with Heterogeneous Graph for Interaction-Aware Trajectory Prediction. arXiv:2012.0555 05032 (es), December 2020. URL. http://arxiv. org/abs/2012.0500 arXív: 2012.05032.
Seong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis- Philipp Morency. Diverse and admissible trajectory forccasting through mul- timodal context understanding. Lecture Noles in Computer Science, pp. 282 298, 2020. ISSN 1611-33 1611-3349. doi: 10.1007/978-3 00778--030-- 080005822--82211 17. URL http:t//t//...oo doi 1077 0010//9978--000088882-----1
1007
Philip Polack, Florent Altché, Brigitte d' d'Andréa Novel, and Arnaud de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEF Intelligent Vehicles Symposium (IV), pp. 812 818, 2017. doi: 10.1109/IVS.2017799555
Davis Rempe, Jonah Philion, Lconidas J. Guibas, Sanja Fidler, and Or Litany. Generating Useful Accident- Prone Driving Scenarios via a Learned Traffic Prior. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Nicholas Rhinchart, Kris M. Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Nicholas Rhinehart, Rowan Mcallister, Kris Kitani, and Sergey Levine. PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Scttings. 2019 IEEE/CVF International Conference Computer Vision (ICCV), Oct 2019. doi: (0.1109/iccv..o 2019.00.0000091.....
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian; Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints. 2019 TEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/ 2019..011.00144..
Adam Šcibior, Vasilcios Lioutas, Danicle Roda, Peyman Batoni, and Frank Wood. Imagining The Road Ahead: Multi-Agent Trajectory Prediction via Disferentiable Simulation. In 2021 IEEE 24rd International Conference Intelligent Transportation Systems (ITSC), 2021.
Edward Smith, Scott Fujimoto, Adriana Romero, and David Meger. GEOMetrics: Exploiting geometric
structure for graph-encoded objects. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. TrafficSim: Learning To Simulate Realistic Multi-Agent Behaviors. In IEEE Conference on Computer Vision and Pallern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 10400 10409. Computer Vision Foundation / IEEE, 2021.
Charlie Tang and Russ R Salakhutdinov. Multiple Futures Prediction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (ods.), Advances in Neural Information Pro- cessing Systems, volume 32. Curran Associates, Inc., 2019.
Wei Zhan, Liting Sun, Di Wang, Haojic Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Kömigshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moON Dataset in Interactive Driving Sce- narios with Semantic Maps. arXie: 191:00910000088 fcs, cessj, 2019.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov. TNT: Target-driveN Trajectory Prediction. ar::i::008.00 08294 [es], pp. 12, August 2020. URL http://arxivy www.aaawwvv.r.oraaaeeeee 2000 08294 arXiv: 2008.08294.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-Agent Tensor Fusion for Contextual Trajectory Prodiction. 2019 IEEE/CVF Con- ference on Compuler Vision and Pallern Recognilion (CVPR), Jun 2019. doi:
Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. Iou loss for 2d/3d object detection. 2019 International Conference on 3D Vision (3DV), Sep 2019. doi: 10.1109/3dv. 2019.00019. URL //dxx doi. org/10
/0/0 .1.11..1.... 1 1109/1000 /0///33V2222 2019. 200190001190000
Deyao Zhu, Mohamed Zahran, Li Erran Li, and Mohamed Elhoseiny. Motion forecasting with un- likelihood training in continuous space. In 5th Annual Conference Robot Learning, 2021. URL
https://openrevieee net:/forumm.dd=4225555
J. Ziegler, P. Bender, M. Schreiber, H. Lategahn, T. Strauss, C. Stiller, T. Dang, U. Franke, N. Appenrodt, C. G. Keller, E. Kaus, R. G. Herrtwich, C. Rabe, D. Pfeiffer, F. Lindner, F. Stein, F. Erbs, M. Enzweiler, C. Knöppel, J. Hipp, M. Haucis, M. Trepte, C. Brenk, A. Tamke, M. Ghanaat, M. Braun, A. Joos, H. Fritz, H. Mock, M. Hein, and E. Zeeb. Making Bertha Drive An Autonomous Journey
Historic Route. IEEE Intelligent Transportation Systems Magazine, 6(2):8-20, 2014. ISSN 1941-1197. doi: 10.1109/MITS 2014.2306552, Conference Name: IEEE Intelligent Transportation Systems Magazine.
14


Reviews:
[{"review_id": "tR3CboxaLw", "reviewer": "Reviewer_34E5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n* The paper is well-written and very easy to follow. Figure 1 helps understand the pipeline of the proposed method.\n* The topic is interesting and the motivation is clear. Generating naturalistic behavior in simulations helps evaluate and train autonomous driving algorithms more efficiently.\n\n### Weaknesses:\n* The proposed sampling procedure is trivial and not efficient. More efforts need to be spent on the algorithm design to increase the contribution. Explained in the Requested Changes.\n* Experiments are not through and some baselines are missing. Explained in the Requested Changes.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In general, this paper aims at solving the problem of driving behavior modeling.\nThey present a new method that fine-tunes a pre-trained foundation model to novel road maps where human demonstrations are not available. Their target is to generate naturalistic driving behaviors without infractions such as collisions and off-road invasion.\n\nSpecifically, this paper has the following contributions:\n* Propose a fine-tune pipeline to solve the OOD problem with little effort. Their method can quickly generate naturalistic driving behaviors in an unseen road map. \n* Design two metrics to represent infractions of collision and off-road invasion. These two metrics are accurate and differentiable, which helps efficiently train their models.\n* Conducted experiments to show that the finetuned method achieves less collision rate and off-road invasion rate. They also explore the influence of sampling methods on performance.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Major concerns (critical):\n\n* **Source of OOD.** I think the mismatch discussed in this paper is mainly caused by the map image. The bird-view image of CARLA is very different from INTERSECTION. However, this problem might be solved by training with broader datasets or using other representations than bird-view images. What's the opinion of the authors on this question?\n* **Sampling is not efficient.** The reject sampling procedure seems trivial and too expensive as the authors said. Since this is the core part of the proposed algorithm, I doubt the proposed method is efficient. There could exist alternative methods to achieve this goal more practically and efficiently, for example, advanced sampling methods like MCMC and Multilevel splitting.\n* **Unfair quality evaluation of behavior.** In section 3.2, the authors discussed how to evaluate the quality of human-like driving behavior. It is indeed difficult to evaluate whether a policy can generate naturalistic behavior. In the experiment part, the authors chose to use the distribution of acceleration as a surrogate metric, which I think may not be fair enough. The autopilot method used in CARLA is a very simple model, which usually takes a constant acceleration as input in most cases. Human drivers have very diverse behaviors, therefore have a wider range of acceleration. In addition, it is acceptable that ITRA has different distributions from human drivers due to the learning process, but the distribution of TITRATED is far away from the human driver and ITRA, indicating that TITRATED cannot generate naturalistic behavior (under the acceleration metric). I suggest the authors use a more complicated metric to evaluate the quality, for example, the likelihood of another generative model trained on very broad maps.\n* **Results are not convincing.** In Table 2, the two metrics are represented with $\\times 10^{-4}$, then the absolute difference seems very small. The ITRA method without finetuning seems already has a low collision rate and off-road rate. \n* **Missing baselines.** The authors only compare ITRA and TITRATED in their experiments. Therefore, the conclusion heavily depends on the performance of the ITRA model. There are other methods that can generate the behavior of NPC in traffic scenarios, which should also be explored to investigate the importance of the foundation model. In addition, the authors may need to consider other baselines for dealing with the OOD mismatch, for example, using reinforcement learning methods to train agents on CARLA scenarios or using domain adaptation methods that are widely investigated in deep generative models.\n\n### Clarifications and minor things:\n\n* On page 2, “Alternatively, we can view this process as an instance of amortized inference, where the observations are known in advance, but the model is further conditioned, here on constraints.” Some explanations of amortized inference would help understand this sentence.\n* On page 3, “… for each agent i and time t is obtained via differentiable rendering…” It is unclear why a differentiable renderer is necessary here. How does this renderer work? My understanding is that the bird-view image can backpropagate gradient only to x, y, and $\\psi$.\n* Provide more details about the ELBO?\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hUfVKmRl4e", "reviewer": "Reviewer_hW55", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Very well written paper. Easy to follow, concise, and polished.\n* Well motivated by a practical issue in generating realistic simulated agents.\n* The experiments were thoughtfully constructed and fairly extensive, supporting the claims made in the paper.\n* The proposed solution is practical and intuitive.\n* Fig. 2 is a highly effective illustration of the benefits of the proposed approach.\n* The authors include a fair discussion of the limitations of the proposed TITRATED methodology.\n\nWeaknesses:\n* According to my understanding, the ITRA baseline was not fine-tuned on the CARLA data. If this understanding is correct, then it is unsurprising that the baseline does not do well and causes infractions upon transfer to a new setting, since as the authors mention, neural networks generally do not perform well on out-of-distribution data. To better showcase the benefits of the proposed methodology in producing fewer infractions in new settings, an ITRA baseline that is fine-tuned on held out CARLA data should be included in the comparison. If my understanding is not correct, and the ITRA baseline is fine-tuned on CARLA data, then this should be made more clear in the paper.\n* There is no ablation study that illustrates the effect of the additional loss terms beyond simply fine-tuning on the filtered dataset. Similarly, there is no indication whether fine-tuning with the additional loss terms, without filtering the dataset, would also result in low infraction performance.\n* Some design choices lack details or context: \n    - On page 2, 'We generate infraction-free trajectories in target locations'. What about along the trajectory? According to my understanding from Eqs. (6) and (7), infractions are considered along the entire trajectory.\n    -  On page 3, it was not entirely clear whether the road structure is included in the input to the network. \n    - In the paragraph following Eq. (6), it is unclear what is the approximation made by Zhou et al. (2019) and consequently how the proposed use of this technique is novel in the behavior modeling context.\n* Missing dedicated literature review. In particular, I would ideally like to see a discussion on transfer learning and learning with constraints.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a practical solution to the problem of transferring existing human behavior models to new environments without incurring unrealistic behaviors. The paper is motivated by the task of generating human-like behaviors in simulated agents for autonomous driving, specifically for CARLA. The authors remark that direct transfer of pre-trained models often results in unrealistic driving infractions, such as off-road driving and collisions. The proposed methodology, TITRATED, addresses this problem in two steps. First the data in the target environment is processed to remove any infractions predicted by a pre-trained behavior model using rejection sampling. Then, the pre-trained behavior model is fine-tuned on this filtered data with additional loss terms to discourage infractions. In this manner, the learned distribution over behaviors is pseudo-conditioned to not produce infractions in the new environment.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "My requested changes are based on the weaknesses described above. For more details, please see my response in the previous question.\n\n1. Baseline against a version of ITRA fine-tuned on held out CARLA data without filtering out infractions to demonstrate the benefits of the proposed two-step TITRATED approach. (Critical)\n\n2. Ablation studies: (a) only fine-tune with loss terms on CARLA data without filtering out infractions and (b) fine-tune on filtered data without the additional loss terms. (Would strengthen the paper)\n\n3. Clarify the details and points of confusion listed in weaknesses above. (Highly encouraged)\n\n4. Add a literature review section that clearly contextualizes the presented work in terms of existing literature in, for example, transfer learning and learning with constraints. Since the current manuscript is two pages under the page limit, there is sufficient space for this discussion. (Highly encouraged)\n\n5. The references should be proofread (e.g., to ensure the year is not entered twice in a citation, conference proceedings are formatted consistently, the conference is listed instead of ArXiv whenever possible). (Nice to have)", "broader_impact_concerns": "The authors provide a thoughtful consideration of the limitations of the TITRATED approach in the Discussion section, specifically, due to TITRATED inherently reducing or even eliminating infractions, dangerous situations may be missing from simulated scenarios. They also discuss the potential bias introduced by removing infractions from the data used for fine-tuning. In my view, this discussion is sufficient to address the intention of a separate Broader Impact statement, although it could be a nice addition if there is space in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JSRWupk3Zp", "reviewer": "Reviewer_TsWW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths summary\n+ The performance evaluation w.r.t. to the prior method is sufficient for us to confidently conclude a significant and substantial performance improvement.\n+ The broader goal of building 'NPC' agents for CARLA / other simulators is worthwhile and of interest to the community.\n\n# Weaknesses summary\nThere are several important weaknesses.\n- This paper is missing discussion of and comparison to related work (there's no separate related work section, which isn't ideal). There are closely related existing methods core to the proposed method: there exists methods for incorporating sample infraction information into the training objective for vehicle motion prediction, including [A, B, C, D]. At the very least, verbal comparison to these related works is necessary, in order for the paper to be clear about its contributions.  \n- This paper is missing a baseline in which there is no separation between the models p(s|A) and p(s), but, unlike ITRA, also uses infraction metrics, but unlike the proposed method, just uses them to train p(s). Perhaps this isn't possible given the nature of the pretraining dataset, but I would like the authors to consider and comment.\n- This paper is missing a clear learning algorithm that uses the data generated by Alg 1 in the objective function (Eq 10). Therefore, the proposed method itself is unclear.\n\n# Updated weaknesses summary\nMy weakness concerns have all been resolved following the revisions and the discussion in the author response thread attached to this review.\n\n## Related work\n- [A] \"ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst\" https://arxiv.org/abs/1812.03079v1 arXiv 2018\n- [B] \"R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting\" ECCV 2018 https://link.springer.com/chapter/10.1007/978-3-030-01261-8_47\n- [C] \"Diverse and admissible trajectory forecasting through multimodal context understanding\" ECCV 2020 https://link.springer.com/chapter/10.1007/978-3-030-58621-8_17 \n- [D] \"Motion Forecasting with Unlikelihood Training in Continuous Space\" CoRL 2021 https://openreview.net/forum?id=4u25M570Iji ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for vehicle motion forecasting that incorporates feedback from infraction metrics to build a finetuned model. The method is motivated from the perspective of learning an amortized inference model. An experimental evaluation in CARLA shows several things:\n\n- 1) relative to the non-finetuned model, the infraction metrics are much lower (on average, 76% reduction in collision rate, and 87% reduction in off-road rate).\n- 2) increasing the amount of samples considered during rejection sampling results in better performance, which the paper then claims could be extrapolated to hold for other methods that lead to better approximate inference.\n- 3) The acceleration profiles of various methods in a few difference scenes show that the proposed method and the method upon which it is based do better than the CARLA autopilot in matching the distribution of human accelerations.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please address the three significant weaknesses, as I see all as critical to the acceptability of the paper.\n\n# Updated requested changes.\nMy requested changes have been implemented, so none remain.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method that fine tunes and adapts a foundational model to the situations where there is missing data. While done in the autonomous driving setting, the paper makes an important contribution to using the available and incomplete data in the space of the safety critical applications.

The authors have made significant improvements to the original manuscript, addressing all the reviewers comments. The ablation studies and positioning working within the related works are especially helpful. As a result, the reviewers agree and I concur that this work would be welcome contribution to the research community and might inspire future work in this important problem space. 
Decision: Accept (poster)