EscIRL: Evolving Self-Contrastive IRL for Trajectory Prediction in Autonomous Driving
Authors: Siyue Wang
Zhaorun Chen
Zhuokai Zhao
Chaoli Mao
Yiyang Zhou
Jiayu He
Albert Sibo Hu
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Trajectory Prediction
Autonomous Driving

Abstract: While deep neural networks (DNN) and inverse reinforcement learning (IRL) have both been commonly used in autonomous driving to predict trajectories through learning from expert demonstrations, DNN-based methods suffer from data-scarcity, while IRL-based approaches often struggle with generalizability, making both hard to apply to new driving scenarios. To address these issues, we introduce EscIRL, a novel decoupled bi-level training framework that iteratively learns robust reward models from only a few mixed-scenario demonstrations. At the inner level, EscIRL introduces a self-contrastive IRL module that learns a spectrum of specialized reward functions by contrasting demonstrations across different scenarios. At the outer level, ESCIRL employs an evolving loop that iteratively refines the contrastive sets, ensuring global convergence. Experiments on two multi-scenario datasets, CitySim and INTERACTION, demonstrate the effectiveness of EscIRL, outperforming state-of-the-art DNN and IRL-based methods by 41.3% on average. Notably, we show that EscIRL achieves superior generalizability compared to DNN-based approaches while requiring only a small fraction of the data, effectively addressing data-scarcity constraints. All code and data are available at https://github.com/SiyueWang-CiDi/EscIRL.

ESCIRL: Evolving Self-Contrastive IRL for Trajectory Prediction in Autonomous Driving
Siyue Wang" Zhaorun Chen 22 Zhuokai Zhao', Chaoli Mao', Yiyang Zhou
Jiayu Hel Albert Sibo Hu-
CIDI Lab, UUniversity of Chicago, 'UNC-Chapel Hill
Abstract: While deep neural networks (DNN) and inverse reinforcement learn- ing (IRL) have both been commonly used in autonomous driving to predict trajec- tories through learning from expert demonstrations, DNN-based methods suffer from data-scarcity, while IRI--basdd approaches often struggle with generalizabil- ity, making both hard to apply to new driving scenarios. To address these issues, we introduce ESCIRL, a novel decoupled bi-level training framework that iter- atively learns robust reward models from only a few -ixed-scenario demonstra- tions. At the inner level, ESCIRL introduces a self-contrastive IRL. module that learns spectrum of specialized reward functions by contrasting demonstrations across different scenarios. At the outer level, EsCIRL employs evolving loop that iteratively refines the contrastive sets, ensuring global convergence. Experi- ments on two multi-scenario datasets, CitySim and INTERACTION, demonstrate the effectiveness of EsCIRL, outperforming state-of-the-arr DNN and IRL-based methods by 41.3% on average. Notably, we show that EscIRL achieves superior generalizability compared DNN-based approaches while requiring only small fraction of the data, effectively addressing scareity constraints. All code and data are available htttts://githii com/ wuun://yyyeaan---aa wwt.yeeewwaaaaaaaaaaaeeeaaia EEDi/EscIRL.
Keywords: Reinforcement Learning, Trajectory Prediction, Autonomous Driving
Introduction
Trajectory prediction has been one of the crucial components of autonomous system [1], allowing the self-driving vehicle to forecast the movements of itself and nearby entities. Given the rise of modern machine learning (ML) and more specifically deep learning (DL) [2, 3, 4]. numerous deep neural networks (DNN)-based methods [5] have been developed for trajectory prediction tasks [6. 71, including more recent works based on Long Short-Term Memory (LSTM) [8. 9, 10] and transform- [11, 1, 12, 13), which take advantage of the more complex and capable DNNs to better capture and understand the intricate temporal and spatial dependencies inherent in various driving scenar- ios. However, while these approaches can learn a more complex policy representation and are more robust to diverse driving scenarios. their larger-scaled network architectures often require substan- tial amounts of data to train [14], which poses concerns in underfitting and limits them from being adopted under data-scarce environments [15] or low-resourced computation conditions [16, 17].
Meanwhile, inverse reinforcement learning (IRL)-based approaches [18, 19, 20] have also found great success in predicting trajectories through inferring cost or reward functions from expert demon- strations, and then using these functions to guide the behavior of self-driving vehicles in unseen driv- ing environments. More specifically, given the infinite numbers of potential cost functions that could explain the demonstrations [21], existing IRL approaches typically employ structural priors for these functions, such as a linear combinations of features [22], or constant derivative regularization [23], reduce the large number of possible cost functions [24].
However, under these assumptions, the learned IRI. controllers often suffer from the curse of gen- cralizability and tend to only explain the demonstrations locally, making the learned controllers
'Equal Contribution. 1 Corresponding Authors: Siyue Wang, Jiayu He, Albert Sibo Hu (wang.sy, he.jy. sibo ) @cidi.ai. Work was done during Zhaorun Chen's 's visit at CIDI Lab.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

difficult to adapt to new environments [25]. In other words, the inferred cost or reward functions can only be applied to unseen driving conditions whose types of traffic situations (e.g. left turn at the red light) must have been included in the demonstrations, hurting the system's performance and ro- bustness across broader range of situations. To handle the real-life complex environments, existing IRL frameworks often pre-classify the mixed-scenario datasets and learn numerous cost functions where each corresponds to one driving scenario. Such rule- strategies not only increase the un- necessary difficulty in training. but also degrade generalizability. making the system prone unseen environments and causing safety concerns. As far as we are concerned, no existing sample-based IRL method can learn global cost functions for general-purposed trajectory prediction tasks [24].
The limitations of both DNN- and IRL-based approaches highlight the necessity to develop a novel data-efficient yet robust and generalizable algorithm for trajectory prediction in autonomous driv- ing. To this end, in this paper we propose EscIRL: an Evolving Self-Contrase IRL approach that utilizes decoupled bi-level training framework to efficiently learn generalizable cost function from diverse trajectory distributions without the need of pre-labeled classifications or additional classi- fiers. More specifically, after initializing with a set of pseudo-controller parameters, on the inner level, EsCIRL employs a self-contrastive IRL (SCIRL) algorithm to increase the likelihood of ex- pert demonstrations within similar distributions, i.e., positive set, while reducing expert likelihood in adverse/dissimilar distributions, i.e. negative set. And on the outer level, EscIRL utilizes an evolv- ing framework iteratively optimize the contrastive sets from the inner level to introduce diversity while ensuring overall convergence. Finally, a continuous controller network (CCN) is trained to capture the mapping from specific trajectory context to the earlier initialized pseudo-parameter
To summarize, four key contributions this paper include: 1) introducing ESCIRL, a novel ap- proach that addresses a common challenge in trajectory prediction tasks faced by both DNN- and IRL--baeed methods, i.e., to ensure generalizability while being data-efficient; 2) a self-contrastive IRL (sCIRL) algorithm that learns spectrum of robust reward functions from contrasted expert demonstrations without additional labels classifications, significantly improving generalizabil- ity: 3) an evolving framework that iteratively refines the positive and negative sets from SCIRL to enhance convergence while encouraging exploration - we argue that such training framework com- bining SCIRL and evolving loop may bring benefits to other IRL applications as well; and 4) exten- sive experiments on two multi-scenario datasets including CitySim [26] and INTERACTION [27], showing that EscIRL outperforms existing SOTA methods by an average of 41.3%.
2 Methodology
In this section, illustrate the details of proposed Evolving Self- Celf-Contrastive IRL (EscIRL). Specifically, EsCIRL aims efficiently learn generalizable cost function from diverse trajectory distributions in a self-supervised manner, where it first employs self-contrastive IRL. (SCIRL) algo- rithm to disentangle similar and dissimilar distributions without requiring any priors. Then ESCIRL adopts an outer-loop evolutionary algorithm to gradually learn the priors via natural selection [28] to interpret the multi-scenario distribution for robust trajectory prediction. An overview of the pro- posed pipeline is shown in Fig. 1. We first summarize the problem statement in $2.1, and then proceed explain the two major components of ESCIRL, which are the inner-level self-contrastive IRL (scIRL) module and the outer-level evolving loop in $2.2 and $2.3 respectively.
2.1 Problem Statement
Let and u represent the states and actions of the vehicle respectively, we formulate the dynamics f(-) of each vehicle as:
k+1 = S(a", wt)
Next, trajectory& & within the spatial-temporal domain is defined as a sequence of states and actions, N-1,, where the superseripts indicate the discretized planning
i.e., { = {a =0,at o², ot,u,,, w........N. N1 step and N denotes the total length of the planning horizon.
Given a set of driving demonstrations Dp {&r} comprising of various individual trajectories as i .,2,....
...,.. N, and a set of parameters of reward function R, the probability of the expert

---

Context Map Encoder
Parametric Space
Dynamic Stacked Context
ECTRE
Ereg
sample
Continuous Controller
Contrastive IRL
Evolution Loop
Select Top 10%
Raw Trajectories
Motion Extractor
Contrastive
Validation Set
Set
Cross Over
Mutation
Figure 1: ESCIRL overview. Given trajectory dataset, we first converts it into Frenet coor- dinates using low-dimensional motion extractor. Next, we initialize its population by randomly sampling a pair of positive and negative set for each trajectory entry in the dataset. Then, ap- ply ScIRL to each contrastive set, and obtain a set of IRL parameters w.r.t. each trajectory in the dataset. With the of IRL parameters, EsCIRL fits unified continuous control network (CCN) to adaptively predict control parameters given the dynamic stacked context (e.g. vehicle states, map geometry) and select the top 10% of the population for crossover and mutation produce the subse- quent population. This evolving process is repeated for a maximum of K times or until convergence.
demonstrations Dp is commonly [29] defined as:
P(Dp w)
(Pr 6 R F F(Dx) Zn
II
(R. (E:)
Zu
Zu
where Zy denotes the set of feasible trajectories, and ,3 specifies the demonstration proximity [30], which is set to throughout this paper unless otherwise noted. The goal of an (maximum entropy) IRL algorithm [29] is infer the underlying reward function that maximizes Eq. (2), which the likelihood of all expert demonstrations, with the assumption that trajectories
exponentially more likely with higher cumulative rewards [31]. More specifically, assume that the trajectory space can be approximated with discretized samples Zu, = EM e (E!").")) where & E 'Di is candidate trajectory in the sampled trajectory set Di for demonstration &;. Thus the goal of IRL is to find set of parameters w" which maximizes the expert demonstration among the set of sampled trajectories. Mathematically, we have
N
w- arg max
N log PwD,lw) arg max
Σ
log P(C,(w)
In the context of trajectory planning and prediction, wt will then be used select the best trajectory from the sampled trajectory set Dm-
Self- Contrannratttve IRL
Following [19, 32], we employ linear-structured reward function with respect to F. which is chosen feature space. Specifically, we follow [32] and define three categories of features separately for efficiency, safety and comfort. Besides, we propose two novel interactive features to capture the driver's desiderata in complex intersections. Finally, we define twenty-two features in total, more details on the feature design are in Appendix C.
While existing works typically partition datasets first into specific scenarios that clearly dis- tinctive from each other such dividing the dataset into subsets of in-lane following [22], lane-
3

---

changing [32], or lane-merging [33], so that the inferred reward function (w" in Eq. (3)) is stable across the demonstrations within the in-domain scenario, we instead consider the complete multi- distribution dataset without any partition to learn a unified controller network that outputs such w" to handle all the diverse driving scenarios. Learning such unified controller network brings more challenges than simply solving for Eq. (3), which produces only a single vector of the control param- eters. Notice that this is fine for existing works as they have implicit assumption that the expert demonstration obey unified distribution [22] due to partition. However, such assumption is no longer valid in our case we consider multi-distributions in the expert demonstrations. Blindly fol- lowing such assumption would lead to decreased performance and lower robustness [24]. Therefore, to handle the multi- multi-mod in terms driving conditions and preserve the distribution diversity in the demonstration dataset, we propose self-contrastive IRL. (SCIRL) algorithm. Inspired by [34), we augment Eq. (2) with an auxiliary contrastive loss term, which can be represented as:
EsCIRL (w) LIRL (w) + Contrastive
(4)
where y is hyper-parameter and is set to throughout this paper unless otherwise noted. Different from [34], ScIRL considers both positive (similar policy) and negative (dissimilar policy) set, with CIRL only being conducted the positive set, i.e.
K Σ
M
1 (w) = K+
w F(G) - log Σ of F(E"") m=0
(5)
and we design Commtssiive as a hinge loss that seeks to ensure that the expert demonstration has higher probability in the positive set than the negative set by margin Mathematically, have:
Coomtsttve (w) max (P- p+ +c,0)
where p+ and P- the probability of each expert demonstration in the positive and negative set respectively, More specifically, for an arbitrary demonstration Dp. we have
FFET)
d F(ET)
Pj KJ
P
Emt=o e Fler
KJ
m =0 e F(ET"
where m², mmmmm,a ,7, and &T represent the demonstration m and sampled trajectory & in the positive (+) and negative (-) subset for trajectory E,. K2* and Kj denote the number of demonstrations in the positive and negative set. Based on a common assumption that each demonstration & E Dp only represents one certain scenario, we define a separate pair of contrastive set for each &.. In other words, instead of applying LIRL on the overall dataset as in Eq. (2), we define a specific pair of positive/negative (contrastive) sets for each demonstration & in the dataset Dp More details illustrated in Algorithm 1. As result, ScIRL effectively preserve the multi-modality (multi- driving conditions) across diverse driving scenarios and optimize expert behaviors on more fine- grained subspace. However, considering the general case where no priors are available to pre- determine these sets, we introduce an evolutionary algorithm to iteratively refine the contrastive sets self-supervised manner.
2.3 Evolving Framework
Since SCIRL does not rely on any priors to pre-define the contrastive sets for each trajectory E₃- we propose an evolving framework, which is considered as the luter-level of EscIRL integrating SCIRL into iterative closed-loop optimization. In this way, we effectively accumulate the pri- determine contrastive sets for each trajectory via natural selection [28], so that convergence and global optimality can be guaranteed. As shown in Fig. 1, for each raw trajectory, we first trans- form it to the Frenet state-space [35] using a rule-based motion extractor. Next, for each instance in the initial population, we set g+ = & and & = Ø for each trajectory in the dataset and then apply SCIRL to obtain a set corresponding reward parameters w; as initialization.
Continuous control network. To capture the underlying relationship between specific driving context including vehicle states (both spatial and kinematic), map geometry, and its expert policy

---

Algorithm Updating CCN with ESCIRL
Require: dataset D {& with N trajectories & feature function F; safety-constrained sample policy Ty; number of points in contrastive set kt = learning rate a, convergence threshold
Ensure: global cost function G; sample policy π.* Randomly initialize wi, क, for all i f1..... ......N. .N]
2: Project trajectory & into feature space: F; = F((:)
3: for each trajectory Te, à fI.....NY NN do
4: Sample M trajectories under policy
5: Find K closest and K furthest points by sorting distance matrix (d(w),,;)))
6: Compute contrastive IRL loss EsCIRL for wi:
7: Update wi using gradient descent: Wi Wi a (wIRL((
8: Update π; using policy gradient (if m, learnable)
9: end for
10: Fit CCN to predict {wi);=1 from the initial s of trajectory {&t (
11: Predict reward parameter wi pred with CCN
12: Update CCN using gradient descent with loss function (uf wc)
distribution, we propose to learn unified continuous control network (CCN) to adaptively predict the corresponding parameters of the reward function (as in Eq. (2)) for the ego vehicle. The architec- ture of the proposed CCN is shown in Fig. 2a, with three streams of network inputs: the static map geometry, mapping of vehicle spatial states (e.g. coordinate, heading), and mapping of vehicle kine- matic states (e.g. velocity, accelerations, jerk). The output is the corresponding control parameters wi of the reward function.
Since both the map geometry and vehicle spatial states are static, we stack them to be dual-channel mapping imput. Then we design Siamese structure [36] in CCN to separately encode the vehicle kinematic information (i.e. kinematic input), and the static map with vehicle spatial information (i.e. spatial input). Notably, we define the intermediate mapping output of the convolution for spatial in- put to be an attention map [37], indicating which specific parts of the context are more responsible for the current predictions. On the other hand, the convolution for kinematic input encompasses a richer representation of the vehicles' dynamic states. After obtaining both convolution outputs, fuse the two mappings using a Hadamard Product and then concatenate it again with the ego in-
vehicle's states to further augment the low-dimensional representation of formation. Finally, the concatenated vector is mapped to the SCIRL parameters space through a fully-connected layer. During training, the loss of CCN is defined as:
LeOn (w (w= =
1 Σ log (ogghchh()a (aash(( wrred wil))
(8)
where wree and wi refers the predicted reward parameter and ground truth respectively. And cosh(x) = e"+e- In general, we argue that this attention- design allows for an adaptive rep- 2
resentation of the driving context and establishes an informative mapping from the driving context to the controller space (i.c. parameter space of reward functions). Furthermore, we provide an analysis on interpretability for the proposed CCN structure in $4.4. The procedure for updating CCN within EsCIRL detailed in Algorithm
Evolving contrastive sets. As shown in Fig. 1, after SCIRL converges in each iteration, we se- lect the top 10% of the population based on averaged displacement error (ADE) from the in-domain validation set, and further crossover and mutate them to produce the subsequent populations. Specif- ically for crossover, we sample Ig mutual demonstrations from the intersection set of the top 10% contrastive sets and sample + JE) demonstrations from the difference set of the union and in- tersection sets. Then we mutate the demonstrations in each contrastive sets by uniform probability of 5% to obtain each population in the next generations. This process is repeated until convergence (the decrease of ADE falls below threshold) for maximum K times, as shown in Fig, 2b.
5

---

hedemadd
Fax pae
Averaged Collsion
12
grre
1.0
0.-
convbb uone0 R.
0.0
Concaronnte
-05 Stage-1 Stage=2
Stage-3
Stage-4
Stapesd
Stage-d
Stage?
Evolution Stage
(a)
(b)
Figure 2: (a): we first encode the vehicle state σ; into multi-channel context map M E ReXW with с channels of size (W, W) to separately involve the spatial and kinematic information of the vehicles and surrounding map geometry. (b): the white dot represents candidate of the contrastive pairs, where each corresponds to a policy; the y-axis shows its corresponding averaged number of collision after convergence. The red dashed line indicates zero averaged collision.
3 Experiment
Datasets. To evaluate the capability of ESCIRL on complex multi-scenario driving modeling tasks, we use INTERACTION [27] and City [26] datasets. INTERACTION a comprehensive dataset of interactive vehicle trajectories collected from various traffic scenarios (e.g. roundabout, inter- sections) around the world. And CitySim is drone-based vehicle trajectory collection featuring vehicle trajectories extracted from 1,140 minutes of drone videos recorded across 12 different loca- tions, capturing variety of road geometries to facilitate comprehensive analysis and applications. Specifically, we extract 20,000 trajectories in total for both dataset. All the trajectories from both datasets are converted to Frenet state-space [35] using rule-based motion extractor.
Baselines. We compare the performance of EscIRL against five principled trajectory prediction approaches, including three IRL-based methods that demonstrate better data-efficiency, and two DNN-based methods which excel especially in multi-scenario complex driving conditions. More specifically, the three IRL baselines include Opt-IRL [38], which learns a unified reward function via maximum entropy objective; CIOC [39], which employs Laplace approximation to estimate rewards in a continuous feature space; and GCL [24], a model-free approach that adopts DNN to learn the reward and policy in an end-to-end manner without predefined heuristics. Additionally, we compare two DNN-based models that have demonstrated superior performance trajectory prediction tasks: RNN-based models, where (e.g.140, 41, 421), and transformer-basee (TF-based) models (e.g.[12, 43, 44]). For both model structures, we adopt the default hyperparameter configurations recommended in respective works.
CNN-LSTM structure in our implementation
Metries. Existing metrics including averaged feature distance (AFD), average displacement error (ADE), final displacement error (FDE), human probability (HP), and averaged number of collisions (AC) are used our experiments. Specifically. HP is evaluated by the likelihood that the predicted trajectory aligns with the expert trajectory. is important to note that not all metrics applicable to every method. Specifically, AFD is unavailable for GCL, RNN-based, and transformer-based methods, as these approaches do not rely on feature engineering. Similarly, HP cannot be applied to DNN-based methods, as they lack a straightforward evaluation for this metric.
Results. To verify EscIRL's performance under a scarcely labeled dataset which incorporates multiple diverse distributions, we randomly sample 2000 trajectories for both dataset and assess EscIRL and the five baselines w.r.t. the five aforementioned metrics. The results are shown in Ta- ble 1. We can denote that ESCIRL outperforms all the other approaches by significant margin on both datasets. We attribute the reasons for the strong performance of EscIRL to the fact that it can efficiently decompose the mixed-scenario expert distributions under limited priors and adaptively

---

Table 1: Results comparing EsCIRL against three IRL-based and two DNN-based approaches in- cluding Opt-IRL, CIÒC, GCL, RNN-based, and transformer based methods. The result denoted as scIRL refers to the first iteration of EscIRL without the subsequent evolving process.
Interaction
CitySim
Method
AFD
ADE
FDEL
HPL
ACL
AFD
ADEL
FDE
HPL
ACU
Opt-IRL
1.39
0.76
2.04
8.70
3.33 2.60
1.86
4,28
1.0e22200
+.33
CIOC
1.68
1.07 +005
3.10
8.9e-3+0.0
2.75 3.32-2636
3.57 20.70
7.98=355
0.01
1.75+m.92
GCL
3.08 2001
4.08
8.5c-520.0
5_2
2.89
40.02
5.45
1.30-42000
1.75
RNN-based
2.12
5.66#142
4.20
+..1
5,98=0.50
3,33
TF-hased
3.59 10.44
7.12
7.33
4.12 8.30
1.88
Ours
SCIRL
1.46
1000
1.365
1.02
1.24
1.62 1002
1.0910.00
2.60
0.02 10.00
0.45
ESCIRL 0.45gam 0.48
1.25
0.01
0g000
1.51 1.01 +0.00 2.45gn00
0.02
0.00
evolve to optimize the control parameters for spectrum of reward functions, thus better explain- ing the diverse expert feature space (low AFD), and consequently resulting in low displacement error (ADE) and averaged collision (AC). Besides, EsCIRL achieves nearly zero safety violation (in terms of AC) on both dataset while other algorithms frequently collide with the environment ve- hicles. Notably, even without the evolving framework, SCIRL alone still performs better than all the baselines, while ESCIRL. further improves the results by iteratively improving the contrastive sets. Additionally, we also note that due to training data scarcity, both the RNN-based and transformer- based method performs poorly on these datasets due to underfitting, which indicates the superiority of EscIRL in efficiently learning the multi-distribution features under the data scarcity challenge.
4 Ablation and Component Interpretation
4.1
Generalizability
To better assess the generalizability of Es- CIRL, we conduct an additional experi- ment where first train CNN in ESCIRL on 2000 random samples from the INTER- ACTION training set, and then test on the CitySim test set. The results are re- ported in Table 2. Results indicate that Es- CIRL outperforms both baselines includ- ing the RNN-based and transformer- based methods which have a much larger net-
Table 2:
Comparison analysis of generalizability. We train each method on the Interaction dataset and test on the overall City dataset.
Method Opt-IRL
AFD
ADE
FDE
AC
2.74+0.9
3,56+002
7.85 4.16
CIOC
3.12
3.44 7.36 4.97
GCL
3.54
7.88
4.55 1.05
TF-based
4.68
8.98
7.11
RNN-
3.77
4.23
3,80
ESCIRL
1.55 1.52
2.85 0.53
work to acquire better robustness. We attribute the efficiency and lightweight of EscIRL to its decoupled training process, where we first obtain the ground truth labels between trajectories and reward function parameters, and fit a CCN to capture this mapping w.r.t. driving context, without directly learning the intricate relationship between the driving context and outcome trajectories.
4.2 Scalability
ExdRi
EidRR
Besides generalizability, another crit- ical aspect in evaluating trajectory prediction methods is the scalability, which focuses on assessing method's performance when the quantity of available expert demonstrations limited (data scarcity) as well as very large (data abundance). As shown in Fig. 3, we vary the number of ex- pert demonstrations in training from 500 to 12,000, and observe both in- domain (train and test dataset being the same) and oft-of-domain (train
RNN basod
TFaased
based
Opt-bt
ADE
Thening delle
200
Training
1500 2000
(a) in-domain scalability
(b) oft-o-- scalability
Figure 3: We assess ADE w.n.t. training data scale. We pre- pare the training set to include random demonstrations from 500 to 12,000 instances from the INTERACTION dataset. In (a) these models are tested on the in-domain samples from INTERACTION test set, while in (b) they are tested on the oft-of-domain Citysim test set.
7

---

on INTERACTION and test on CitySim, as in $4.1) performance of the approaches including our proposed ESCIRL. Results demonstrate that ESCIRL has excellent scalability, especially in the data-scarcity scenarios, where EsCIRL significantly outperforms other methods. When the number of available expert demonstrations goes up, ESCIRL is also able to improve its performance. Fur- thermore, we posit that EscIRL would further benefit in scenarios of data abundance, particularly when the CCN is enhanced with large-scale, possibly transformer-basee neural network.
Evolving Process
The trajectory prediction performance during the evolving process is shown in Fig. 2b. Both average displacement error (ADE) and average collision (AC) are tracked during this process. The results demonstrate progressive increase in safe behaviors throughout the evolving process, affirming the robustness of our proposed ESCIRL.
4.4 Interpretation Analysis of CCN
Our aim in designing the CCN architecture, as detailed in $2.3, to maintain the high in- terpretability characteristic of our proposed ESCIRL. To achieve this, more specifically, we incorporate the attention mechanism [37] to dynamically adjust the focus on various in- puts such map geometry and vehicle kinematic data based their relevance to the cur- rent driving scenario, enabling the ego vehicle to accurately predict and adapt its trajectory. In this section, we visualize this at- tention to interpret the model deci- sion making. As shown in Fig. 4, we visualize scenario from the INTER- ACTION dataset in which the ego ve- hicle on highway, surrounded by multiple vehicles, with the road ori- ented downwards. We can observe that the CCN utilizes integrated spa- tial and kinematic maps to discern critical information such as the po- sitions and movements of adjacent and leading vehicles in the same lane, aligning with the vehicle's intent to follow the lane. We believe that this high level of interpretability in the CCN is attributed to its structured design, which may include layered networks that process spa- tial and temporal data separately to optimize response times and accuracy. Additionally, our unique decoupled training approach allows each component the system to trained under varied sce- narios, enhancing the model's performance and reliability in complex driving environments.
(a) spatial map (b) kinematic map (c) attended map
Figure 4: Visualization of hidden layers CCN. (a) and (b) depicts the first convolution layer of the spatial and kine- matic input. (c) depicts the attended mapping by fusing the third hidden layer of both spatial and kinematic streams. Lighter color indicates greater values. The fused map effec- tively captures the intention of the ego vehicle by attending more on the flanking and leading vehicles the same lane.
Conclusion and Discussion
In this paper, we present ESCIRL, novel approach that advances trajectory prediction au- tonomous driving through evolving self-contrastive IRL. EscIRL not only enhances prediction accuracy and robustness against diverse driving scenarios, outperforming existing methods by large margin, but also demonstrates superior generalizability against DNN-based approaches which are known for better generalizability, while maintaining data efficiency as the other IRL-based ap- proaches. In other words, EscIRL demonstrates advantages from both worlds, which have not been seen other existing works yet. We attribute the superior performance of ESCIRL to the unique bi-level optimization framework that blends both self-contrastive IRL and an evolving algorithm to offer compelling balance of the trade-off. Moving forward, future work integrating more com- plex sensor data would be an interesting way refine EscIRL's understanding of more detailed real-world contexts. Additionally, given the successful demonstration of the feasibility for learning generalizable reward functions, it would be meaningful to further explore more unified and adaptive framework for trajectory prediction in the field of autonomous driving.

---

Acknowledgments
We thank Yi Yang at CIDI Lab and all the reviewers for their valuable comments and suggestions.
References
[1] H. Wu, T. Phong, C. Yu, P. Cai, S. Zheng, and D. Hsu. What truly matters in trajectory prediction for autonomous driving? arXiv preprint wrXiv:2306.15366 2023.
[2] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521 44444)))33364444 2015.
[3] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
[4] I. H. Sarker. Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions. SN Computer Science, 2(6):420, 2021.
[5] Y. Huang and Y. Chen. Autonomous driving with deep learning: A survey of state-of-art technologies. arXiv preprint wwXi:220060009), 2020.
[6] Liu, X. Mao, Y. Fang, D. Zhu, and M. Q.-H Meng. A survey on deep-learning approaches for vehicle trajectory prediction in autonomous driving. In 202/ IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 978-985. IEEE, 2021.
[7] V. Bharilya and N. Kumar. Machine learning for autonomous vehicle's trajectory prediction: A comprchensive survey, challenges, and future research directions. Vehicular Communications, page 100733, 2024.
[8] L. Xin, P. Wang, C.-Y. Chan, Chen, S. E. Li, and B. Cheng. Intention-aware long horizon trajectory prediction of surrounding vehicles using dual Istm networks. In 2018 21st Inter- national Conference on Intelligent Transportation Systems (ITSC), pages 1441-1446. IEEE, 2018.
[9] S. Wang, P. Zhao, B. Yu, W. Huang, and H. Liang. Vehicle trajectory prediction by knowledge- driven Istm network in urban environments. Journal of Advanced Transportation, 2020:1-20, 2020.
[10] L. Rossi, A. Ajmar, M. Paolanti, and R. Pierdicca. Vehicle trajectory prediction and generation using lstm models and gans. Plos one, 16(7):e0253868, 2021.
[11] A. Postnikov, A. Gamayunov, and G. Ferrer. Transformer based trajectory prediction. arXiv preprint arXiv:2112.043300 2021.
[12] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and Moutarde. Home: Heatmap output for future motion estimation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 500-507. IEEE, 2021.
[13] Z. Li and H. Yu. Trajectory prediction for autonomous driving using transformer network. arXiv preprint trXiv:2402.1650/ 2024.
[14] M. Gheisari, G. Wang, and M. Z. A. Bhuiyan. A survey on deep learning in big data. In 2017 IEEE international conference on computational science and engineering (CSE) and IEEE in- ternational conference on embedded and ubiquitous computing (EUC), volume 2, pages 173- 180. IEEE, 2017.
[15] Datar. Navigating data scarce environments in computer vision. ITNOW, 65(4))60-61 61, 2023.
[16] L. Liu, S. Lu, R. Zhong, B. Wu, Yao. Q Zhang, and W. Shi. Computing systems for autonomous driving: State of the art and challenges. IEEE Internet of Things Journal, 8(8): 1469, 2020.

---

[17] J. Qian, L. Zhang, Q. Huang, X. Liu, X. Xing, and X. Li. A self-driving solution for resource- constrained autonomous vehicles in parked areas. High-Confidence Computing, 4(1):100182, 2024.
[18) S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
[19] J. Fischer, C. Eyberg, M. Werling, and M. Lauer. Sampling- baatingg--adddd-- inverse reinforcement learn- ing algorithms with safety constraints. In 2021 IEEE/RSJ International Conference Intelli- gent Robots and Systems (IROS), pages 791-798. IEEE, 2021.
[20] S. Adams, T. Cody, and P. A. Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review, 5566)43377-4346 2022.
[21] Q. Zou, Li, and R. Zhang. Inverse reinforcement learning via neural network driver behavior modeling. 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1245-1250. IEEE, 2018.
[22] Z. Huang. J. Wu, and C. Lv. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning. IEEE transactions on intelligent transportation systems, 23(8):10239-1022
2021.
[23] R. Boney, N. Di Palo, M. Berglund, A. Ilin, J. Kannala, A. Rasmus, and H. Valpola. Regu- larizing trajectory optimization with denoising autoencoders. Advances in Neural Information Processing Systems, 32, 2019.
[24] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. International conference on machine learning, pages 49-58. PMLR, 2016.
[25] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306. 16927, 2023.
[26] O. Zheng, M. Abdel-Aty, L. Yue, A. Abdelraouf, Z. Wang, and N. Mahmoud. Citysim: A drone-based vehicle trajectory dataset for safety oriented research and digital twins. arXiv preprint arXiv:2208. 11036, 2022.
[27] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, et al. Interaction dataset: An international, adversarial and co- operative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.03088, 2019.
[28] T. Bäck and H.-P. Schwefel. An overview of evolutionary algorithms for parameter optimiza- tion. Evolutionary computation, 1(1):1-23, 1993.
[29] B. Ziebart, A. L. Maas, A. Bagnell, A. K. Dey, Maximum entropy inverse reinforce- ment learning, Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[30] Z. Wu, L. Sun, W. Zhan, C. Yang, and M. Tomizuka. Efficient sampling-base based maximum entropy inverse reinforcement learning with application autonomous driving. IEEE Robotics and Automation Letters, 5(4):5355-553 2020.
[31] J. Von Neumann and O. Morgenstern. Theory of games and economic behavior, 2nd rev. 1947.
[32] J. Ying and Y. Feng. Full vehicle trajectory planning model for urban traffic control based on imitation learning. Transportation research record, 2676(7):186-19)) 2022.
[33] L. Sun, W. Zhan, and M. Tomizuka. Probabilistic prediction of interactive driving behavior via hierarchical inverse reinforcement learning. In 2018 21st International Conference Intelligent Transportation Systems (ITSC), pages 2111-2117. IEEE, 2018.
10

---

[34] A. v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /807.03748, 2018.
[35] N. Clelland. From Frenet to Cartan: the method of moving frames, volume 178. American Mathematical Soc., 2017.
[36] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, t.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] M. Kuderer, S. Gulati, and W. Burgard. Learning driving styles for autonomous vehicles from demonstration. In 2015 IEEE international conference on robotics and automation (ICRA), pages 2641-2646. IEEE, 2015.
[39] S. Levine and V. Koltun. Continuous inverse optimal control with locally optimal examples. arXiv preprini arXiv: /206.4617, 2012.
[40] N. Deo and M. M. Trivedi. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1468-1476, 2018.
[41] Altché and A. de La Fortelle. An lstm network for highway trajectory prediction. 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), pages 353- 359. IEEE, 2017.
[42] G. Xie, A. Shangguan, R. Fei, W. Ji, W. Ma, and X. Hei. Motion trajectory prediction based on a cnn-lstm sequential model. Science China Information Sciences, 63:1-21, 2020.
[43] L. Feng, M. Bahari, K. M. B. Amor, É. Zablocki, M. Cord, and A. Alahi. Unitraj: A unified framework for scalable vehicle trajectory prediction. arXiv preprint arXiv:2403.15098, 2024.
[44] Gilles, S. Sabatini, Tsishkou, Stanciulescu, and F. Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In 2022 international conference on robotics and automation (ICRA), pages 9107-9114 IEEE, 2022.
[45] K. Brown, K. Triggs-Campbelly and M. J. Kochenderfer. Modeling and prediction of human driver behavior: A survey, arXiv preprint vrXiv:2006.08832 pages 4-5,, 2020.
[46] J. Fang, F. Wang, Shen, Z. Zheng, J. Xue, and T.-s. Chua. Behavioral intention prediction in driving scenes: A survey. arXiv preprint 1rXiv22211.00383 2022.
[47] S. Paravarzar and B. Mohammad. Motion prediction on self-driving cars: review. arXiv preprint arXiv:2011.03633, 2020.
[48] R. Kadwani, J. Pereira, A. Thakur, and K. Wani. Object detection and pattern analysis in an autonomous car. In Applications of Advanced Computing in Systems: Proceedings of Inter- national Conference on Advances in Systems, Control and Computing, pages 13-21. Springer, 2021.
[49] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):8005, 2000.
[50] B. Hegde, M. O'Keefe, Muldoon, Gonder, and C.-F. Chang. Real-world driving features for identifying intelligent driver model parameters. Technical report, National Renewable En- ergy Lab. (NREL,,,), Golden, CO (United States), 2021.

---

[51] A. Shiledar, M. Villani, G. Rizzoni, E. A. Adinolfi, A. Pandolfi, A. Paolino, and C. Pianese. modified enhanced driver model for heavy-duty vehicles with safe deceleration. Technical report, SAE Technical Paper, 2023.
[52] B. Zhang, Zhong, and W. Cai. data-driven approach for pedestrian intention prediction in large public places. In Proceedings of the 2022 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, pages 33-36, 2022.
[53] T. Westny. Data-Driven Interaction-Awarr Behavior Prediction for Autonomous Vehicles. Phl) thesis, Linköping University Electronic Press, 2023.
[54] Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.04225 2024.
[55] O. Derbel, T. Peter, H. Zebiri, B. Mourllion, and M. Basset. Modified intelligent driver model for driver safety and traffic stability improvement. IFAC Proceedings Volumes, 46(21):744- 749, 2013.
[56] S. Albeaik, A. Bayen, M. T. Chiri, X. Gong, A. Hayat, N. Kardous, A. Keimer, S. T. McQuade, B. Piccoli, and Y. You. Limitations and improvements of the intelligent driver model (idm). SIAM Journal on Applied Dynamical Systems, 2163):186-1189 2022.
[57] L. Alzubaidi, J. Bai, A. Al-Sabaawi, Santamaria, A. Albahri, B. S. N. Al-dabbagh, M. A. Fadhel, M. Manoufali, J. Zhang, A. H. Al-Timemy, al. A survey deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. Journal of Big Data, 10(1):46, 2023.
[58] Z. Chen, Y. Du, Z. Wen, Y. Zhou, C. Cui, Z. Weng, H. Tu, C. Wang, Z. Tong, Q. Huang, et al. Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation? arXiv preprint arXiv:2407.06 048442 2024.
[59] Z. Mo. R. Shi, and X. Di. A physics-informet deep learning paradigm for car-following models. Transportation research part C: emerging technologies, 130: 2021.
[60] P. Karle, M. Geisslinger, J. Betz, and M. Lienkamp. Scenario understanding and motion pre- diction for autonomous vehicles vehicles-revieww review and comparison. JEEE Transactions on Intelligent Transportation Systems, 13110)::6962-1699
2022.
[61] Y. Gan, H. Xiao, Y. Zhao, E. Zhang, Z. Huang, X. Ye, and Ge. Mgtr: -ulti-granular transformer for motion prediction with lidar. arXiv preprint arXiv:2312.02409, 2023.
[62] M. Siami, M. Naderpour, and J. Lu. A mobile telematics pattern recognition framework for driving behavior extraction. IEEE Transactions on Intelligent Transportation Systems, 22(3): 1459-1472, 2020.
[63] X. Na and D. Cole. Theoretical and experimental investigation of driver noncooperative-game steering control behavior. IEEE/CAA Journal of Automatica Sinica, 8(1):189-205, 2020.
[64] F. Leon and M. Gavrilescu. A review of tracking and trajectory prediction methods for tonomous driving. Mathematics, 96))6600 2021.
[65] X. Di and R. Shi. A survey on autonomous vehicle control in the era of mixed-autonomy: From baysic to ai-guided driving policy learning. Transportation research part C: emerging technologies, 125:103008 2021.
[66] T. Gindele, Brechtel, and R. Dillmann. A probabilistic model for estimating driver behaviors and vehicle trajectories in traffic environments. In 13th International IEEE Conference Intelligent Transportation Systems, pages 1625-1631. IEEE, 2010.
12

---

[67] S. Hoermann, D. Stumper, and K. Dietmayer. Probabilistic long-term prediction for au- tonomous vehicles. In 2017 EEEE Intelligent Vehicles Symposium (IV), pages 237-243. IEEE, 2017.
[68] N. Deo, A. Rangesh, and M. M. Trivedi. How would surround vehicles move? unified framework for maneuver classification and motion prediction. IEEE Transactions on Intelli- gent Vehiclex, 3(2):129-140, 2018.
[69] M. Geng, Z. Cai, Y. Zhu, X. Chen, and D.-H. Lee. Multimodal vehicular trajectory prediction with inverse reinforcement learning and risk aversion urban unsignalized intersections. IEEE Transactions on Intelligent Transportation Systems, 2023.
[70] A. Alahi, K. Goel, Ramanathan, Robiequet, Fei-Fei, and Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[71] Sun, Y. Li, H.-S. Fang, and C. Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEF/CVF International Con- ference on Computer Vision, pages 13250-13259, 2021.
[72] J. Li, H. Ma, 7. Zhang, J. Li, and M. Tomizuka. Spatio-temporal graph dual-attention net- work for multi-agent prediction and tracking. IEEE Transactions on Intelligent Transportation Systems, 23: 10556-1099, 2021.
[73] X. Mo, Z. Huang, Y. Xing, and C. Lv. Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network. IEEE Transactions on Intelligent Transportation Sys- tems, 23(():9554-92)) 9567, 2022.
[74] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255-22252222 2264, 2018.
[75] Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, Isola, A. Maschinot, C. Liu, and D. Kr- ishnan. Supervised contrastive learning, Advances in neural information processing systems, 33:18661-18673, 2020.
[76] O. Henaff.
Data-efficient image recognition with contrastive predictive coding. In Interna- tional conference on machine learning, pages 4182-4192. PMLR, 2020.
[77] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[78] M. N. Azadani and A. Boukerche. A novel multimodal vehicle path prediction method based on temporal convolutional networks. IEEE Transactions on Intelligent Transportation Systems, 23(12):5534-255393 2022.
[79] H. Cui, V. Radosavljevic, F.-C Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pages 2090- 2096. IEEF, 2019.
[80] A. Kawasaki and A. Seki. Multimodal trajectory predictions for urban environments using geometric relationships between a vehicle and lanes. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9209. IEEE, 2020.
[81] L. Fang, Q. Jiang, J. Shi, and B. Zhou. Tpnet: Trajectory proposal network for motion predic- tion. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pages 6797-6806, 2020.
13

---

[82] R. Sun, S. Hu, H. Zhao, M. Moze, F. Aioun, and F. Guillemard. Human Juman-like highway trajectory modeling based on inverse reinforcement learning. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1482-1489. IEEE, 2019.
[83] G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
[84] A. Nilim and El Ghaoui. Robust control of markov decision processes with uncertain tran- sition matrices. Operations Research, 53(5):780-798, 2005.
[85] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[86] Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li. Agentpoison: Red-teaming Ilm agents via poisoning memory or knowledge bases. arXiv preprint arXiv:2407.12722 2024.
[87] Z. Chen, Z. Zhao, T. He, B. Chen, X. Zhao, L. Gong, and C. Liu. Safe reinforcement learn- ing via hierarchical adaptive chance-constraint safeguards. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024.
[88] Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[89] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: (00.:/8800061/// 58, 2018.
[90] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pages 6036-6045. PMLR, 2019.
14

---

Appendix
A
Related Works
A.1 Driving Behavior Modeling
Driving behavior modeling is essential in autonomous driving, focusing on trajectory prediction and encompassing various methodologies and assumptions [45]. Key areas include behavioral inten- tion prediction (BIP) [46], motion prediction [5, 47], and pattern analysis [48]. BIP predicts future road user actions using historical and contextual data, utilizing parametric models like the intelli- gent driver model (IDM) [49, 50, 51] and data- methods [52, 53, 54]. Parametric models are simple and efficient [55] but may not capture complex dynamics [56], while data-driven approaches handle complex patterns well [46] but struggle with interpretability and adaptability to new data sce- narios [57]. Motion prediction enables autonomous vehicles to anticipate other users' movements, enhancing safety and efficiency [47, 58]. involves physics- models [59] for simple scenarios and DL methods for complex environments [60, 61]. Pattern analysis extracts features from human driving data to identify driving traits and styles [62], integrating advanced techniques like game the- ory [63] for deeper insights. Our research advances these concepts by mathematically formulating decision-makin in self-driving cars reward functions, which aid in control and planning, and deriving parameters from driving data to improve trajectory prediction accuracy and interpretability.
A.2 Trajectory Prediction
Trajectory prediction in autonomous systems is classified into model-based and data-driven ap- proaches [64]. Model-based methods use physics equations and stochastic models [65], such as Bayesian networks [66], Monte Carlo simulations [67], and Gaussian mixture models [68], to predict vehicle movements. While highly interpretable, their application often limited to less uncertain environments like highways [69]. Data-driven methods, on the other hand, learn from historical and spatial data using recurrent [70, 71], convolutional [40], graph [72, 73] neural networks, transform- ers [11, 12, 13], and generative adversarial networks [74], which excel in handling complex temporal and spatial data. Recent studies also explore self-supervised contrastive learning [34], proven effec- tive in supervised [75] and unsupervised [34, 76, 77] settings, to improve trajectory prediction. This work introduces an action-based contrastive loss that incorporates pedestrian action data, enhancing the accuracy and context-sensitivity complex environments.
A.3 Robust IRL
Both model-based and data-driven trajectory prediction methods struggle with complexity and re- quire extensive data, often producing only average predictions that miss the variability in human driving [78]. Multimodal methods address this by considering multiple possible paths [79, 80, 81), but they can be unstable and overlook specific driving characteristics. IRL and imitation learn- ing aim to more closely mimic human driving by learning from observed behaviors, yet they are limited by their reliance on environmental models for simulating interactions [22, 82, 33). Robust IRL, developing from robust Markov Decision Process (MDP) frameworks [83, 84, 85, 86, 87], fo- cuses on enhancing decision-making robustness. Our work diverges from traditional approaches by addressing dynamics mismatches rather than just learner-expert mismatches, significant issue in Generative Adversarial Imitation Learning (GAIL) [88] and its variants [89, 90]. On the other hand, ESCIRL offers robust, generalizable solution that enhances policy robustness and applicability by handling these mismatches effectively.
B Experiment Settings
B.1 Maximum Entropy Inverse Reinforcement Learning (MaxEnt-IRL)
Recalling the definition in Eq. (1), a and w represent the states and actions of the vehicle respec- tively, we formulate the dynamics f(-) of each vehicle as defined in Eq. (1), A driving trajec- tory { within the spatial-temporal domain comprises a sequence of states and actions, such that E [o", u", _t, ...,.... .....,,o-- N .,.. N-1, J., where N denotes the length of the planning horizon. Given set of driving demonstrations D, (5.). indexed by 1,2,.... N, the maximum maxtmummentrrop
15

---

IRL [29] problem seeks to infer the underlying reward function that maximizes the likelihood of these expert demonstrations, under the assumption that trajectories are exponentially more likely with higher cumulative rewards. This assumption formalized by the Boltzmann noisily-rational model P(E) [31] as shown in (10), where w is the parameters of reward function R and 8 is a hyper-parameter to specify demonstration proximity [30], which we consistently adopt 8 1 in this study. Notably, while previous works typically partition datasets into particular scenarios such as in-lane following [22]. lane-changing [32], and merging [33] to assume a stable reward function across demonstrations, we engage with the complete dataset instead. In our study, the dataset con- tains multi-modal driving policies with distinguished distributions, and we aim to learn a continuous controller network to handle diverse driving scenarios presented in the dataset.
Our methodology employs linear-structured reward function that is characterized by specifically chosen feature space F()), which is defined in relation the trajectories E:
R(E:w) = w" F(E)
(9)
As result, the probability, likelihood, of the demonstration set is given by:
P(DpWw)= ())
II Zw
N
e R. (E:)
II
N
1 (R.)
(10)
==1 Zw
1=1
In this context, Zu denotes the set of all feasible trajectories that correspond to the initial and termi- nal conditions delineated by Ç. The primary objective is to ascertain the optimal parameter vector w" that enhances the average log-likelihood of the observed demonstrations, as shown below:
wn arg max
log p(Dplw))
(11)
N
arg max
Σ log P(E/W)
(12)
From (4) and (5), we can see that the key step in solving the optimization problem in (5) is the calculation of the partition factors Zu. In sampling based methods, Zw for each demonstration is
approximated via the sum over samples in the sample set (5TT), m = 1, 1,2,...; M, and & is denoted as c::
M
Zu
(13)
m==0
Thus, the objective function in (5) becomes:
I(w)= = N
log (()/())
im
e RR_
log
(14)
N
Σ M
m== e (E!)
1=1
(R) - -log
e (ET)
N
m=0
The derivative is thus given by:
3 N
L =
F(())
(15)
N 1=1
M
Rew (rl') F((
F(E)) =
(16)
ΣM
(E!T)
where F(E) defines the expected feature counts over all samples given w.
Note that an additional 4 regularization over the parameter vector is introduced in the training process to compensate for possible errors induced via the selected set of features.
16

---

C Feature Design for ESCIRL
For any trajectory& È oij=0 with T frames of states, where o, [Ti Ma. O2, 86, dis Uzl. we project it into feature space as follows.
C.1 Features for comfort
f1
(U441 - vi)2
(17)
T
1=0
T-3
1
f2
U142 - 02)
(18)
T
1=0
We introduce fi, f2 to prevent dramatic velocity changes. Essentially, fi calculates the average of squared differences in velocity between consecutive states, which is measure of the smoothness of velocity over time. f2 is similar to f1, but it takes the average squared difference in velocity over every two time steps instead of every one. This means it measures the change velocity over a longer time span. By minimizing these two features, we can avoid jerky movements which may cause uncomfortable riding experience.
T-2
fa = T 1
(Wrap(0441 - -2)))
(19)
1=0
0+ π
Wrap(0) = 2T
(20)
2x
fs sums up the average squared change of heading angles over consecutive steps. Here, the Wrap function ensures the heading angle change is bounded in [-n, m). It minimized to ensure smooth trajectory.
C.2 Features for efficiency
T fo
2
2
(21)
T
fo sums up the squared velocity over the whole trajectory. By minimizing fu. the model penal- izes abnormally high velocity, thus ensuring more fuel efficient and environment friendly driving dynamic.
T-1 Σ
Uiiit
fo
(22)
Uhmit
Uliii the velocity limitation of the current road. To ensure the high efficiency of trajectory plan- ning, we may want the vehicle's speed be fast without violating traffic rules. By minimizing So. can ensure the vehicle's 's speed is close to the speed limit.
f4
(8T-1 -
(23)
1=0
fs T
1
Σ
(Wrap(arctan(
9T-1 Vi
0,))
(24)
TT-1 Dz
T T-1
fo =
1
(25)
T k=0
17

---

T-1 T + R
fio
(26)
1 and 1 represent the ego vehicle whether to merge left or right, according to its destination.
T
fn
(max( (mmxx(mmngiin margin
send 0) + 1)
(27)
T
2=
J12
1 T
(e dineR
11) margin and 0) + 1)
(28)
gurgel and d'argetR are the distance from the ego vehicle the left or right road's center line if should merge left right. smargin is constant distance to measure the urgency of changing lanes. gend is the distance from the current pose to the endpoint of the current road.
C.3 Features for safety
1 T
f7 T (
Σ
(29)
T-1
fs Wrap((; - rrond))
(30)
fs measures the deviance vehicle's ' heading angle from the road's angle. Minimizing this feature keeps the vehicle on the right track.
f13
1 7
Σ
(p-8p)
(31)
1 T
fu T 1=0
1d
[8
(32)
1 T-1
Jis
7
Σ 1=0
d E (B-Bn)
(33)
((E-SF), d(E-Sz), and (E- SR), is the distance from the ego vehicle the nearest surrounding vehicle in front, left, right region respectively, and denoted as inf if no surrounding vehicle exits in such region.
1 T-1
(K-Cr) (pp-
f10
(34)
T 1=0
1 T-1
(B-0 01)
18, -1
f17
(35)
T 1=0
1 T-1
(R-CR) (82-CR)
f18
(36)
T
A collision point between the ego vehicle and the nearest surrounding vehicle in the front, left, or right region would be respettivll. We chose the Constant Velocity model prediction
18

---

detectable
E: ego vehicle (im)
region
S: surrounding vehicles C
blind region
C: collision point
Asa
d
R: right; L: left; froron
d: distance
As: predicted trajectory
Figure 5: Interactive feature design
method, which is the simplest method. d(E-Cr), d(E-Cr), and d(E-Cn) is the distance from ego vehicle to the collision point with the nearest surrounding vehicle the front, left, and right region, while d(S)-Cr), d(Sz-Cz) and -C2) is the distance of the surrounding vehicle. A sample shown in Figure 5.
P-1
f19
Σ
Npoll
(37)
T
1=0
Ncoll is the number of collisions between ego vehicle and surrounding vehicles.
D
Continuous Controller Network (CCN)
D.1 Structure Design
Given a trajectory (T:), a parameter W; would be calculated through Contrastive IRI. We designed a Continuous Control Network (CCN) to map the relationship between the first states of trajectories and CIRL parameters, which provides a continuous mapping method for any given state.
Let D {(oP, ) be dataset for cate-parameter pairs, where the of is the initial state of each trajectory. We first encode the initial state into a multi-channel context map M E x0xx with e channels and size (200, 200). The first channel is road information near the ego vehicle; the second channel is the spatial information about the ego vehicle and surrounding vehicles, containing their positions, headings, and shapes; other channels are the kinematic information about the ego vehicle and surrounding vehicles, containing velocity, acceleration, distance to road center, etc. The kinematic information about the ego vehicle is also collected ego kinematic data.
The structure of the Continuous Control Network is shown in Figure 2a. The kinematic channels of the context map are fed into two layers' 2D convolutional neural network with kernel size (6, 6), stride 2, and padding 2, and a 2D max pooling layer in the middle with kernel size (2, 2) and stride The output of the last layer is 32 channels with the size (25, 25), denoted as an attention map. The context map's 's road channel and spatial channel fed into three layers' 2D convolutional neural network with kernel size (4, 4), stride 2, and padding The output of the last layer (original map) also 32 channels with the size (25, 25) and fused with the attention map Hadamard Product. The original and fusion maps are extracted into feature vectors through the fully connected layers. The ego kinematic data is also extracted into feature vector and concatenated with the other two feature vectors. Finally, the prediction reward parameters pred mapping to the given initial state predicted from the final FC layer by feeding the concatenated feature vector as input.
To update the parameters of CCN, the loss function is denoted follows:
.N log log(cosh(Juag pore wil))
LOON (wored) (w) N
(38)
19

---

Sim Dotaset
-66
Figure 6: The interactive scenarios of CitySim dataset (e.g. intersection, roundabout) that we inves- tigate this study [26].
D.2 Dataset
While a vast array of datasets exists for autonomous vehicles, they often encompass restricted range of real-world scenarios. To address the limitations in variety and depth of data, particularly acute in regions with less technological infrastructure, our model incorporates a contrastive process that enhances its capability to handle diverse real-world driving conditions with limited feature sets.
To realize this, the datasets selected for training characterized by extensive variability and rich set of annotations necessary for accurately generating labels for subsequent training stages. Adhering to these selection criteria, we have curated suite of datasets that provide comprehensive coverage of the myriad situations encountered on the roads.
INTERACTION. The INTERACTION dataset encompasses four categories of interactive driving scenarios collected from 11 different locations, provides rich, diversified backdrop that reflects the variability of driving behaviors across the globe with driving scenarios from different countries and cultural contexts. This inclusivity ensures that models developed tested against this dataset can be more easily adapted to the unique conditions of developing countries, mitigating the bias often found in datasets predominantly sourced from high-resource environments.
CitySim. The selection of the CitySim dataset is predicated on its unparalleled contribution to safety-oriented research and applications within the domain of autonomous driving, particularly un- der the constraints of developing countries where safety-critical events and infrastructure limitations pose unique challenges. CitySim's comprehensive and meticulously curated vehicle trajectory data set the stage for innovative machine learning approaches that are both resource-efficient and highly applicable to real- weal-world scenarios.
These datasets instrumental in overcoming the prevalent data scareity and contribute more robust and versatile model training process. They ensure that the resulting algorithms capa- ble of interpreting and reacting to a wide array of traffic scenarios, making the advancements in autonomous vehicle technology more accessible and effective across different regions.
20


Reviews:
[{"review_id": "pf51L2Y3HQ", "reviewer": "Reviewer_RjKv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This papert introduce a novel decoupled bi-level training framework that iteratively learns robust reward functions from complex multi-scenario expert demonstrations. This innovative approach has been experimentally verified using a public dataset", "summary_of_recommendation": "This paper is well-written and introduces a new algorithm that has performed well compared to previous work. However, the limitations are not well discussed. Although there was a discussion about generalizability, the algorithm needs to be assessed on more datasets to make a stronger argument about the generalizability of the proposed algorithm", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "Please provide an explanation of how the baseline algorithms are chosen for comparison. Are there any other algorithms that could perform better?\n\nComments by the authors are acceptable. But suggested changes need to be fully addressed if accepted for the publication,", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkJHP3V89I", "reviewer": "Reviewer_vT2J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Evolving Self-Contrastive Inverse Reinforcement Learning (ESCIRL), a novel approach that integrates DNN and IRL for trajectory prediction in autonomous driving. ESCIRL employs a decoupled bi-level training framework, with a self-contrastive IRL module at the inner level and an evolving loop at the outer level. The approach aims to address data scarcity and generalizability issues inherent in DNN and IRL methods. Experimental results on the CitySim and INTERACTION datasets demonstrate that ESCIRL outperforms state-of-the-art DNN and IRL methods by 82.5% and 58.7% on average.", "summary_of_recommendation": "The proposed innovative framework presents a significant advancement in trajectory prediction for autonomous driving by addressing the limitations of current DNN and IRL methods, particularly in data efficiency, robustness, and generalizability. The experiment setting, results, ablation study, and interpretation are convincing.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. What are the limitations of the proposed solution, e.g., complexity, training process?\n2. Has a similar framework been seen in other Robotics or ML applications? Are there any literature on the framework?\n3. Further highlight and elaborate on the novel parts of the proposed framework (Fig 1.) vs. existing solutions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kmEpyj3yM", "reviewer": "Reviewer_iJ2t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents EscIRL, an algorithm for trajectory prediction which aims to combine the data-efficiency of IRL with the robustness and generalizability of supervised learning approaches to trajectory prediction. The approach models a dataset of trajectory demonstration as a set of distinct policies, and aims to jointly learn the specific sets, as well as a reward function describing behavior in each set. In contrast to previous work, the authors make no assumptions on how to define these sets, but rather learn them directly on data using an evolutionary framework.", "summary_of_recommendation": "I found the idea novel, results compelling, and would support acceptance assume authors are able to clarify details of their evolutionary approach.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "- Could you describe in more detail how the evolving strategy works?\n- How sensitive is the approach to the choice of $K^+$ and $K^-$? How are these values chosen?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The strengths of this paper are clarity of overall presentation, motivation, and presentation of results.  There were several weaknesses identified by the reviewers in the submission. However, during the rebuttal stage the authors have successfully addressed the reviewers' concerns and clarified the details improving the quality of the paper.
Decision: Accept