EscIRL: Evolving Self-Contrastive IRL for Trajectory Prediction in Autonomous Driving

Authors: siyue  wang
zhaorun  chen
zhuokai  zhao
chaoli  mao
yiyang  zhou
jiayu  he
albert sibo hu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 15:51:32

Abstract: While deep neural networks (DNN) and inverse reinforcement learning (IRL) have both been commonly used in autonomous driving to predict trajectories through learning from expert demonstrations, DNN-based methods suffer from data-scarcity, while IRL-based approaches often struggle with generalizability, making both hard to apply to new driving scenarios. To address these issues, we introduce EscIRL, a novel decoupled bi-level training framework that iteratively learns robust reward models from only a few mixed-scenario demonstrations. At the inner level, EscIRL introduces a self-contrastive IRL module that learns a spectrum of specialized reward functions by contrasting demonstrations across different scenarios. At the outer level, ESCIRL employs an evolving loop that iteratively refines the contrastive sets, ensuring global convergence. Experiments on two multi-scenario datasets, CitySim and INTERACTION, demonstrate the effectiveness of EscIRL, outperforming state-of-the-art DNN and IRL-based methods by 41.3% on average. Notably, we show that EscIRL achieves superior generalizability compared to DNN-based approaches while requiring only a small fraction of the data, effectively addressing data-scarcity constraints. All code and data are available at https://github.com/SiyueWang-CiDi/EscIRL.

Main paper: 
 introduction: 
 Trajectory prediction has been one of the crucial components of autonomous system [1] , allowing the self-driving vehicle to forecast the movements of itself and nearby entities. Given the rise of modern machine learning (ML) and more specifically deep learning (DL) [2, 3, 4] , numerous deep neural networks (DNN)-based methods [5] have been developed for trajectory prediction tasks [6, 7] , including more recent works based on Long Short-Term Memory (LSTM) [8, 9, 10] and transformers [11, 12, 13] , which take advantage of the more complex and capable DNNs to better capture and understand the intricate temporal and spatial dependencies inherent in various driving scenarios. However, while these approaches can learn a more complex policy representation and are more robust to diverse driving scenarios, their larger-scaled network architectures often require substantial amounts of data to train [14] , which poses concerns in underfitting and limits them from being adopted under data-scarce environments [15] or low-resourced computation conditions [16, 17] .Meanwhile, inverse reinforcement learning (IRL)-based approaches [18, 19, 20] have also found great success in predicting trajectories through inferring cost or reward functions from expert demonstrations, and then using these functions to guide the behavior of self-driving vehicles in unseen driving environments. More specifically, given the infinite numbers of potential cost functions that could explain the demonstrations [21] , existing IRL approaches typically employ structural priors for these functions, such as a linear combinations of features [22] , or constant derivative regularization [23] , to reduce the large number of possible cost functions [24] . However, under these assumptions, the learned IRL controllers often suffer from the curse of generalizability and tend to only explain the demonstrations locally, making the learned controllers difficult to adapt to new environments [25] . In other words, the inferred cost or reward functions can only be applied to unseen driving conditions whose types of traffic situations (e.g. left turn at the red light) must have been included in the demonstrations, hurting the system's performance and robustness across a broader range of situations. To handle the real-life complex environments, existing IRL frameworks often pre-classify the mixed-scenario datasets and learn numerous cost functions where each corresponds to one driving scenario. Such rule-based strategies not only increase the unnecessary difficulty in training, but also degrade generalizability, making the system prone to unseen environments and causing safety concerns. As far as we are concerned, no existing sample-based IRL method can learn global cost functions for general-purposed trajectory prediction tasks [24] .The limitations of both DNN-and IRL-based approaches highlight the necessity to develop a novel data-efficient yet robust and generalizable algorithm for trajectory prediction in autonomous driving. To this end, in this paper we propose ESCIRL: an Evolving Self-Contrastive IRL approach that utilizes a decoupled bi-level training framework to efficiently learn generalizable cost function from diverse trajectory distributions without the need of pre-labeled classifications or additional classifiers. More specifically, after initializing with a set of pseudo-controller parameters, on the inner level, ESCIRL employs a self-contrastive IRL (SCIRL) algorithm to increase the likelihood of expert demonstrations within similar distributions, i.e., positive set, while reducing expert likelihood in adverse/dissimilar distributions, i.e., negative set. And on the outer level, ESCIRL utilizes an evolving framework to iteratively optimize the contrastive sets from the inner level to introduce diversity while ensuring overall convergence. Finally, a continuous controller network (CCN) is trained to capture the mapping from specific trajectory context to the earlier initialized pseudo-parameters.To summarize, four key contributions of this paper include: 1) introducing ESCIRL, a novel approach that addresses a common challenge in trajectory prediction tasks faced by both DNN-and IRL-based methods, i.e., to ensure generalizability while being data-efficient; 2) a self-contrastive IRL (SCIRL) algorithm that learns a spectrum of robust reward functions from contrasted expert demonstrations without additional labels or classifications, significantly improving generalizability; 3) an evolving framework that iteratively refines the positive and negative sets from SCIRL to enhance convergence while encouraging exploration -we argue that such training framework combining SCIRL and evolving loop may bring benefits to other IRL applications as well; and 4) extensive experiments on two multi-scenario datasets including CitySim [26] and INTERACTION [27] , showing that ESCIRL outperforms existing SOTA methods by an average of 41.3%. 
methodology: 
 In this section, we illustrate the details of our proposed Evolving Self-Contrastive IRL (ESCIRL). Specifically, ESCIRL aims to efficiently learn a generalizable cost function from diverse trajectory distributions in a self-supervised manner, where it first employs a self-contrastive IRL (SCIRL) algorithm to disentangle similar and dissimilar distributions without requiring any priors. Then ESCIRL adopts an outer-loop evolutionary algorithm to gradually learn the priors via natural selection [28] to interpret the multi-scenario distribution for robust trajectory prediction. An overview of the proposed pipeline is shown in Fig. 1 . We first summarize the problem statement in §2.1, and then proceed to explain the two major components of ESCIRL, which are the inner-level self-contrastive IRL (SCIRL) module and the outer-level evolving loop in §2.2 and §2.3 respectively.Let σ and u represent the states and actions of the vehicle respectively, we formulate the dynamics f (•) of each vehicle as:EQUATIONNext, trajectory ξ within the spatial-temporal domain is defined as a sequence of states and actions, i.e., ξ = {σ 0 , u 0 , σ 1 , u 1 , . . . , σ N -1 , u N -1 }, where the superscripts indicate the discretized planning step and N denotes the total length of the planning horizon.Given a set of driving demonstrations D p = {ξ i } comprising of various individual trajectories as i = 1, 2, . . . , N , and a set of parameters ω of a reward function R, the probability of the expert to adaptively predict control parameters given the dynamic stacked context (e.g. vehicle states, map geometry) and select the top 10% of the population for crossover and mutation to produce the subsequent population. This evolving process is repeated for a maximum of K times or until convergence.demonstrations D p is commonly [29] defined as:EQUATIONwhere Z ω denotes the set of feasible trajectories, and β specifies the demonstration proximity [30] , which is set to 1 throughout this paper unless otherwise noted. The goal of an (maximum entropy) IRL algorithm [29] is to infer the underlying reward function that maximizes Eq. ( 2), which is the likelihood of all expert demonstrations, with the assumption that trajectories are exponentially more likely with higher cumulative rewards [31] . More specifically, assume that the trajectory space can be approximated with discretized samples Z ω ≈ M m=0 e βRω(ξ m i ) , where ξ m i ∈ D i m is a candidate trajectory in the sampled trajectory set D i m for demonstration ξ i . Thus the goal of IRL is to find a set of parameters ω * which maximizes the expert demonstration among the set of sampled trajectories. Mathematically, we haveEQUATIONIn the context of trajectory planning and prediction, ω * will then be used to select the best trajectory from the sampled trajectory set ξ * i ∈ D m .Following [19, 32] , we employ a linear-structured reward function with respect to F, which is a chosen feature space. Specifically, we follow [32] and define three categories of features separately for efficiency, safety and comfort. Besides, we propose two novel interactive features to capture the driver's desiderata in complex intersections. Finally, we define twenty-two features in total, more details on the feature design are in Appendix C.While existing works typically partition datasets first into specific scenarios that are clearly distinctive from each other such as dividing the dataset into subsets of in-lane following [22] , lane-changing [32] , or lane-merging [33] , so that the inferred reward function (w * in Eq. ( 3)) is stable across the demonstrations within the in-domain scenario, we instead consider the complete multidistribution dataset without any partition to learn a unified controller network that outputs such w * to handle all the diverse driving scenarios. Learning such unified controller network brings more challenges than simply solving for Eq. (3), which produces only a single vector of the control parameters. Notice that this is fine for existing works as they have an implicit assumption that the expert demonstration obey a unified distribution [22] due to partition. However, such assumption is no longer valid in our case as we consider multi-distributions in the expert demonstrations. Blindly following such assumption would lead to decreased performance and lower robustness [24] . Therefore, to handle the multi-modality in terms of driving conditions and preserve the distribution diversity in the demonstration dataset, we propose a self-contrastive IRL (SCIRL) algorithm. Inspired by [34] , we augment Eq. ( 2) with an auxiliary contrastive loss term, which can be represented as:EQUATIONwhere γ is a hyper-parameter and is set to 1 throughout this paper unless otherwise noted. Different from [34] , SCIRL considers both positive (similar policy) and negative (dissimilar policy) set, with L IRL only being conducted on the positive set, i.e.EQUATIONand we design L Contrastive as a hinge loss that seeks to ensure that the expert demonstration has a higher probability in the positive set than the negative set by a margin ϵ. Mathematically, we have:EQUATIONwhere P + and P -are the probability of each expert demonstration in the positive and negative set respectively. More specifically, for an arbitrary demonstration ξ ∈ D p , we haveEQUATIONwhere m + , m -, ξ + i , and ξ - i represent the demonstration m and sampled trajectory ξ in the positive (+) and negative (-) subset for trajectory ξ j . K + j and K - j denote the number of demonstrations in the positive and negative set. Based on a common assumption that each demonstration ξ i ∈ D p only represents one certain scenario, we define a separate pair of contrastive set for each ξ i . In other words, instead of applying L IRL on the overall dataset as in Eq. ( 2), we define a specific pair of positive/negative (contrastive) sets for each demonstration ξ j in the dataset D P More details are illustrated in Algorithm 1. As a result, SCIRL can effectively preserve the multi-modality (multidriving conditions) across diverse driving scenarios and optimize expert behaviors on a more finegrained subspace. However, considering the general case where no priors are available to predetermine these sets, we introduce an evolutionary algorithm to iteratively refine the contrastive sets in a self-supervised manner.Since SCIRL does not rely on any priors to pre-define the contrastive sets for each trajectory ξ j , we propose an evolving framework, which is considered as the outer-level of ESCIRL integrating SCIRL into an iterative closed-loop optimization. In this way, we can effectively accumulate the priors to determine contrastive sets for each trajectory via natural selection [28] , so that convergence and global optimality can be guaranteed. As shown in Fig. 1 , for each raw trajectory, we first transform it to the Frenet state-space [35] using a rule-based motion extractor. Next, for each instance in the initial population, we set ξ + i = ξ i and ξ - i = ∅ for each trajectory in the dataset and then apply SCIRL to obtain a set of corresponding reward parameters ω i as initialization.Continuous control network. To capture the underlying relationship between a specific driving context including vehicle states (both spatial and kinematic), map geometry, and its expert policy Algorithm 1 Updating CCN with ESCIRL Require: a dataset D = {ξ i } N i=1 with N trajectories ξ i ; feature function F; safety-constrained sample policy π i ; number of points in contrastive set K + , K -, learning rate α, convergence threshold ϵ Ensure: a global cost function G; sample policy π * 1: Randomly initialize ω i , π i for all i ∈ {1, . . . , N } 2: Project trajectory ξ i into feature space:F i = F(ξ i ) 3: for each trajectory τ i , i ∈ {1, . . . , N } do 4:Sample M trajectories under policy π i 5:Find K + closest and K -furthest points by sorting distance matrix {d(ω i , ω j )} M j=1 6:Compute contrastive IRL loss L SCIRL (ω i ) for ω i :7:Update ω i using gradient descent:ω i ← ω i -α∇ ωi L SCIRL (ω i ) 8:Update π i using policy gradient (if π i is learnable) distribution, we propose to learn a unified continuous control network (CCN) to adaptively predict the corresponding parameters of the reward function (as in Eq. ( 2)) for the ego vehicle. The architecture of the proposed CCN is shown in Fig. 2a , with three streams of network inputs: the static map geometry, mapping of vehicle spatial states (e.g. coordinate, heading), and mapping of vehicle kinematic states (e.g. velocity, accelerations, jerk). The output is the corresponding control parameters ω i of the reward function.Since both the map geometry and vehicle spatial states are static, we stack them to be a dual-channel mapping input. Then we design a Siamese structure [36] in CCN to separately encode the vehicle kinematic information (i.e. kinematic input), and the static map with vehicle spatial information (i.e. spatial input). Notably, we define the intermediate mapping output of the convolution for spatial input to be an attention map [37] , indicating which specific parts of the context are more responsible for the current predictions. On the other hand, the convolution for kinematic input encompasses a richer representation of the vehicles' dynamic states. After obtaining both convolution outputs, we fuse the two mappings using a Hadamard Product and then concatenate it again with the ego vehicle's states to further augment the low-dimensional representation of vehicle-environment information. Finally, the concatenated vector is mapped to the SCIRL parameters space through a fully-connected layer. During training, the loss of CCN is defined as:EQUATIONwhere ω pred i and ω i refers to the predicted reward parameter and ground truth respectively. And cosh(x) = e x +e -x. In general, we argue that this attention-based design allows for an adaptive representation of the driving context and establishes an informative mapping from the driving context to the controller space (i.e. parameter space of reward functions). Furthermore, we provide an analysis on interpretability for the proposed CCN structure in §4.4. The procedure for updating CCN within ESCIRL is detailed in Algorithm 1.Evolving contrastive sets. As shown in Fig. 1 , after SCIRL converges in each iteration, we select the top 10% of the population based on averaged displacement error (ADE) from the in-domain validation set, and further crossover and mutate them to produce the subsequent populations. Specifically for crossover, we sample I E mutual demonstrations from the intersection set of the top 10% contrastive sets and sample (K +/--I E ) demonstrations from the difference set of the union and intersection sets. Then we mutate the demonstrations in each contrastive sets by a uniform probability of 5% to obtain each population in the next generations. This process is repeated until convergence (the decrease of ADE falls below a threshold) or for a maximum K times, as shown in Fig. 2b .  
experiment: 
 Datasets. To evaluate the capability of ESCIRL on complex multi-scenario driving modeling tasks, we use INTERACTION [27] and CitySim [26] datasets. INTERACTION is a comprehensive dataset of interactive vehicle trajectories collected from various traffic scenarios (e.g. roundabout, intersections) around the world. And CitySim is a drone-based vehicle trajectory collection featuring vehicle trajectories extracted from 1,140 minutes of drone videos recorded across 12 different locations, capturing a variety of road geometries to facilitate comprehensive analysis and applications. Specifically, we extract 20,000 trajectories in total for both dataset. All the trajectories from both datasets are converted to Frenet state-space [35] using a rule-based motion extractor.Baselines. We compare the performance of ESCIRL against five principled trajectory prediction approaches, including three IRL-based methods that demonstrate better data-efficiency, and two DNN-based methods which excel especially in multi-scenario complex driving conditions. More specifically, the three IRL baselines include Opt-IRL [38] , which learns a unified reward function via maximum entropy objective; CIOC [39] , which employs a Laplace approximation to estimate rewards in a continuous feature space; and GCL [24] , a model-free approach that adopts DNN to learn the reward and policy in an end-to-end manner without predefined heuristics. Additionally, we compare two DNN-based models that have demonstrated superior performance in trajectory prediction tasks: RNN-based models, where we use a CNN-LSTM structure in our implementation (e.g. [40, 41, 42] ), and transformer-based (TF-based) models (e.g. [12, 43, 44] ). For both model structures, we adopt the default hyperparameter configurations recommended in respective works.Metrics. Existing metrics including averaged feature distance (AFD), average displacement error (ADE), final displacement error (FDE), human probability (HP), and averaged number of collisions (AC) are used in our experiments. Specifically, HP is evaluated by the likelihood that the predicted trajectory aligns with the expert trajectory. It is important to note that not all metrics are applicable to every method. Specifically, AFD is unavailable for GCL, RNN-based, and transformer-based methods, as these approaches do not rely on feature engineering. Similarly, HP cannot be applied to DNN-based methods, as they lack a straightforward evaluation for this metric.Results. To verify ESCIRL's performance under a scarcely labeled dataset which incorporates multiple diverse distributions, we randomly sample 2000 trajectories for both dataset and assess ESCIRL and the five baselines w.r.t. the five aforementioned metrics. The results are shown in Table 1. We can denote that ESCIRL outperforms all the other approaches by a significant margin on both datasets. We attribute the reasons for the strong performance of ESCIRL to the fact that it can efficiently decompose the mixed-scenario expert distributions under limited priors and adaptively evolve to optimize the control parameters for a spectrum of reward functions, thus better explaining the diverse expert feature space (low AFD), and consequently resulting in low displacement error (ADE) and averaged collision (AC). Besides, ESCIRL achieves nearly zero safety violation (in terms of AC) on both dataset while other algorithms frequently collide with the environment vehicles. Notably, even without the evolving framework, SCIRL alone still performs better than all the baselines, while ESCIRL further improves the results by iteratively improving the contrastive sets. Additionally, we also note that due to training data scarcity, both the RNN-based and transformerbased method performs poorly on these datasets due to underfitting, which indicates the superiority of ESCIRL in efficiently learning the multi-distribution features under the data scarcity challenge. 2 . Results indicate that ES-CIRL outperforms both baselines including the RNN-based and transformer-based methods which have a much larger network to acquire better robustness. We attribute the efficiency and lightweight of ESCIRL to its decoupled training process, where we first obtain the ground truth labels between trajectories and reward function parameters, and fit a CCN to capture this mapping w.r.t. driving context, without directly learning the intricate relationship between the driving context and outcome trajectories. Besides generalizability, another critical aspect in evaluating trajectory prediction methods is the scalability, which focuses on assessing method's performance when the quantity of available expert demonstrations is limited (data scarcity) as well as very large (data abundance). As shown in Fig. 3  
evolving process: 
 The trajectory prediction performance during the evolving process is shown in Fig. 2b . Both average displacement error (ADE) and average collision (AC) are tracked during this process. The results demonstrate a progressive increase in safe behaviors throughout the evolving process, affirming the robustness of our proposed ESCIRL.Our aim in designing the CCN architecture, as detailed in §2.3, is to maintain the high interpretability characteristic of our proposed ESCIRL. To achieve this, more specifically, we incorporate the attention mechanism [37] to dynamically adjust the focus on various inputs such as map geometry and vehicle kinematic data based on their relevance to the current driving scenario, enabling the ego vehicle to accurately predict and adapt its trajectory. In this section, we visualize this attention to interpret the model decision making. As shown in Fig. 4 , we visualize a scenario from the INTER-ACTION dataset in which the ego vehicle is on a highway, surrounded by multiple vehicles, with the road oriented downwards. We can observe that the CCN utilizes integrated spatial and kinematic maps to discern critical information such as the positions and movements of adjacent and leading vehicles in the same lane, aligning with the vehicle's intent to follow the lane. We believe that this high level of interpretability in the CCN is attributed to its structured design, which may include layered networks that process spatial and temporal data separately to optimize response times and accuracy. Additionally, our unique decoupled training approach allows each component of the system to be trained under varied scenarios, enhancing the model's performance and reliability in complex driving environments. 
conclusion and discussion: 
 In this paper, we present ESCIRL, a novel approach that advances trajectory prediction in autonomous driving through evolving self-contrastive IRL. ESCIRL not only enhances prediction accuracy and robustness against diverse driving scenarios, outperforming existing methods by a large margin, but also demonstrates superior generalizability against DNN-based approaches which are known for better generalizability, while maintaining data efficiency as the other IRL-based approaches. In other words, ESCIRL demonstrates advantages from both worlds, which have not been seen in other existing works yet. We attribute the superior performance of ESCIRL to the unique bi-level optimization framework that blends both self-contrastive IRL and an evolving algorithm to offer a compelling balance of the trade-off. Moving forward, future work integrating more complex sensor data would be an interesting way to refine ESCIRL's understanding of more detailed real-world contexts. Additionally, given the successful demonstration of the feasibility for learning generalizable reward functions, it would be meaningful to further explore more unified and adaptive framework for trajectory prediction in the field of autonomous driving.IRL [29] problem seeks to infer the underlying reward function that maximizes the likelihood of these expert demonstrations, under the assumption that trajectories are exponentially more likely with higher cumulative rewards. This assumption is formalized by the Boltzmann noisily-rational model P (ξ) [31] as shown in (10) , where ω is the parameters of reward function R and β is a hyper-parameter to specify demonstration proximity [30] , which we consistently adopt β = 1 in this study. Notably, while previous works typically partition datasets into particular scenarios such as in-lane following [22] , lane-changing [32] , and merging [33] to assume a stable reward function across demonstrations, we engage with the complete dataset instead. In our study, the dataset contains multi-modal driving policies with distinguished distributions, and we aim to learn a continuous controller to handle diverse driving scenarios presented in the dataset.Our methodology employs a linear-structured reward function that is characterized by a specifically chosen feature space F(•), which is defined in relation to the trajectories ξ:EQUATIONAs a result, the probability, or likelihood, of the demonstration set is given by:EQUATIONIn this context, Z ω denotes the set of all feasible trajectories that correspond to the initial and terminal conditions delineated by ξ. The primary objective is to ascertain the optimal parameter vector ω * that enhances the average log-likelihood of the observed demonstrations, as shown below:EQUATION= arg maxEQUATIONFrom ( 4) and ( 5), we can see that the key step in solving the optimization problem in ( 5) is the calculation of the partition factors Z ω . In sampling-based methods, Z ω for each demonstration is approximated via the sum over samples in the sample set {ξ m i }, m = 1, 2, . . . , M , and ξ i is denoted as ξ 0 i :EQUATIONThus, the objective function in (5) becomes:EQUATIONThe derivative is thus given by:EQUATIONEQUATIONwhere F(ξ) defines the expected feature counts over all samples given ω.Note that an additional l 1 regularization over the parameter vector ω is introduced in the training process to compensate for possible errors induced via the selected set of features. 
 
 Appendix: For any trajectory ξ = σ i T -1 i=0 with T frames of states, where σ i = [x i , y i , θ i , s i , d i , v i ], we project it into feature space as follows.EQUATIONEQUATIONWe introduce f 1 , f 2 to prevent dramatic velocity changes. Essentially, f 1 calculates the average sum of squared differences in velocity between consecutive states, which is a measure of the smoothness of velocity over time. f 2 is similar to f 1 , but it takes the average squared difference in velocity over every two time steps instead of every one. This means it measures the change in velocity over a longer time span. By minimizing these two features, we can avoid jerky movements which may cause uncomfortable riding experience. C.2 Features for efficiencyEQUATIONEQUATIONEQUATIONf 0 sums up the squared velocity over the whole trajectory. By minimizing f 0 , the model penalizes abnormally high velocity, thus ensuring a more fuel efficient and environment friendly driving dynamic.EQUATIONv limit is the velocity limitation of the current road. To ensure the high efficiency of trajectory planning, we may want the vehicle's speed to be fast without violating traffic rules. By minimizing f 6 , we can ensure the vehicle's speed is close to the speed limit.EQUATIONEQUATIONEQUATION)EQUATIONI L and I R represent the ego vehicle whether to merge left or right, according to its destination.EQUATIONEQUATIONd targetL and d targetR are the distance from the ego vehicle to the left or right road's center line if it should merge left or right. s margin is a constant distance to measure the urgency of changing lanes. s end is the distance from the current pose to the endpoint of the current road.C.3 Features for safetyEQUATIONEQUATIONf 8 measures the deviance of vehicle's heading angle from the road's angle. Minimizing this feature keeps the vehicle on the right track.EQUATIONEQUATIONEQUATIONd (E-S F ) , d (E-S L ) , and (E -S R ), is the distance from the ego vehicle to the nearest surrounding vehicle in front, left, or right region respectively, and denoted as inf if no surrounding vehicle exits in such region.EQUATIONEQUATIONEQUATIONA collision point between the ego vehicle and the nearest surrounding vehicle in the front, left, or right region would be predicted respectively. We chose the Constant Velocity model as our prediction is the distance of the surrounding vehicle. A sample is shown in Figure 5 .Δs ! Δ𝑠 "! 𝑑 !#$! 𝑑 "!#$! 𝑑 !#"! Δ𝑠 "" 𝑑 !#$" 𝑑 ""#$" 𝑑 !#"" 𝑑 !#"#EQUATION)N coll is the number of collisions between ego vehicle and surrounding vehicles.D.1 Structure Design Given a trajectory (τ i ), a parameter ω i would be calculated through Contrastive IRL. We designed a Continuous Control Network (CCN) to map the relationship between the first states of trajectories and CIRL parameters, which provides a continuous mapping method for any given state.Let D = {(σ 0 i , ω i )} N i=1 be a dataset for state-parameter pairs, where the σ 0 i is the initial state of each trajectory. We first encode the initial state into a multi-channel context map M ∈ R c×200×200 with c channels and size (200, 200) . The first channel is road information near the ego vehicle; the second channel is the spatial information about the ego vehicle and surrounding vehicles, containing their positions, headings, and shapes; other channels are the kinematic information about the ego vehicle and surrounding vehicles, containing velocity, acceleration, distance to road center, etc. The kinematic information about the ego vehicle is also collected as ego kinematic data.The structure of the Continuous Control Network is shown in Figure 2a . The kinematic channels of the context map are fed into two layers' 2D convolutional neural network with kernel size (6, 6) , stride 2, and padding 2, and a 2D max pooling layer in the middle with kernel size (2, 2) and stride 2. The output of the last layer is 32 channels with the size (25, 25) , denoted as an attention map. The context map's road channel and spatial channel are fed into three layers' 2D convolutional neural network with kernel size (4, 4), stride 2, and padding 1. The output of the last layer (original map) is also 32 channels with the size (25, 25) and fused with the attention map by Hadamard Product. The original and fusion maps are extracted into feature vectors through the fully connected layers. The ego kinematic data is also extracted into a feature vector and concatenated with the other two feature vectors. Finally, the prediction of reward parameters ω pred mapping to the given initial state is predicted from the final FC layer by feeding the concatenated feature vector as input.To update the parameters of CCN, the loss function is denoted as follows:EQUATIONFigure 6 : The interactive scenarios of CitySim dataset (e.g. intersection, roundabout) that we investigate in this study [26] .While a vast array of datasets exists for autonomous vehicles, they often encompass a restricted range of real-world scenarios. To address the limitations in variety and depth of data, particularly acute in regions with less technological infrastructure, our model incorporates a contrastive process that enhances its capability to handle diverse real-world driving conditions with limited feature sets.To realize this, the datasets selected for training are characterized by extensive variability and a rich set of annotations necessary for accurately generating labels for subsequent training stages.Adhering to these selection criteria, we have curated a suite of datasets that provide comprehensive coverage of the myriad situations encountered on the roads.The INTERACTION dataset encompasses four categories of interactive driving scenarios collected from 11 different locations, provides a rich, diversified backdrop that reflects the variability of driving behaviors across the globe with driving scenarios from different countries and cultural contexts. This inclusivity ensures that models developed or tested against this dataset can be more easily adapted to the unique conditions of developing countries, mitigating the bias often found in datasets predominantly sourced from high-resource environments.CitySim. The selection of the CitySim dataset is predicated on its unparalleled contribution to safety-oriented research and applications within the domain of autonomous driving, particularly under the constraints of developing countries where safety-critical events and infrastructure limitations pose unique challenges. CitySim's comprehensive and meticulously curated vehicle trajectory data set the stage for innovative machine learning approaches that are both resource-efficient and highly applicable to real-world scenarios.These datasets are instrumental in overcoming the prevalent data scarcity and contribute to a more robust and versatile model training process. They ensure that the resulting algorithms are capable of interpreting and reacting to a wide array of traffic scenarios, making the advancements in autonomous vehicle technology more accessible and effective across different regions. 


Decision: The strengths of this paper are clarity of overall presentation, motivation, and presentation of results.  There were several weaknesses identified by the reviewers in the submission. However, during the rebuttal stage the authors have successfully addressed the reviewers' concerns and clarified the details improving the quality of the paper.

Reviews: review: title: title: This paper is generally well-written and structured. However, there are several areas that need attention and clarification.
paper_summary: paper_summary: This papert introduce a novel decoupled bi-level training framework that iteratively learns robust reward functions from complex multi-scenario expert demonstrations. This innovative approach has been experimentally verified using a public dataset
main_review: main_review: This paper is generally well-written and structured. However, there are several areas that need attention and clarification.

There is a formatting error in lines 77-79 when referencing sections 2.1-2.3 (§2.2). Please correct this to ensure proper referencing. Additionally, in line 82, why the function is defined as  𝑓(⋅)

In line 119, provide an explanation for why and how the parameter γ is set to 1. It is important to justify this choice to help readers understand its significance and impact on the results. 

Furthermore, in line 124, clarify why "higher probability in the positive set than the negative set" is chosen. Explain the rationale behind this decision and its implications for the overall methodology.

To improve the flow of the paper and aid in the understanding of the content discussed, include Figure 2(a) before Section 2.3.

Additionally, in lines 190-201, provide a detailed explanation of how the baseline algorithms are chosen for comparison. Discuss the criteria used for their selection and why these particular algorithms are relevant to the study.
 Finally, offer a thorough explanation and references for the metrics chosen to evaluate the algorithms in lines 202-207.
questions: questions: Please provide an explanation of how the baseline algorithms are chosen for comparison. Are there any other algorithms that could perform better?

Comments by the authors are acceptable. But suggested changes need to be fully addressed if accepted for the publication,
limitations: limitations: 1
review_summary: review_summary: This paper is well-written and introduces a new algorithm that has performed well compared to previous work. However, the limitations are not well discussed. Although there was a discussion about generalizability, the algorithm needs to be assessed on more datasets to make a stronger argument about the generalizability of the proposed algorithm
title: title: ESCIRL: Evolving Self-Contrastive IRL for Trajectory Prediction in Autonomous Driving
paper_summary: paper_summary: The paper introduces Evolving Self-Contrastive Inverse Reinforcement Learning (ESCIRL), a novel approach that integrates DNN and IRL for trajectory prediction in autonomous driving. ESCIRL employs a decoupled bi-level training framework, with a self-contrastive IRL module at the inner level and an evolving loop at the outer level. The approach aims to address data scarcity and generalizability issues inherent in DNN and IRL methods. Experimental results on the CitySim and INTERACTION datasets demonstrate that ESCIRL outperforms state-of-the-art DNN and IRL methods by 82.5% and 58.7% on average.
main_review: main_review: The paper introduces the Evolving Self-Contrastive Inverse Reinforcement Learning (ESCIRL) framework to address data scarcity and generalizability issues in the state-of-the-art DNN and IRL methods in trajectory prediction tasks. ESCIRL is a decoupled bi-level training framework with a self-contrastive IRL (SCIRL) module at the inner level and an evolving loop at the outer level. Experimental results on the CitySim and INTERACTION datasets demonstrate that ESCIRL outperforms state-of-the-art DNN and IRL methods by an average of 68.2%.

### Quality:
The paper explained the limitations of the existing DNN- and IRL-based approaches and proposed a well-thought-out framework to address these common challenges. The author provided details of the workflow, the algorithm, and the mathematical description for each component. The proposed solution was compared against five SOTA trajectory prediction approaches on two multi-scenario datasets and evaluated with multiple metrics. The results demonstrated the effectiveness of the proposed solution. In addition, the author conducted a detailed ablation study and interpreted each component. The findings are convincing.

(weakness) It would be helpful to highlight the incremental technical differences between the proposed and existing solutions in more detail, especially on what the SOTA solutions have already achieved.

### Clarity:
the paper is well-written and easy to follow.

### Originality:
Although the components in the framework have been studied in various applications, according to the authors, the proposed setting has not been seen in the state-of-the-art solutions in trajectory prediction tasks.

(weakness) It would be helpful to provide more literature on the proposed framework and components and elaborate on how previous works inspired the idea. 

### Significance:
Based on the difference between the proposed SOTA solutions and the experiment results, the proposed solution seems to improve trajectory prediction tasks significantly. 

(weakness) The authors need to justify why such a training framework combining SCIRL and evolving loop may bring benefits to other IRL applications. Has a similar framework been seen in other applications?
questions: questions: 1. What are the limitations of the proposed solution, e.g., complexity, training process?
2. Has a similar framework been seen in other Robotics or ML applications? Are there any literature on the framework?
3. Further highlight and elaborate on the novel parts of the proposed framework (Fig 1.) vs. existing solutions.
limitations: limitations: 2
review_summary: review_summary: The proposed innovative framework presents a significant advancement in trajectory prediction for autonomous driving by addressing the limitations of current DNN and IRL methods, particularly in data efficiency, robustness, and generalizability. The experiment setting, results, ablation study, and interpretation are convincing.
title: title: Novel idea with compelling results in data-scarce trajectory prediction
paper_summary: paper_summary: This paper presents EscIRL, an algorithm for trajectory prediction which aims to combine the data-efficiency of IRL with the robustness and generalizability of supervised learning approaches to trajectory prediction. The approach models a dataset of trajectory demonstration as a set of distinct policies, and aims to jointly learn the specific sets, as well as a reward function describing behavior in each set. In contrast to previous work, the authors make no assumptions on how to define these sets, but rather learn them directly on data using an evolutionary framework.
main_review: main_review: Strengths:
- The paper is generally well written and generally presents ideas clearly. 
- The algorithm is well motivated, and presents a novel approach to apply IRL to settings where the assumption that the demonstrations are explained by a single reward function does not hold.
- The experiments are comprehensive and successfully address the motivation of the paper, demonstrating the performance in data-scarce regimes, the improved generalization to held-out datasets, and how the algorithm scales to larger amounts of data. The results are compelling, showing impressive generalization in low-data settings while still scaling well with larger data. I would have liked to see this scaling experiment extended beyond the 12000 sample limit, as it appears that the transformer based HOME was still showing strong returns on scaling the dataset.

Weaknesses:
- The evolving framework was not explained very clearly. I would have liked to see the strategy laid out clearly in the algorithm block, but it was not clear to me that Algorithm 1 included it. What exactly is being mutated and crossed over? How is the number of contrastive sets chosen? 
- The paper is presented as targeting trajectory prediction, yet Algorithm 1 describes learning a policy $\pi$, which is not discussed in the body of the paper. 
- Driving trajectories may be multimodal, i.e., in a single state, there may be equally valid choices for future behavior of vehicles. For this reason, many trajectory forecasting papers sample multiple trajectories from the model and present minADE/FDE metrics over the top $k$ samples, to demonstrate how well the forecasts cover the range of possible behavior. Adding these metrics would strengthen the paper, as would a discussion of how this approach handles multimodality in future trajectory prediction.


Comments:
- Define $\mathcal{F}$ used in equation (2).
- Section 2.3, what does it mean for $\xi_{i}^- = \emptyset$? If it means that the negative sample set is chosen to be the empty set, then consider using notation to define these sets explicitly (rather than just defining the elements $\xi_i^-$ and cardinality $K^-$.
questions: questions: - Could you describe in more detail how the evolving strategy works?
- How sensitive is the approach to the choice of $K^+$ and $K^-$? How are these values chosen?
limitations: limitations: 1
review_summary: review_summary: I found the idea novel, results compelling, and would support acceptance assume authors are able to clarify details of their evolutionary approach.