Kati8/ICLR_2023_2025 huggingface dataset:
authors: Matt Barnes
authors: Matthew Abueg
authors: Oliver F. Lange
authors: Matt Deeds
authors: Jason Trader
authors: Denali Molitor
authors: Markus Wulfmeier
authors: Shawn O'Banion
reviews: 
Comments: This paper addresses the problem of route optimization. Given a set of demonstrations of chosen navigation routes that optimize a set of unknown preferences (e.g., concerning distance, traffic, pollution), the goal is to learn a model such that suitable routes can be suggested for (possibly unseen) route-destination pairs. The authors address this problem with inverse reinforcement learning, in which the goal is to learn the reward function underlying these preferences. Equipped with the reward function, routes can be suggested e.g. via finding the highest cumulative reward path between the source and destination.

The authors present a set of improvements over standard IRL algorithms, concerning an improved initialization of the MaxEnt algorithm,  learning separate reward functions per geographical region, and trading off between expensive stochastic rollouts and cheaper deterministic planners. The method is evaluated on a global dataset of routes in several cities, showing that the method compares favorably with other IRL algorithms.

## Post-response update
I am updating the score to 6 as a result of the discussion. I think the benefits of publishing the findings of this work outweigh the shortcomings.
Strengths: **S1**. The work successfully scales IRL to a large, real-world setting, indeed representing (to the best of my knowledge) the largest-scale evaluation of IRL.

**S2**. Furthermore, it provides an interesting perspective on the inherent challenges of global route optimization, for example regarding the "locality" of the learned policies, suggesting individuals navigate differently in different cities. This may have wider implications in other domains e.g. transportation science, neuroscience.
Weaknesses: **W1**. Methodological contributions: with the exception of the MaxEnt initialization findings, I am unsure of the value of the methodological developments. The geographical split into multiple experts and the graph compression are, in my opinion, both straightforward. I think simplicity is desirable, but the contribution is oversold. 

**W2**. Generalizability and reproducibility: given the repeated nods to engineering and deployment constraints, how generalizable and reproducible are the results? How many organizations face global scale routing optimization? While the achieved improvements are definitely impressive in terms of e.g. customer satisfaction, the contribution to the scientific community is not clear-cut, especially given that code and data (I assume) will not be released. Reproducibility and code / data availability are not even mentioned in passing.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper proposes MaxEnt++, an adaptation of the classical MaxEnt algorithm, to handle very large route instances with hundreds of millions of states and demonstration trajectories. Their techniques include MaxEnt++, a MaxEnt algorithm with a DIJKSTRA component, a new policy formulation that the authors call Receding Horizon Inverse Planning (RHIP), and a graph compression technique to reduce memory usage. The algorithm was then tested with a routing dataset of 200M states, showing some improvements compared to the standard MaxEnt and other baselines.
Strengths: The paper addresses an interesting problem. Learning with very large-scale routing datasets would have significant applications in modern transportation systems. The techniques used in the paper (except for MaxEnt, as I will discuss in the Weaknesses) are sound and relevant. The algorithm seems to work well (but again, the experiments lack comparisons with more scalable IRL algorithms, as I will discuss later).
Weaknesses: My biggest concern is that the paper primarily revolves around MaxEnt, which was developed about 15 years ago and is now very outdated. In the introduction, the authors state that MaxEnt is limited in its scalability, which is true. Recent literature on IRL has introduced many advanced algorithms to address this issue. For instance, Adversarial IRL [1] and IQ-Learn [2], value DICE [3] are well-known recent IRL algorithms that are much more scalable. Therefore, it is crucial to focus on these algorithms instead of the outdated MaxEnt.

[1] Fu, Justin, Katie Luo, and Sergey Levine. "Learning robust rewards with adversarial inverse reinforcement learning." ICLR 2018. 

[2] Garg, Divyansh, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. "IQ-Learn: Inverse Soft-Q Learning for Imitation." Advances in Neural Information Processing Systems 34 (2021): 4028-4039.

[3] Kostrikov, Ilya, Ofir Nachum, and Jonathan Tompson. "Imitation learning via off-policy distribution matching." ICLR 2019

I notice that the related work section exclusively references older papers and appears to be outdated. It would be beneficial for the authors to give greater consideration to more recent developments in the field of IRL/imitation learning.

This should be noted that the routing task is deterministic, so both online and offline IRL/imitation learning algorithm can be applied. The authors should look at relevant works and make a complete comparison.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper mainly focuses on scaling inverse reinforcement learning (IRL) for route optimization by learning the reward function from expert demonstrations. The application scenario is a popular route recommendation platform that should be able to generalize globally. Given a dataset of expert trajectories, in this approach, a reward function is learned from these demonstrations and this reward then guides an action selection policy from the start state to the destination. 

Building on prior work in IRL, particularly MaxEnt IRL, the authors propose an initialization strategy that leads to faster convergence, called MaxEnt++. Next, they generalize these and other IRL algorithms in their proposed framework called RHIP (Receding Horizon Inverse Planning) that trades-off using an expensive stochastic policy upto a horizon H with a cheap deterministic planner afterwards. Additionally, a number of parallelized computation and graph compression techniques are implemented to further improve the scalability of their algorithm for the application setting. Experiments on held-out validation trajectories show the superior performance of their method compared to prior work in IRL for quality route recommendations.
Strengths: 1. The authors address a well-motivated and useful application to show the statistically significant gains obtained from scalable IRL in route recommendation. The techniques that worked for this task have been clearly explained, along with explanations and evidence for some techniques that didn't work. 

2. The proposed method unifies several prior IRL algorithms through the RHIP framework for trading-off quality of route recommendation with convergence speed. This helps improve understanding of the similarities and differences in these approaches.   

3. Several ablation studies have been performed for different graph compression techniques and reward modeling approaches that help establish the significance of the experimental results.
Weaknesses: 1. The experimental results are not from real-time execution of the proposed method and utilizes static features of the road network for route optimization. Incorporating dynamic features, for example varying traffic flow throughout a day, planned or unplanned diversions and road closures etc. would increase the difficulty of obtaining a scalable DP approach.  

2. The reward function is learning a scalar value, whereas in the real world for applications like route optimization, it should intuitively be a multi-objective optimization problem. It is not immediately clear whether such possibilities would fit into the proposed algorithmic framework.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good