Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper explores the problem of active IRL, where the goal is to explore the environment to infer a good reward function. The paper addresses the Active IRL problem without assuming access to the transition model of the environment or the expert's policy. The paper proposes the Active Exploration for IRL (AceIRL) method and analyzes two sampling strategies. Finally, the paper evaluates AceIRL in 4 environments comparing normalized regret and sample efficiency to baselines.

strengths_and_weaknesses: Strengths:
* AceIRL tackles the important problem of Active IRL without any generative model. No assumption of a generative model is an important improvement over baselines like TRAVEL. In my understanding, AceIRL is the first method for Active IRL without a generative model.
* Two versions of AceIRL are studied and theoretically analyzed with sample complexity bounds. 
* The paper characterizes the goals of Active IRL with Definitions 1 and 2. Furthermore, the paper studies the impact of using learned generative models for the expert and transition model on the recovered reward in the finite horizon setting.
* The paper is the first to study Active IRL in the context of finite-horizon, undiscounted MDPs.

Weaknesses:
* In the related work, I would like to see more comparisons to how active IRL relates to the standard IRL problem. For example, what is the relationship between Active IRL methods and other IRL methods such as Apprenticeship IRL [4] or MaxEnt IRL [3]? Furthermore, in contrast to the claim on line 63, many recent IRL algorithms that use function approximation do not assume the underlying transition model is known, such as the f-IRL method [2]. Are these methods not applicable to the active IRL setting?
* How does Theorem 1 differ from Theorem 3.1 of [1] since both characterize the impact of error propagation from learned generative models? 

[1] A. M. Metelli et al. Provably efficient learning of transferable rewards. In Proceedings of International Conference on Machine Learning (ICML), 2021.
[2]  Ni, Tianwei, et al. "f-irl: Inverse reinforcement learning via state marginal matching." arXiv preprint arXiv:2011.04709 (2020).
[3] Ziebart, Brian D., et al. "Maximum entropy inverse reinforcement learning." Aaai. Vol. 8. 2008.
[4] Abbeel, Pieter, et al. "Apprenticeship learning via inverse reinforcement learning." Proceedings of the twenty-first international conference on Machine learning. 2004.
questions: * Why is the sample complexity of AceIRL Greedy higher empirically than Uniform Sampling (gener. model) in the Four Paths and Double Chain environments in Table 1 when the theoretical sample complexity bounds are the same? In Four Paths and Double Chain environments, the sample complexity of AceIRL greedy is over an order of magnitude higher than that of Uniform Sampling (gener. model). 
* How does the Random Exploration baseline work? I do not see it described in the text.
* Why does AceIRL Greedy underperform Random Exploration in the Four Paths environment? This is surprising since AceIRL Greedy tries to minimize the reward estimation error. 
* Why are the results on the environments from [1] so different from the first two in Table 1? The gap between AceIRL and baselines is much larger in the first two environments, and AceIRL greedy outperforms AceIRL Full in some of the settings from [1]. What is the reason for this difference?

[1] A. M. Metelli et al. Provably efficient learning of transferable rewards. In Proceedings of International Conference on Machine Learning (ICML), 2021.
limitations: The paper discusses that the method is limited to discrete state and action spaces, and future work will extend it to continuous states and actions. The paper also states the efficiency limitations of iteratively solving convex optimization problems in the algorithm. Potential negative social impacts are also discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the problem of active learning in explorations for inverse RL. And this paper provides necessary and sufficient conditions for solving such active learning problem.  They theoretical analyze the estimation error of the reward function caused by the error in transition function and expert policy. A novel algorithm AceIRL is proposed in this paper whose idea is to actively explore sample with large uncertainties for learning a better reward function. Some empirical experiment results are presented to support the theories. And the proposed method performs better than naive exploration strategies.  
strengths_and_weaknesses: Strengths: Overall, this paper is well written. The theorems are presented in a clear way. Sufficient number of theoretical analysis are provided to support the main result. I like the idea of applying a similar technique in UCRL to the inverse RL problem for estimating a better reward function. The proposed algorithm makes a lot of sense to me. The problem considered in this paper is promising since it is important to select samples in a efficient way for estimating the reward function. When the reward function is hard to learn, the proposed method has significant advantage over other random sampling based algorithm. 
Weaknesses: The empirical result in this paper is not sufficient to show the benefits of active learning for IRL problems. 
questions: I have a concern regarding the scalability of the proposed algorithm to large state and action spaces MDPs. The proposed transition model and expert policy model are calculated by using the frequency which is actually not sample efficient. And it may lead to a large variance estimation although they are unbiased. Can you provide some explanation for this problem?   

And the idea of focusing on plausibly optimal policies is a good way to reduce sample complexity. But in the real experiments, there is no description for how you choose the optimal policy set. It will be great if you can present some details for your experiment. 

In the experiments, only sample complexity is considered. It will be good if you can include the comparison of estimation errors. 

limitations: Yes. The authors provides adequate information for this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The idea is to drive exploration to build the ideal set of demonstrations so that any IRL algorithm can reconstruct a reward function that leads to a policy as close as the optimal one, with a fixed limited number of examples.
The authors first set up a PAC criteria that their algorithm must verify in order to converge.
Then, they analyse three strategies
- A (oracle) uniform sampling as baseline
- A greedy exploration that picks transitions that reduce the IMMEDIATE uncertainty about the value associated with ANY policy.
- A smarter exploration that picks transitions that reduce the FUTURE uncertainty about the value associated with THE OPTIMAL policy.

They compare theoretical complexities and empirical performances of these different strategies.
strengths_and_weaknesses: The paper is nicely written, the methodology is well driven and brings a solid solution to the problem. 
I did not go through the proof, but the mathematical claims look intuitively correct. 
The empirical results are reflecting the theoretical findings, and they are verified over 50 seeds. 

I would only have minor critics:

1) The abbreviation PAC (Probability Approximately Correct) is never written without abbreviation (so I had to check the meaning in another paper).
2) In Theo. 1, A and V  are associated with which MDP’s reward? (the ground truth or the approximation?)
3) Detail: in lemma 3, “theorem 4” should be “theorem 1”.
4) L209: if you use the word “straightforward”, you have to at least explain why it is straightforward (in this case, triangular inequality with bellman equations?)
5) L256 Why is it “surprising”?

questions: Cf minor critics above.
limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: this paper proposes AceIRL, which is an IRL algorithm with active explorations in unknown environments. Authors give some novel analysis on this newly proposed method and empirically, AceIRL outperfroms some naive exploration methods. 
strengths_and_weaknesses: Strengths:

novle algorithm to explore the environment and the expert policy to infer a good reward signal.

provides two exploration strategies for AceIRL, and the algorithms are sound


Weaknesses:

the experiments seem not enough for evaluation. why not conduct experiments on some complex environment like MuJoCo?
questions: why not conduct experiments on some complex environment like MuJoCo?
limitations: No
review_title: null

Meta-review: metareview: This submission is solid as a work that introduces a novel IRL approach and analyzes its theoretical underpinnings. Its evaluation is done on very much toy problems though, and the metareviewer suggests reinforcing it with more complicated benchmarks, e.g., low-dimensional MuJoCo-based ones used in D4RL, to demonstrate that it works well even in continuous-state/-action settings.
confidence: Certain
Decision: Accept