Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The submission proposes a novel IRL algorithm.  
Instead of prior methods optimizing maximum entropy objective or adversarial objective for reward learning, the proposed method (BC-IRL) seeks reward function that makes a policy updated from the reward be close to expert behavior.
To optimize such an objective, the few-step updated policy parameters are bi-level optimized with respect to reward parameters.
Learned reward function shows improved generalization ability than existing IRL methods and thus it can able to train more robust policy.
strength_and_weaknesses: ### Strength
- The proposed method meaningfully improves the performance of previous imitation learning methods.
Both the generalization quality and robustness of learned policy are consistently outperforming prior imitation learning methods.  
- The qualitative results clearly depict the benefit of the method and how it can improve generalization power.
- Paper is well written and easy to follow.

### Weakness
- As clearly described in the paper, BC-IRL should be used with a backbone RL algorithm that can update policy using reward parameters.
This limitation will exclude a meaningful portion of conventional RL algorithms.
- Evaluations are done in a relatively short-horizon and less diverse set of the domain. Few more results on other domains can strengthen the experimental support.
clarity,_quality,_novelty_and_reproducibility: This paper has high clarity and quality.
All text and graphics are well-polished and structured.
Code, hyperparameters, and experimental details are provided for reproduction.
summary_of_the_review: This paper is very clearly written and the proposed method is strongly supported by experimental results.
There is a methodological limitation but it doesn't outweigh the benefit of sharing this idea with the community.
Thus, I vote to accept this paper, and I'd be convinced stronger if I can see results in a few more challenging evaluation domains.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new form of IRL in which the learned reward is not based on the state occupancy, but is instead computed in order to lead a policy-gradient learner at imitating the demonstration.
This is done with a meta-learning approach, in which the inner loop updates the policy (with a policy-gradient objective) and the outer loop updates the reward (with a BC objective).
They observe that such a reward that is not based on occupancy is better at generalising behaviours and is more robust to changes of the initial distribution.

strength_and_weaknesses: The method is novel, easy to understand and implement and the results looks impressive.

I however have a concern regarding the main tool-experiment as shown in Figure 1 and 2.
Even if the reward is only on the states that are occupied by the demonstration (resulting in a X-shape), the learned behaviour with such a reward should still learn to reach the highest rewarding states. 

For example, a Value-function associated with such a reward would diffuse values outside of the X cross.
In this experiment there is no value-function, but they use PPO that directs policies updates with generalized advantage estimators which behave exactly like a value-function (it is a non-biased estimator). Therefor the fact there is no reward outside the cross should not significantly affect the optimal behaviour.

I would understand that the reward obtained by BC-IRL is much more dense, and so accelerates the learning. But AIRL well implemented should not fail on this task (at least with enough learning steps). Does AIRL ends up learning the good behaviour after a (much) longer training?


clarity,_quality,_novelty_and_reproducibility: The paper is well written, the method is simple to understand and well described. 
I have never seen such an approach for IRL before.

summary_of_the_review: I think this paper is definitely novel and shows impressive improvements of SOTA approaches, but I have a strong concern regarding the implementations and behaviour of the baselines. In that doubt I'm giving a low score, but I am ready to significantly increase it if authors adresse my comments with a convincing explanation.


After reading authors answer and revisions, I've increased my socre.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Providing informative rewards is crucial for effective reinforcement learning. Prior IRL methods overfit to demonstrations and fail to learn generalizable rewards. To combat this issue, this paper proposes BC-IRL, which uses gradient-based bi-level optimization to learn the reward. The authors evaluate on two continuous control tasks against IRL and imitation learning methods.
strength_and_weaknesses: Strengths:
- The paper proposes a novel algorithm for learning rewards with bi-level gradient-based optimization.

Weaknesses:
- Unclear in what sense the authors are referring to when they use the term “generalization”. In the experiments, the generalization seems like it is limited to slightly different start and goal distributions. This should be made clearer.
- GAIL is a method that trains a policy that matches the state-action distribution of the expert data, it should be clarified how the goal here is different since this method also aims to learn a policy that matches the behavior of the expert.
- Experimental evaluation is very limited. Only 2 tasks of a similar nature are evaluated, and the tasks are reaching, which is very simple. Also, the environments have a low-dimensional state space, despite the simplicity of the tasks.
- How does this compare to AIRL or GAIL where the discriminator learned uses mixup regularization or spectral norm? These are 2 common techniques (among others) for making the discriminator less brittle. Much more experimentation is needed to conclude how much more generalizable the rewards learned by BC-IRL are.

clarity,_quality,_novelty_and_reproducibility: Clarity and quality need to be greatly improved, see above for more detailed discussion of weaknesses.
summary_of_the_review: Due to issues with clarity and quality, this paper does not seem ready for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes a new technique based on bi-level optimization for inverse RL.

Strengths:
* Really novel technique for IRL based on bi-level optimization
* The proposed technique mitigates the overfitting issue experienced by many other inverse RL techniques
* Meaningful improvement iover state of the art RL
* Paper is well written and easy to follow.

Weaknesses:
* Limited experimental results on a few environments
* Questionably low results for some of the baselines

Overall, this is very interesting work that makes an important contribution to the state of the art of inverse RL with an innovative approach that mitigates overfitting that plagues previous inverse RL techniques.
justification_for_why_not_higher_score: * Limited experimental results on a few environments
* Questionably low results for some of the baselines
justification_for_why_not_lower_score: * Really novel technique for IRL based on bi-level optimization
* The proposed technique mitigates the overfitting issue experienced by many other inverse RL techniques
* Meaningful improvement iover state of the art RL
* Paper is well written and easy to follow.
Decision: Accept: notable-top-25%