Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for learning soft constraints in sequential decision making problems that can be modeled as MDPs, under the assumption that the entire MDP model is known (both transition probabilities and the reward function), but a constraint on the state-action space exists that is known only to a decision maker who can provide optimal decisions that respect this constraint, and the objective is to recover the constraint from the observed decisions of the expert. Empirical evaluation on several domains demonstrates that the proposed method is far superior to estimating the constraint than two existing methods. 
strength_and_weaknesses: The paper proposes a principled way of addressing the problem, borrowing from the computational principle and machinery of inverse reinforcement learning (IRL) methods, and the empirical evaluation appears to be convincing. The limitation that there is only one constraint is probably not significant, as this constraint could potentially be an arbitrarily complex function on state/action pairs. (Unlike many IRL algorithms, where the learned reward function is linear in the vector of parameters.) It appears that the chosen parametric form of the constraint function is a neural net with two hidden layers of 64 units each, ReLU activations, and final output with a sigmoid activation function (Table 5). Why this form, and is it really sufficient to represent arbitrary functions? What about the constraints in the chosen experimental environments, how well does it represent its constraints? Maybe learning the constraints directly from the known functions by means of supervised learning could answer this question, and also effectively provide a lower bound on the MSE that this method can provide?  

Another thing I was not clear about is the effect of the initial form of the constraint (that is, c in Equation 3 on the very first iteration of the algorithm). Was it completely random in the computational experiments, and does its value matter? In other words, does the algorithm always converge to a global minimum, or there are local minima?

Some minor typos:
P.1: "are therefore are" -> "therefore are"
P.3: "in an other line of work" -> "in another line of work"


clarity,_quality,_novelty_and_reproducibility: The proposed method appears to be novel, and very different from the few previous methods known for solving this class of problems. The paper is written very well and is largely self-contained. I would suggest, though, to explain in plain text the logic behind the constraints in the experimental environments. For example, what is the meaning of the constraints on the cart pole systems in Fig. 10, and what is the optimal policy under them? It is difficult to figure it out just by looking at the plots of the constraint function. (And, it does not help that the coordinate axes do not have labels or dimensions.)
summary_of_the_review: The paper proposes a novel method for recovering soft constraints in sequential decision problems, where the MDP model is known, and expert demonstrations are available under the unknown constraint. The empirical evaluation demonstrates that the method has much lower MSE in recovering the unknown constraint that two existing baseline methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel framework to learn soft constraints for ICL. The paper gives a theoretical formulation with proof of convergence, and then provides a practical algorithm with relaxation on the theoretical formulation and several techniques. Experiments on synthetic and real-world environments show improvements of the proposed method compared to baselines. Some ablation studies are provided to verify the design choices of the algorithm.
strength_and_weaknesses: Strength:
The paper provides a novel framework with a theoretical formulation and a practical algorithm with reasonable relaxations. It also provides empirical results to show the improvements.

Weaknesses:
1. For the definition of 'soft' constraint, as described in future work, I also feel that the constraint through expectation is not as useful as constraint through probability for stochastic environments. Can the author give more explanation of where the current setting could be applied in real life?

2. It would be great if the author could explain more about how this 'soft' constraint is guiding the algorithm design. 

2. The experiment results seems not strong enough to me. There is only one real-work environment and the setting is also relatively simple. It would be good to provide more complex experiment setups (e.g. robotics manipulations).
clarity,_quality,_novelty_and_reproducibility: The overall clarity, quality, and novelty are good. Can't comment on the reproducibility.
summary_of_the_review: I think the paper provides an interesting framework with a practical algorithm. Each part of the paper is 'good' but not 'great' with the weaknesses mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an inverse RL algorithm for recovering soft constraints from expert trajectories. The algorithm takes a nominal reward function, and expert trajectories as input, and outputs a recovered constraint function that best matches the behavior of the expert trajectories. The proposed algorithm interleaves between two steps: 1) solving a constrained RL problem with the latest constraint function, and 2) adjusting the constraint function to maximize the constraint value among all the previously obtained policies, subject to that expert trajectories satisfying the constraint. A theorem is provided to prove the convergence to the expert policy and a practical algorithm is provided that shows empirically successful results on three constraint learning tasks. The proposed method shows better performance in terms of constraint satisfaction as well as similarity to expert trajectories than baseline methods on the selected tasks.
strength_and_weaknesses: Strengths:
The problem of learning constraints is an important one and a solution to this problem can be impactful
The theoretical framework is helpful in gaining insight of the algorithm as well as providing theoretical justifications for the proposed algorithm
The proposed method shows decent performance on the selected tasks

Weaknesses:
The requirement of known reward function and single constraint is understandable but could limit the applicability of the algorithm to real-world problems.
The tasks used in the work are relatively simple.
The proposed computational algorithm seems quite complex with various tricks, such as the reweighting scheme, the two-stage constrained optimization with different weights, etc. This makes it potentially difficult to be applied to a wider variety of tasks, especially ones that are notably more complex.

clarity,_quality,_novelty_and_reproducibility: good
summary_of_the_review: My high-level concerns can be seen in the weaknesses section. One more detailed comment: the baseline method ICRL was applied to several more difficult continuous control tasks. It would more convincing if the proposed method was compared on similar tasks with the baselines.

Also, it is mentioned that the proposed method requires more training time than the baseline methods but I didn't find any numbers (I might have missed it). Could you provide some more details about how much slower it is?

In general the proposed method seems reasonable and attempts to solve an important problem. If the concerns above could be addressed it can be a good contribution to the ML community.


==============
post rebuttal: I have updated the score from 5 to 6 after reading other reviews and the authors' rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of inverse constraint learning, whose goal is to learn the unknown (soft) constraints that the expert demonstrations obey in a given expert dataset, assuming that the reward function is known. The authors propose a novel method named ICL, which learns a constraint function in a way that the total constraint is bounded for the expert demonstrations but maximized for agent behaviors. They demonstrate that their proposed method outperforms two baselines on three environments.
strength_and_weaknesses: **[Strengths]**
- The paper is well-written and easy to understand.
- The proposed approach is theoretically well-motivated and sensible.
- The experiments include ablation studies.

**[Weakness]**
- The main weakness of the paper is its limited empirical evaluation. The authors tested their method only on two toy environments (GridWorld and CartPole) and the HighD environment, which also seems relatively simple as it has a 1D action space (and presumably a 2D state space). Given the complexity of the method, it would be desirable to include more extensive comparisons on diverse, more challenging domains (e.g., constrained MuJoCo environments as in the ICRL paper).

**[Questions]**
- Is the constraint function $c$ in the paper a per-step function (i.e., $c(s_t, a_t)$) or a per-trajectory function ($c(\tau)$)? If the former is the case, there should be infinitely many ways to assign per-step constraints to the expert trajectories because the only constraint is that their sum should be bounded by $\beta$. How does the method deal with this?
- I didn't understand the CartPole figures in Fig. 10. What do the "a=0" and "a=1" lines mean? What are the axes?
- How is the state space defined in the HighD environment? How does the agent observe the cars in front of it? How are the expert data collected?
clarity,_quality,_novelty_and_reproducibility: **[Clarity & Quality]**
- The paper is mostly well-written and easy to understand.
- Detailed descriptions of the expert demonstrations used for their experiments are lacking.
- It would be better to include the experimental details (in Appendix C.1) in the main paper.

**[Novelty]**
- The proposed method is novel to my knowledge.

**[Reproducibility]**
- The authors have released their code.
summary_of_the_review: While their proposed method is well-motivated and sensible, I cannot recommend acceptance in its current form due to its limited experimental evaluation. I would be happy to raise my score if the authors address my concerns above.

-------------(11/16 Update)-------------

I raised my score from 5 to 6 as the response addressed my initial concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses a variant of the inverse reinforcement-learning problem in the context of constrained MDPs.  In this setting, there is a generic known reward function but the interesting problem is to infer the constraints that the demonstrator is operating under.  

I believe this is an important setting which may be more useful for robotics than "standard" IRL, because constraints are both more generic (keeping my waterglass upright can be done with all kinds of different robots and strategies) and composable (if I can infer two constraints from two different sets of demonstrations, I can independently solve for a strategy that tries to satisfy both of them.)  The paper is well written and reviewers find it to be novel.  The authors provided a very nice list of applications in which this setting is appropriate.

Initially there were weaknesses with respect to the experimental validation, but the reviewers suggested additional experiments and the authors carried them out effectively.  Some additional concerns about writing and formulation were largely addressed in rebuttals.



justification_for_why_not_higher_score: The paper doesn't seem to be sufficiently ground-breaking to justify an oral presentation.  
justification_for_why_not_lower_score:  I think the setting is a good one and the method seems smart and the experiments good.  The reviewers were on balance positive.

Poster would also be fine, but I feel it should be accepted.
Decision: Accept: notable-top-25%