Toward Exploratory Inverse Constraint Inference with Generative Diffusion Verifiers

Authors: Runyi Zhao, Sheng Xu, Bo Yue, Guiliang Liu
Primary area: reinforcement learning
Keywords: Inverse Reinforcement Learning, Generative Diffusion Model

Abstract:
An important prerequisite for safe control is aligning the policy with the underlying constraints in the environment. In many real-world applications, due to the difficulty of manually specifying these constraints, existing works have proposed recovering constraints from expert demonstrations by solving the Inverse Constraint Learning (ICL) problem. However, ICL is inherently ill-posed, as multiple constraints can equivalently explain the experts' preferences, making the optimal solutions not uniquely identifiable. In this work, instead of focusing solely on a single constraint, we propose the novel approach of Exploratory ICL (ExICL). The goal of ExICL is to recover a diverse set of feasible constraints, thereby providing practitioners the flexibility to select the most appropriate constraint based on the practical needs of deployment. To achieve this goal, we design a generative diffusion verifier that guides the trajectory generation process using the probabilistic representation of an optimal constrained policy. By comparing these decisions with those made by expert agents, we can efficiently verify a candidate constraint. Driven by the verification feedback, ExICL implements an exploratory constraint update mechanism that strategically facilitates diversity within the collection of feasible constraints. Our empirical results demonstrate that ExICL can seamlessly and reliably generalize across different tasks and environments. The code is available at https://github.com/ZhaoRunyi/ExICL.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730328123971

thread_type:
official_review

<thread_object>
creator: reviewer_56r9

<content>
summary:
The authors consider inverse constraint learning, and improve on previous work by constructing an algorithm that can generate a set of constraints, and verify those constraints by applying techniques developed in diffusion modelling for RL. In particular, the authors construct a guidance term that is the gradient of a set of feasiblity terms, which they can use for on the fly verification of the proposed feasilibity functions, thereby eliminating a costly second optimization loop. The authors test their proposed algorithm on a variety of RL benchmarks.

strengths:
- The technique makes clever use of the advanatages found in diffusion techniques: being able to modify the policy at run time by applying guidance terms
- The paper strikes a good balance of building a new method out of existing elements.

weaknesses:
The authors seem to omit some details of their mechanism, which I think are quite crucial to the paper. These are:
- how is reward treated? Is a separate reward model that is (1) differentiable, and (2) conditioned on diffusion time (i in the author's notation) trained following Janner et al? These details are not present in Alg. 1, but are necessary to evaluate the gradient p_Mc in eqns (9) and (10). 
- It is also not made clear whether in (9) and (10) the feasibility functions and reward are made to condition on diffusion time i, as I would expect it should since only tau_i is available at i.
- After algorithm 1 is complete, how is the final policy constructed for the experiments? Perhaps this is as simple as running eqn. (9) and (10) a final time, but this is not specified either. 
- After algorithm 1 completes, how are constraints chosen by the practitioner as the abstract says? How do the authors choose what constraints they apply when sampling their final evaluations? This is stated in the abstract but is not discussed in the paper at all. 
- How is constrained data collected? Is there an expert that already includes the constraint?

Minor:
- A few scattered grammar errors could be addressed

questions:
See also questions under "weaknesses"
- Do the authors have some intuition why their method seems to outperform baselines significantly for HalfCheetah, marginally for Limited-Walker and only ties for Blocked-Ant?
- In the MuJoCo experiments, is the reward presented in Table 2 the feasible reward? I.e. are rewards truncated after a constraint has been violated? It seems that that would be the more inveresting metric to report, I would recommend the authors report that metric, and if they already do so make it clear it is that metric.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668931729

thread_type:
official_review

<thread_object>
creator: reviewer_zyaa

<content>
summary:
The paper tackles the safe reinforcement learning problem using a diffusion model and guidance to train a set of feasibility functions. Unlike traditional inverse constraint learning, which is difficult to verify whether a candidate constraint is feasible and returns a single constraint, the paper's algorithm rapidly recovers a diverse set of constraints once the diffusion model is trained on expert data. The paper's algorithm outperforms baselines on constrained mazes and Mujoco experiments regarding performance and sample efficiency.

strengths:
- The idea of amortizing the ICL loop cost by pre-training a diffusion model is interesting.
- The paper provides convincing empirical results that show the superiority of their method compared to the baselines of their experiments, both for reward and cost. It also investigates how reliable feasibility functions are on expert non and non-expert data.
- While they have not directly demonstrated the advantages of having multiple constraint candidates returned by the algorithm (aside from possibly making search more efficient), this seems like a practical feature to have for real world use cases.

weaknesses:
- While the authors list computational concerns as one of the advantages Ex-ICL has over ICL, they do not conclusively show Ex-ICL's computational advantage. Figure 6 shows that Ex-ICL is more sample efficient in constraint inference, but a true test of computational efficiency should also take into account diffusion model training time.
- The experiments on maze and Mujoco are comprehensive but are fairly simple. For example, the baseline paper [1] includes a more realistic experiment on traffic scenarios.
- There's not enough detail in the main paper or appendix on methodology (how is \phi parameterized?)

[1] Guorui Quan, Zhiqiang Xu, & Guiliang Liu (2024). Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation. In Forty-first International Conference on Machine Learning.

questions:
- How are you selecting the constraint out of the constraint pool discovered by Ex-ICL for the experiment section?
- Why does Figure 4's Ex-ICL figure have so much larger variance for bad trajectory cost value than other methods?
- How sensitive are the results to exploration coefficient \delta and exploration round m? Also, would it be instructive to showcase model performance for Ex-ICL that only searches over a single \delta?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701035911

thread_type:
official_review

<thread_object>
creator: reviewer_x5jn

<content>
summary:
This paper proposes ExICL to tackle Inverse Constraint Learning problem, which aims to recover a diverse set of feasible constraints through an exploratory constraint update mechanism. The designed generative diffusion verifier utilizes the guided sampling strategy to verify the feasibility of explored constraints. This paper also aims to guarantee the robustness of feasible constraints discovery by accurately estimating the cost of noisy trajectory.

strengths:
1. Introduction clearly states the current issues in Inverse Constraint Learning and the related works section is complete.
2. The experiments are comprehensive, demonstrating the effectiveness of the proposed approach.

weaknesses:
1. The contributions claimed in this paper are not apparent to me. Contents in 4.1 is quite close to what has been proposed in [1], and the non-convex objective theorem is inherited from [2]; the ambiguity of how things are defined in section 4.2, 4.3 impairs the significance of contributions again. There are many math notations are not defined or briefly mention. I will list each of them below in the question section. I found it confusing and hard to see how the idea works.
2. Again, theorem 4.1 seems related to some existing conclusion from Paternain's paper [2], and this theorem is critical as it supports the zero duality gap for non-convex objective. The theorem stated in this paper is not quite the same as what is shown in [2], as the constraints here are not constant but are functions, but constants in [2]. There is supposed to be a connection shown here to support the theorem or a direct proof. A typo follows the theorem in Equation (9): $\lambda\epsilon$ might be missing at the end in the exponential term.

[1]  Janner, Michael, et al. "Planning with Diffusion for Flexible Behavior Synthesis." International Conference on Machine Learning. PMLR, 2022.
[2] Paternain, Santiago, et al. "Constrained reinforcement learning has zero duality gap." Advances in Neural Information Processing Systems 32 (2019).

questions:
1. My biggest confusion is about how the reward and cost are defined, respectively. Usually reward is defined as the negative cost if cost is positive, but in this paper, it seems not. Can you explicitly show how they are defined and how different they are?
2. In section 4.2, on line 286, how is $\phi_\omega(s_t^i, a_t^i, i)$ defined? 
3. In section 4.3, can you explicitly give the expressions for dist$[1, \phi_\omega(s_t, a_t)$ and dist$[\tilde\phi_\omega(s_t, a_t), \phi_\omega(s_t, a_t)])$?
4. In algorithm 1, ``Updating $\lambda$ by minimizing the loss $\mathcal{L} = \lambda \mathbb{E}_{\hat\tau\sim \tilde{p}_M}[c(\tau) - \epsilon]$, why is no reward term involved here to update $\lambda$? Another question related to this in Table 2: there is a significant discrepancy between the magnitudes of the Reward and Cost. Could you provide some insight into this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732266661385

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Updates and Global Responses

comment:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely appreciate your valuable feedback and insightful guidance. Your comments have been instrumental in significantly improving our work. In response, we have incorporated detailed clarifications, expanded explanations, additional experimental results, and improved figures into our revised manuscript (changes are highlighted in blue). A summary of the major updates is provided below:

1. More Details to Cost Value Model: Addressing the suggestions of Reviewers X5Jn and zyAA, we have clarified the U-Net-based architecture, the use of noisy trajectory inputs, the generation of pair-wise feasibility and trajectory cost value outputs, and the noise-robust nature of the model for guiding the denoising process.

2. Clarification on Constraints Selection: As requested by Reviewers zyAA and 56R9, we have detailed the $\delta$-based constraint selection process used in our experiments and further elaborated on the relationship between sparsity, the regularization parameter $\delta$, and its role in practical constraint selection.

3. Adding Experiment under Realistic Environment: Following the suggestion of Reviewer zyAA, we conducted additional experiments in a realistic autonomous driving environment to showcase our method's ability to handle complex real-world tasks.

4. More Experiments and Analysis on Cost Model's Performance: Addressing Reviewer zyAA’s suggestion, we have included additional experimental results illustrating the changing trend of cost model predictions as a function of the number of exploration rounds. In addition, to address the questions of Reviewer 56R9, we have provided a more in-depth analysis of the reasons behind the large variance observed in the exploratory cost models' predictions and the varying performance gap between our method and the baselines.

5. In-depth Explanation of Setting and Theory: Responding to Reviewer X5Jn's questions, we have provided a comprehensive explanation of the cost and reward definitions within our framework and detailed the duality optimization theory underlying our method's design.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732432721246

thread_type:
official_comment

<thread_object>
creator: area_chair_opsk

<content>
title:
Encouraging the reviewers to participate in discussion

comment:
Hello, I encourage the reviewers to participate in discussion with the authors! Please recall that this phase of the discussion period ends on  Nov 26th, AoE. 

To the authors eager to see author responses, I don't think there is anything to worry about yet. 3 days remain in the response period. Moreover it is currently the weekend (Saturday), and is thus understandable that the reviewers are not yet checking their OpenReview clients :)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper uses Diffusion models to develop new algorithms for in context reinforcement learning. While the sub-techniques are not novel, and there are no new theoretical contributions, authors generally appreciated the synthesis of existing ideas into a new application. Experiments showed promise, but reviewers felt that the domains could have been extensive, and had concerns that the proposed method did not uniformly outperform state of art. A more extensive set of experiments would be appreciated. 

Overall, this paper is a nice application of existing ideas, and all three reviewers are okay with acceptance. However, no single reviewer feels incredibly strongly given the limitations mentioned above. Hence, I lean towards acceptance as a poster.