Action-Constrained Imitation Learning

Authors: Chia-Han Yeh, Tse-Sheng Nan, Risto Vuorio, Shao-Hua Sun, Ping-Chun Hsieh
Primary area: reinforcement learning
Keywords: action-constrained reinforcement learning, imitation learning, safety

Abstract:
Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications.
In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where an action-constrained imitator aims to learn from a demonstrative expert with larger action space.
The fundamental challenge of ACIL lies in the unavoidable mismatch of occupancy measure between the expert and the imitator caused by the action constraints. We tackle this mismatch through $\textit{trajectory alignment}$ and propose DTWIL, which replaces the original expert demonstrations with a surrogate dataset that follows similar state trajectories while adhering to the action constraints. Specifically, we recast trajectory alignment as a planning problem and solve it via Model Predictive Control, which aligns the surrogate trajectories with the expert trajectories based on the Dynamic Time Warping (DTW) distance. Through extensive experiments, we demonstrate that learning from the dataset generated by DTWIL significantly enhances performance across multiple robot control tasks and outperforms various benchmark imitation learning algorithms in terms of sample efficiency.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729860512314

thread_type:
official_review

<thread_object>
creator: reviewer_yerc

<content>
summary:
This paper proposes action-constrained imitation learning (ACIL), a new imitation learning algorithm for action-constrained imitators to learn from demonstrations. The authors propose DTWIL to solve this problem by first replacing the original expert demonstrations with a surrogate dataset that follows similar state trajectories and then recasting trajectory alignment as a planning problem and solving it via Model Predictive Control. Through experiments in both navigation and locomotion tasks, they show the effectiveness of proposed method.

strengths:
1. The problem formulation of this paper is novel for tackling action-constrained imitation learning. It tackles the problem by generating demonstration data that adheres to the action constraints.
2. The quantitative experiment results are good.

weaknesses:
1. The motivation requires to be further explained. I am not fully convinced that action-constrained imitation learning is an important problem in a wider range of tasks. The author should give more examples for this point. 

2. Also, what is the difference between action-constrained imitation learning and cross-embodiment imitation learning? How do they tackle this problem? The author should elaborate on this more in the introduction/related works sections.

3. From my point of view, the proposed method is based on such an important assumption: the surrogate demonstrations generated by Trajectory Alignment (Section 4.1) must solve the task (or achieve high rewards), so that the imitator can solve the task by doing BC on this data. However, this assumption is not always true for general imitation learning tasks. The author should provide more analysis for this assumption to show what kinds of tasks meet this assumption and what tasks do not. Also, how does the "box constraints" coefficient affect this assumption? For example, if the box constraint of the maze task becomes *action<-0.5 and action >0.5*, can the method also generate good surrogate demonstrations?

4. Why do the IRL and LfO methods (such as GAIL) require a +P operation? Can they directly use the constrained action space for IRL/LfO?

5. Although the proposed method has better interaction sample efficiency than IRL/LfO methods, the author should also show if IRL/LfO methods can solve this task with more interactions, and how many interactions they need to train a successful policy.

6. I doubt if directly using MPC with constrained action space can solve the task with the proposed DTW metric as the (negative) cost function. 

7. The stability of the proposed method for other tasks is doubtful. The proposed method requires task-specific operations to make the method effective such as normalization (Section 4.1.1) and time-step alignment (Section 4.2), as well as the $\beta$ hyperparameter.

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729947366228

thread_type:
official_review

<thread_object>
creator: reviewer_obyj

<content>
summary:
The paper proposes a new setting of action-constrained imitation learning where an action-constrained imitator aims to learn from a demonstrative expert with larger action space. The authors show that using behavior cloning to imitation the behavior in the dataset followed by deploying the actions with constraints is insufficient to perform the tasks in their experiments. Accordingly, they propose DTWIL for improving the policy performance in such action constratined settings.

strengths:
- The paper address a new domain of action constrained imitation learning to enable agents to address environmental or physical constraints when performing a task without have the constraints reflected in the expert demonstrations.
- The algorithms uses DTW-based trajectory alignment for modifying the expert trajectories with the agent constraints and perform BC on this modified dataset to obtain policy. The authors also include Actor Regularized Control to improve the alignment’s effectiveness.
- The paper carries out experiments on 3 simulated tasks and provides ablation studies across hyperparameters to justify their design choices.

weaknesses:
- I am a bit confused about the action constrained setting that the paper operates in. For the experiments shown in the paper, the demonstrations are collected with the same agents in the same environments as at test time. So why should additional action constraints be added during deployment when they are not need during demonstration collection? It would be great if the authors could provide a real world example of a scenario needing the introduction of such action constraints during deployment.
- I am a little confused about the algorithm. From what I understand, given some expert demonstrations, these demonstrations are modified through online interactions with the environment. Some questions based on this - (1) In Algorithm 1 Line 3, does each iteration correspond to a trajectory rollout or one step of action in the environment?, (2)  From Eq. 2, it seems like the action sequence optimization is done over a planning horizon. However, Line 6 in Algorithm 1 makes it seem like the whole trajectory is aligned in one go. So does this trajectory alignment involve multiple alignment steps and actions in the environment? (3) In Line 5 of Algorithm 1, its seems like a new forward dynamics model is trained for each iteration on the updated training data. How long does this take? I reckon this might make training slower. (4) In case each iteration corresponds to one environment action step, is a new trajectory sampled at each iteration after having taken action(s) based on a different trajectory in the previous iteration? It would be great if the authors could provide some clarifications regarding this.
- There is a missing reference in line 231.
- In Table 1, for HalfCheetah HC+O, BC+P is the best  performing method but DTWIL is still boldened. Similarly, in Table 3 for DTW-S, Hopper Box-Sync outperforms Hopper Box but Hopper Box is boldened. This is confusing to the reader and beats the purpose of boldening the results.

questions:
It would be great if the authors could address the questions/concerns mentioned in Weaknesses. I am willing to increase my score once these concerns have been addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730152766099

thread_type:
official_review

<thread_object>
creator: reviewer_wdkr

<content>
summary:
This paper addresses the setting of action-constrained imitation learning, an imitation learning problem where the learned agent must use an action space that is more restricted than the one used by the expert demonstrator. It proposes DTWIL, a method that uses a combination of Model Predictive Control, Dynamic Time Warping, and Behavioral Cloning, to train agents in this setting. The paper shows that the method outperforms several baselines across 3 tasks.

strengths:
The problem setting introduced by the paper is interesting and not widely studied (at least to the best of my knowledge). The paper is well-written and easy to follow. The experiments section compares against a large number of baselines, all of which are reasonable, and the results showcase the effectiveness of the approach.

weaknesses:
There are a few critical weaknesses that should be addressed.

**Limited Experiment Results.** While I appreciate the thoroughness of the results in terms of the number of baselines, the number of tasks shown are small and do not have much variety. For example, Half-Cheetah and Hopper are both locomotion tasks, and relatively simple in complexity (RL approaches can solve these tasks very efficiently from scratch). It would be great to see more settings such as robot manipulation tasks -- there are a wide number of suitable datasets and benchmarks available today (e.g. [robosuite](https://robosuite.ai/), [RLBench](https://sites.google.com/view/rlbench), [ManiSkill](https://www.maniskill.ai/home), and others). It is important to show that the method is general-purpose and easy to apply to many scenarios. Similarly, BC+P seems like a very strong baseline (from Table 1) -- seeing results across more tasks and settings to highlight the value of proposed method would paint a more complete picture.

**Method Limitations.** The ARC (Section 4.2) seems like a hack, and could require per-task tuning, which is undesirable. It is also unclear if it's a good idea to always use blended actions up to a certain timestep, compared to other alternatives for incorporating expert actions, like using the notion of residual additive actions (for example, https://arxiv.org/abs/1812.03201). Including more tasks could help show that one set of parameters works well across multiple settings. It also seems like this method is only suitable for imitating a single specific trajectory, in contrast to typical scenarios where an agent must deal with a variety of initial conditions (such as a robot that needs to manipulate objects that start in diverse configurations on a table from episode to episode). Section 6 mentions "as long as the agent is able to be initialized to the same starting state as the expert" -- this is a severe, and often impractical assumption to make. Finally, it seems like the method implicitly assumes full state observability (e.g. privileged information) compared to partially observed settings (raw sensor data such as images or depth sensing), which also impedes the practicality of the approach.

**Some Writing Issues.** Section 4 does not adequately describe the overall approach of how BC is used with MPC. This needs more details -- I had to figure this out from the Algorithm 1 pseudocode. It also isn't clear why BC is needed versus directly using MPC for control -- some further experiments might help point out the value of using BC.

**Minor Issues.**

- I suggest organizing Related Work further, with a bolded title for each paragraph at least.
- line 231, undefined Figure reference
- Section 5 beginning - "both offline baselines online baselines"

questions:
See "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730355852983

thread_type:
official_review

<thread_object>
creator: reviewer_viqr

<content>
summary:
This paper proposes a novel problem setting called Action-Constrained Imitation Learning (ACIL), where an imitation learner's action space is a subset of an expert's action space. The paper proposes DTWIL, which (1) generates feasible trajectory for the imitator by using Model Predictive Control (MPC) with Dynamic Time Warping (DTW) as its objective, and (2) performs behavior cloning from the generated trajectories. Experimental study showed that, existing imitation learning (IL) methods with a naive projection approach suffer from low performances in ACIL settings, and DTWIL outperforms these methods.

strengths:
- The proposed paradigm, ACIL, is an important research direction. It must have many potential applications and a large group of potential audiences (originality, significance).
- The high level idea of DTWIL seems reasonable and novel, and the experimental study showed that it indeed performs better than baselines in ACIL problems (originality, quality, significance).

weaknesses:
Besides the strengths above, the presentation quality of the paper is not good in general. The followings are my concerns.

### Major concerns
- The exposition of DTWIL is not self-completed and clear enough. For example,
  - At L.231, there is a statement "The pseudo code for trajectory alignment is presented in __??__" and the pseudo code for __trajectory alignment__ is missing in the paper.
  - The definition of DTW distance in Eq. (2) is not provided, though its high level idea is stated in page 4.
  - Since trajectory alignment algorithm is missing, it is not clear how the action constraints are handled practically. I conjecture that Eq. (2) is solved by a constrained optimization problem.
  - The definition of $\bar{S}(A,f_\theta)$ is not concrete. Does this sequence start from $\bar{s}_{t_{\rm pg}}^e$?
  - How $t_{\rm pg}$ is updated in practice? Figure 3 explains only discrete cases. For continuous spaces, I suppose that we need to compute the distance of states by some metric and determine by a threshold, which must be an additional hyper parameter to be tuned.

- I am not convinced of the validity of Actor Regularized Control (ARC). In my understanding, the expert actions before projection, $a^e$, are sampled from the dataset. On the other hand, $a^{\rm sampled}$ are computed for states in the generated trajectory. Therefore, the states for which $a^e$ and $a^{\rm sampled}$ are sampled are different by construction. Mixing these different-state-dependent actions seems not valid.

- Ablation study is not comprehensive enough. Since the paper describes the importance of excluding the final expert state in Figure 3, the reader may expect its experimental impact.

- The time complexity of the naive DTW algorithm is O(NM), where N and M are the lengths of the two input sequences. I suppose that DTWIL has a drawback in the computational complexity compared to the baselines, which might hinder the applicability of DTWIL to real world applications. I think that it is necessary to compare the computational time with baseline methods.


### Minor concerns
- For Maze2d, state-dependent constraint is not stated.
- Section 5.6 and Appendix A.3 look exactly the same.


## After reading author responses and revised manuscripts
- As of 11/25: Thank you for revising the paper. The presentations are improve largely. I raise the score from 3 to 5.
- As of 12/03: Thank you for clarifications. I raise the score from 5 to 6.

questions:
### Questions
- How the action constraints are handled practically in DTWIL? Is Eq. (2) solved by a constrained optimization problem?
- How ARC in Eq. 3 is justified?

### Suggestion
- Please address my concerns, especially in the exposition of DTWIL, and update the manuscript in Discussion Stage. In the current form of the paper, the contributions are not clear. Considering the strengths of this paper, I am happy to re-evaluate after revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies a novel problem formulated as action-constrained imitation learning where the imitation learning agent has access to only a subset of the demonstrator's actions. The proposed learning algorithm leverages dynamic time warping to measure discrepancies between trajectories, generates surrogate data with actions from the learner's action space using model predictive control, and then train a behavior cloning policy with the surrogate data. 

The reviewers agree the problem studied in this paper is novel and can have a high impact on robotics applications, but raised concerns about the assumption the proposed algorithm makes, as well as the lack of sufficient evaluation on benchmark tasks and comparison with state-of-the-art baselines. Reviewers suggest that incorporating additional empirical results and testing in realistic task settings would strengthen the paper.

In summary, while the paper presents a novel problem and a plausible solution, further empirical validation and broader evaluations are necessary to fully establish its efficacy and applicability.