Action-Constrained Imitation Learning
Authors: Chia-Han Yeh
Tse-Sheng Nan
Risto Vuorio
Wei Hung
Hung Yen Wu
Shao-Hua Sun
Ping-Chun Hsieh
Venue: ICML 2025 poster (2025)
Keywords: Imitation Learning; Action Constraints
Primary area: reinforcement_learning
TL;DR: We study a new problem setting termed Action-Constrained Imitation Learning and propose a method  to mitigate the mismatch of occupancy measures between the expert and the learner caused by the action constraints.

Abstract: Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications.
In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where an action-constrained imitator aims to learn from a demonstrative expert with larger action space.
The fundamental challenge of ACIL lies in the unavoidable mismatch of occupancy measure between the expert and the imitator caused by the action constraints. We tackle this mismatch through trajectory alignment and propose DTWIL, which replaces the original expert demonstrations with a surrogate dataset that follows similar state trajectories while adhering to the action constraints. Specifically, we recast trajectory alignment as a planning problem and solve it via Model Predictive Control, which aligns the surrogate trajectories with the expert trajectories based on the Dynamic Time Warping (DTW) distance. Through extensive experiments, we demonstrate that learning from the dataset generated by DTWIL significantly enhances performance across multiple robot control tasks and outperforms various benchmark imitation learning algorithms in terms of sample efficiency.

Action-Constrained Imitation Learning
Chia-Han Yeh
Tse-Sheng Nan
Risto Vuorio
Wei Hung' Hung- Yen Wu Shao-Hua Sun"
Ping-Chun Hsieh
Abstract
cumulative rewards through interactions with environments. However, in many real-world applications, interacting with environments can be costly or dangerous, and designing an effective reward function that consistently encourages the desired behavior in all situations could pose a significant challenge. In such cases, imitation learning (IL) (Pomer- leau A, 1991: Ho Ermon, 2016) offers a compelling alternative. Rather than learning from reward function through trial and error, IL learns policy directly from set of pre-collected expert demonstrations, which are transition data logged from near-optimal policy.
Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications. In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where action-constrained imitator aims to learn from demonstrative expert with larger action space. The fundamental challenge of ACIL lies in the unavoidable mismatch of occupancy measure between the expert and the imitator caused by the action constraints. We tackle this mismatch through frajectory alignment and propose DTWIL,, which replaces the original expert demonstrations with surrogate dataset that follows similar state trajectories while ad- hering to the action constraints. Specifically, we recast trajectory alignment as a planning problem and solve it via Model Predictive Control, which aligns the surrogate trajectories with the expert trajectories based on the Dynamic Time Warping (DTW) distance. Through extensive experiments, we demonstrate that learning from the dataset generated by DTWIL significantly enhances performance across multiple robot control tasks and outperforms various benchmark imitation learning algorithms in terms of sample efficiency. Our code is publicly available at https: github. wwt.cooo//oo/.. ACRI-Basee nes.
In many real-world tasks, imposing constraints that define feasible sets of actions to ensure the safe and proper func- tioning of agents necessary. Classic examples include optimal allocation of network resources under capacity con- straints (Xu et al., 2018; Gu et al., 2019; Zhang et al., 2020) and robot control under kinematic limitations that prevent damage to the physical structure of robots (Pham et al., 2018; Gu et al., 2017; Jaillet & Porta, 2012; Tsounis et al., 2020). While there has been substantial research on action- constrained reinforcement learning (ACRL) (Kasaura et al., 2023; Lin al., 2021; Braahmaage et al., 2023; Chen et al., 2024; Hung et al., 2025), surprisingly, little attention has been given to its imitation learning counterpart. Hence, in this work, we propose a novel problem, Action-Constrained Imitation Learning (ACIL), which concerns learning an agent under action constraints from a constraint-free expert demonstration set, e.g., learning control a robot arm with limited power supply from trajectories collected from more powerful robot arm (i.e., higher torque limit).
Can we adopt existing ACRL methods to address the ACIL problem? To ensure that the actions generated by the policy adhere to specific constraints during both training and evalu- ation, most existing ACRL methods incorporate a projection layer on top of the policy network (Chow et al., 2018; Liu et al., 2020; Gu et al., 2017). However, such approach cause problems in IL. Most IL approaches aim to minimize the discrepancy between the occupancy measure of the ex- pert demonstrations and that of the imitator (Pomerlcau & A, 1991; Ho Ermon, 2016). When expert actions are outside the feasible action set, the projection layer can prevent the imitator from accurately matching the occupancy measure of the expert, especially in cases with more restrictive action
1. Introduction
Reinforcement learning (RL) finds policies that maximize
'Wor conducted while the University of Oxford. 'Equal contribution National Yang Ming Chiao Tung Univer- sity, Hsinchu, Taiwan UUniversity of Illinois at Urbana-Champaign, Ilinois, United States 'Reflection AI Natioona Taiwan University, Taipei, Taiwan. Correspondence to: Ping- Cnn- Hsich <pingh- sich Seh@nccu..du.t>>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Action-Constrained Imitation Learning
By
Strang expert
vx
Weak imitator
Action space
(a) Expert-imne inpeettimitttor Performance Gap
(b) Expert
BC Policy
(d) Ours
Figure 1. (a) illustrates the performance gap due to the mismatch between the action spaces of the strong expert and the weak imitator. (b)
shows the expert trajectory in Maze2d goal-reaching task, which serves as the reference for imitation. (c) demonstrates the trajectory the action-constrained BC policy, which fails to execute a timely left turn due to the restricted action space. (d) depicts the trajectory generated by DTWIL alignment, where the agent adapts to the constrained action space by adjusting its pacc, enabling it to accurately follow the expert trajectory.
sets. This issue leads a problem we term "occupancy measure distortion." Figure 1 illustrates the issue of occu- pancy measure distortion caused by the mismatch between the action spaces of a strong expert and a weak imitator. The expert's trajectory highlights the optimal behavior, as shown the data. However, the constraindd behavior cloning policy fails to reproduce this trajectory due to the restricted action space, leading suboptimal performance and collisions.
arm manipulation tasks, excelling in sample efficiency and robustness occupancy measure distortion.
2. Related Work
Actton-Constrained Reinforcement Learning (ACRL).
ACRL studies learn a policy under action constraints. Kasaura al. (2023) provides benchmarks for evaluating existing ACRL approaches. Pham et al. (2018); Bhatia et al. (2019); Dalal et al. (2018) ensure safe and compliant behav- ior by incorporating differentiable projection layer at the end of the policy network to meet action constraints. How- ever, Lin et al. (2021); Brahmanage et al. (2023) highlight issues with this approach, particularly the zero gradient and longer training times, and propose alternative methods. No- tably, Brahmanage et al. (2023); Chen et al. (2024) employ normalizing flows to directly generate actions that comply with the constraints, thereby cireumventing the drawbacks associated with projection layers. Hung et al. (2025) pro- poses an efficient approach that significantly reduces the need for quadratic programs by using acceptance-rejectin sampling and constructing an augmented Markov decision process. Yet, adapting existing ACRL techniques to ACIL is non-trivial.
The most effective way to eliminate occupancy measure distortion is to ensure that both the expert and the imitator share the same feasible action set. However, since we can- not access the expert's 's policy to generate aetion-coostraaiee expert demonstrations, we can only align to the expert's tra- jectories with constrained actions and learn from the align- ment. This alignment is crucial to enable the imitator to accurately maintain consistency with the demonstrated tra- jectories. To address this challenge, we introduce Dynamic Time Warping Imitation Learning (DTWIL), an algorithm specifically designed to bridge this gap by generating sur- rogate aetion-constrained demonstrations and learning corresponding policy that respects the constraints of the imitator. By aligning constrained trajectories with expert demonstrations, DTWIL provides robust solution to miti- gate occupancy measure distortion.
Learning from Demonstration. IL focuses on deriving policy using only the information from expert demon- strations, and this paradigm is also termed Learning from Demonstration (LfD). BC (Pomerleau & A, 1991) ap- proaches this by treating policy as a state-to-action map- ping, learning in a supervised manner. Adversarial Imita- tion Learning (AIL), on the other hand, focuses on matching the state-action distribution between expert and imitator through adversarial training. GAIL. (Ho & Ermon, 2016) is foundational method in this domain, using discrimina- tor to distinguish between expert and imitator transitions, and providing rewards based on this discrimination. Var- ious AlL extensions (Kostrikoy et al., 2019a;b; Lai et al.,
Our contributions can he summarized as follows: (1) We for- malize the Action-Conne Constrained Imitation Learning (ACIL) problem and identify the challenge of occupancy measure distortion. (2) We propose DTWIL, a novel framework for generating surrogate expert demonstrations and learning action-constrained policies. (3) We reformulate trajectory alignment as planning problem, leveraging MPC for con- strained trajectory generation. Moreover, we employ the DTW distance as the similarity metric and design practical implementations to enhance alignment efficiency. (4) Our experiments show that DTWIL outperforms the baseline IL algorithms in MuJoCo locomotion, navigation, and robot
2

---

Action-Constrained Imitation Learning
2024) improve upon GAIL,, tailoring the method different environments and goals. A comprehensive review of IL techniques can be found in (Zare et al., 2024), but ACIL, where a capability gap exists between the expert and the learner, remains unexplored.
dicating the conditional distribution of the next state s' S given the current state E S and action E A: μa (SS) is the initial state distribution; Y E [0, 1) is the discount factor. For each state E S, C(s) S A denotes the feasible action set induced by the action constraints. Equivalently, under an AC-MDP, no actions that fall outside the feasible set C(s) can be employed to the environment at both train- ing time and test time, Moreover, we impose no assumption the structure of C(s), i.e., C(s) can be arbitrary subset of the action space. Notably, the above setting of AC-MDP is standard in the ACRL. literature (Lin et al., 2021; Kasaura et al., 2023; Brahmanage et al., 2023).
Learning from Observation (Lff)., An alternative ap- proach to avoid the undesirable effects of the projected pol- icy results after imitating expert actions is to learn only from expert observation data, which falls under the scenario of LfO (Lee et al., 2021; Huang et al., 2024). GAIFO (Torabi et al., 201 8b) and IDDM (Yang et al., 2019) follow the principles of GAIL by training a state-only diseriminator. OPOLO (Zhu et al., 2020) further improves on this by relax- ing the on-policy requirement, speeding up the learning pro- cess. BCO (Torabi et al., 2018a 8a) directly learns an inverse dynamics model to infer missing expert actions from obser- vations and then applies BC to learn policy. CFIL (Freund et al., 2023) and DIFO (Huang et al., 2024) use generative model to capture state or state-action distributions. There also exists a line of work that focuses on state trajectory alignment. Liu al. (2019); Gangwani & Peng (2020) aim to address the transition dynamics mismatch between the expert and the learner. Radosavovic et al. (2021) matches the state distributions in the context of dexterous manipula- tion, and Boborzi et al. (2023) proposes a non-adversarie framework for state-only imitation.
Action-constrained imitation learning. In ACIL, an action-constrained imitator is trained to perform similarly to an unconstrained expert from a set of expert demon- stration trajectories D. generated under expert policy Te S - (AA). For ease of exposition, we let T. and Te denote the sets of state-action trajectories that consist of unconstrained actions and constrained actions, respectively. Due to the gap in the action capability between the expert and the imitator, is generally infeasible for the imitator to exactly reproduce an expert state-action trajectory To E T or match the state-action distribution (also known occu- pancy measure) of the expert. As result, the standard IL methods, which focus on matching the state-action distribu- tions, are not directly applicable in the action-constrained setting.
However, none of the existing LfO methods addresses the capability gap between the expert and the imitator. They attempt to mimic the trajectories induced by unconstrained actions, although such trajectories be fundamentally infeasible for an action-constrained imitator to reproduce.
Therefore, in ACIL, our focus on more relaxed imita- tion learning setting: An action-constrained imitator learns to produce a state sequence as similar to the expert state sequence as possible. That is, in ACIL, the behavioral simi- larity is captured through state sequence similarity. Specifi- cally, we use (T)) = (so, (1,,81,"" ...) to denote the state se- quence induced by a state-action trajectory T E T and let our := (st,..., ,8t) denote the partial sequence of (T)). Similarly, we use a(r) to denote the action se- quence included in r. Moreover, we let S denote the set of all possible state sequences (either of finite or countably infinite steps). For any pair of state sequences o' E S (possibly of different lengths), we use d(o,, o') to denote the discrepancy between o and o', and the d(-,-) :) is to be con- figured in the sequel. Our goal to learn a feasible imitator policy म² E Ie that minimizes the expected discrepancy from the expert, i.e.,
3. Action-Constraiied Imitation Learning: Problem Statement and Challenges
In this section, we formalize the proposed ACIL problem and delineate the main challenges in achieving ACIL.
For any set Z, we use A(2) to denote the set of all probability distributions over Z. Given any subset X R" and any vector La projection of æ onto the set X. For any sequence (oo, "),, use Tue to denote the partial sequence (Za, ,..).
Notation.
R", we let (x) denote the
T == arg min E ENp [d(a(T)) ~n]. (1) rElle
3.1. Problem Statement
Action-constrained Markov decision process. We model the environment as an action-constrained Markov de- cision process (AC-MDP) defined by a tuple M := (S,ATT, T,,, C)C, where S and A are the state space and action space, respectively: : transition dynamics of the environment, with T(8'1s, a) in-
3.2. Challenges of ACIL
Due to the action constraints, solving ACIL naturally in- volves the following two salient technical challenges com-
pared to the conventional unconstrained IL settings:
S x A - A(S) denotes the
Aligning state sequences of different lengths. To mimic
3

---

Action-Constraneed Imitation Learning
an expert state sequence of T time steps, the imitator could possibly require more than T steps to generate similar state sequence, mainly due to the limited action capability, For example, as illustrated by the Maze2D navigation task in Figure 1, the imitator in Figure 1d takes several more steps than the expert to reach the goal state, despite their similarity in the state sequences. This issue would be more significant as the action constraints become tighter.
Stage 2: Employ any IL method with surrogate demon- strations. Given the surrogate dataset Daur that involves only feasible actions, one can simply leverage any off-the- shelf IL algorithm to imitate the surrogate demonstrations to enforce the action constraints. In the subsequent exper- iments, we showcase this flexibility by using both BC and inverse RL methods to achieve effective ACIL.
Trajectory misalignment under projection. To enforce the action constraints required by the imitator, one direct approach is to employ an off-the-shelf unconstrained IL algorithm and apply an additional action projection step, which identifies the nearest feasible action to the original unconstrained action generated by the policy. Notably, the action projection step has been widely adopted by the ACRL algorithms, either post-processing subroutine (Lin et al., 2021; Kasaura et al., 2023) or as a differentiable layer at the policy output for end-to-end training (Pham et al., 2018; Bhatia et al., 2019). Despite its effectiveness in ACRL, action projection can lead to severe trajectory misalignment as the action discrepancies could quickly compound the state discrepancies. Such trajectory misalignment has also been illustrated in Figure Ic. The misalignment issue further highlights that ACIL and ACRL are fundamentally different problems in spite of their high-level resemblance. As a result, ACIL cannot be tackled simply by action projection, and an alternative solution is needed.
Through this framework, nicely decouple the constraint satisfaction from the downstream imitation learning method.
4.2. Trajectory Alignment via Model Predictive Control
Trajectory alignment as trajectory optimization. To con- struct the surrogate demonstrations (i.e., Stage described in Section 4.1), we propose to recast trajectory alignment as a planning problem and solve it by trajectory optimiza- tion. Specifically, for each expert state-action trajectory Te = (so, 0g. $1, (1,...) in De, the problem of finding a surrogate trajectory of feasible actions can be formulated as
Tsur = arg min d(a((T) TETe
(2)
where d(..-) is the state sequence discrepancy defined in Section 3.1. If we focus on the sequence of actions
in Tsur (denoted by ai....... -), the problem in (2) can be reformulated as trajectory optimization problem
4. Methodology
Oo:K- ==
arg min d(a()),
(3)
r=a())ecK K21
In this section, formally introduce the proposed trajec- tory alignment method for addressing ACIL. We start by presenting the overall framework and then describe the two main modules, namely MPC-based trajectory optimization and dynamic time warping.
where the minimization is over all feasible action sequences. Notably, directly solving (3) can be difficult for two reasons: (i) The optimal length of the surrogate trajectory (i.e., K* in (3)) can be hard to determine priori. (ii) Even if K* is known, it is known that directly handling a large planning problem of large length K in a non-adaptive manner can be rather sub-optimal.
4.1. Surrogate Expert Demonstrations
To imitate the expert state sequence and enforce the action constraints simultaneously, we propose to decompose the imitation process of ACIL into two stages:
Model predictive control for alignment. To adaptively solve (3), we leverage MPC, which addresses (3) by se- quentially solving finite-horizon planning subproblems with the help of forward dynamics model. Specifically, we use MPC to determine the surrogate action at each step se- quentially. For each step of Tsur. MPC configures fixed planning horizon H and approximately minimizes a finite- horizon objective function d(a(('),, o Hr (T=)) by first (i) generating a set of H H-step synthetic rollouts (7'} with the help of forward dynamics model, (ii) selecting the rollout with the smallest di d(a(r"), o2.+ H (Te)), and then (iii) assign- ing of to be the first action of the selected rollout. This of would be applied to the environment to obtain the next state. This design of taking only the first action ensures that each step of the alignment process can better adapt to the remaining part of the demonstration.
Stage 1: Generate action-constrained surrogate demonstrations from unconstrained expert demon- strations. Given the original expert demonstrations De, the main purpose here is to construct a surrogate dataset Danr such that similar state sequences can be reproduced solely by using feasible actions. Specifically, for each state-action trajectory T = (so, do, 81, 01,88,,11 ..) in De. we 00,
create a surrogate trajectory Taur- = (s6, a/. s1, ...) that satisfies: (i) s = so: (ii) dg E C(s(), for all i; (iii) The discrepancy of the state sequences induced by 7 and Turr is small.
4

---

Action-Constrained Imitation Learning
4.3. Dynamic Time Warping as the Alignment Criterion
Expert Demonstration De (t',r), r*) Align r
To substantiate the ACIL objective function in Equation (1) and the trajectory optimization problem in Equation (3), we leverage the DTW distance (Hiroaki & Chiba, 1978) as the state sequence discrepancy metric. Recall from Section 3.2 that one main challenge of ACIL is the need to align state sequences of different lengths, and DTW naturally addresses this issue as a distance that can "warp" time over the space of time sequences.
Environment
Projection
a2
Layer
4
Policy
si
π
Alignment Target
Feasible action of
Inference
Off-the-Sheelf Imitation
Learning Method
We describe the concept of DTW in the context of ACIL as follows: Consider any two state sequences o, of E S of lengths m and n. Let A be the pairwise lz distance matrix of size m X n, where Aij = l|o; - d/1l2. We use A to denote an m X n alignment matrix, which satisfies the following three properties: (i) A is a binary matrix, i.e., it only has elements 0 or 1; (ii) The ones in A shall induce path from the top-left the bottom-right of the matrix; (iii This path only involves three possible moves, ie., right, down, and lower right. We also let A be the set of all such matrices. Then, the DTW distance between a and o' is defined as
8t
Target
Surrogate Expert Demonstration Dsur
MPC
Learned Model
(T onr Faur)
Figure 2. An illustration of the DTWII. framework. MPC aligns the action-constrained trajectories with expert demonstrations, gen- erating surrogate demonstrations for downstream imitation (e.g.. behavior cloning). The trained policy is then deployed during inference with actions projected onto the feasible action space.
segment of the expert state sequence for alignment would be o +1) (T'). Suppose the current MPC timestep is L, the current progress is Egg. and the planning horizon is H. Then, MPC approximately finds an H-step planning trajectory (4)
dDTW(O, o') min AEA
r' such that durw (a(')) ( (re (1) is minimized by generating set of synthetic rollouts with the help of the learned dynamics model.
where DF denotes the Frobenius inner product. Moreover, it is known that the DTW distance in Equation (4) can be found by dynamic programming through the following recursion
These surrogate expert demonstrations are then used to train policy via any off-the-shelf II. method, e-g., BC). An overview of DTWIL is provided in Figure 2, and the pseudo code can be found in Algorithm and Algorithm 2.
dorw dbz) = lo,, - 8اار2 + min (dorw (oo.w(oo:-1100 oay)s dDTw(rous; 1)) doTw(ro:-11, o. }
(5)
Remark 4.1. In typical RL planning tasks, the objective func- tion of MPC is maximize the cumulative reward (Chua et al., 2018; Hansen al., 2022). By contrast, in the ACIL setting, we use DTW distance as the criterion in aligning the state sequences of the expert and the surrogate one.
where the minimum operation reflects the three possible moves (i.e., right, down, and lower right) described above.
4.4. Putting Everything Together: DTWIL
Remark 4.2. To facilitate the rollout generation in MPC, we also incorporate the cross-entropy method (CEM) optimizer, which iteratively refines the search for optimal actions by sampling, evaluating, and updating the distribution of candi- date actions (Chua al., 2018). Moreover, to ensure that the candidate synthetic rollouts in MPC are all feasible, we employ rejection sampling to enforce the action constraints. Other details about our MPC implementation can be found in Appendix A.
We are ready to present the the proposed algorithm, namely Dynamic Time Warping Imitation Learning. We substan- tiate the generation process of surrogate demonstrations described in Section 4.1 by integrating MPC-based trajec- tory alignment with the DTW distance. Specifically: (i) Alignment target: We choose one expert trajectory r' E D. at a time as the alignment target. (ii) MPC-based align- ment with DTW: At the beginning of an alignment episode, MPC initialized to the 0-th state of the selected expert trajectory. After initialization, MPC aligns with the tar- get trajectory at each step sequentially until the episode is finished. To utilize DTW as the criterion, we introduce progression parameter, tpg. which indicates the timestep of the expert state with which the action-constrained agent is currently aligned. For example, if the current progress at fpy and the planning horizon is set to H. the targeted
4.5. Practical Implementation
1.5.1. PROGRESSION MANAGEMENT
The progression parameter, fpg. is initialized to 0 at the start of every trajectory alignment, indicating that the alignment begins from the O-th state of the expert trajectory. At the be- ginning of each alignment step, we update 1ng by analyzing
5

---

Action-Constrained Imitation Learning
the warping map determine how many expert states the agent's action has advanced. Consequently, if the agent's first planning state, $1, is not sufficiently close to the next expert state, s1, it is more likely to be matched with the cur-
Expert Trajectory
Expert Trajectory
60
Agent Trajectory
Agent Trajectory
rent expert state, ~6- Figure 3 shows how this advancement value is determined. The value of 'p is updated after every MPC step. For how this mechanism affects the alignment, please refer Appendix A.6.
92
(a) Progression ment=0
Advance-
Progression Advance-
ment
Algorithm Dynamic Time Warping Imitation Learning
1: Input: Expert demonstration De = (T' |=1 number of episodes MM
Figure 3. Since the MPC controller executes only the first planning step iteration, we focus on the number of expert states the agent advances after the initial action do. The figure shows two DTW warping path cases (patches in gray). In Figure 3a, the agent transitions from 8o #1 while staying aligned with s causing no progression (fn unchanged). In Figure 3b, the agent advances to the next expert state, updating fpg to fpz +
2: Surrogate demonstration dataset Sur + {
3: Dynamies model training dataset 'D De
4: for iteration
1 to M do 5: Select an expert trajectory r' from De
6: Train the dynamics model ensemble with D
7:
Tur <- Trajectory Alignment(r")
8:
- D-UUT
9:
if dDdT (Tar r²) dprw (DucliD ( then
expert actions with sampled actions to serve as guidance, mechanism that we refer to Expert Regularized Control (ERC). Specifically, the first few actions used to roll out the planning trajectories in the MPC controller become the weighted average of the actions sampled and the correspond- ing segment of the actions of the experts. The details the implementation can be found Appendix A.7.
10: Daur sur
11: end if
12:
end for
13: Learn an imitator policy from sur by an off-the-shelf
imitation learning algorithm
For other implementation details, please refer to Ap- pendix A.4 and Appendix A.5.
Algorithm 2 Trajectory Alignment
1: Input: Planning horizon H, ERC horizon here, i-th De, action
expert trajectory 7³ ((st, (ff,.a)) at) lengthe! constraint C.
length=l
5. Experiments
2: Output: Turr + !
We evaluate our proposed method in various continuous con- trol domains, including navigation, locomotion, and robot arm manipulation, subject to a variety of action constraints.
3: Agent's s initial state 80 < sb. lpg +- 0
4: for time step t = 0, 1,... do
5: for Actions sampled 0 from CEM, to NSam-
ples do
5.1. Setup
6: Apply E.RC(au+tH, H, here
Environments. To evaluate our method, we conducted experiments on four benchmark tasks: Maze2d, HalfChee- tah, Hopper, and Table- Table-Wiping. Maze2d-Medium et al., 2020), point-mass agent navigates 2D maze from random start location to goal, with 2-dimensional action space [a,, 02] E [-1.0, 1.0). The state informa- tion includes and representing the agent's velocity in the x- and y-directions. We collected 100 demonstra- tions, yielding 18,525 state-action pairs. HalfCheetah (Brockman et al., 2016), bipedal chectah runs forward by applying torque through 6-dimensional action space [a1 a2....,......................... .a] E [-1.0, 1.0). The state includes w;, the angular velocity of each joint. For this task, we use 5 ex- pert demonstrations of 1000 steps each. In Hopper (Brock- man et al., 2016), robot hops forward by controlling a 3 3-dimensional action space [aj, 02 ag] E [-1.0, 1.0]. Sim- ilarly, the state contains Wi, the angular velocity of each
7: Propagate 7' from 8; using dynamics models
8: Evaluate Q1:t-If as dprw(at dorw(aa('), ot tt H)) (r)))
(Fu
9: Update CEM distribution
10: end for
11:
Execute of
The < Tuur (s;
13: if Progression has advanced in the warping path then
14:
t.g tps +1
15: end if
16:
end for
4.5.2. EXPERT REGULARIZED CONTROL
In environments that require precise movements, even small errors can cause significant disturbances. Inspired by Actor Regularized Control (ARC) (Sikchi al., 2021), we mix
6

---

Action-Constraine Imitation Learning
CFIL-s & CFIL--a (Freund et al., 2023) utilize flow-based model to capture state or state-action dis- tributions. CFIL-S learns from state-only sequences (LfO), while CFIL-sa learns from both states and ac- tions (LFD)..
Figure 4. We evaluate the impact of action constraints DTWII. and haseline methods across three environments: Maze2d- Medium-vl, HalfCheetah-v3, Hopper-v2, and Robosuite Table- Wiping task.
DIFO (Huang et al., 2024) learns a diffusion discrimi- nator and learns the policy with the diffusion reward.
Note that we project output actions feasible sets with projection layer to avoid constraint violation for all methods.
Table Experiment environments and constraints
Environment
Task
Constraint
5.2. Experimental Results
M+B
las]
0.1,
Vi
Maze2d
Since we aim to investigate sample efficiency, we only al- low 50K environment steps during the training of all the online methods, including ours, on all tasks. All results are evaluated with randomly initialized starting states. Follow- ing this, the best best-performing model from each algorithm during these interactions was selected for final evaluation. This ensures that the results reflect the effectiveness of each method in small-sample regime.
M+0
Σi
|0,0,]
$ 0.5
H+B
la:1 < 0.9, Vi
Hopper
H+M
Ei_, max wya, 0) S 10
HC+B
las] 0.5, Vi
HalfCheetah
HC+0
Σi: wya, S 10
W+B
la:] 0.3, vi
Table-Wiping
+ a 1 < 0.5,
W+L2
+ og
+
0.05
The experimental results in Table 2 indicate that online al- gorithms, such as GAIL and OPOLO, face two primary challenges: poor sample efficiency and action constraints, leading to consistently suboptimal performance in all tasks. CFIL also struggles to achieve competitive performance under limited sample conditions. Despite their ability to in- teract with the environment, these methods fail to bridge the expert-im ispert-imitator performance gap within the given number of interaction steps, resulting in persistently low scores, While BCO shows competitive performance in simpler tasks like Maze2d M+O, it falls short in more complex environments.
joint. For Hopper, we use 5 expert demonstrations of 1000 steps each for training. In Table-Wiping from Robosuite (Zhu et al., 2025), robot arm controlled by 6-dimensional action space aims to wipe a stained table. The expert dataset consists of ten 500-step demonstrations. The constraints considered for each environment are summarized in Table 1, including box constraints (+B), state-dependent power limit constraints (+M and +O), and L. constraints (+L.)..
Baselines. We compare our method to various baselines in learning from demonstration (LID), i.e., state-action se- quences, and learning from observation (LfO), i.e., state- only sequences.
In contrast, DTWIL, which learns from surrogate expert data and adopts a BC approach to learn the policy, performs well in all tasks. By learning from the surrogate demon- strations to match the expert trajectories and using BC for policy learning, DTWIL manages to replicate expert per- formance while maintaining sample efficiency. As result, it successfully reproduces expert-like trajectories across tasks, without being adversely affected by the constraints that cripple other methods. The results of training the var- ious baseline methods for sufficient steps are included in Appendix A.3.
BCO (Torabi et al., 2018a) is an LfO method, learning an inverse dynamies model to infer actions from state- only data and applying BC learn policy.
GAIL (Ho & Ermon, 2016) is an LfD method that uti- lizes a generative adversarial network (GAN) to infer the underlying reward function.
GAIfO (Torabi et al., 2018b) resembles the idea of GAIL, but its discriminator only learns from state tran- sitions (s, 8') instead of state-action pairs (s, a).
5.3. Ablation Study
DTW distance and l distance as alignment criteria. We investigate the impact of using different alignment criteria, specifically comparing & distance and DTW distance, on the performance of the DTWIL framework. The & distance evaluates the pointwise differences between trajectories, while the DTW distance captures temporal misalignment, providing a more flexible measure for trajectory alignment.
OPOLO (Zhu et al., 2020) is an off-policy LfO method and is among the most effective LfO techniques.
SAIL (Liu et al., 2019) learns a reward function to align the agent and the expert state trajectories.

---

Action-Constrained Imitation Learning
Table Evaluation performance of the proposed method and baseline algorithms across various tasks, with results expressed the mean and standard deviation calculated from three seeds.
Task
Metric
GAIL
BCO
GAIO
OPOLO
CFIL.s
CFIL-sa
SAIL
DIFO
DTWII.(Ours)
M+B
retur
0.22 0.0
0.14 0.05
0.07 0.02
0.2 0.06
0,23 = 0.06
0
0,45 0.06
0.27 0.04
0.77 0.04
dorw
10.8
7.8 1 0.5
0.2
8.2 0.5
10.5 0.6
10.9 1.8
16.0 a 0.61
0.1
4.0 0.1
0.14 0.05
0.88 0.06
0.19 0.08
0.64 0.13
0.45 0.12
0.47 0.10
0,46 0.15
0.45 0.09
0.87 + 0.04
M+O
dorw
0.8
8.8 1 0.5
0.2
8.8 0.3
10.8 1: 1.1
10.2 0.7
1
1.0
3,6 1 0.1
HC+B
retur
-163 47
.4
-74 +
-015 + 390
-172 738
-95 515
1222 + 260
8
2669 4
dorw
0.0
58.7 + 11.9
39.5
37.9
56.2 11.8
26.5
43.7 1,9
7.4 0.0
HC+0
return
66
61
-163 33
-0
1422 1830
1674 1316
2666 +
-87 5:
2637 26
dorw
0.7
43.1 + 17.8
76.8 6.5
35.1 4.2
27.9 1.9
43.0 21.6
19.7 0.1
44.4 1.8
7.8 + 0.4
H-B
return
59
219 20
197 30
1068 952
866 249
1485 + 677
1375
331
2844 57
doow
3.. 1.6
33.1 2.1
33,2 d: 1.4
33.7 2.8
32.9 2.8
29.8 à: 4.2
21.7
34.3 d: 1.9
5.8 + 0.1
81
224 32
206 19
228
1443 + 547
1553 1096
1273 495
333
2873 240
H+M
dorw
37.1 0.4
33.1 2.2
32.4 0.9
31.8 a 0.9
29.4 2.3
31.0 a 4.2
24.9 0.8
33.3 à: 1.2
6.6 + 1.5
1
59
30
29
1
61
6
61 +
W+B
doow
2.6
29.6 1.2
36.3 2.7
2
39.2
38.4 5.5
2:.1
37.5 4.3
11.8 2.3
W+L
return
19
91 17
2
44
0
1
58
7+1
dorw
3.7
22.9 1.3
30.1 2.9
31.1 4.3
34.6
36.4 +
19.9 0.8
41.6
9.6 0.4
We conduct experiments HalfCheetah and Hopper. For each task, we train policies using surrogate demonstrations generated with both alignment criteria and evaluate the poli- cies on their ability to reproduce expert-like behavior.
Table 3. Comparison of policy performance using & distance and DTW distance as alignment criteria across different tasks.
Task
&
DTW
The results, summarized Table 3, demonstrate that the DTW distance significantly outperforms &g distance as an alignment criterion. Policies trained using D'W--lligned sur- rogate demonstrations achieve higher task performance and exhibit trajectories that more closely resemble expert behav- iors. This highlights the importance of a flexible alignment metric in addressing temporal distortions and occupancy measure discrepancies in cetion-constrained environments.
HC+B
2157.52
60.84
2669.41 41 # 4.56
HC+0
2279.12
51,18
2637.34 & 26.82
H+B
1054,49
# 227.64
2844.68 + 57.77
H+M
1245.61 47.05
2873.88 # 240.46
Table 4. Performance comparison of BC and DTWIL.
Task
BC
DTWIL (Ours)
DTWIL-Angmeented IL. key advantage of our approach is its versatility the surrogate demonstrations generated by DTWIL can be integrated with any online or offline IL. method. Table 4 compares our method to the original BC under action constraints, illustrating that DTWIL, which ap- plies BC learning with surrogate data, consistently achieves superior performance across multiple tasks. Furthermore, we apply CFIL the surrogate data and observe notable performance boost, as shown in Figure 5. These results demonstrate that the surrogate data effectively mitigates occupancy measure distortion, a fundamental challenge in action-constrained imitation learning. They also suggest that our approach is broadly applicable and can enhance the performance of various IL methods constrained environ- ments.
M+B
0,61 0.05
0.77 = 0.04
M+0
0,81 0.810000 0.05
0.87 0.04
H+B
2204.83 + 753.32
2844.68 57.77
H+M
1233.96 211.87
2873.88 240.46
HalfCheetah-v2 Box
maze2d-medium-vl Box
0.25
0.20
Returms
Returns
0.15
0.10
0.05
-1000
0.00
Interation Steps (x10*)
Interation Steps (x10*)
6. Conclusion
CFIL-s
CFIL-s+DTWIL
ACIL has the potential to greatly influence real-world robot training, where constrained action spaces arise from power limits, mechanical imperfections, or wear and tear chal-
Figure 5. Performance comparison CFIL with and without DTWIL- surrogate data across multiple tasks.
8

---

Action-Constrained Imitation Learning
lenges that previous methods have effectively addressed. In this paper, we identify occupancy measure distortion key issue when learning from expert demonstrations un- der action constraints. We further introduce DTWIL, the first ACII method, which leverages the DTW distance for alignment to generate surrogate expert demonstrations for downstream IL methods. DTWIL outperforms projection- based methods, demonstrating that a dedicated algorithm for the ACIL problem is both effective and necessary.
ing. Advances in Neural Information Processing Systems, 2024.
Chow, Y. Nachum, O., Duenez-Guzman, E., and Ghavamzadeh, M. A Lyapunov-based approach to safe re- inforcement learning. In Advances in Neural Information Processing Systems, 2018
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning a handful of trials using proba- bilistic dynamics models. Advances in Neural Informa- tion Processing Systems, 2018.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y. Safe exploration in continuous action spaces. httv::801.0875:1 2018.
Freund, G. J., Sarafian, E., and Kraus, A coupled flow ap- proach imitation learning. In International Conference on Machine Learning, 2023.
Acknowledgements
This material is based upon work partially supported by the National Science and Technology Council (NSTC), Taiwan, under Contract No. NSTC 113-2628-E-A49-022 and the Higher Education Sprout Project of the National Yang Ming Chiao Tung University and the Ministry of Ed- ucation, Taiwan. We also thank the National Center for high-performance Computing (NCHC) for providing com- putational and storage resources. We appreciate the finan- cial support from the Featured Area Research Center Pro- gram within the framework the Higher Education Sprout Project by the Ministry of Education (NTU- 1 140900901). Shao-Hua Sun supported by the Yushan Fellow Pro- gram by the Ministry of Education, Taiwan.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv:2004.072 19, 2020.
Gangwani, T. and Peng. J. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002.1 11899, 2020.
Gu, L.,, Zeng, D., Li, W., Guo, S., Zomaya, A. Y., and Jin, H. Intelligent VNF orchestration and flow scheduling via model-assisted deep reinforcement learning. IEEE Journal Selected Areas in Communications, 2019.
Gu, S., Holly, E., Lillicrap, T., and Levine, S. Deep rein- forcement learning for robotic manipulation with asyn- chronous off-policy updates. In IEEE International Con- ference on Robotics and Automation, 2017.
References
Bhatia, A., Varakantham, P., and Kumar, A. Resource constrained deep reinforcement learning. In Interna- tional Conference on Automated Planning and Schedul- ing, 2019.
Hansen, N. A., Su, H., and Wang, X. Temporal difference learning for model predictive control. In International Conference on Machine Learning, 2022.
Hiroaki, S. and Chiba, S. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transac- tions on Acoustics, Speech, and Signal Processing. 1978.
Boborzi. D., Straehle. C.-N., Buchner, J. S., and Mikelsons, L. Imitation learning by state-only distribution matching. Applied Intelligence, 53(24):30865-300 -00866 2023.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. Advances in Neural Information Processing Sys- tems, 2016.
Brahmanage, J., Ling, J., and Kumar, A. FlowPG: Action- constrained Policy Gradient with Normalizing Flows. Ad- vances in Neural Information Processing Systems, 2023.
Huang, B.-R., Yang, C.-K., Lai, C.-M., Wu, D.-.,, and Sun, S.H.. Diffusson imitation from observation. In Neural Information Processing Systems, 2024.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,
Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv: Axiiv:606.001 /606.01540, 2016.
Hung, W. Sun, S.-H., and Hsich, P.-. Efficient action-constrained reinforcement learning via acceptance- rejection method and augmented mdps. In International Conference on Learning Representations, 2025.
Chen, C., Karunasena, R., Nguyen, T., Sinha, A., and Varakantham, P. Generative modelling of stochastic ac- tions with arbitrary constraints in reinforcement learn-

---

Action-Constranned Imitation Learning
Jaillet, L. and Porta, J. M. Path planning under kinematic constraints by rapidly exploring manifolds. IEEE Trans- actions Robotics, 2012.
Sikchi, H., Zhou, W., and Held, D. Learning off off-policy with online planning. In Conference of Robot Learning, 2021.
Torabi, F., Warnell, G., and Stone, Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, 2018a.
Kasaura, K., Miura, S., Kozuno, T., Yonetani, R., Hoshino, K., and Hosoe, Y. Benchmarking actor-critic deep rein- forcement learning algorithms for robotics control with action constraints. Roboties and Automation Letters, 2023.
Torabi, F., Warnell, G., and Stone, P. Generative ad- versarial imitation from observation. arXiv preprint arXiv: 1807.06 2018b.
Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Addressing sam-
Tsounis, V., Alge, M., Lee, J., Farshidian, F,, and Hutter, M. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, 2020.
Tompson, J. inperiminator-actor-cooo ple inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Rep- resentations, 2019a.
Xu, Z., Tang, J., Meng, J., Zhang, W., Wang. Y., Liu, C. H., and Yang, D. Experience- networking: A deep rein- forcement learning based approach. In IEEE Conference on Computer Communications 2018
Kostrikov, Nachum, O., and Tompson, Imitation learn- ing via off- policy distribution matching. In International Conference on Leurning Representations, 2019b.
Lai, C.-M., Wang, H.-C., Hsieh, P.-C.,, Wang, Y.-C. F., Chen, M.-H., and Sun, S.-H. Diffusion-rewardd adversarial imita- tion learning. In Neural Information Processing Systems, 2024.
Yang, C., Ma, X., Huang, W., Sun, F., Liu, H., Huang, J., and Gan, C. Imitation learning from observations by minimizing inverse dynamics disagreement. Advances in Neural Information Processing Systems, 2019.
Lee, Y., Szot, A., Sun, S.-H., and Lim, J. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems, 2021.
Zare, M., Kebria, P. M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent develop- ments, and challenges. IEEE Transactions on Cybernet- ics, 2024.
Lin, J.-L., Hung. W., Yang, S.-H., Hsich, P.-C., and Liu, X. Escaping from zero gradient: Revisiting action- constrained reinforcement learning via Frank-Wolfe pol- icy optimization. In Uncertainty in Artificial Intelligence, 2021.
Zhang. J., Ye, M., Guo, 7., Yen, C.-Y., and Chao, H.J. CFR- RL: Traffic engineering with reinforcement learning in SDN. IEEE Journal on Selected Areas in Communica- tions, 2020.
Zhu, Y., Wong, J., Mandlekar, A., Martin-Martin, R., Joshi, A., Lin, K., Maddukuri, A., Nasiriany, S., and Zhu, Y. robosuite: A modular simulation framework and bench- mark for robot learning. 2025.
Liu, A.. Shi, G., Chung, S.-J., Anandkumar, A., and Yue, Y. Robust regression for safe exploration in control. In Learning for Dynamics and Control, 2020.
Liu, F., Ling, Z.. Mu, T., and Su, H. State alignment- based imitation learning. arXiv preprint arXiv: 19Xiv 10947, 2019.
Zhu, Z., Lin, K., Dai, B., and Zhou, J. Off-policy imi- tation learning from observations. Advances in Neural Information Processing Systems, 2020.
Pham, T.-H., De Magistris, G., and Tachibana, R. Optlayer Practical constrained optimization for deep reinforce- ment learning in the real world. In IEEE International Conference on Robotics and Automation, 2018
Pomerleau and D. Efficient training of artificial neural networks for autonomous navigation. Neural computa- tion, 1991.
Radosavovic, I., Wang, X., Pinto, L., and Malik, J. State- only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7865-7871. IEEE, 2021.
10

---

Action-Constrained Imitation Learning
A. Detailed Implementation of DTWIL
A.1. CEM Optimizer
Our implementation of the CEM optimizer closely follows the approach used in PETS (Chua et al., 2018), where a momentum term added into the update calculations, and bounds are imposed on the standard deviations in addition the standard CEM optimization.
Specifically, if a distribution at CEM iteration i, ( of), is updated toward target distribution N (Mtrger o lugee). the resulting updated distribution at iteration + (N 244)), will be given by:
(( NN((uu aqu + (1 a) Mtargets aof + (1 a a E [0, 1],
(6)
and the value of of is further constrained by Lw. where w represents the minimum distance from μ4 to the bounds of the feasible action space.
Moreover, to adapt the CEM optimizer for action-constrained setting, we employ rejection sampling ensure that all sampled actions strictly adhere to the predefined constraints.
A.2. Dynamics Model
In this work, train an ensemble of probabilistic neural networks model the system's dynamics. Specifically, we utilize ensembles of five dynamics models, where the 6th model, fog. is parameterized by 0g. Each network in the ensemble is trained to minimize the negative ligeliikl of the predicted outcomes, optimizing the following objective:
N
((,)) = Σ
log fo, (sn+1 (Sn, an).
Referring the ensembles used in PETS (Chua et al., 2018), define network to output Gaussian distribution with di- agonal covariance parameterized by 0 and conditioned on sn and an, i.e.: f = Pr(s 8s, at) = (No00(8 az), (o oz)). In this specific case, Equation (7) becomes:
N
(o(
Σ jua, (Sn, an) - 5nt1l Emp 2 (sns an) [M9, (Sns an) - sn+1] | + log det Ea, (8ns an), 1=1
(8)
The next states are obtained in the same manner as TSco described in PETS.
Additionally, to mitigate the risk of over-fitting that can occur when a dynamics model is trained solely on expert trajectories, we augment the training data with online agent experiences and iteratively retrain the dynamics models.
A.3. Training Curves for Baseline Methods with Additional Steps
In Section 5.2, we presented the performance of DTWIL and various baseline methods when interacting with the environment for up to 50K steps, focusing on sample efficiency. In Figure 6, we showcase the training curves of baseline methods over 500K steps, which is times the original limit. These results reveal that methods like CFII. and OPOLO can train effective policies on multiple tasks when granted sufficient interaction steps. However, compared DTWIL, which requires only the training of an MPC dynamics model to generate surrogate expert demonstrations, these online LfO methods demand significantly more interaction steps, highlighting their inefficiency relative to DTWIL.
A.4. DTW Input Normalization
To address variations scale across different dimensions, normalize both the planned trajectory and the corresponding expert trajectory segment before computing the DTW distance. Specifically, apply min-max normalization, which linearly transforms each dimension so that its values are scaled to fall within consistent range. This is achieved by subtracting the minimum value and dividing by the range (maximum value minus minimum value) of the expert trajectory for each dimension. We analyze the impact of this normalization. Table 5 shows an ablation study on HalfCheetah and Hopper with their respective box constraints. We observe performance drop in both environments when this normalization step is omitted from DTWIL. This because, without normalization, DTW becomes disproportionately influenced by

---

Action-Constrained Imitation Learning
HalfCheetah-v2 Box
Hopper-v2 Box
maze2d-medium-V Box
Table-Wipinn Box
100
2000
3000
0.8
Returns
1000
Returns
2000
Returns
Returns
0.4
40
1000
4
2
Interation Steps (x10²)
Interation Steps (x109)
Interation Steps (x109)
Interation Steps (x109)
HalfCheetah-v2 HC+0
Hopper-v2 H+M
maze2d-medium--1 M+0
Table-Wiping W+L
3000
3000
80
Returns
Returns
2000
Returns
Returns
0.4
40
1000
0.2
-1000
0
0.0
Interation Steps (x10²)
Interation Steps (x10²)
Steps (x10P)
Interation Steps (x10P)
CFIL-s
OPOLO
GAIL
DIFO
CFIL-sa
GAIFO
SAIL
DTWIL (Ours)
BCO
Figure 6. Training curves for baseline methods over 500K interaction steps across multiple tasks.
dimensions with larger scales, leading to poor generalization. Conversely, when the states normalized in advance, DTW treats each dimension equally, resulting in more effective warping.
Table 5. Impact of DTW input normalization performance. "Wo N indicates results obtained without applying DTW input normalization.
Task
HalfCheetah Box
HalfCheetah Box w/o N
Hopper Box
Hopper Box w/o N
ReturnSS
2576.20 = 61.62
1667.46 51.13
2527.63 # 572.53
608. 18 208.20 281.13 31.88
Return-BC
2669.41 = 4.56
1893.90 71.56
2844.68 + 57.77
A.5. Excluding the Final Expert State
Notably, when constructing the warping path, the final expert state in the segment is excluded from the matching calculation to prevent unintended progression when the agent exhibits minimal movement across consecutive actions. Specifically, when two trajectories have an equal number of states, DTW often tends to align states in a strictly 1-to-1 manner, which can mislead progression. By excluding the final expert state, the DTW algorithm is encouraged to create a 2-to-1 alignment during the matching process. Given constrained actions, which naturally take smaller steps than expert actions, this 2-to-1 aligmment often occurs in the initial few states. This concept illustrated in Figure Including the final expert state (Figure 7a) leads to 1-to-1 alignment since both trajectories have the same number of states. Excluding it Figure 7b prevents state from advancing, yielding more desirable matching.
As demonstrated in Table 6, this adjustment significantly enhances performance in the Maze2d-Medium environment under box constraints. Specifically, excluding the final expert state when determining the DTW warping path improves the returns obtained during both the trajectory alignment phase and the subsequent behavioral cloning (BC) phase. These results validate the effectiveness of the proposed modification stabilizing and optimizing the alignment process.
12

---

Action-Constrained Imitation Learning
Expert Trajectory
Expert Trajectory
$1
$2
$3
$1
$2
$3
Table 6. Results comparison of whether the final expert state is excluded when calculating the warping path in Maze2d-Medium under the box constraint.
Agent Trajectory
Agent Trajectory
So
$1
.1
Excluded
Not Excluded
$2
$2
DTW-S
2.99 0.75
2.99 = 0.82
$3
Return-S
0.76 = 0.00
0.69 + 0.00
Return-BC
0.77 = 0.04
0.72 + 0.03
(a) Final expert state is not excluded
(b) Final expert state is excluded
Figure 7. Effect excluding the final expert state on the DTW warping path.
A.6. Progression Management
In this section, we compare two approaches to progression management. The first is asynchronous progression, where the parameter fng is updated in tandem with the warping path. This method is used in our algorithm. The second is synchronous progression, where 'ng increases by 1 with each step of the imitator, matching the expert's pace. Given that agents with action constraint typically take longer to replicate expert behavior, asynchronous progression more sensible. Table 7 presents the full experimental results for both methods. DTW-S denotes the DTW distance between the generated surrogate trajectories and the expert trajectories, Return-S indicates the average return of the surrogate expert data, and Retum-BC represents the average return of BC policy trained this surrogate expert data. While the differences on HalfCheetah minimal, asynchronous progression significantly outperforms on Hopper.
Table Comparison of results between asynchronous and syn- chronous progression methods.
Table 8. Comparison of results with and without ERC applied during action sampling Hopper.
Task
Metric
Asynchronous
Synchronous
DTW-S
15.17
15.06 = 0.12
HC+B
Return-S
2576.20 61.62
2590.31 # 24.07
Without ERC
With ERC
Retum-BC
2669.41 4.56
2594.28 29.80
Return-S
820.7 4 84.8
2527.6 + 572.5
DTW-S
11.70 6.02
27.68 = 0.26
Return-BC
889.7 + 5.4
2844.7 = 57.8
H+B
Return-S
2527.63 572.53
418.73 89.35
Retum-BC
2844.68 S7.77
153.52 = 1.20
A.7. Implementation Details of ERC
In DTWIL, ERC acts as a stabilizer to guide sampled actions in environments requiring precise control. To implement this, first extract specific segment of tha from the expert actions a", where has is the horizon over which expert actions are blended, Then, given the dynamics model ensembles fa(s, a), a specific weight / E [0, 1]. and the projection function I'(ss)(o) which projects an action a onto a specific constrained action space C(s). ERC guides the trajectory generation with the following functions:
For h = 0,,,,.,
agioo I C((s (af (a)
* u
+ (1 B) 0%
sampled
if <= here
ah =
a
sampled
if h here
8h+1
So(Sn, an)
(9)
where an is the th action step in an H-step planning trajectory, a% optimizer, and 8 is the 1tth state of the planning trajectory.
sampled is the htth action directly sampled from a CEM
13

---

Action-Constrained Imitation Learning
The performance of our algorithm in environments where agents are highly susceptible deviations--uchh as Hopper, where falling results in early termination-is significantly enhanced by incorporating ERC. Table 8 demonstrate clear performance difference: without ERC, the agent frequently falls, leading significantly lower rewards and shorter trajectories. In contrast, incorporating ERC stabilizes the agent's 's behavior, allowing it to generate surrogate trajectories of appropriate length and maintain consistent performance throughout the task. This highlights the importance of ERC in enabling robust and reliable imitation under cetion-constrained settings, Refer to Appendix A.8 for detailed hyperparameter tuning.
A.8. Hyperparameters in ERC
We explore the influence of the hyperparameter 8, which regulates the balance between expert actions and MPC-sampled actions in the ERC method. Additionally, we examine the effect of the horizon length here. which determines how many steps to blend MPC-sampled actions with expert actions. We conducted experiments on the Hopper with H+M constraints, varying 3 from 0 to 0.2 and here from 0 to 20, while keeping all other hyperparameters fixed at their optimal values identified in prior tuning. As shown in Table 9, setting / 0.05 results in the highest performance. A lower 3 leads to instability in the sampled actions, while higher values negatively impact the MPC optimization process. Regarding here. value of 5 provides the best results. Extending the horizon does not improve performance, as expert actions taken too far in the future become less informative due to the action constraints.
Table 9, Impact of varying 3 and here values performance in the Hopper task with H+M constraints. The table highlights the optimal balance between expert actions and MPC sampling, showing the best performmig configurations for stability and action guidance.
B = 0
A = 0.02
8 = 0.05
B = 0.1
9 0.2
Return-S
820.71 84.78
1492.97 = 144.35
2527.63
572.53
1657.47 7 286.44
670.72 328.28
Return- BBC
889.65 = 5.39
1138.85 = 56.35
2844.68 57.77
2167.30 = 360.73
723.95 345.70
here 0
Nare
Nere 10
hac 20
Return-S
820.71 = 84.78
2527.63 + 572.53
2425.25 + 370.40
2166.99 + 351.04
ReturnBCC
889.65 = 5.39
2844.68 57.77
2686.85 # 135.64
2616.09 102.90
A.9. Number of MPC Steps per Planning
Our MPC implementation executes only the first step of the planned trajectory after each action sampling. This design allows the alignment process adapt more precisely to the remaining expert trajectory every timestep. However, re-planning at every step can be computationally expensive. A common strategy to reduce the time complexity is execute multiple steps per planning cycle, thereby amortizing the planning cost over several environment steps. This introduces a trade-off between computational efficiency and alignment fidelity. We conducted an ablation study by varying the number of execution steps taken before re-planning. We report the results in Table 10e results indicate that as the number of steps increases, the alignment performance degrades. In particular, in the Hopper environment, poor alignment resulting from infrequent re-planning significantly increases the risk of falling.
Table 10. Ablation study the number of steps executed per MPC planning cycle. Executing more steps before re-planning reduces alignment quality, leading to degraded performance.
Steps \
Task
HC+B
HC+0
H+B
H+M
step
2669 4
2637 + 26
2844 57
2873 4 240
steps
2427 +
810 2324
753 + 43
5 steps
1270 464
2
5
579 + 10
A.10. Different Level of Constraints
We have shown that the imitator may suffer from occupancy measure distortion under action constraints. Beyond the type of constraint, the tightness of the constraint bounds also plays a critical role, as it directly determines the size of the imitator's feasible action space. If the constraints are loose, their impact on the learner's ability to replicate expert trajectories is minimal. In the extreme case where the constraints do not restrict any of the expert's actions, direct imitation learning
14

---

Action-Constrained Imitation Learning
still achieve strong performance. In main experiments, adopt moderately tight constraints ensure that they meaningfully affect the learning process. In this section, further investigate the performance of our method under both looser and tighter constraint levels. Corresponding results shown in Table 11. The experimental results show that across nearly all cases, DTWIL consistently outperforms baseline methods, demonstrating robustness to varying levels of constraint tightness.
Table I. Ablation study on the strength of action constraints. vary the constraint bounds across Maze2D (M+B) and HalfCheetah (HC+O) environments. Results show that DTWIL maintains strong performance under both looser and tighter constraint settings. indicating robustness to constraint tightness.
Task (Constraint Level)
CFIL-s
CFIL-sa
BCO
OPOLO
DTWIL (Ours)
M+B (0.3)
0.20 a 0.14
0.21 0.03
0.44
0.44
0.86 + 0.04
M+B (0.1) - paper
0.22 0
0.23
0.14
0.20
0.77 = 0.04
M+B (0.07)
0.05 0.02
0.06
0.16
0.14
0.68 = 0.01
HC+O (15)
2065 + 714
2796 + 247
13
673
2785 + 17
HC+0 (10) paper
1422 + 1830
1674 1316
6
-9
2637 + 26
HC+0 (7)
671
354 1587
-130
99 941
1408 + 25
A.11. Number of Expert Demonstrations
To evaluate the generalization ability of method, conduct ablation study by varying the number of expert demonstrations used for training. Specifically, we investigate whether DTWIL can still learn effective policies when provided with fewer expert trajectories, which limits coverage of the imitial state distribution.
As shown in Table 12, DTWIL consistently achieves strong performance across different demonstration sizes, indicating that it can generalize well even with limited access expert data. This suggests surrogate demonstrations generated by our method capture essential structure for learning, beyond merely replicating observed trajectories.
Table 12. Ablation study the number of expert demonstrations. We vary the number of expert trajectories provided for training evaluate the generalization ability of DTWIL. Results show that DTWIL maintains strong performance even with limited demonstrations, indicating its robustness and sample efficiency.
Number of Demos
Task
HC+B
H+M
5 Demos
2669 4
2873 = 240
Demos
2585
2994
1 Demo
2618 28
2685 +: 457
15


Reviews:
[{"review_id": "HLh67uSLtN", "reviewer": "Reviewer_NcAj", "summary": "The manuscript proposes DTWIL, a data pre-processing mechanism for translating state-action trajectories between an expert and a weaker, action-constrained learner. After synthesizing this translated data, using an MPC with a dynamic time-warping (DTW) distance measure and a forward dynamics model to provide feasible actions for a state sequence that is close to the experts, the framework encourages the use of any off-the-shelf imitation learning method.\n\n## update after rebuttal\n\nI appreciate the authors' answers to some of my questions; I have increased my score by one point. However, despite the author response, I still retain concerns over fair comparisons with Huang et al., 2024, which boasts strong performance on the same tasks, while using a similar formulation as what the present manuscript considers. Rather than subjecting the DiFO baseline to a simple projection layer — a strategy that the present manuscript itself argues will 'cause issues in IL' (L36;C2) such as 'severe trajectory misalignment' (L161;C2) — DiFO should be allowed to be a stronger baseline, in order to more effectively motivate the proposed approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful comments and valuable suggestions. We provide our point-by-point response as follows.\n\n> L162-164;C1 — Wouldn’t it be more accurate to say that there exist trajectories within \\mathcal{T} that cannot be entirely replicated by the learner, since it contains one or more actions that are outside the learner’s feasible set. For one thing, this doesn’t preclude corresponding states to be inadmissible for the learner. Another thing is that these trajectories only make up a subset of all the trajectories in \\mathcal{T}. I don’t know where the intuition behind the use of the term “generally”, here, comes from.\n\nThanks for the insightful comment. Our use of \"generally\" was intended to convey that when the action constraints are very tight, it is typically difficult for the learner to replicate the expert trajectory. However, we acknowledge that your suggested phrasing more precisely captures the idea that certain trajectories in $\\mathcal{T}$ cannot be entirely replicated due to infeasible actions. We appreciate your suggestion and will revise this part for better clarity.\n\n\n> Why wouldn’t a diffusion policy alleviate all of these issues? The manuscript should compare with Huang et al., 2024: https://arxiv.org/pdf/2410.05429\n> Huang et al., 2024 was cited briefly, but surprisingly not used as a baseline, since it pursues the same problem and even has the same evaluation.\n> L116-120;C2 — Why does LfO underperform so much, if it’s trying to do the same thing? Why not compare to DIfO?\n\nLfO also aims to imitate the state trajectory, but it tries to mimic the trajectory induced by **unconstrained actions**—trajectories that the constrained imitator cannot perfectly replicate. This limitation is precisely why we generate surrogate expert demonstrations before applying imitation learning, ensuring that the learner has feasible trajectories to follow. DIFO is also an LfO approach, and suffers from the same problem. We added a projection layer after DIFO's policy output and tested it across various action-constrained tasks. \nThe results below show that across all tasks, DIFO performs poorly and fails to achieve competitive results.\n\n| Method \\ Task Name | M+B | M+O | HC+B | HC+O | H+B | H+M | W+B | W+L2 |\n| -------- | -------- | -------- | -------- | -------- |  -------- | -------- | -------- |  -------- | \n| **DIFO**               | 0.27 ± 0.04 | 0.45 ± 0.09 | 8 ± 60 | -87 ± 55 | 331 ± 29  | 333 ± 32 | 6 ± 1 | 7 ± 1 |\n| **DTWIL**  | **0.77 ± 0.04** | **0.87 ± 0.04** | **2669 ± 4** | **2637 ± 26** | **2844 ± 57** | **2873 ± 240** | **61 ± 3** | **70 ± 4** |\n\n\n> Section 4 — Should specify what dynamics model is chosen, in the main paper content\n\nThank you for the suggestion. We will move the relevant paragraph from A.2 to the main content.\n\n> L192-197;C2 — The manuscript states, \"... (iii) assigning a⇤t to be the first action of the selected rollout. This a⇤t would be applied to the environment to obtain the next state. This design of taking only the first action ensures that each step of the alignment process can better adapt to the remaining part of the demonstration.\" Common heuristic, but costly in planning complexity. What happens if more actions are executed before replanning?\n\nWe appreciate the reviewer's suggestion. This approach can indeed improve time complexity. We conducted an ablation study on varying the number of steps executed before replanning. The results show that when MPC executes multiple steps per planning cycle, the alignment performance deteriorates as the step count increases. In the Hopper environment, poor alignment further increases the risk of falling.\n\n| Steps \\ Task Name | HC+B | HC+O | H+B | H+M |\n| -------- | -------- | -------- | -------- | -------- |\n| 1 step     | **2669 ± 4** | **2637 ± 26** | **2844 ± 57** | **2873 ± 240** |\n| 3 steps     | 2427 ± 0 | 2324 ± 28 | 810 ± 21 | 753 ± 43 |\n| 5 steps     | 1270 ± 464 | 2123 ± 6 | 579 ± 2 | 579 ± 10 |\n\n\n> General: The manuscript should properly use inline citations\n\nThank you for the suggestion. We will carefully check the inline citations for the final version.\n\n \n> L115;C1: \", using\" —> \"uses\"\n> Pham et al., 2018b — duplicate entry\n> L175;C1: \"describing\" —> \"describe\"; \"We start .. and then describe\"\n> 184;C2: \"help a\" —> \"help of a\"\n> Equation 1 — Notation error: \\tao_e has not been defined as the expert trajectory yet; however, \\tao was introduced as the expert state-action trajectory on L163\n\nThank you for catching these typos. We will fix them in the final version.", "claims_and_evidence": "L162-164;C1 — Wouldn’t it be more accurate to say that there exist trajectories within \\mathcal{T} that cannot be entirely replicated by the learner, since it contains one or more actions that are outside the learner’s feasible set. For one thing, this doesn’t preclude corresponding states to be inadmissible for the learner. Another thing is that these trajectories only make up a subset of all the trajectories in \\mathcal{T}. I don’t know where the intuition behind the use of the term “generally”, here, comes from.", "methods_and_evaluation_criteria": "Sections 3.2, 4.2 — Why wouldn’t a diffusion policy alleviate all of these issues? The manuscript should compare with Huang et al., 2024: https://arxiv.org/pdf/2410.05429\n\nSection 4 — Should specify what dynamics model is chosen, in the main paper content", "theoretical_claims": "N/A — no proofs or theoretical claims", "experimental_designs_or_analyses": "Section 5.1 — Old baselines; missing DIfO from Huang et al., 2024, which addresses the same problem with the same set of experiments\n\nL192-197;C2 — The manuscript states, \"... (iii) assigning a⇤t to be the first action of the selected rollout. This a⇤t would be applied to the environment to obtain the next state. This design of taking only the first action ensures that each step of the alignment process can better adapt to the remaining part of the demonstration.\" Common heuristic, but costly in planning complexity. What happens if more actions are executed before replanning?", "relation_to_broader_scientific_literature": "Huang et al., 2024 — method, idea, implementation supersedes the present manuscript\n\nKasaura et al., 2023; Lin et al., 2021; Brahmanage et al., 2023; Chen et al., 2024 — ACRL approaches that inspires the proposed method\n\nLee et al., 2021; Huang et al., 2024; Torabi et al., 2018b; Yang et al., 2019; Zhu et al., 2020; Torabi et al., 2018a — similar training objectives, problem definitions, settings", "essential_references_not_discussed": "Huang et al., 2024 was cited briefly, but surprisingly not used as a baseline, since it pursues the same problem and even has the same evaluation", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "General: The manuscript should properly use inline citations\n\nL115;C1: \", using\" —> \"uses\"\n\nPham et al., 2018b — duplicate entry\n\nL175;C1: \"describing\" —> \"describe\"; \"We start .. and then describe\"\n\n184;C2: \"help a\" —> \"help of a\"\n\nEquation 1 — Notation error: \\tao_e has not been defined as the expert trajectory yet; however, \\tao was introduced as the expert state-action trajectory on L163", "questions_for_authors": "L116-120;C2 — Why does LfO underperform so much, if it’s trying to do the same thing? Why not compare to DIfO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HlBsp117UK", "reviewer": "Reviewer_Fdv6", "summary": "This paper explores a new task, Action-Constrained Imitation Learning, that the imitator may have the constrained action space compared with export. Then, this paper analyze the challenge of this task and propose a methods to solve this challenge. Specially, this paper first employ MPC to build a surrogate dataset, where the trajectories in this dataset are similar with the export and with same action space with imitator.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and effort in reviewing our work and raising valuable questions. We provide our point-by-point response as follows.\n\n> Analyzing the rationality of constraint design and the strength of constraints is crucial for exploring this new task. Conducting more experiments can contribute to this point.\n\nThanks for the suggestion. We would like to emphasize that the constraint design follows the standard ACRL literature [1-3], where similar constraints have been considered. Specifically, box and L2 constraints directly limit the numerical range of actions, while state-dependent constraints regulate the agent's power consumption. We provide results on how DTWIL and baseline methods perform under different constraint strengths in the later sections.\n\n[1] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.\n\n[2] Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe, “Benchmarking actor-critic deep einforcement learning algorithms for robotics control with action constraints,” Robotics and Automation Letters, 2023.\n\n[3] Jyun-Li Lin,Wei Hung, Shang-Hsuan Yang, Ping-Chun Hsieh, and Xi Liu, “Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization,” UAI 2021.\n\n> Why are the constraints designed according to the content in Figure 01, and what are the reasons for such a design? \n\nWe believe you are asking about the constraint used in the Maze2d toy example in Figure 1. In this case, we use a box constraint, limiting action values within [−0.5, 0.5]. Our goal is to illustrate, in a simple environment with a basic constraint, how standard IL algorithms may cause the learner to deviate from the intended trajectory under action constraints, leading to task failure. We did not delve too deeply into this example, as it primarily serves as an intuitive illustration to highlight the challenges of action-constrained imitation learning.\n\n> How does the proposed method perform under different constraints? If the constraints are very weak, can the direct learning still achieve good results? If the constraints are very strong, will the proposed method also be unable to handle them? Is the proposed method only effective within a relatively narrow range of constraints?\n\nIf the constraints are loose, their impact on the learner's ability to replicate the expert trajectory is minimal. Therefore, if the constraints are loose enough such that they have no effect on reproducing the expert trajectory, direct learning can still achieve good results. In our experiments, we select moderately tight constraints to ensure they significantly impact learning. Here, we show how DTWIL performs under tighter and looser constraints.\n\n| Task (Constraint Level) \\ Method | CFIL-s | CFIL-sa | BCO | OPOLO | DTWIL |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| M+B (0.3)  | 0.20±0.14 | 0.21±0.03 | 0.44±0.34 | 0.44±0.16 | **0.86 ± 0.04** |\n| M+B (0.1) - paper  | 0.22 ± 0 | 0.23 ± 0.21 | 0.14 ± 0.05 | 0.20 ± 0.06 | **0.77 ± 0.04** |\n| M+B (0.07)    | 0.05 ± 0.02 | 0.06 ± 0.01 | 0.26 ± 0.10 | 0.34 ± 0.13 | **0.68 ± 0.01** |\n|||||||\n| HC+O (15) | 2065 ± 714 | **2796 ± 247** | -151 ± 13 | 47 ± 673 | **2785 ± 17** |\n| HC+O (10) - paper  | 1422 ± 1830 | 1674 ± 1316 | 6 ± 31 | -9 ± 80 | **2637 ± 26** |\n| HC+O (7)  | 54 ± 671 | 354 ± 1587 | -130 ± 9 | 99 ± 941 | **1408 ± 25** |", "claims_and_evidence": "Yes,  the task is meaningful and the claims is convincing.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation can effectively verify the claims.", "theoretical_claims": "Yes, the building process of surrogate dataset by MPC is correct.", "experimental_designs_or_analyses": "Yes, the experimental designs are reasonable.", "relation_to_broader_scientific_literature": "This paper is related to constrained RL.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-written overall, with technical points and experiments clearly articulated.\nThis task is meaningful and interesting.\nThe framework is feasible for this task, and its performance surpasses previous methods, according to the outcomes in the paper.\n\nWeaknesses:\nAnalyzing the rationality of constraint design and the strength of constraints is crucial for exploring this new task. Conducting more experiments can contribute to this point.\n\nQuestion:\nWhy are the constraints designed according to the content in Figure 01, and what are the reasons for such a design?\nHow does the proposed method perform under different constraints? If the constraints are very weak, can the direct learning still achieve good results? If the constraints are very strong, will the proposed method also be unable to handle them? Is the proposed method only effective within a relatively narrow range of constraints?", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4K4AfrIN33", "reviewer": "Reviewer_G927", "summary": "This paper focues on a problem setting called Action-Constrained Imitation Learning. This setting involves an imitator agent that learns from an expert demonstrator, but the imitator has a more restricted action space than the expert. This difference in action capabilities creates a mismatch in the occupancy measure, making standard imitation learning methods ineffective.\n\nTo address this, the authors propose a new method called Dynamic Time Warping Imitation Learning. DTWIL works in two stages:\n\n1. Surrogate Demonstration Generation: It replaces the original expert demonstrations with a \"surrogate\" dataset. This new dataset contains trajectories that are similar to the expert's state trajectories but adhere to the imitator's action constraints. This alignment is achieved by using MPC and Dynamic Time Warping distance. The MPC treats trajectory alignment as a planning problem, generating feasible actions for the imitator, while DTW serves as the similarity metric between the surrogate and expert trajectories.\n\n2. Imitation Learning: Once the surrogate dataset is created, any standard imitation learning algorithm (like BC or IRL) can be used to train the imitator policy.\n\nExperiments on robot control tasks show that DTWIL outperforms other imitation learning methods in this constrained setting.\n\n## update after rebuttal\n\nThe authors addressed my major by adding an additional baseline, so I raised my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive feedback.\n\n>This paper overlooks an important set of baselines—state-based imitation learning [1,2,3,4]. In fact, the problem setting of \"Action-Constrained Imitation Learning\" is not entirely new and has been extensively discussed in these works. However, this submission does not discuss these papers.\n\nThank you for your suggestion. We will include a discussion on this line of work in the main text. [1] and [2] focus on handling transition dynamics mismatches in state-only IL. [3] also uses state distribution matching to align the agent’s state trajectories with the expert's, and it’s more specific for dexterous manipulation tasks. [4] proposes a non-adversarial LfO approach. These works align the state-trajectories induced by **unconstrained actions**—trajectories that the constrained imitator cannot perfectly replicate, thus failing to achieve optimal performance. We found SAIL [1] the most relevant to our problem formulation, so we chose SAIL as a baseline. To adapt SAIL to the ACIL setting, we projected the actions generated by the SAIL policy onto each task’s feasible set before interacting with the environment, like other baselines. The table shows the return of SAIL on benchmark tasks. SAIL is only comparable to DTWIL in the HC+O and W+B task, while in all other tasks, it fails to achieve good returns.\n\n|Task Name|M+B|M+O|HC+B|HC+O|H+B|H+M|W+B|W+L2|\n|-|-|-|-|-|-|-|-|-|\n|SAIL|0.45±0.06|0.46±0.15|1222±260|**2666±55**|1375±790|1273±495|**61±9**|58±3|\n|**DTWIL**|**0.77±0.04**|**0.87±0.04**|**2669±4**|**2637±26**|**2844±57**|**2873±240**|**61±3**|**70±4**|\n\n>I did not find a detailed explanation of how the action constraints are implemented in the simulator. If the weak imitator outputs an action that violates the action constraints, is it discarded or somehow mapped back to a valid action?\n\nWe follow the standard approach in the ACRL literature to enforce action constraints [5-7]. Specifically, we use a projection layer that ensures all output actions are mapped to the feasible set before interacting with the environment. This is mentioned in L364–L366, C1, and the caption of Figure 2. We appreciate the reviewer’s feedback and will add further clarification in the main text.\n\n[5] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.\n\n[6] Kasaura, Kazumi, et al. \"Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints.\" IEEE Robotics and Automation Letters 8.8 (2023): 4449-4456.\n\n[7] Lin, Jyun-Li, et al. \"Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization.\" Uncertainty in Artificial Intelligence. PMLR, 2021.\n\n>The proposed method employs MPC to generate action-constrained surrogate demonstrations. However, MPC relies on the ability to reset the environment to specific states, which is feasible in simulation but not transferable to real-world scenarios.\n\nThe reviewer raises a valid concern regarding MPC's reliance on environment resets when using true dynamics for planning. However, our method mitigates this limitation by leveraging a learned dynamics model for planning, as done by various existing MPC methods [8-9]. Instead of requiring resets, we roll out possible trajectories within this model, making our approach more adaptable to real-world scenarios. Further details can be found in PETS [10].\n\n[8] Draeger, Andreas, Sebastian Engell, and Horst Ranke. \"Model predictive control using neural networks.\" IEEE Control Systems Magazine 15.5 (1995): 61-66.\n\n[9] Kamthe, Sanket, and Marc Deisenroth. \"Data-efficient reinforcement learning with probabilistic model predictive control.\" International conference on artificial intelligence and statistics. PMLR, 2018.\n\n[10] Chua, Kurtland, et al. \"Deep reinforcement learning in a handful of trials using probabilistic dynamics models.\" Advances in neural information processing systems 31 (2018).\n\n>One implicit assumption of this paper is that the state contains information strongly correlated with task success. However, a more common scenario involves access to only a partial state, meaning the task is a POMDP. It is unclear whether the proposed method would still perform effectively in such cases.\n\nSince even standard MDP settings are not extensively discussed in prior work, we take the first step towards tackling action-constrained imitation learning and leave the challenges of partial observability to future work. One possible direction to address this issue is to learn a latent representation that captures task-relevant information, enabling the agent to infer missing state information.\n\n>In Table 1, the meanings of \"H+B,\" \"M+O,\" \"W+L2,\" etc., are not explained. I assume \"B\" refers to \"box constraints,\" correct?\n\nThe reviewer is correct here. \"B\" refers to \"box constraints.\" We will add explanations to the main text.", "claims_and_evidence": "I think most of the claims in this submission are supported by evidence. However, there are some concerns regarding their experimental evaluation, which will be discussed in sections below.", "methods_and_evaluation_criteria": "Yes, they make sense to me.", "theoretical_claims": "This submission does not include proofs.", "experimental_designs_or_analyses": "- This paper overlooks an important set of baselines—state-based imitation learning [1,2,3,4]. In fact, the problem setting of \"Action-Constrained Imitation Learning\" is not entirely new and has been extensively discussed in these works. However, this submission does not discuss these papers. I think at least one of these methods should be **included as a baseline**. Without such a comparison, it is not entirely convincing that the proposed DTWIL method offers clear advantages over these existing approaches.\n\n- I did not find a detailed explanation of how the action constraints are implemented in the simulator. If the weak imitator outputs an action that violates the action constraints, is it discarded or somehow mapped back to a valid action? This implementation detail could be crucial to certain design choices. Please let me know if I have overlooked this information in the paper.\n\n\n[1] Liu, Fangchen, et al. \"State Alignment-based Imitation Learning.\" International Conference on Learning Representations.\n\n[2] Gangwani, Tanmay, and Jian Peng. \"State-only Imitation with Transition Dynamics Mismatch.\" International Conference on Learning Representations. \n\n[3] Radosavovic, Ilija, et al. \"State-only imitation learning for dexterous manipulation.\" 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.\n\n[4] Boborzi, Damian, et al. \"Imitation learning by state-only distribution matching.\" Applied Intelligence 53.24 (2023): 30865-30886.", "relation_to_broader_scientific_literature": "See \"Essential References Not Discussed\".", "essential_references_not_discussed": "As mentioned in \"Experimental Designs or Analyses,\" there is a line of work focusing on imitating states in the demonstrations rather than actions. This concept is also a core idea of this submission. Therefore, I believe these works should be discussed and incorporated into the experimental comparisons.", "other_strengths_and_weaknesses": "### Strengths\n\n- The proposed method DTWIL separates the concerns of constraint satisfaction (addressed during surrogate data generation) and imitation learning (which can then leverage any standard IL algorithm), offering greater flexibility in algorithm design.\n- The introduction of Dynamic Time Warping (DTW) as a distance metric makes sense to me. DTW is particularly well-suited for comparing trajectories of varying lengths, which is essential in the ACIL setting.\n- The writing is generally clear and accessible. The figures and tables are neat.\n  \n\n### Weaknesses\n\n- The proposed method employs MPC to generate action-constrained surrogate demonstrations. However, MPC relies on the ability to reset the environment to specific states, which is feasible in simulation but not transferable to real-world scenarios. In contrast, many state-based imitation learning methods mentioned earlier do not have this requirement.\n- One implicit assumption of this paper is that the state contains information strongly correlated with task success (e.g., the velocity of a cheetah or the position of the target object). Under this assumption, minimizing the discrepancy between the state sequences induced by the expert and the imitator is reasonable. However, a more common scenario involves access to only a partial state, meaning the task is a POMDP. It is unclear whether the proposed method would still perform effectively in such cases. Discussions addressing this point would be valuable.\n- In Table 1, the meanings of \"H+B,\" \"M+O,\" \"W+L2,\" etc., are not explained. I assume \"B\" refers to \"box constraints,\" correct?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MskbqzqWQf", "reviewer": "Reviewer_7mJi", "summary": "This work proposes Action-Constrained Imitation Learning (ACIL), a new problem setting in which an imitator learns from a constraint-free expert while operating under action constraints. The key challenge is the infeasible expert actions issue, and the authors solve the mismatch issue by trajectory alignment. The proposed method is DTWIL, which generates surrogate expert demonstrations by aligning constrained trajectories with expert ones using MPC and DTW distance. Experiments on MuJoCo and Robosuite tasks demonstrate superior sample efficiency and performance over standard IL methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful comments and suggestions. We provide our point-by-point response as follows.\n\n> The author should make ablation results about the number of expert demos, and also discuss the different levels of action restrictions for the imitator.\n\nBased on the reviewer’s suggestions, we conducted ablation studies on both the number of expert demonstrations and different levels of action restrictions for the imitator, testing these across two distinct environment settings. The results are shown in the tables below. Our findings demonstrate that our method remains strong under varying constraint levels and can generalize well to datasets with fewer expert demonstrations.\n\nTable 1: **Ablation Study on Different Expert Demos**\n| Number of Demos \\ Task Name | HC+B | H+M |\n| -------- | -------- | -------- |\n| 5 Demo     | 2669 ± 4 | 2873 ± 240 |\n| 3 Demos     | 2585 ± 26 | 2994 ± 35 |\n| 1 Demos     | 2618 ± 28 | 2685 ± 457 |\n\nTable 2: **Ablation Study on Different Constraint Levels**\n| Task (Constraint Level) \\ Method | CFIL-s | CFIL-sa | BCO | OPOLO | DTWIL |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| M+B (0.3)  | 0.20 ± 0.14 | 0.21 ± 0.03 | 0.44 ± 0.34 | 0.44 ± 0.16 | **0.86 ± 0.04** |\n| M+B (0.1) - paper  | 0.22 ± 0 | 0.23 ± 0.21 | 0.14 ± 0.05 | 0.20 ± 0.06 | **0.77 ± 0.04** |\n| M+B (0.07)    | 0.05 ± 0.02 | 0.06 ± 0.01 | 0.26 ± 0.10 | 0.34 ± 0.13 | **0.68 ± 0.01** |\n|||||||\n| HC+O (15) | 2065 ± 714 | **2796 ± 247** | -151 ± 13 | 47 ± 673 | **2785 ± 17** |\n| HC+O (10) - paper  | 1422 ± 1830 | 1674 ± 1316 | 6 ± 31 | -9 ± 80 | **2637 ± 26** |\n| HC+O (7)  | 54 ± 671 | 354 ± 1587 | -130 ± 9 | 99 ± 941 | **1408 ± 25** |\n\n\n> The experiment on robotics is table-wiping, which is too simple. The authors are encouraged to conduct some manipulation tasks, at least picking and placing tasks.\n\nWe appreciate the reviewer's suggestion. However, we have observed that existing and even state-of-the-art ACRL works have not tackled robotic arm tasks either [1-3].Pick-and-place tasks are indeed more complex due to higher-dimensional action spaces and additional constraints such as grasp stability and object interactions[4]. Since this work is the first step in addressing ACIL, we focus on simpler settings and leave the challenge of extending ACIL to more complex robotic manipulation tasks for future work.\n\n[1] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.\n\n[2] Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe, “Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints,” Robotics and Automation Letters, 2023.\n\n[3] Hung, Wei, Shao-Hua Sun, and Ping-Chun Hsieh. \"Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs.\" arXiv preprint arXiv:2503.12932 (2025).\n\n[4] Lobbezoo, Andrew, Yanjun Qian, and Hyock-Ju Kwon. \"Reinforcement learning for pick and place operations in robotics: A survey.\" Robotics 10.3 (2021): 105.\n\n> No generalization experiments are discussed, which is significant for this setting.\n\nWe are unsure about the specific meaning of \"generalization experiments\" mentioned in this comment. However, we believe this may refer to the ablation study on varying numbers of expert demonstrations. In our experiments, we have tested two different environment settings, and the results are presented in Table 2 above.\n\n> Why does DTW outperform the l2 distance to a large degree, can you give more proofs in theorems? Is it the optimization issue or the convergence issue?\n\nDTW distance can be viewed as a special case of the Wasserstein distance, as discussed in Sec 2.1 in [5]. Specifically, when applied to discrete measures, the Wasserstein and DTW distances share the same form of optimization problem. In contrast, L2 distance computes element-wise differences without considering the overall alignment. Consequently, in sequence matching tasks, optimizing with L2 distance can lead to issues since it does not align with the underlying objective [6-7].\n\n[5] Samuel Cohen, Giulia Luise, Alexander Terenin, Brandon Amos, and Marc Peter Deisenroth, “Aligning Time Series on Incomparable Spaces,” AISTATS 2021.\n\n[6] David Schultz and Brijnesh Jain, “Nonsmooth Analysis and Subgradient Methods for Averaging in Dynamic Time Warping Spaces,” Pattern Recognition, vol. 74, pp. 340-358, 2018.\n\n[7] Rohit J. Kate, “Using Dynamic Time Warping Distances as Features for Improved Time Series Classification,” Data Mining and Knowledge Discovery, vol. 30, no. 1, pp. 1-22, 2015.", "claims_and_evidence": "DTWIL is claimed to mitigate the occupancy measure mismatch by aligning trajectories. Experimental results across multiple tasks, such as Maze2d, Hopper, HalfCheetah, and Robosuite, show that DTWIL outperforms baseline methods in terms of task performance and sample efficiency.\n\nThe authors argue that DTW is a more effective metric for trajectory alignment compared to the traditional l2 distance. Ablation studies confirm that policies trained with DTW-aligned trajectories achieve better performance.", "methods_and_evaluation_criteria": "The proposed approach involves generating surrogate expert trajectories that adhere to the imitator's action constraints. This is formulated as a planning problem solved via MPC, with DTW used to measure trajectory similarity. And the evaluation metric is common.", "theoretical_claims": "No theorems.", "experimental_designs_or_analyses": "1. The experiments are designed to assess the effectiveness of DTWIL across various tasks with differing levels of action constraints.\n\n2. The impact of using DTW v.s. l2 distance for trajectory alignment is analyzed, demonstrating the advantage of DTW.", "relation_to_broader_scientific_literature": "The paper addresses a gap in imitation learning research by focusing on scenarios where the imitator operates under action constraints not present for the expert.\n\nThe use of DTW for aligning trajectories introduces a novel application of this technique in the context of imitation learning, differing from traditional methods that may not account for temporal variations in trajectories.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The introduction of ACIL addresses practical challenges in scenarios where imitators have inherent action limitations. The problem setting is interesting and practical.\n\n2. The application of DTW for trajectory alignment in imitation learning is a novel approach that effectively handles temporal discrepancies between expert and imitator trajectories.\n\nWeaknesses:\n1. The author should make ablation results about the number of expert demos, and also discuss the different levels of action restrictions for the imitator.\n\n2. The experiment on robotics is table-wiping, which is too simple. The authors are encouraged to conduct some manipulation tasks, at least picking and placing tasks.\n\n3. No generalization experiments are discussed, which is significant for this setting.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Why does DTW outperform the l2 distance to a large degree, can you give more proofs in theorems? Is it the optimization issue or the convergence issue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Action-Constrained Imitation Learning (ACIL), where an imitator learns from an expert but operates under stricter action constraints. Standard imitation learning fails due to mismatched occupancy measures between the expert and the constrained learner. To address this, the authors propose DTWIL, a method that generates a surrogate dataset aligning constrained actions with expert trajectories using Model Predictive Control (MPC) and Dynamic Time Warping (DTW) distance. Experiments demonstrate that DTWIL improves performance across multiple robot control tasks, outperforming existing imitation learning methods in terms of sample efficiency.

The paper introduces a new problem setting relevant for safe learning, and a novel approach, evaluated with reasonable and interesting experimental settings. 

Incomplete evaluation against the most pertinent baselines is perhaps the main concern. In the AC's judgment, the concerns raised in the review regarding comparison to an "imitation-from-observation" approach are largely addressed through the comparisons to DiFO and SAIL. Of these, SAIL (2019) was designed to permit imitation with different action spaces.


SAIL is an old baseline however, and there is at least one more recent and apparently better-performing approach than SAIL for "imitation learning across *mismatched* expert and student action spaces": Ma et al 2022, SMODICE: https://arxiv.org/abs/2202.02433. 
One would ideally like to see both a clear discussion of this family of works in the paper, and ideally also a comparison.

The AC would also recommend discussing the literature on addressing the "imitation gap" problem, such as Weihs et al, 2024. https://proceedings.neurips.cc/paper/2021/hash/9fc664916bce863561527f06a96f5ff3-Abstract.html
Decision: Accept (poster)