When a Robot is More Capable than a Human: Learning from Constrained Demonstrators
Authors: Xinhu Li
Ayush Jain
Zhaojing Yang
Yigit Korkmaz
Erdem Biyik
Venue: ICLR 2026 Poster (2026)
Keywords: Inverse Reinforcement Learning
Learning from Observations
Learning from Constrained Expert Demonstrations
Robot Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: Inverse RL method to learn from and surpass expert demonstrations provided under interface constraints.

Abstract: Learning from demonstrations enables experts to teach robots complex tasks using interfaces such as kinesthetic teaching, joystick control, and sim-to-real transfer. However, these interfaces often constrain the expert's ability to demonstrate optimal behavior due to indirect control, setup restrictions, and hardware safety. For example, a joystick can move a robotic arm only in a 2D plane, even though the robot operates in a higher-dimensional space. As a result, the demonstrations collected by constrained experts lead to suboptimal performance of the learned policies. This raises a key question: Can a robot learn a better policy than the one demonstrated by a constrained expert? We address this by allowing the agent to go beyond direct imitation of expert actions and explore shorter and more efficient trajectories. We use the demonstrations to infer a state-only reward signal that measures task progress, and self-label reward for unknown states using temporal interpolation. Our approach outperforms common imitation learning in both sample efficiency and task completion time. On a real WidowX robotic arm, it completes the task in 11 seconds, 10x faster than behavioral cloning.

Published as a conference paper at ICLR 2026
WHEN A ROBOT IS MORE CAPABLE THAN A HUMAN: LEARNING FROM CONSTRAINED DEMONSTRATORS
Xinhu Li' Ayush Jain 1.21 Zhaojing Yang! Yigit Korkmaz' Erdem Bıyak'
Thomas Lord Department Computer Science, University of Southern California Meta AI DDepartment of Computer Science & Engineering, University of California San Diego 1ixinhu98@gmall com, ayush j24 memeta com, chy070@ucses edu, (ykorkmaz, biyik; @usc.edu
ABSTRACT
Learning from demonstrations enables experts to teach robots complex tasks using interfaces such kinesthetic teaching, joystick control, and sim-to-real transfer. However, these interfaces often constrain the expert's 's ability to demonstrate optimal behavior due to indirect control, setup restrictions, and hardware safery, For example, joystick can move robotic arm only in a 2D plane, even though the robot operates in higher-dimension collected by constrained experts lead to suboptimal performance of the learned policies. This raises key question: Can a robot learn better policy than the one demonstrated by constrained expert? We address this by allowing the agent to go beyond direct imitation expert actions and explore shorter and more efficient trajectories. We use the demonstrations infer a state-only reward signal that measures task progress, and self-label reward for unknown states using temporal interpolation. Our approach outperforms common imitation learning in both sample efficiency and task completion time. On a real WidowXX robotic arm, it completes the task in 12 seconds, 10x faster than behavioral cloning. We provide real-robot videos and additional resources on our project website: https://sitees.oogg//e goo.gl...om/vew///oossrrneeeeepp
space. As result, the demonstrations
Human Expert
collects demonstrations
under the constraints
of joysttick interface
More capable robot
learns shorter path
beyond the constrained demonstrations
Figure 1: A human expert constrained by a mode-switching joystick produces segmented trajectories. A robot employing LfCD-GRIP executes smooth and efficient motions beyond the demonstrations.
INTRODUCTION
Imitation learning (IL) and inverse reinforcement learning (IRL) are powerful frameworks to acquire complex robotic behaviors from expert demonstrations (Argall et al., 2009; Abbeel et al., 2010; Arora Doshi, 2021; Hussein et al., 2017; Ravichandar al., 2020; Bıyık et al., 2022). However, in practice, human operators often constrained by the control interface, occluded viewpoints, or
Ayush Jain performed this work his personal capacity outside of their work Meta AL.

---

Published as a conference paper at ICLR 2026
physical precision, preventing them from demonstrating optimal behaviors. For example, 6-DoF arm teleoperated with a joystick interface (Herlant et al., 2016; Losey, 2020) requires mode-switching to control different axes. This restricts their ability to perform smooth, simultancous, and multi-axis motions. As a result, the demonstrations exhibit slow, segmented demonstration trajectories (Figure 1). In contrast, the robot is capable of fast, fluid, and coordinated actions across all degrees of freedom.
This discrepancy raises the key problem of Learning from Constrained Demonstrations (LfCD): Can an agent learn from constrained demonstrations and discover more efficient behaviors once those constraints are lifted? Traditional approaches struggle because IL (Schaal, 1999) mimics the suboptimal actions, while IRL (Ho & Ermon, 2016) infers reward functions that reflect the same constraints. While work on learning from suboptimal demonstrations (Gao et al., 2018; Brown et al., 2019) addresses imperfect or noisy expert behavior, LfCD applies to competent experts whose demonstrations are constrained by the interface, leading to goal-directed but inefficient trajectories.
The LfCD problem poses three key challenges to enable agents to explore beyond constrained expert imitation. (1) Since expert actions are restricted by the interface, the IRL reward should be decoupled from the expert action, defined for state-state transitions rather than state-action. (2) Since demonstrations cover only part of the state space, a learning agent must identify which explored states have reliable reward estimates. (3) Even for the novel states encountered during exploration, the agent requires generalizable reward signal.
To address these challenges, we introduce LICD with Goal-proximity Reward InterPolation (LICD- GRIP). (1) To decouple rewards from constrained expert actions, our key insight is to use state-only measure of progress toward the goal. We adopt a goal proximity reward (Lee et al., 2021; Bae et al., 2024), trained along expert demonstration trajectories via backward temporal decay from the goal. However, such estimates do not generalize to observations beyond the demonstration distribution. (2) To ensure reliable reinforcement learning. LICD-GRIP includes a confidence estimator that identifies expert-like observations where goal proximity reward is valid. (3) Finally, to assign rewards in novel states never visited by the experts, interpolates proximity values along agent-collected trajectories. propagating task progress smoothly between the states covered in demonstrations. Intuitively, we use expert-like like observations as anchors and interpolate proximity values across the agent's 's own trajectories to propagate the notion of task progress beyond the constraints.
We evaluate LFCD-GRIP across a range of discrete and continuous control tasks in both navigation and manipulation domains. Empirical results show that LfCD-GRIP consistently outperforms baseline IL and IRL approaches by finding efficient task solutions, particularly in scenarios where demonstrations constrained. For instance, in real-world pick-and-plaat task using WidowX arm, LICD-GRIP reduces task completion time from 100 seconds (under IL. to just 12 seconds. This demonstrates the effectiveness of our method in enabling robots to go beyond the constraints of the demonstrations.
Our contributions in this work are as follows:
1. We introduce the problem of learning from constrained demonstrations (LfCD) to highlight that expert demonstrations are often constrained in practice, limiting the quality of learned behaviors.
2. We propose I. (CCD-GRIP. an IRI. framework that extends goal-proximity rewards beyond expert demonstrations with a confidence-basee proximity interpolator over an agent's 's online learning.
3. We show LICD-GRIP enables more efficient and generalizable policies across multiple domains, outperforming existing IL and IRL baselines under various constrained expert demonstrators.
2 RELATED WORK
Imitation Learning and Learning from Observations. Imitation learning (IL) enables robots to acquire behaviors by mimicking expert demonstrations without requiring access an explicit reward function (Argall et al., 2009; Hussein et al., 2017; Schaal, 1999). The most basic form of II. is behavioral cloning (BC) (Pomerleau, 1988), which treats imitation as supervised learning problem by directly mapping states to actions. Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) introduced an adversarial training framework that matches the state-action distribution of expert demonstrations. These methods replicate the expert's actions, which is suboptimal in LfCD.
Imitation learning from observations addresses the case when expert action labels are unavailable (Liu et al., 2018; Torabi et al., 201 8a; Yang et al., 2019; Liu et al., 2020; Ko et al., 2024; Wen et al., 2024). For instance, GAII, from Observations (GAIfO) (Torabi et al., 2018b) extends GAIL. to
2

---

Published as a conference paper at ICLR 2026
learn solely from state transitions. It mitigates action space mismatch but still replicates expert- like state transitions. In contrast, LfCD-GRIP only follows the expert's ''s demonstrated intended goal and discovers improved policies that leverage the robot's 's full capabilities.
Learning from Suboptimal Demonstrations. Expert demonstrations can be noisy or suboptimal due to limited skill or inconsistent behavior (Choi al., 2019; Yang et al., 2022; Zhu al., 2022; Yu et al., 2023; Gao et al., 2019; Zhang et al., 2021). T-REX (Brown et al., 2019) and D-REX (Brown et al., 2020) infer reward functions from suboptimal demonstrations by ranking trajectory segments, while Self-Supervised Reward Regression (SSRR) (Chen et al., 2021 1) learns a reward by injecting noise into expert trajectories and using noise levels as self-supervised ranking signals. In contrast, LfCD assumes the demonstrator arrives at the correct goal with actions optimal within their constrained action space, but is limited by real-world factors such as interface restrictions or safety constraints. Moreover, our method explicitly leverages the goal-directe nature of tasks to enable policy improvement beyond the constraints, which not leveraged by prior works on learning from suboptimal demonstrations.
Demonstrators. Approaches like Raychaudhuri et al. (2021); Hu et al. (2021); Dasari et al. (2019); Zakka et al. (2022); Xu et al. (2023) learn from experts operating in different action spaces than the robot, due to mismatched embodiment or control interfaces. They typically learn alignment functions or explicit mappings that translate expert demonstration actions into actions feasible within the robot's action space. For example, Pross-Domain Imitation Learning (CDIL) (Raychaudhuri et al., 2021) employs optimal transport to align action distributions before imitation. Similarly, robot-aware control (Hu al., 2021) models the dynamics of both the expert and the robot to bridge the action space mismatch. These techniques aim to reproduce the expert's behavior, whereas we aim to discover better policies that exploit the agent's wider action space.
Learning from
Coal-Proximity Reward Learning from Demonstrations. Proximity-based IRL (Lee et al., 2021; Ma et al., 2022) trains a goal proximity function from demonstrations to provide shaped, dense, action-free free rewards that measure task progress. However. it fails to generalize to states observed in the agent's online exploration beyond the demonstration distribution. As a result, the agent receives low reward in unexplored states, limiting its ability to discover more efficient policies than those demonstrated. LFCD-GRIP addresses this by interpolating proximity values along agent trajectories in online rollouts. Using high-confidence states anchors, construct smoother, more reliable rewards that guide exploration beyond constrained demonstrations.
LFCD PROBLEM FORMULATION
We formulate the LfCD problem Markov decision process (MDP) (Sutton, 1984), defined by the tuple A, R. P. ?)) where S is the state space, A is the action space, R:S x A X +R is the reward function, P(s' a) is the transition distribution, Po is the initial state distribution, and E [0, 1) is the discount factor, A policy [aa) s) defines a distribution over actions conditioned on the current state. The objective is to find a policy that maximizes the expected discounted return,
max EE (50. (O DOUT
rtt R(st, ar Se+1)
(I)
where 7' is the episode length. Our setting focuses on goal-reaching problems, where the primary objective is to complete a task by arriving at a desired terminal state. A key structural property of such tasks is that progress toward the goal can be measured in a state-dependent manner, independent of the specific actions taken. This structural assumption enables us to decouple reward learning from expert actions under constraints. Without access to the reward function R, this objective is achieved by learning from a dataset of expert demonstrations De = {TI, , TK). ). where Tk {so, ao..........................aa.. ST We work in a generalized formulation where access actions is not necessary and state trajectories = {so, $1, ST} are sufficient to define the objective.
Particularly in LICD, expert demonstrations are collected under action space constraints that limit the expert's ' available actions at each state. We denote this potentially unknown constrained action space as (s) S A. indicating that at state s, the expert can only choose from actions a E A (s). In contrast, the learning agent i.e., the robot, has access to the full action space A, and can potentially learn policies that outperform the constrained expert by utilizing actions unavailable during demonstration
"Our method trivially extends to the cases where the robot's action space is also state-dependent as long A (8) S (s) for all E S.
3

---

Published as a conference paper at ICLR 2026
4 APPROACH: LFCD WITH GOAL-PROXIMITY REWARD INTERPOLATION
To address the problem of learning from constrained demonstrations, we develop the Goal-proxing Reward InterPolation (LFCD-GRIP) framework, which extends proximity- IRL. with confidence- guided reward propagation. Our approach builds on the insight that expert actions restricted by the interface, but their demonstrations still contain reliable signals task progress. LICD-GRIP therefore (i defines goal-proximity reward that depends only on states, decoupling reward from suboptimal expert actions, (ii) introduces a confidence estimation module to identify which states states-whethen whether from expert data agent rollouts provide trustworthy proximity values, and (iii) incorporates trajectory-wise interpolation mechanism that propagates the reliable proximity values to novel states encountered during exploration. Together, these components enable the agent to explore efficient behaviors toward the goal and surpass the constraints of human demonstrators (Figure 1).
4.1 GOAL-PROXIMITY AS ACTION-FREE REWARD
Proximity- based IRL (Lee al., 2021) defines rewards based on the estimated proximity of state to the task goal, rather than relying on expert actions. This formulation assumes that expert demonstrations are optimal and collected without constraints. The proximity function fs(s) is trained with two complementary objectives: (1) expert states are assigned exponentially decayed proximity values, such that states closer the goal receive higher values, and (2) agent rollouts are pushed toward zero to avoid overgeneralization The combined loss is
L BANDe (fols)) 8t 8²
E EANDD (fo(s:))
where E (0, 1) is the temporal decay factor, T the trajectory length, D the expert dataset, and DD' the dataset of agent rollouts. We denote 4% as the expert supervision loss, and D online regularization loss, for reference in later sections.
Proximity- based IRL alternates between updating the proximity network and training the policy with rewards derived from it. The reward label for a state transition is the reduction in goal proximity:
Aprox (8t, $1+1) = (+(+))1) - (s()
The policy my is trained to maximize the expected cumulative reward:
max Exg Σ (pe $6+1) = max E
Eo((s)1)- - fo(s))
(4)
While framework is compatible with any reinforeement learning algorithm, proximal policy optimization (PPO) (Schulman et al., 2017) for all experiments. Proximity-baseed IRL provides dense, progress- based reward signal independent of expert actions, making well-suited for learning from demonstrations collected under constrained action spaces.
All agent explored states assigned low proximity, de- incentivizing exploration. While effective in unconstrained settings, Proximity-based IRL struggles when expert demonstrations collected under constrained action spaces. This limitation stems from the objective for online states, C", which penalizes proximity predictions on all states outside the demonstration wwtibbuttoonnncuuuin those that could enable shorter paths the goal. As result, discourages exploration and prevents the agent from discovering more efficient solutions.
Limitation of proximity-base -ased IRL:
To overcome this, we propose to provide meaningful proximity values for -uttof-distribution obser- vations with expert demonstrations. The key idea is to propagate proximity estimates from reliable observations- those with confident and well-calibrated predictions to uncertain online observations. As training progresses and more reliable observations are identified, this propagation extends to broader region of the observation space. This process introduces two main challenges: (1) identify- ing which observations have reliable proximity predictions, and (2) assigning proximity values to uncertain observations based on their relationship to confident anchors. We address both through two core components: a confidence estimation module and a trajectory wise interpolation mechanism.
4.2
CONFIDENCE ESTIMATION MODULE
This module aims to identify reliable observations that can serve as anchors for proximity propagation. We first treat expert observations as reliable, as their proximity values are predefined based on temporal
4

---

Published as a conference paper at ICLR 2026
Algorithm LICD-GRIP
Require:
Expert dataset Dr, decay factor ö, rollout budget N, RL algorithm (e.g.. PPO)
1: Initialize proximity network Jø. policy m
2: Pretrain S using expert loss 2
3: for iteration = to N do
4: Collect rollouts D using policy mo
5: Proximity Model Training:
6: Estimate confidence for each 8, E D using MCD
7: Identify confident endpoints and construct sub-trajectories Deamf
8:
Generate interpolated proximity targets for intermediate states from Deamf
9:
Compute proximity loss: L8 C L3 + L + L
Lymmonf
10:
Update Se using gradient descent on LCRIP
11:
Policy Training:
Compute rewards for policy buffer with Rynox (8t, 8141) fo(0)+1) So((t)
13: Update policy via RL using proximity rewards
14:
end for
distance to the goal. However, we must also distinguish reliable observations among online collected samples, which lie outside the expert distribution.
Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016) provides practical solution. By enabling dropout inference time and performing multiple stochastic forward passes, can estimate the uncertainty of the proximity predictions. The variance of these predictions reflects model uncertainty, with lower variance indicating higher confidence:
K Σ
(onfidence.(s5) = -Va((f((te) =
(s))
fo(s.))
K
k=1
where fo(st)
1
Σ
(k)
(st).
K
and (s.) denotes the output of the proximity network on the k-th forward pass with dropout.
In practice, we pretrain the proximity network on expert demonstrations using the expert supervision loss Cg. which results in low-variance (i.e., high-confidence) predictions on those expert observations. To classify whether an online-collected state high-confidence or not, we compute a dynamic threshold at each iteration based the expert states. Specifically, the confidence threshold is set as maximum proximity variance observed among expert states. Any online state with lower variance than this threshold is treated high-confidence. This design guarantees that all expert states always included as high-confidence anchors throughout training, while also allowing online states with similarly low uncertainty to be used for proximity interpolation.
4.3 PRAL-PROXIMITY INTERPOLATION MECHANISM
Once high-confidents observations are identified, we propagate their proximity values nearby low-confidence observations. Here, nearby refers to temporal rather than spatial proximity. To enable this propagation, we identify sub-trajectories where both endpoints are high-confidence, and use them as anchors for interpolation. Concretely, when two high-confidence obser- vations lie on the same trajectory, the intermediate states are assigned smoothly interpolated proximity values, as illustrated in Figure 2. For each intermediate state 8t, we define proxim- ity target fe by linearly interpolating in the bog-prxximity space between the sub-trajectory' 's start and end points:
Pt Pscart
Taub (Pond - -ezurr)
Pend
Petart
Figure 2: Proximity is interpolated between high-confidence anchors.
f. 8Paart vin (Pow Paan)
(5)
where Tush is the temporal length of the sub-trajectory, and Pstart and Pend are the log-scale goal- proximity distances at its endpoints.
5

---

Published as a conference paper at ICLR 2026
MiniGrid-LKCD
Maze2d
Fetch-Push
Fetch-Pick
WidowX-Pich
Figure We use various manipulation and navigation tasks with different kinds and degrees of constrained expert demonstration datasets.
To stabilize training in the early stages, we introduce an annealing strategy that gradually increases the reliance on interpolated values. each training iteration, we sample a masking probability Pitr E [0, 1] that starts at and linearly decays to over time. With probability p. we mask the interpolated targets, assigning them zero proximity. This encourages conservative learning early on, and enables progressive generalization training proceeds. The resulting loss for interpolated states becomes:
(1- (1-mar) matr) (fo(si)- mar (fo(st)))
Cenm ES
(6)
where matr Bernoulli(par)) is stochastic mask applied independently each intermediate state, and Doonf denotes online sub- sub-trajectories with high-confidence start and end states. This masking strategy allows the model to interpolate proximity values only when becomes confident enough, ensuring smooth propagation without introducing premature bias from uncertain data. We include comparison to the no-masking variant in Appendix C.
For all remaining states those not covered by confident sub-trajectories--w retain the original assumption of zero proximity:
L
unscnn
(Dr (Dew) (fa(s:))²
The full training objective for the proximity function in LfCD-GRIP is:
CRRIP = C. + coonf + C
(8)
The complete LfCD-GRIP training loop, including proximity updates and policy optimization, provided in Algorithm 1. In summary, LFCD-GRIP trains agent with PPO by inferring goal-proximity reward complemented with confidence estimation and proximity interpolation from constrained expert demonstrations.
5
EXPERIMENTS
We investigate the effectiveness of LFCD-GRIP through the following experiments: (1) Can LfCD- GRIP learn to produce optimal trajectories that other methods fail to discover? (2) Does LICD- GRIP outperform standard IL methods and state-of-the-an approaches for learning from suboptimal demonstrations under constrained experts?
(3) Can LfCD-GR GRIP lead to policies that leverage actions unavailable to the expert? (4) How do LfCD-GRIP and baselines perform expert constraints become more severe? (5) What the practical impact of LLCDDGGUL for real-robot applications?
Baselines and Ablations. We compare LICD-GRIP against common imitation learning and inverse RI. baselines, state-of-the-an method for learning from suboptimal demonstrations, and ablations of our approach to validate the technical contributions. The key comparison metric is having short trajectory length of achieving the goals and consistently achieving successes.
BC directly maps observations to actions via supervised learning on expert demonstrations.
GAIL trains a discriminator to distinguish expert observation-action pairs from those generated by the learning agent, using the discriminator reward function.
GATfO removes the need for expert actions by matching cate-trannitton distributions.

---

Published as a conference paper at ICLR 2026
SSRR learns reward by ranking demonstrations with injected noise and using noise severity proxy for suboptimality. is a state-of satie--ff-hea-rrt the-art art method for learning from suboptimal demonstrations.
Proximity Froximity--assee IRL) learns reward function based on the temporal distance to the goal and trains the agent via RL.
Proximity-Dree is ablation of Proximity- IRL with dropout layers enabled, but without confidence estimation or interpolation. This baseline isolates the contribution of our proposed modules from the regularizing effect of dropout.
LFCD-GRIP augments Proximity-Dooe -roo with confidence estimation and interpolation to propagate reliable proximity values to unseen observations.
5.1 LFCD-GRIP DISCOVERS SHORTCUT TRAJECTORIES TO GOAL IN MINIGRID-LFCD
To evaluate whether LfCD-GRIP can recover optimal trajectories beyond those demonstrated, we design simple but illustrative MiniGrid en- vironment (Brockman et al., 2016; Chevalicr-Boiste et al., 2023). The agent always starts in the top-left cor- ner with the goal fixed in the bottom- right. Expert demonstrations, con- strained to the four cardinal directions, traverse only the top row and right- most column. In contrast, the agent allowed to move all eight direc- tions, including diagonals. This asym- metry introduces a shorter diagonal path that lies well outside the expert distribution. As shown in Figure 4, only LfCD-GRIP discovers this optimal shortcut by propagating goal proximity values to unseen but reachable states, while all baselines remain confined to the demonstrated path, yielding longer average trajectories.
MiniGriddCComstranned
Expert
BC
CARC
Trajectory Length
Prodmnit GAlL
Expert
LICD-GRIP
Figure 4: MiniGrid-LfCD Results. (left) The expert follows the blue path to the green goal, while LFCD-GRIP takes the red shorteut; (right) average episode length across methods.
5.2 QUANTITATIVE RESULTS ON CONSTRAINED EXPERT DEMONSTRATIONS
We further validate LICD-GRIP on various navigation and manipulation tasks, as shown in Figure 3: MAZE2D (Fu al., 2020; Kumar al., 2020), FETCHPICK (Plappert et al., 2018), FETCHPUSH (Plap- pert et al., 2018). Details for all environments are provided in Appendix A.
Action Space Constraints. To simulate limited control interfaces, constrain the expert's ac- tion space during demonstration collection. The full action space for continuous environments is normalized [-1, 1] in each dimension. The specific constraints for each environment are follows:
MAZE2D. The robot uses 2D accelerations with experts actions constrained to [-0.1, 0.1]. FETCHPICK. The robot actions control the 3D continuous Cartesian displacements (æ., y. 2) and binary action for the gripper. Expert actions are constrained to |-0.1, 0.1].
FETCHPUSHI The action space matches that of FetchPick except the gripper, which is disabled.
Expert actions are constrained to [00.55, 0.05].
This setup emulates realistic scenarios which robots are capable of high-speed motion, but expert demonstrations are collected under constrained control for safety and reliability.
For each environment, we compare all methods under two settings: (1) the UnconsttainedExxprt setting, where the agent and the expert share the same constrained action space, except in Maze2D, where both the agent and expert use the full action space. This intentional exception allows us to contrast Maze2D with Minigrid; and (2) the ConstrainedExpper setting, where the agent has access to the full action space while the expert demonstrations are generated under constraints. This dual setup assesses LFCD-GRIP's performance gains when constraints are introduced in expert demonstrations.
7

---

Published as a conference paper at ICLR 2026
Maze2d
Fetch-Pick
Fetch-Push
100
Trajectory Length
150
140
Fetch-Pick Constrindd
Fetch-Push Tush-Constrained
Trajectory Length
100
2.80
150
40
1.90
Expert
BC
Pronimity
Procinity-Drop
LCD.CRIPPCUs))
Figure 5: Average episode length across UnconstrainedExper settings (top) and ConstrainedExpert settings (bottom). LICD-GRIP consistently outperforms all baselines constrained settings by finding short trajectory length solutions consistently, and remains robust in unconstrained ones.
Results Discussion. For all environments, we run experiments with four random seeds, and each evaluation checkpoint averages results over 160 episodes. We report the average episode length across all evaluation trajectories from the final trained policy, including unsuccessful attempts. This length metric measures the policy's optimality and ability to leverage the unconstrained action space for faster goal completion. In Figure 5, LECD-GRIP achieves strong performance across all environments in both settings. Although it remains competitive in UnconstrainedExpper, other baselines when the agent operates with an expanded action space in ConstrainedExperr. In the Maze2D-Constrainee setting, although the bar for Proximity-base -based IRL appears visually similar to LICD-GRIP in the figure, LICD-GRIP completes the task in 100 transitions on average-reducing episode length by over 10% compared to Proximity-based IRL, which requires 113 transitions.
consistently outperforms
These results support central claim: LICD-GRIP enables agents go beyond expert constraints by learning -rooxmmtyy reward function, rather than mimicking constrained expert behavior.
5.3 ANALYSIS: DOES LFCD-GRIP LEVERAGE
(OOC) ACTIONS?
To assess whether LFCD generalizes be- yond constrained demonstrations, we analyze the proportion of actions selected by each agent that fall outside the expert's action space in the Maze2D environment. Table reports both the success rate and the ratio of out-of-constraint (OOC) actions. LICD-GRIP achieves a 100% success rate while selecting OOC actions 100% of the time. contrast, GAIL and BC favor in- distribution actions, and GAIfO, despite using OOC actions entirely, fails to achieve high task success. These results underscore the importance of reward generalization, not just action diversity. This analysis further validates that our method is action-independent, as it successfully exploits actions beyond expert constraints achieve optimal performance.
Buseline
Rate
Action
GAIL
69%
71%
BC
12%
69%
GATO
51%
100%
LICD-GRIP
100%
100%
Table 1: Success rate and OOC action ratio in CoonsiD-oostrriined LfCD-GRIP achieves 100% success while effectively leveraging OOC actions.
5.4 ANALYSIS: LFCD-GRIP PERFORMANCE WITH MORE SEVERE EXPERT CONSTRAINTS
We evaluate LfCD -RII under two constraint levels in the FetchPush environment, In the relaxed case, the constraint widened to [-0.7, 0.7]. allowing more expressive expert behavior. the severe case (Severity 2), the expert's action space is limited to [-0.05, 0.05), simulating highly restricted demonstrations. We compare against representative subset of baselines: BC (supervised imitation), Expert (expert performance), and Proximity based IRL (closest to method). We omit GAIL and GAIfO, as they never surpass expert performance and behave similarly to the Expert baseline. SSRR excluded because it fails to learn successful policy under constrained demonstrations.
8

---

Published as a conference paper at ICLR 2026
Time Elapsed
6
(-
BC
Others
Suncesss
GRIP
Figure 7: Real-robot rollouts of the WidowX-Pie X-iick task. Only BC learns meaningful policies, while LICD-GRIP completes the task 10x faster than BC.
Figure shows LfCD-GRIP maintains strong performance across both constraint levels, whereas baselines such BC and Proximity-based IRL degrade substantially under severe constraints. These results demonstrate that LfCD- GRIP works effectively across varying degrees of expert action space constraints, and is able to find short path solutions to the goal consistently by utilizing the agent's s exploration efficiently.
Expert
Trajectory Length
Proxiiity LACD-GRIP(Oums)
201
5.5 WIDOWX-PICK:
Severity Level
SIMULATION AND REALTROBOT EXPERIMENT
Figure 6: Varying constraint severity shows the increasing benefit of LfCD- GRIP over baselines. Severity means constraint [-0.05, 0.05).
We evaluate LICD-GRIP on the WidowX-Pick task, both in simulation and on the real WidowX 250s robotic arm. We use a mode-switching joystick interface (Losey, 2020) to collect demonstrations, which allows control of only one axis at a time. This creates a natural constraint in the expert's action space, yielding constrained demonstrations, Training in simulation (Figure 8) shows that LICD-GRIP outperforms all baselines, achieving substantially shorter trajectories. Except for BC, the other baselines fail to learn meaningful policies, while BC remains limited by the expert's constrained behavior.
We then deploy the Iearned policy the real-world WidowX-Pick setup (Figure 7). While BC reproduces expert-like behavior, fails utilize the full action space and executes slowly, requiring 100 seconds per trial.
WidowX-Pick
300
Trojectory Length
contrast, LICD-GRIP generates efficient trajectories that completes the task 10x faster, in just 12 seconds. These results demonstrate that our method transfers effectively real hardware and enables better-than chaere performance by overcoming action space constraints.
200
180
CONCLUSION & LIMITATIONS
Figure 8: WidowX-Pick Simulation. Only BC and LFCD-GRIP succeed, with LFCD-GRIP being more efficient.
We address the challenge of learning from constrained expert demonstrations, where expert lacks access the robot's is full action space due to interface embodi- ment limitations. We propose LfCD-GRIP, a framework that learns generalizable progress- aware reward function

---

Published as a conference paper at ICLR 2026
via interpolation, enabling agents extrapolate beyond constrained demonstrated behavior and discover more efficient policies. Extensive experiments show that LiCD-GRIP outperforms baselines, highlighting the promise of decoupling reward learning from constrained expert actions.
reward assumes that task progress can be measured with respect to specific goal state. While well suited for goal-reaching tasks, this limits applicability settings without clearly defined terminal conditions. Extending LfCD-GRIP to such tasks remains an important direction for future work. Estimating progress in multi-task scenarios also remains challenging, as our approach currently relies on goal-conditioned proximity estimates tailored to individual demonstrations. Generalizing progress signals across tasks with semantically varied goals will require advances in representation learning and reward modeling.
Limitations. The proximity-
Future Work. While we demonstrate LICD-GRIP in the setting of constrained experts, the core idea of interpolating proximity-base rewards along agent's own trajectories is broadly applicable. In particular, this mechanism enables reward generalization data provides only partial coverage of the state space, which in turn enables more efficient paths to the goals than those observed demonstrations. We therefore view constrained experts as one practically important instance of a more general problem: learning progress-aware rewards from sparse or biased expert coverage, and plan to investigate our approach more broadly.
unseen states whenever the expert
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility, We provide the source code in the supplementary material, with a README file containing commands for all experiments. We describe the simulation environments in Ap- pendix A, the real-robot setup in Appendix B, and the training details in Appendix E and Appendix
LLM USAGE
We used large language models (LLMs) to assist with grammar correction and rewording. No model-generated content was used for scientific claims, experiments, or core contributions. All ideas and analyses are original and developed by the authors.
ACKNOWLEDGEMENTS
We acknowledge funding by the Airbus Institute for Engineering Research (AIER) for this project. We thank the USC Center for Advanced Research Computing (CARC) for providing us with compute resources. We thank Ziyi Liu for valuable discussions and for helping improve the clarity and presentation of the paper.
BIBLIOGRAPHY
Pieter Abbeel, Adam Coates, and Andrew Y. Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 29(13), 2010.
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5), 2009.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Junik Bae, Kwanyoung Park, and Youngwoon Lee. Tldr: Unsupervised gond-conditioned rl via
temporal distance-aware representations. arXiv preprint arXiv:2407.08464 2024.
Erdem Biyık, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45 67, 2022.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.1 015400000001.... 2016.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrr demonstrator imitation learning via automaticaall--ankee demonstrations. In Conference on robot learning, pp. 330-359, PMLR, 2020.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pp. 1262-1277. PMLR, 2021.
Maxime Chevalier-Boisee Bolun Dai, Mark Towers, Rodrigo Perez- Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Systems, 36:73383-7339. 2023.
Sungjoon Choi, Kyungjae Lee, and Songhwai Oh. Robust learning from demonstrations with mixed qualities using leveraged gaussian processes. IEEE Transactions on Robotics, 354))-664-776, 2019.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-roba learning. arXiv preprint orXiv: /215, 2019.
11

---

Published as a conference paper at ICLR 2026
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yang Gao, Huazhe Xu, Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv: 2018.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning
from imperfect demonstrations, 2019. URL https: /tte:///rrxee http://..rrg/ab///eeee8 05313.
Laura V Herlant, Rachel M Holladay, and Siddhartha S Srinivasa. Assistive teleoperation of robot arms via automatic time-optimal mode switching. In 2016 11th ACM/IEEE International Conference Humon-Robot Interaction (IIRI), pp. 35-42. IEEE, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv:1606.03476, 2016.
arXiv preprint
Edward S Hu, Kun Huang, Oleh Rybkin, and Dinesh Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. arXiv preprint arXiv:2107.09047 2021.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50(2):35, 2017.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. The Twelfth International Conference on Learning Representations, 2024.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191, 2020.
Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph Lim. Generalizable imitation learning from observation via inferring goal proximity. Advances in neural information processing systems, 34:16118 18-16130, 2021.
Fangehen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. In International Conference on Learning Representations, 2020.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation:
Learning imitate behaviors from raw video via context translation. 2018 IEEE international conference on robotics and automation (ICRA), pp. 1118-1125. IEEE, 2018.
Dylan Losey. Personalizing robots through learned latent actions. https::/youte J-00EXBi J1 2020. Accessed: 2025-04-27.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXh::2210.00030 2022.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi goal reinforcement learn ing: Challenging roboties environments and request for research. arXiv preprint arXiv: 000.8800004444 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, (1):297-330, 2020.
12

---

Published as a conference paper at ICLR 2026
Dripta Raychaudhuri, Sujoy Paul, Jeroen Vanbaar, and Amit K. Shyy-hhoohhrr.... Cross- imitation from observations. In International conference on machine learning, pp. PMLR. 2021.
Stefan Schaal. Is imitation learning the route humanoid robots? Trends in cognitive sciences, 3(6): 233-242, 1999.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Mas- sachusetts Amherst, 1984.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807.0000..61 058, 2018b.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. In Roboties: Science and Systems (RSS), 2024.
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on robot learning, pp. 3536-3555, PMLR, 2023.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Ad- vances in neural information processing systems, 32, 2019.
Mengjiao Yang, Sergey Levine, and Ofir Nachum. TRAIL: Near-optimal imitation learning with
suboptimal data. International Conference on Learning Representations (ICLR), 2022.
Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty- Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, 2023.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pp- 537-546. PMLR, 2022.
Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-awann imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing Systems, 14:12040-2 2021.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Self-adaptive imitation learningg Learning tasks with delayed rewards from sub-optimal demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):9269-922 2022.
13

---

Published as a conference paper at ICLR 2026
Appendix
A ENVIRONMENT DETAILS
We conduct experiments across four environments: MINIGRID-LFCD, MAZE2D, FETCHPICK, and FETCHPUSH. Below describe their environment dynamics, state/action spaces, and demonstration collection protocols. Note that the full action space for continuous environments is always normalized to [-1,1] 1] in each dimension.
Minigrid-LfCD. This environment is a grid-based navigation task with discrete spatial observations. Each cell in the grid is encoded as a one-hot vector representing one of four categories: wall, empty space, the agent, or the goal. The layout of the grid remains fixed throughout all episodes. The environment fully observable, with observations of size [19 19 x 4J.
The agent always starts in the top-left corner, and the goal is located in the bottom-right corner. The full action space includes 8 discrete movement actions: the four cardinal directions (up. down, left, right) and four diagonals. In contrast, the expert is constrained to only the 4 cardinal directions, simulating limited action capabilities.
The expert dataset consists of single demonstration, generated using a breadth- first search (BFS) planner that computes the shortest path under the constrained action space. This setup ensures that the demonstration is optimal given the expert's limitations, while allowing the agent to potentially discover shorter paths using the full action space.
Maze2D. The agent is point mass that navigates through a 2D maze by controlling its (x, v) acceleration in continuous space. The state includes the agent's ' 2D position, velocity, and the goal position. We slightly modify the standard ma ze2d 1ee-dd-meiiee um -1 environment from D4RL (Fu et al., 2020) by reducing the maximum episode length from 600 to 400 steps.
The full action space A is 2-dimensional continuous space, where each dimension controls acceler- ation in the æ or direction. the constrained setting, the expert's s actions are clipped to a restricted range [-0.1, 0.1], reducing movement magnitude and limiting directional flexibility.
For training, we collect two datasets: 800 expert demonstrations using the full action space, and 800 expert demonstrations under the [-0.1, 0.1] constrained action space, both using the planner provided by D4RL.
FetchPick and FetchPush. These manipulation tasks adapted from the OpenAl Gym Fetch environments (Plappert et al., 2018), where a 7-DoF arm controls its end-effector in 3D space with an additional continuous dimension for the gripper (which is ineffective in FETCHPUSH). The 16- dimensional state vector includes the relative position of the goal to the object, the end-effector to the object, and the robot's joint configuration. Following prior findings from Proximity- IRL (Lee et al., 2021), we exclude velocity information from the state input, which improves performance for learning-from -oom---seraao-ooiiiiio approaches.
For both environments, the full action space A is 4-dimensional continuous space, representing Cartesian displacements x, y. and = directions of the end effector, along with a gripper control signal, which is fixed for FetchPush. In the constrained setting, the expert is limited to actions within a smaller bounded region, [-0.1, 0.1] for FETCHPICK and [-0.05, 0.05] for FETCHPUSH, reducing dexterity and making successful grasps more challenging.
For both FETCHPICK and FETCHPUSH, we collect 400 constrained demonstrations using a scripted policy that moves the gripper above the object, descends to grasp/push it, and transports it to the goal.
B WIDOWX DESCRIPTION
Environment. We use the ManiSkill simulator to collect expert demonstrations and pretrain the policy, followed by sim-to-real transfer to the Widow 250s hardware. The task requires the robot to grasp cube placed on a surface and lift it slightly above that surface to succeed. To simplify the task and reduce orientation complexity, the robot's end-effector fixed in a nownward-pointiee orientation. Additionally, simulation and the real-world hardware, we use -iwdeeensiioall observation space instead of visual inputs. Specifically, the observations provided to the robot include the gripper's 's end-effector position,
mitigate challenges in precisely replicating visual setups between
14

---

Published as a conference paper at ICLR 2026
the gripper opening state, the cube's position and whether the cube is grasped. To ensure clarity of the cube's ss state for the agent, the cube is initialized randomly at one of three fixed, predetermined positions at each environment reset.
Expert Action Space. Expert demonstrations are collected in simulation using a Machenike G5 controller. To minimize accidental inputs and ensure precise control, map discrete movements to the controller's s directional pad (D-pad) and ABXY buttons. Specifically, the D-pad is used to command horizontal movements-leff,, right, forward, and backward-while the A and Y buttons control vertical movement (up and down). The B and X buttons control the opening and closing of the gripper. Human demonstrators thus issue discrete, single-axis commands sequentially, restricting simultaneous multi-axis control and limiting the range and complexity of demonstrated actions.
Agent Action Space. The robot agent operates in continuous 4-dimensional action space: three degrees for Cartesian movement and one for gripper actuation. Unlike the human expert, the agent can perform smooth and simultaneous multi-axis movements, enabling more efficient trajectories and improved manipulation behaviors.
C FURTHER ABLATION: No MASKING
To evaluate the importance the masking strategy in LfCD- GRIP, conduct an ablation study in which the masking probability is removed- -.eeedddi....a..... the interpolated values are always used training targets for intermediate states. This variant is evaluated on the BetchPPck-Constrainig environment.
150
125
Trajectory Length
100
As shown in Figure 9, removing the masking leads the agent to become overconfident its early interpolations. This results in reward propagation through unreliable states, ultimately pre- venting the policy from generalizing and achieving successful task completion. These findings highlight the importance of gradual interpolation: masking helps regulate learning by lim- iting reward propagation only confident regions in the early stages of training.
25
Masking
w/o Masking
Figure 9: Ablation of the masking strategy for interpolated values.
D FULL TRAINING
CURVES FOR ALL SIMULATION EXPERIMENTS
This section presents the complete training curves for all baselines across the simulation experiments in Figure 10. Unlike the main paper, which reports only the final converged values, these curves illustrate the learning dynamics and stability of each method throughout training.
Note: For BC and SSRR, which use different X-axes, we represent their results with horizontal line indicating final performance.
15

---

Published as a conference paper at ICLR 2026
Maze2d
Maze2d Constrained
600
Trajectory Length
480
Trajectory Length
500
360
240
200
120
100
Env Steps
Env Steps
Je6
FetchPick
FetchPick Constrained
150
150
Trajectory Length
Trajectory Length
120
90-
60
0.0
1.5
3.0
45
6.0
0.0
15
3.0
4.5
6.0
Env Steps
tx?
Env Steps
FetchPush
FetchPush Constrained
500
Trajectory Length
Trajectory Length
400
300
300
200
100
100
Env Steps
107
Env Steps
107
MiniGrid
WidowX
100
300
Trajectory Length
Trajectory Length
60
180
120
0.0
0.5
1.0
1.5
2.0
0.00
0.25
0.50
0.2%
1.00
Env Steps
Env Steps
SSRR
GAIO
GAIL
Proximity
Procimity-Drop
LICD-GRIP(OUrs)
Figure 10: RL Training Curves: UnconstrainedExpper settings (left) and EonstrainedEspert settings (right) except Minigrid and WidowX. Both of them belong to ConstrainedExpert settings
16

---

Published as a conference paper at ICLR 2026
E NETWORK ARCHITECTURES
Actor and Critic Networks. The actor and critic networks share the same architecture, differing only the final layer: the actor outputs an action distribution, while the critic outputs a scalar value estimate. For MiniGrid-LfCD, C3 2, 16) ReLU- MaxPool(2.2) CONV(3, 2, 32) ReLU CONV 2, 64), followed by two fully connected layers of size 64. Here, CONV (k, s) c) denotes convolutional layer with kernel size k stride s, and с output channels.
use convolutional encoder with the following structure:
For all other environments, layer size 256. For continuous control tasks, the final layer the actor MLP outputs both the mean and standard deviation of a Gaussian distribution over actions. We use ReLU activations for MiniGrid-LfCD and tanh activations elsewhere.
separate 3-layer MLPs for the actor and critic, each with hidden
Goal Proximity Function and Discriminator. The proximity function and discriminator networks adopt the same encoder architectures as the policy networks. For image-based inputs, we use the same convolutional encoder above, followed by single hidden layer of size 64. For other tasks, we use 3-layer MLP with 64 hidden units. For uncertainty estimation, we maintain an ensemble of proximity networks.
F TRAINING DETAILS
For all baselines (except BC), we train policies using PPO (Schulman et al., 2017). A full list of training hyperparameters for each environment is provided in Table 2.
Hyperparameter
Minigrid-LffCD Maze2I
FetchPick
FetchPush
WidowX
PPO related
Entropy Coefficient
1e-2
le-2
le-3
1e-3
le-4
learning Rate
le-3
le-3
le-4
le-4
le-3
Epochs per Update 4
10
10
Mini-batcher Rollout Size
4
4
32
32
32
le4
le4
4096
4096
4096
Proximity Function related
Discount Factor
0.95
0.95
0.99
0.99
0.95
learning Rate 8
1e-3
le-3
1e-3
1e-4
le-3
Batch Size Epochs for Pre-training
32
32
128
128
128
2
5
2
500
Table 2: Policy-specific Hyperparameters
G DIFFERENCES FROM SUBOPTIMAL DEMONSTRATORS
Section 2 discusses learning from suboptimal demonstrations and explains that LfCD assumes constrained-but-coom but-competent demonstrators instead of noisy or inconsistent experts. Here, we elaborate the distinction more explicitly. Conceptually, REX, SSRR) assume that expert and agent share the same action space, that the expert is unconstrained but imperfect, and that near-optimal behavior is already present in the data; their objective is to denoise or rank trajectories to recover the best actions hidden in suboptimal demonstrations. In contrast, LfCD assumes competent expert who is structurally constrained by the interface (e.g.. switchingg joystick), while the robot later operates setting, the agent's optimal policy lie beyond anything the expert physically demonstrate. so the goal to extract a goal-directed, state-only notion of progress from constrained trajectories and extrapolate efficiency beyond them. Table 3 summarizes these differences.
methods (c.g.. T-REX, D-
strictly larger action space. In this
17

---

Published as a conference paper at ICLR 2026
Table 3: Key differences between settings with suboptimal demonstrators and constrained demonstra- tors.
Property
Suboptimal demonstrators
Constrained demonstrators
Action space
Shared between expert and agent.
Expert acts in restricted action subspace; agent has a larger space.
Expert behavior
Expert behaviors are noisy/inconsistent; some trajectories are better than others.
Expert is optimal under interface con- straints (e.g.. moode-witchingg joystick).
Agent policy
Assumes the agent's optimal policy is present in the expert data under noise.
Agent's optimal policy can lie beyond what the expert demonstrates
Problem objective
Denoise/rank trajectories to uncover the best actions in suboptimal data.
Extrapolate beyond demonstrated behav- ior to outperform the expert.
Learning signal
Relative quality of trajectories (rankings, noise levels).
Goal-reaching state-only progress to be independent of constraints.
Suitable application
Unconstrained but imperfect experts in shared action space.
Interface- conttrnn perts with action-space mismatch.
H TIME PENALTY Do NOT ENABLE REWARD EXTRAPOLATION
natural alternative to interpolation mechanism is augment proximity- -based IRL with global time penalty, encouraging shorter trajectories. We analyze this modification and show that temporal penalization does address the fundamental limitation of reward extrapolation beyond expert coverage.
Early Termination Bias. In environments that terminate early upon failure, time penalty en- courages the agent to fail quickly in order to minimize cumulative penalty. contrast, LFCD-GRIP continues to reward progress toward the goal and remains effective regardless episode length.
Uniform Penalization of Transitions. A global time penalty treats all transitions equally, failing to distinguish between productive and unproductive steps. As result, even sub-trajectories containing meaningful progress may be undervalued embedded in longer episodes. LFCD-GRIP, by contrast, identifies and leverages high-quality sub-trajectories through interpolation, preserving valuable learning signals.
Interaction with Dynamic Reward Learning. Selecting appropriate penalty scale is non-trivial. If the penalty is too large, it can overwhelm the learned reward signal. Since proximity-based IRL uses dynamic, evolving reward during training, this interaction becomes difficult calibrate in practice.
Empirical Evaluation. Empirically, ran an ablation on Maze2D comparing Proximity-IRL with various time penalty magnitudes. None matched the performance of LfCD- further supporting the limitations of simple temporal penalization.
Method
LICD-GRIP
Proximity
+0.1 penalty
+0.01 penalty
+0.001 penalty
Avg. Episode Length
101
112
119
111
112
Table 4: MazeeD-Constrainee Adding global time penalty Proximity-IRL does not recover the
efficiency achieved by LFCD-GRIP.
I HYPERPARAMETER SENSITIVITY ANALYSIS
To assess the robustness of LICD-GRIP, we conduct hyperparameter sensitivity analysis across four key hyperparameters: (1) size of the expert dataset(12.5%) 25%, 50%, 100%). (2) number of pretraining iterations(2, 5, 10), (3) temporal decay factor ă used in proximity value calculation(O.9, 0.95, 0.99), and (4) the number of Monte Carlo samples K used for confidence estimation(3, 5, 10). The first experiment is conducted MAZF2D, and the remaining three MINIGRID.
18

---

Published as a conference paper at ICLR 2026
For MINIGRID, the result reported in the main paper is 25.2 average episode length. The optimal path has length 24, while the constrained expert's trajectory length is 32. All baselines average above 32, meaning they fail to surpass the constrained expert. In contrast, LFCD-GRIP achieves over 25% improvement.
For MAZE2D, reported result is average trajectory length of 103, significantly outperforming the constrained expert whose average is 264.37. Also obviously outperforms the best baseline, Proximity-IRL, with an average trajectory length of 113.
As shown Table 5, except for pretrain iteration number = 10 and K = 3, LICD-GRIP demonstrates strong and consistent performance across different hyperparameter choices.
Table 5: Hyperparameter Sensitivity Analysis
Expert Dataset Size
Trajectory Length
Pretrain Iteration Number
Trajectory Length
25 %
104
2
25.2
50 %
103
5
25.5
100
105
10
33.5
Trajectory Length
K
Trajectory Length
0.9
25
3
29.5
0.95
25.2
5
25.2
0.99
25.0
7
25.3
10
24.1
19


Reviews:
[{"review_id": "t2cTmAHlwA", "reviewer": "Reviewer_25Mi", "summary": "This paper proposes LfCD, a learning from demonstration method that specializes scenarios where the demonstrators are \"constrained\" compared to the robot. LfCD extends the IRL framework by 1) using a state-only, learned proximity reward and 2) a mechanism to enhance fidelity of the proximity reward on OOD observations. The core idea of 2) is to provides reward target computed from linearly interpolating confident states to the less confident ones.", "questions": "Similar to the weakness section above. \n\n- Can the presented approach apply to non-smooth, or high-dimensional state space, or how can one extend the method to make it work on them?\n- How does the presented approach improve over the baseline over various \"constrained\" level of the demonstrator?\n- Can we get a more detailed analysis on the sensitivity of the hyperparameters of the method, such as the confidence/non-confidence threshold, data / compute of the pretraining of the network vs training etc.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and presented. The algobox provides a clear overview of the presented approach.\n+ The core idea of creating pseudo supervision using confident predictions makes a lot of sense", "weaknesses": "- It seems the presented approach only works in setups where the state space is 1) observable / low-dimensional and 2) smooth. If 1) or 2) does not hold, linear interpolation might not be a proper way to obtain the supervision targets. \n- While the presented approach makes sense, I don't quite see how it relates to the motivation of expert being constrained. The proposed method seems rather generic.\n- Similarly, I would like to see how the presented approach improves over the baseline, over various levels of the demonstrator being \"constrained\".\n- I'd like to see a more detailed analysis on the sensitivity of the hyperparameters of the method, such as the confidence/non-confidence threshold, data / compute of the pretraining of the network vs training etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiLjCz1TG9", "reviewer": "Reviewer_pvLM", "summary": "The paper focuses on the setting where the demonstrator performance is hindered due to constraints in observations/actions.\nThe paper questions whether it is possible to create a policy, without said constraints, to outperform the constrained demonstrator using the generated demonstrations.\nThe paper proposes to learn a goal-proximity reward with a confidence-based proximity interpolator, that consists of a confidence estimator and a trajectory-wise interpolator.\nThe former estimates the confidence through Monte-Carlo dropout and aims to identify reliable observations, and the latter estimates low-confidence observations through interpolating the values between reliable observations.\nThe paper then conduct experiments on four simulated tasks and a real-world manipulation picking task, and demonstrate that the proposed method identifies out-of-constraint actions that result in shorter trajectories.", "questions": "- About Eq. 5: Is log-scale goal proximity distances the same as $log(r_{end}) - log(r_{start})$?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The introduced problem is well motivated, especially in the robotic setting where the demonstrator may have less degree-of-freedoms compared to the actualy embodiment.\n- The proposed idea is straightforward and easy-to-follow, and the writing is clear.\n- The analyses on MiniGrid environment and out-of-constraint actions are great to demonstrate that the proposed method identifies better actions beyond the demonstrator.", "weaknesses": "I am happy to increase my score after these comments are addressed.\n- While the confidence estimation module will provide low variance on \"reliable observations\", the uncertainty might come from another source, e.g., having multiple demonstrators gathering data.\n\t- As we scale the number of demonstrators, this might become a degenerate problem where none of the demonstrations contain high-confidence states.\n- There are a handful of design/hyperparameter choices that I am unsure if it's well justified/experimented.\n\t- Decaying strategy for masking\n\t- What's considered as high confidence?\n\t- The choice of interpolation strategy\n\t- There are no sensitivity analysis on any hyperparameters, especially on the introduced modules\n\t\t- $\\delta$, $K$, etc.\n- Nit: It would be great if the paper shows the expert performance on all figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhHEvTj3bV", "reviewer": "Reviewer_jvYJ", "summary": "This paper proposes a framework for Learning from Constrained Demonstrations, Goal-proximity Reward InterPolation (LfCD-GRIP). LfCD-GRIP extends proximity-based IRL with confidence-guided reward propagation. The authors' formulation of LfCD is interesting, specifically focused on scenarios where the demonstrator is optimal within their constrained action space but where the robot has access to a broader set of actions. The methodology is well-designed and straightforward. The authors test their approach against several baselines and ablations of their proposed framework, and find that LfCD-GRIP has several benefits.", "questions": "1. Can the authors comment on why SSRR performs so poorly across these domains? From my knowledge of that framework, I cannot see a specific reason on why that framework should underperform GAIL by such a large margin.\n2. How does the second term in Equation 2 help avoid overgeneralization?\n3. Can you reply to the weaknesses noted above?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper has an interesting formulation of LfCD. I could see many applications where a robot may need to learn from LfCD.\n+ The paper is well-written and clear.\n+ The paper has ample results and a deployment on a real-world robot.", "weaknesses": "- The authors should better explain the difference between their framework and those that learn from suboptimal demonstrations. To my knowledge, works on learning from suboptimal demonstration often have similar motivation and can be applied to the same problem statement posed in this paper. This would also help better highlight the novelty in the proposed work.\n- The results have very large standard deviations, and it is unclear whether LfCD-GRIP is actually outperforming other frameworks. A statistical significance analysis and explanation regarding large standard deviations would be beneficial.\n- The key result noted in the intro (100 seconds to 12 seconds) seems an overclaim given the large standard deviation in Figure 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vh1YrYdh1N", "reviewer": "Reviewer_Q8E8", "summary": "This paper presents LfCD-GRIP, a new way to do imitation learning/learning from demonstration that considers the fact that the input expert demonstrations likely have some amount of \"constraint\" to them. The proposed approach seeks to learn better policies by using \"unconstrained\" actions. This is done by estimating the \"confidence\" in the \"proximity\" of different states to the goal, finding high-confidence states, and interpolating the proximity of the intermediate states. In this way a more generalized model of proximity can be used to guide the PPO RL loss function, which allows learning to visit states and take actions that may not have shown up in the expert demonstrations. The authors show results in simulation and on a real robot against several baselines, showing faster task performance, particularly in the setting where expert actions were constrained.", "questions": "* Exactly how are \"high-confident\" states determined? I want to make sure I understood the code correctly.\n* What is the intuition around how Proximity-Drop performs almost universally worse than Proximity? Did you test LfCD-GRIP without Drop - it seems like it might perform better?\n* Conceptually, why wouldn't a technique like Proximity be able to learn to take diagonal actions in MiniGrid?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality\n- The approach of identifying high-confidence good states and then interpolating between them is novel to me\nQuality\n- The paper compares against several baselines and has ablations\nClarity\n* The graphs and visualizations are clear\n* The paper is generally well-written\n* The literature review is appropriately broad and deep for a problem like this \nSignificance\n- All expert demonstrations are constrained in some way, so this general idea and the problem being tackled has broad applicability across imitation learning.", "weaknesses": "- The contribution of the paper, while novel, is fairly small. Proximity-based approaches are already published work (like the cited Lee et al. 2021). LfCD-GRIP only performs slightly better than the Proximity approach, with the small changes of estimating proximity confidence. Proximity interpolation is done by time, which is the same as Lee et al. 2021.\n- There are so many figures in the paper that there is not much room for discussion of results.\n* The results are not presented in a way that clearly shows that the proposed approach is better than prior work.\n\t* Graphs include error bars, but there is no indication of confidence levels or statistical significance.\n\t* No comparison to Ma et al. 2022, which was listed in the lit review and seems it seeks to solve almost the exact same problem.\n- The explanation of the proposed approach is incomplete.\n\t* Line 251 says \"we identify sub-trajectories where both endpoints are high-confidence, and use them as anchors for interpolation\", but there is no description of how high-confidence endpoints are determined. I had to go to the code to determine how it is done, and it's not a fixed threshold but more complicated. This is an important part of the approach and should be described in detail.\n* The experiments could more clearly illustrate the problem and proposed solution.\n\t* Most of the examples of a \"constrained\" expert are simply limiting the magnitude of the possible motion. This seems like an almost trivial constraint to overcome by using any amount of reward shaping during the RL process, like penalizing total time taken. However, I don't see evidence that this was tested against.\n\t* An experiment where the constraint was relaxed gradually (i.e. increasing the possible action interval in steps) to see that the proposed approach improves as the constraint is relaxed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)