A Dual Approach to Imitation Learning from Observations with Offline Datasets

Authors: harshit  sikchi
caleb chuck ůt
amy  zhang
scott  niekum
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 07:16:40

Abstract: Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board.

Main paper: 
 introduction: 
 Imitation Learning [1] promises to leverage a few expert demonstrations to train performant agents. This setting is also motivated by literature in behavioral and cognitive sciences [2, 3] that studies how humans learn by imitation, for example when mimicking other humans or watching tutorial videos. While learning from a small number of examples is often the motivation, many imitation learning methods [4, 5, 6, 7, 8] typically assume the impractical setting where the learning agent is allowed to interact with the environment as often as needed. We posit that the main reason humans can imitate efficiently is due to their knowledge priors from previous interactions with the environment; humans are able to distill skills from prior interactions to solve a desired task. Examples of expert behavior are commonly available through the ever-increasing curated, multi-robot or cross-embodied, datasets and even through tutorial videos. However, leveraging these expert datasets efficiently presents two challenges: (a) The expert data often comes in the form of observation trajectories lacking action information (e.g. tutorial videos in the observation space of agent, cross-embodiment demonstrations, etc.) (b) The learning agent should be able to leverage its collected dataset of environment interactions to mimic expert behavior. This collected dataset may not contain expert transitions and are referred to as suboptimal datasets or datasets of arbitrary quality. These challenges serve as our key motivation to bring imitation learning closer to these practical settings. We consider ˚Equal contribution 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. the setup of offline imitation learning from observations, where the agent has access to an offline dataset of its own action-labeled transitions of arbitrary quality and is provided with potentially few task-relevant expert demonstrations in the form of observation trajectories.Learning from Observations (LfO) has been widely studied [9, 10, 11, 12] in the online setting, where the agent is allowed to interact with the environment. A common denominator across LfO methods is the use of learned one-step models to compensate for missing expert actions. However, in the offline setting learning accurate one-step models with limited data is challenging and can result in compounding errors during downstream policy learning. These models have either taken the form of a discriminator to predict single-step expert rewards or Inverse Dynamics Models (IDM) to predict expert actions. Methods that learn a discriminator to distinguish the states or state-next states between expert and the suboptimal policy data seek to match visitation distributions of expert and the agent [8, 11, 13, 4, 14] . The learned discriminator serves as a pseudo-reward for the next step of policy optimization. In the offline setting, with limited data, the discriminator is susceptible to overfitting and any errors will compound during RL when treating the discriminator as an expert reward function [15] . A negative side-effect of using discriminator-based distribution matching in LfO is also its reliance on minimizing a proxy upper bound rather than the true objective [11, 13] . The other popular family of algorithms for LfO involves learning an IDM [16, 17] , where the agent uses the offline data to predict actions from consecutive states and uses it to annotate the expert trajectories with actions. The policy is then extracted via behavior cloning on inferred expert actions. Aside from the well-known compounding error issue with behavior cloning (the errors in learned IDM only serve to exacerbate the issue), this approach discards the wealth of recovery behaviors that could be learned from offline datasets to better imitate the expert. Thus, the key question is: Can we derive an efficient, lightweight yet principled off-policy algorithm for learning from observations that (a) learns from offline datasets of arbitrary quality, (b) bypasses the learning of intermediate one-step models, and (c) does not resort to minimizing loose upper bounds?In this work, we frame Imitation Learning from Observations as a modified distribution matching objective between joint state-next state visitations of the agent and expert that enables leveraging off-policy interactions. The distribution matching objective can be written as a convex program with linear constraints. Using the principle of duality, we propose Dual Imitation Learning from Observations or DILO, which converts the distribution matching objective to its dual form, exploiting the insight that the next state leaks information about missing actions. DILO no longer requires knowing expert actions in the agent's action space and instead requires sampling multiple consecutive states in the environment. An overview of our method can be found in Figure 1 . DILO presents three key benefits over prior work: (1) DILO is completely off-policy and optimizes for exact distribution matching objective without resorting to minimizing upper bounds (2) DILO learns a multi-step utility function quantifying the effect of going to a particular next-state by minimizing long term divergence with the expert's visitation distribution, avoiding the compounding errors persistent in methods that learn intermediate one-step models. (3) DILO solves a single-player objective, making the learning stable and more performant. Our experimental evaluation on a suite of MuJoCo [18] environments with offline datasets from D4RL [19] and Robomimic [20] shows that DILO achieves improved performance consistently over the evaluation suite. We demonstrate that DILO scales to image observations seamlessly without extensive hyperparameter tuning. Finally, DILO shows improved real robot performance compared to prior methods, which are observed to be more sensitive to the quality of the suboptimal dataset. 
related work: 
 Learning from Observations: Imitation Learning from Observations (LfO) considers the setting where the expert trajectories are available in the form of observations but missing action labels. This setting is more practical as performant algorithms developed for LfO can unlock learning from a plethora of video datasets and develop ways to transfer skills across embodiments. Unfortunately, learning from observations alone has been shown to be provably more difficult compared to the setting where expert actions are available [21] . As a result, current methods in LfO restrict themselves to small observation spaces and involve complicated learning algorithms that first train a model using offline interaction data to either predict expert actions [22, 11, 23] or learn a state-only reward function [11, 13] in the form of a discriminator. This learned model is used for subsequent Behavior Cloning, as in BCO [16] , or for RL [13, 11] . As a result, prior methods suffer from compounding errors either during training or deployment. The issue of compounding errors in the offline setting with BC approaches or RL with a learned reward function has been investigated theoretically and empirically in prior works [24, 25, 15] . These errors can be fixed with repeated online interaction but can lead to substantially poor performance in the offline setting.The duality perspective in reinforcement learning has been explored in the early works of [26, 27] and has gained recent popularity in the form of Dual RL [28, 29] and DICE [30, 31, 32, 33, 34, 35] methods. Dual approaches formulate RL as a convex program under linear constraints and leverage the Lagrangian or the Fenchel Rockefeller duality to obtain an unconstrained and principled objective for RL. The appeal of the dual perspective stems from the ability of dual approaches to learn from arbitrary off-policy data without being sensitive to distribution shift or losing sample efficiency as traditional off-policy methods [36, 37] . This behavior is attributed to the fact that dual approaches compute the on-policy policy gradient using off-policy data in contrast to traditional off-policy methods, which perform Bellman backups uniformly over state space. Duality has been previously leveraged in imitation learning from observations [35, 13, 11] by first creating an upper bound to the distribution matching objective of imitation learning such that it resembles a (return maximization) RL objective and then solving it using dual RL algorithms. 
preliminaries: 
 We consider a learning agent in a Markov Decision Process (MDP) [38, 39] which is defined as a tuple: M " pS, A, p, R, γ, d 0 q where S and A denote the state and action spaces respectively, p denotes the transition function with pps 1 |s, aq indicating the probability of transitioning from s to s 1 taking action a; R denotes the reward function and γ P p0, 1q specifies the discount factor. The reinforcement learning objective is to obtain a policy π : S Ñ ∆pAq that maximizes expected return: E π "ř 8 t"0 γ t rps t , a t q ‰ , where we use E π to denote the expectation under the distribution induced by a t " πp¨|s t q, s t`1 " pp¨|s t , a t q and ∆pAq denotes a probability simplex supported over A. f -divergences define a measure of distance between two probability distributions given by D f pP }Qq " E x"Q " f p P pxq Qpxq q ı where f is a convex function.The visitation distribution in RL is defined as the discounted probability of visiting a particular state-action under policy π, i.e d π ps, aq " p1 ´γqπpa|sq ř 8 t"0 γ t P ps t " s|πq and uniquely characterizes the policy π that achieves the visitation distribution as follows: πpa|sq " d π ps,aq ř a d π ps,aq . Our proposed objective is motivated by the recently proposed Dual-V class of Dual RL [28] , a 1 qpps|s 1 , a 1 q, @s P S.(The above objective is constrained and difficult to optimize, but the Lagrangian dual of the above objective presents an unconstrained optimization that results in a performant Dual-RL algorithm.EQUATIONwhere f p pyq " max xPR xx ¨yy ´f pxq s.t x ě 0. Our proposed method builds upon and extend this formulation to an action-free LfO setting.Imitation Learning from Observations: We consider the LfO setting where the expert provides stateonly trajectories: D E " trs 0 0 , s 0 1 , ...s 0 h s, ...rs n 0 , s n 1 , ...s n h su. Our work focuses on the offline setting where in addition to the expert observation-trajectories, we have access to an offline interaction data that consists of potentially suboptimal reward-free ts, a, s 1 u transitions coming from the learning agent's prior interaction with the environments. We denote the offline dataset by d O consisting of {state, action, next-state} tuples and ρps, a, s 1 q as the corresponding visitation distribution of the offline dataset. Distribution matching techniques aim to match the state visitation distribution of the agent to that of expert. Although we use s as a placeholder for states, the method directly extends to fully-observable MDP's where we perform visitation distribution matching in the common observation space of expert and agent. 
dual imitation learning from observations: 
 Classical offline LfO approaches that rely on learning a discriminator and using it as a psuedoreward for downstream RL are susceptible to discriminator errors compounding over timesteps during value bootstrapping in RL [40, 21, 41, 15] . The discriminator is likely to overfit with limited data especially when expert observations are limited or high dimensional. Methods that learn IDM and use behavior cloning (BC) only perform policy learning on expert states and suffer compounding errors during deployment as a result of ignoring the recovery behaviors that can be extracted from offline, even suboptimal datasets [24, 25] . The key idea of the work is to propose an objective that directly learns a utility function quantifying how state transitions impact the agent's long-term divergence from the expert's visitation distribution. We derive our method below by first framing LfO as a specific visitation distribution matching problem and then leveraging duality to propose an action-free objective.To derive our method, we first note a key observation, also leveraged by some prior works [10] , that the next-state encodes the information about missing expert actions as the next-state is a stochastic function of the current state and action. We instantiate this insight in the form of a distribution matching objective. We define ts, s 1 u joint visitation distributions denoted by dπ ps, s 1 , a 1 q " p1 γqπpa 1 |s 1 q ř s0"d0,at"πpstq γ t pps t`1 " s 1 , s t " s|πq. Intuitively, it extends the definition of stateaction visitation distribution by denoting the discounted probability of reaching the {state, next-state} pair under policy π and subsequently taking an action a 1 . Under this instantiation, the LfO problem reduces to finding a solution of:EQUATIONas at convergence, dπ ps, s 1 , a 1 q " dE ps, s 1 , a 1 q holds, which implies dπ ps, s 1 q " dE ps, s 1 q and also dπ psq " dE psq by marginalizing distributions. Unfortunately, the above objective (a) requires computing an on-policy visitation distribution of current policy ( dπ ) (b) provides no mechanism to incorporate offline interaction data (d O ), and (c) requires knowing expert actions in the action space of the agent (a 1 ).We now show how framing imitation (Eq. 3) as a constrained optimization objective w.r.t visitation distributions allows us to derive an action-free objective. First, in order to leverage offline interaction data ρ, we consider a surrogate convex mixture distribution matching objective with linear constraints:max dě0 ´Df pMix β p d, ρq}Mix β p dE , ρqqs.t ř a 2 dps 1 , s 2 , a 2 q " p1 ´γq d0 ps 1 , s 2 q `γ ř s,a 1 PSˆA dps, s 1 , a 1 qpps 2 |s 1 , a 1 q, @s 1 , s 2 P S ˆS.(EQUATIONThe constraints above represent the Bellman flow conditions any valid joint visitation distribution needs to satisfy. The mixture distribution matching objective preserves the fixed point of optimization dπ ps, s 1 , a 1 q " dE ps, s 1 , a 1 q irrespective of mixing parameter β, thus serving as a principled objective for LfO. Mixture distribution matching has been shown to be a theoretically and practically effective way [28, 42] of leveraging off-policy data. Prior works [28, 42] dealing with state-action visitation in the context of imitation learning consider an overconstrained objective resulting in a complex min-max optimization. Our work departs by choosing constraints that are necessary and sufficient while giving us a dual objective that is action-free as well as a simpler single-player optimization.The constrained objective is convex with linear constraints. An application of Lagrangian duality to the primal objective results in the following unconstrained dual objective we refer to as DILO:EQUATIONwhere V is the Lagrange dual variable defined as V : S ˆS Ñ R and f p is a variant of conjugate f ˚defined as f p pxq " maxp0, f 1 ´1pxqqpxq ´f pmaxp0, f 1 ´1pxqqq. We derive DILO objective as Theorem 6.1 in Appendix 6.1 where we also see that strong duality holds and the dual objective can recover the same optimal policy with the added benefit of being action-free. Moreover, we show that the solution to the dual objective in Equation 5, V ˚ps, s 1 q represents the discounted utility of transitioning to a state s from s 1 under the optimal imitating policy that minimizes the f -divergence with the expert visitation [43] (Appendix 6.1.2). Intuitively, this holds as the primal objective in Eq 4 can be rewritten as the reward maximization problem E Mix β p d,ρq rrps, s 1 , a 1 qs with rps, s 1 , a 1 q " ´Mix β p d,ρq Mix β p dE ,ρq f pMix β p d,ρqMix β p dE ,ρq q. This reward function can be thought of as penalizing the policy every time it takes an action leading to a different next state-action than the expert's implied policy in agent's action space.An empirical estimator for the DILO objective in Eq. 5 only requires sampling s, s 1 , s 2 under a mixture offline dataset and expert dataset and no longer requires knowing any of the actions that induced those transitions. This establishes DILO as a principled action-free alternative to optimizing the occupancy matching objective for offline settings.To instantiate our algorithm, we use the Pearson Chi-square divergence `f pxq " px ´1q 2 ˘which has been found to lead to stable DICE and Dual-RL algorithms in the past [7] . With the Pearson chisquare divergence, f p takes the formf p pxq " x ˚`max `x 2 `1˘, 0 ˘´``m ax `x 2 `1˘, 0 ˘´1 ˘2.We outline the intuition of the resulting objective after substituting Pearson chi-square divergence in Appendix 6.1.4.At convergence, the DILO objective does not directly give us the optimal policy π ˚but rather provides us with a utility function V ˚ps, s 1 q that quantifies the utility of transitioning to state s 1 from s in visitation distribution matching. To recover the policy, we use value-weighted regression on the offline interaction dataset, which has been shown [44, 45, 46] to provably maximize the V function (thus taking action to minimize divergence with expert's visitation) while subject to distribution constraint of offline dataset:EQUATIONChoice of d0 ps, s 1 q: Any distribution over state and next-state is implicitly dependent on the policy that induces the next-state. The initial distribution in Eq. 5 forms the distribution over states from which the learned policy will acquire effective imitation behavior to mimic the expert.In our work, we set d0 ps, s 1 q to be the uniform distribution over replay buffer ts, s 1 u pairs, ensuring that the learned policy is robust enough to imitate from any starting transition observed from all the transitions available to us.Algorithm 1: DILO 1: Init V ϕ , π ψ 2: Params: temperature τ , mixture ratio β 3: Let D " p ρ " tps, a, s 1 qu be an offline dataset and D E " ts, s 1 u be expert demonstrations dataset. 4: for t " 1..T iterations do 5:Train V ϕ via Orthogonal gradient update on Eq. 5 6:Update π ψ by minimizing Eq. 6 7: end for Practical optimization difficulty of dual objectives: Prior works in reinforcement learning that have leveraged a dual objective based on Bellman-flow constraints suffer from learning instabilities under gradient descent. Intuitively, in our case, learning instability arises as the gradients from V ps, s 1 q and V ps 1 , s 2 q can conflict if the network learns similar feature representations for nearby states due to feature co-adaptation [47] . Prior works [28] have resorted to using semi-gradient approaches but do not converge provably to the optimal solution [29] . To sidestep this issue, we leverage the orthogonal gradient update proposed by ODICE [29] for the offline RL setting that fixes the conflicting gradient by combining the projection of the gradient of V ps 1 , s 2 q on V ps, s 1 q and the orthogonal component in a principled manner. We refer to the ODICE work for detailed exposition. Our complete practical algorithm can be found in Algorithm 1. 
experiments: 
 In our experiments, first, we aim to understand where the prior LfO methods based on IDM or a discriminator fail and how the performance of DILO compares to baselines under a diverse set of datasets. Our experiments with proprioceptive observations consider an extensive set of 24 datasets. The environments span locomotion and manipulation tasks, containing complex tasks such as 24-DoF dextrous manipulation. Second, we examine if the simplicity of DILO objective indeed enables it to scale directly to mimic expert image observation trajectories. Finally, we test our method on a set of real-robot manipulation tasks where we consider learning from a few expert observations generated by human teleoperation as well as cross-embodied demos demonstrated by humans as videos.We use offline imitation benchmark task from [28, 13] where the datasets are sourced from D4RL [19, 18] . For locomotion tasks, the benchmark uses an offline interaction dataset consisting of 1-million transitions from random or medium datasets mixed with 200 expert trajectories (or 30 expert trajectory in the few-expert setting). For manipulation environments, the suboptimal datasets comprise of 30 expert trajectories mixed with human or cloned datasets from D4RL. The expert demonstrates 1 observation trajectory for all tasks. DILO uses a single set of hyperparameters across all environments. Hyperparameters and additional experimental details can be found in Appendix 6.2.1.Baselines: We compare DILO against offline imitation from observations (LfO) methods such as ORIL [48] , SMODICE [13] as well as offline imitation from action-labelled demonstration (LfD) methods like BC [49] , IQ-Learn [50] and ReCOIL [28] . We choose these imitation learning methods as they represent the frontier of the LfO and LfD setting, outperforming methods like ValueDICE [42] and DemoDICE [34] as shown in prior works. Intuitively, the imitation from action-labeled demonstrations represents the upper bound of performance as they have additional information on expert actions even though sometimes we observe LfO algorithms to surpass them in performance. ORIL and SMODICE first learn a discriminator and, subsequently run downstream RL treating the discriminator as the expert pseudo-reward. on a learned discriminator falls sharply potentially due to overfitting of discrimination that results in compounding downstream errors. DILO gets rid of this intermediate step, completely reducing the problem of LfO to a similar training setup as a traditional actor-critic algorithm. BC methods, representing an upper bound to BCO [16] shows poor performance even without a learned IDM.Learning to mimic expert in the image observation space presents a difficult problem, especially in the absence of a pretrained representations. To evaluate our algorithm in this setting, we consider the Robomimic datasets [20] which gives the flexibility of choice to use image observations or the corresponding proprioceptive states for learning. Our suboptimal datasets comprises of Multi-Human (MH), Machine Generated (MG) datasets from Robomimic without access to expert trajectories. We obtain 50 expert-observation trajectories from Proficient Human (PH) datasets. This setup is more complicated as the agent has to learn expert actions purely from OOD datasets and match expert visitations. We consider the most performant LfO baseline from the previous section SMODICE [13] along with a BCO [16] baseline as BCO has shown success in scaling up to image observations [51] . Setup: Our setup utilizes a UR5e Robotic Arm on a tilted 1.93m ˆ0.76m Wind Chill air hockey table to hit a puck or manipulate tabletop objects. Puck detection utilizes an overhead camera, with additional environment details in Appendix 6.3. The set of tasks in this domain is designed to stress both 1) challenging inverse dynamics through complex striking motions and 2) partial state coverage through the wide variety of possible paddle ˆpuck positions and velocities. While baselines can struggle with compounding errors in one or both of these settings DILO's ability to side-step learning one-step models allow it to scale gracefully to these complexities.Tasks and Datasets: We consider three tasks and 9 datasets for real-world experiments. Our tasks are: where the humans demonstrate using their hands, and the state is detected using an overhead camera.Analysis: Fig. 3 compares the success rate of Learning from Observation algorithms in settings with varying dynamics. Safe Object Manipulation presents a task with easy inverse dynamics modeling since the arm restricts its motion to move through the workspace. Consequently, BCO performs well when provided with good coverage of expert observations (few-uniform), but is still outperformed by DILO as a result of ignoring offline datasets to learn recovery behaviors. SMODICE shows poor performance consistently in tasks with small datasets-i.e. poor coverage. Puck striking presents both easy inverse dynamics and good state coverage, which may explain the comparable performance from BCO and SMODICE against DILO. On the other hand, Dynamic Puck Hitting is challenging both for inverse dynamics, because of the wide range of actions necessary to hit a moving puck, and for state coverage, where the range of possible paddle and puck positions is substantial. Fig. 4 demonstrates an example of learned puck hiting behavior. DILO handles both complexities gracefully, resulting in an impressive success rate over both baselines. 
conclusion: 
 Offline Imitation from Observations provides a solution for fast adaptation of the agent to a variety of expert behaviors agnostic of the agent's action space. In this work, we propose a principled, computationally efficient, and empirically performant solution to this problem. Our work frames the problem as a particular distribution-matching objective capable of leveraging offline data. Using the principle of duality under a well-chosen but sufficient set of constraints, we derive an action-free objective whose training computational complexity is similar to an efficient offline RL algorithm. We show that the proposed method shows improved performance across a wide range of simulated and real datasets, learning from proprioceptive or image observations and cross-embodied expert demonstrations. 4) is given by the DILO objective in Equation 5. Moreover, as strong duality holds from Slater's conditions the primal and dual share the same optimal solution d ˚for any offline transition distribution ρ and any choice of mixture distribution ratio β.We start with the primal objective that matches distributions between the agent's visitation dps, s 1 , a 1 q and expert's visitation d E ps, s 1 , a 1 q. As before ρ denotes the visitation distribution of offline data.EQUATIONwhere for any two distributions µ 1 and µ 2 , Mix β pµ 1 , µ 2 q denotes the mixture distribution with coefficient β P p0, 1s defined as Mix β pµ 1 , µ 2 q " βµ 1 `p1 ´βqµ 2 .Formulating the objective as a constrained objective in agent's visitation distribution d allows us to create a primal objective that is a convex program. This is crucial in subsequently creating a dual objective that is unconstrained and easy to optimize. max dě0 ´Df pMix β pd, ρq}Mix β pd E , ρqq s.t ř a 2 dps 1 , s 2 , a 2 q " p1 ´γqd 0 ps 1 , s 2 q `γ ř s,a 1 PSˆA dps, s 1 , a 1 qpps 2 |s 1 , a 1 q, @s 1 , s 2 P S ˆS.where the constraints above dictate the conditions that any valid visitation distribution dps 1 , s 2 q needs to satisfy and are our proposed modifications to the commonly known bellman flow constraints.Below we outline the derivation of how these specific constraints with the mixture distribution matching objective allows us to create a dual objective that is independent of expert's actions. Applying Lagrangian duality to the above constrained distribution matching objective, we can convert it to an unconstrained problem with dual variables V ps, s 1 q defined for all s, s 1 P S ˆS:max dě0 min V ps 1 ,s 2 q´Df pMix β pd, ρqps, s 1 , a 1 q || Mix β pd E , ρqps, s 1 , a 1 qq `ÿ s 1 ,s 2 V ps 1 , s 2 q ˜p1 ´γqd 0 ps 1 , s 2 q `γ ÿ s,a 1 dps, s 1 , a 1 qpps 2 |s 1 , a 1 q ´ÿ a dps 1 , s 2 , a 2 q ¸(9)EQUATIONwhere the last equation uses a change of variable from s 1 , s 2 to s, s 1 without loss of generality. Using a simple algebraic manipulation below, we can get rid of the inner maximization. We add and subtract the terms shown below:" max dě0 min V ps,s 1 q βp1 ´γqE d0ps,s 1 q " V ps, s 1 q ‰ `βE s,s 1 ,a 1 "d « γ ÿ s 1 pps 2 |s 1 , a 1 qV ps 1 , s 2 q ´V ps, s 1 q ff `p1 ´βqE s,s 1 ,a 1 "ρ « γ ÿ s 2pps 2 |s 1 , a 1 qV ps 1 , s 2 q ´V ps, s 1 q ff ´p1 ´βqE s,a,g"ρ « γ ÿ s 1 pps 2 |s 1 , a 1 qV ps 1 , s 2 q ´V ps, s 1 q ff ´Df pMix β pd, ρqps, s 1 , a 1 q || Mix β pd E , ρqps, s 1 , a 1 qq (11)As strong duality holds using Slater's conditions [52] (see [43] for a detailed account of strong duality in RL under visitation distributions). Using the fact that strong duality holds in this problem we can swap the inner max and min and rewrite an equivalent maximization under the mixture distribution:EQUATIONIn the following derivation, we will show that the inner maximization in Eq 12 has a closed form solution even when adhering to the non-negativity constraints. Let yps, s 1 , a 1 q " E s 2 "pps 1 ,a 1 q rV ps 1 , s 2 qsV ps,s 1 q. max Mix β pd,ρqps,s 1 ,a 1 qě0 E s,s 1 ,a 1 "Mix β pd,ρqps,s 1 ,a 1 q « γ ÿ s 2 pps 2 |s 1 , a 1 qV ps 1 , s 2 q ´V ps, s 1 q ff ´Df pMix β pd, ρqps, s 1 , a 1 q || Mix β pd E , ρqps, s 1 , a 1 qqNow to solve this constrained optimization problem we create the Lagrangian dual and study the KKT (Karush-Kuhn-Tucker) conditions. Let wps, s 1 , a 1 q ∆ " Mix β pd,ρqps,s 1 ,a 1 q Mix β pd E ,ρqps,s 1 ,a 1 q , then the constraint Mix β pd, ρqps, s 1 , a 1 q ě 0 holds if and only if wps, s 1 , a 1 q ě 0 @s, s 1 , a 1 . max wps,s 1 ,a 1 q max λě0 E s,s 1 ,a 1 "Mix β pd E ,ρqps,s 1 ,a 1 q " wps, s 1 , a 1 qyps, s 1 , a 1 q ‰ ´EMix β pd E ,ρqps,s 1 ,a 1 q " f pwps, s 1 , a 1 qq ‰ `ÿ s,s 1 ,a 1 λpwps, s 1 , a 1 q ´0q (13)Since strong duality holds, we can use the KKT constraints to find the solutions w ˚ps, s 1 , a 1 q and λ ˚ps, s 1 , a 1 q.• Primal feasibility: w ˚ps, s 1 , a 1 q ě 0 @ s, a 1 , a 1• Dual feasibility: λ ˚ě 0 @ s, s 1 , a 1• Stationarity: Mix β pd E , ρqps, s 1 , a 1 qp´f 1 pw ˚ps, s 1 , a 1 qq `yps, s 1 , a 1 q `λ˚p s, s 1 , a 1 qq " 0 @ s, s 1 , a 1• Complementary Slackness: pw ˚ps, s 1 , a 1 q ´0qλ ˚ps, s 1 , a 1 q " 0 @ s, s 1 , a 1Using stationarity we have the following:EQUATIONNow using complementary slackness, only two cases are possible w ˚ps, s 1 , a 1 q ě 0 or λ ˚ps, s 1 , a 1 q ě 0.Combining both cases we arrive at the following solution for this constrained optimization:w ˚ps, s 1 , a 1 q " max ´0, f 1 ´1pyps, s 1 , a 1 qq ¯(15)Using the optimal closed-form solution (w ˚) for the inner optimization in Eq. ( 12) we obtain min V ps,s 1 q βp1 ´γqE d0ps,s 1 q " V ps, s 1 q ‰ `Es,s 1 ,a 1 "Mix β pd E ,ρqps,s 1 ,a 1 q " max `0, pf 1 q ´1 `yps, s 1 , a 1 q ˘˘yps, s 1 , a 1 q ´αf `max `0, pf 1 q ´1 `yps, s 1 , a 1 q ˘˘˘‰ ´p1 ´βqE s,a"ρEQUATION)For deterministic dynamics, this reduces to the following simplified objective:min V ps,s 1 qβp1 ´γqE d0ps,s 1 q " V ps, s 1 q ‰ `Es,s 1 ,a 1 "Mix β pd E ,ρqps,s 1 ,a 1 q " max `0, pf 1 q ´1 `yps, s 1 , a 1 q ˘˘yps, s 1 , a 1 q ´f `max `0, pf 1 q ´1 `yps, s 1 , a 1 q ˘˘˘‰ ´p1 ´βqE s,a"ρ " γV ps 1 , s 2 q ´V ps,EQUATION)where yps, a, gq " γV ps 1 , s 2 q ´V ps, s 1 q.6.1.2 What does the utility function V ˚ps, s 1 q represent?Prior work [43] shows that for the regularized RL problem max dě0 E dps,aq rrps, aqs ´αD f pdps, aq || d O ps, aqq s.t ř aPA dps, aq " p1 ´γqd 0 psq `γ ř ps 1 ,a 1 qPSˆA dps 1 , a 1 qpps|s 1 , a 1 q, @s P S.(EQUATIONthe dual optimizes for a Langrangian variable V that represents a regularized optimal value function. This insight directly extends to our work with reward function set to zero, our Lagrangian variable learns only the regularized visitation probabilities under optimal policy.It is easy to see why this is the case using the previous derivation. Following the derivation from the previous section, note that we had rewritten the inner maximization w.r.t the visitation distribution d, thus effectively getting rid of manipulating visitation distributions in the final objective. Our derivation above uses the following substitution shown in Eq 15 that holds as part of the closed form solution w.r.t inner maximization:Mix β pd, ρqps, s 1 , a 1 q Mix β pd E , ρqps, s 1 , a 1 q " max ´0, f 1 ´1pyps, s 1 , a 1 qq ¯(19)where y " γV ps 1 , s 2 q ´V ps, s 1 q. For deterministic dynamics, at convergence, the following holds for all s, s 1 , a 1 where d ˚ps, s 1 , a 1 q ą 0:EQUATIONimplying:pγV ˚ps 1 , s 2 q ´V ˚ps, s 1 qq " f 1 ˆMix β pd ˚, ρqps, s 1 , a 1 q Mix β pd E , ρqps, s 1 , a 1 q ˙" ´ri ps, s 1 , a 1 q (21)The above relation makes the the interpretation of V ˚ps, s 1 q clear. pV ˚ps, s 1 q ´γV ˚ps 1 , s 2 qq denotes the implied reward function r i ps, s 1 , a 1 q under which V ˚computes the maximum cumulative expected return, where a 1 is the action that leads to s 2 . As shown above the the implied reward function r i ps, s 1 , a 1 q " ´f 1 ´Mix β pd ˚,ρqps,s 1 ,a 1 q Mix β pd E ,ρqps,s 1 ,a 1 q ¯is the divergence between expert stationary visitation distribution and agent's stationary visitation that is obtained after taking the action a 1 from s 1 and then acting optimally to match the expert visitation distribution. Note that the function f 1 is non-decreasing as the function f is convex from definition of f -divergences.For χ 2 divergence, the generator function f pxq " px ´1q 2 . f 1 pxq " 2px ´1q and correspondingly f 1´1 pxq " x 2 `1. Substituting f 1´1 pxq in definition of f p :EQUATIONSince x we substitute takes the form of residual residual " γE s 2 "pp¨|s 1 ,a 1 q rV ps 1 , s 2 qs ´V ps, s 1 qq, the below pseudocode shows the implementation of f p for DILO.1 def f_star_p ( self , residual , type = ' chi_square ') : To better understand this objective's behavior we consider the last two terms from Eq 5 in its expanded form below. We ignore the first term as it is simply pushing down Q-values at initial distribution of states, to prevent overestimation when learning from offline datasets. βE s,s 1 ,s 2 " dE q " f p pγV ps 1 , s 2 q ´V ps, s 1 qq ‰ `p1 ´βq ˚Es,s 1 ,s 2 "ρ " f p pγV ps 1 , s 2 q ´V ps, s 1 qq ‰EQUATIONDenote rps, s 1 , a E q " V ps, s 1 q ´γV ps 1 , s 2 q as the implicit expert reward of under a learned Qfunction. The objective presents a clear intuition when we study the objective's behavior in different situations individually: (a) For samples from ρ, the objective pushes down the implicit reward to 0 as shown below:EQUATION(b) For samples from the expert distribution dE , the objective ensures that reward is greater than equal to 2EQUATION)It becomes clear now that DILO is implicitly learning a valid reward function that ensures higher discounted return for the expert compared to the suboptimal dataset by shaping Q-values directly.The algorithm for DILO can be found in Algorithm 1. We base the DILO implementation on the official implementation of pytorch-IQL that is based on IQL [44] . We keep the same network architecture as the original code and do not vary it across environments.Our experiment design is based on the benchmark from [13, 28] but we explain the setup here for completeness.Environments: For the offline imitation learning experiments we focus on 9 locomotion and manipulation environments from the MuJoCo physics engine [18] comprising of Hopper, Walker2d, HalfCheetah, Ant, Kitchen, Pen, Door and Hammer to make a total of 24 datasets. The MuJoCo environments used in this work are licensed under CC BY 4.0 and the datasets used from D4RL are also licensed under Apache 2.0. are given in Table 3 . For policy update, using Value Weighted Regression, we use the temperature τ to be 3 for all environments.Hyperparameters for our proposed off-policy imitation learning method DILO are shown in Table 3 .We use robomimic [20] for our imitation with image observations experiments. The following two environments are used here (the description is taken from their paper and written here for conciseness):Lift: Object observations (10-dim) consist of the absolute cube position and cube quaternion (7-dim), and the cube position relative to the robot end effector (3-dim). The cube pose is randomized at the start of each episode with a random z-rotation in a small square region at the center of the table.Can Object observations (14-dim) consist of the absolute can position and quaternion (7-dim), and the can position and quaternion relative to the robot end effector (7-dim). The can pose is randomized at the start of each episode with a random z-rotation anywhere inside the left bin.Robomimic provides three datasets and two modalities of observation (Proprioceptive, Images) for both environments above. The datasets are denoted by -MH (Multi-human), MG (Machine Generated), PH(Proficient-human). We use the MG and MH datasets as the suboptimal datasets in our task and PH as the source of expert observations. MH and MG datasets consists of 200 trajectories of usually suboptimal nature and we use 50 observation-only trajectory from PH datasets. This tasks is complex by the fact that expert-level actions are mostly unseen in the suboptimal dataset and the agent needs to learn the best actions that matches expert visitation from the suboptimal dataset. We implement all algorithms in the Robomimic codebase without any change in network architecture, data-preprocessing or learning hyperparameters. We tune algorithm specific hyperparameters in a course grid for BCO, SMODICE, and DILO to compare the best performance of methods independent of hyperparameters. For BCO, we tune the inverse dynamics model learning epochs between [1, 5, 10] .For SMODICE, we tuned discriminator learning epochs between [1, 5] , and gradient penalty between [1, 5, 10, 20] . To control overestimation due to learning with offline datasets in DILO we consider a linear weighting λ between the optimism and pessimism terms in Eq 5 inspired by prior work [28] as follows:min Q p1 ´λqβp1 ´γqE d0 " V ps, s 1 q ‰ `λE s,s 1 ,a 1 "Mix β p dE ,ρq " f p pγE s 2 "pp¨|s 1 ,a 1 q " V ps 1 , s 2 q ‰ ´V ps, s 1 qq ‰ ´λp1 ´βqE s,s 1 ,a 1 "ρ " γE s 2 "pp¨|s 1 ,a 1 q " V ps 1 , s 2 q ‰ ´V ps, s 1 q ‰ ,The hyperparameters used for DILO can be found in Table 4 . For the architecture specific hyperparameters we refer the readers to [20] .Our setup for manipulation experiments is inspired by the robot air hockey environment [55] for applying DILO to physical robotics settings. Our setup utilizes a Universal Robotics Sony Playstation Eye, a high framerate camera, which gathers 640 ˆ480 frames at 60 FPS, mounted to the ceiling to have a full view of the table. The paddle that is held by the robot end effector is 9.5cm in diameter and the puck is 6.3cm. In this setup, the negative x direction is oriented along the table, and the y is across the table. The action space for the arm utilizes pose control in x, y through an inverse kinematics controller accessible through the Universal robotics real-time data exchange interface. Utilizing the serving command, the arm is controlled using delta positions clipped between 26cm in the x direction and 13cm in the right direction. These control limits are specified to prevent the robot from triggering force limits, which results in an emergency stop. Actions are taken at a 20Hz frequency to allow for rapid response to dynamic elements, such as hitting a falling puck. The position and velocity of the end effector can be recovered through the real-time data excahnge, but other objects like the puck or the hand require identification. This work utilizes an overhead camera running at 60Hz to locate these objects using a vision pipeline that relies on hue saturation value segmentation followed by object identification. This gives an x c , y c coordinate in camera space, which we convert via OpenCV [56] homography to robot coordinates. This homography is computed by mapping the end effector positions given by the robot sensors to visual locations from the camera.Learning from Observation is a challenging setting, and while DILO makes some key assumptions in order to achieve good performance. First, matching distributions becomes exponentially more challenging in the dimensionality of the state space. In this work, while DILO outperforms baselines BCO generalizes incorrectly at a number of locations producing policies that hit obstacles. DILO learns to mimic expert's intent better demonstrating signs that it has learned to avoid obstacles by the arrows around in the Expert Image observations, it still shows limited performance. Second, while learning from observations opens the door for good performance without expert actions, the expert observation space must match that of the agent. In some video settings, this is not the case, ex. the agent might use a fixed camera when the human is egocentric, or vice versa. Finally, DILO utilizes the conservatism parameter τ to regulate the degree of extrapolation from the algorithm. In some settings, the values can diverge, resulting in V ˚taking on values that might be too large to be used for learning the downstream policy. Adaptively selecting τ to maximize extrapolation while avoiding divergence is an area of active investigation.While DILO outperforms other methods in overall success rate, the failure modes can differ. In general, DILO tends to be conservative in what actions it takes, learning motions that might be slower than BCO or may get stuck before arriving at the goal. In low-dim observation settings, DILO can also exhibit "dead zone" behavior, where the model becomes mostly unresponsive. Below we detail some of the exact error modes in particular tasks:Safe Manipulation: While DILO and BCO have comparable success rates, the two algorithms fail in different ways. BCO tends to take large actions while ignoring obstacles to reach the goal, while DILO takes more conservative actions. Thus, while BCO might fail by knocking over a cup, DILO will tend to fail to reach the goal. Because this is a low data setting, both algorithms BCO and DILO can end up coming close to the cups or brushing them gently. As a side note, SMODICE fails at even reaching the goal in most cases in this task, possibly because of this low data setting.Striking: This domain is challenging because of the narrow data regime, and all methods tend to struggle in similar ways. The most common consequence of low data arises through sensitivity to the x location of the puck (along the table). While intuitively, striking behavior should be relatively invariant for a fixed y (horizontal position on the table), slight variation in x from the dataset can result in a policy that moves the arm in the opposite direction of the puck, probably due to errors in extrapolation. In addition, striking is a dynamic behavior that requires a precise combination of forward and horizontal actions. Even a slight error in the ratio can result in a near miss. Finally, DILO tends to learn more conservative policies and, in some locations, may not not strike the puck with much force. However, because of the low data coverage, this issue is endemic to all the learned policies.Hitting: The primary challenge of achieving a hit in this task is the precise alignment of the paddle to the puck. While DILO performs well, it is not perfectly accurate, resulting in touches that bounce off the side of the paddle. This challenge is endemic to all policies. Additionally, the conservatism of DILO actions appear when it moves under the puck, where it tends to move slowly, and dropping the puck too quickly can result in DILO failing to reach the puck. As a result, while DILO is likely to succeed at the first hit, it can struggle to generate multiple hits because this can require rapid side-to-side movement. These issues are largely endemic to all the learned policies, where SMODICE tends to be even less precise, and BCO struggles to learn to strike, though it can occasionally position under the puck.Visualizations of the failure modes can be seen in the project website.Our proposed method is limited by the assumption of matching visitation distributions in the observation space of the agent and expert rather than a meaningful semantic space, but we hope that with improvement in universal representations, this limitation is lifted by distribution matching in compact representation space. Our work assumes that expert's optimality, but in reality, experts demonstrate a wide range of biases. We leave this extension to future work. Finally, we demonstrate the failure modes of our method and further limitations in Appendix 6.4. 
 
 Appendix: Policy learning rate 3e-4 Value learning rate 3e-4 f -divergence χ 2 max-clip (Value clipping for policy learning) 100 MLP layers (256,256) β (mixture ratio) 0.5 η (orthogonal gradient descent) 0.5 τ (policy temperature) 3Table 3 : Hyperparameters for DILO in imitation from proprioceptive observations.Cups StrawberryWe apply imitation learning from observations on several tasks built on top of the above robot setup. The following experiments are described here:Safe object manipulation: This task involves moving a strawberry to a bowl while avoiding four cups placed in the workspace of the robot. The bowl is placed in the top right corner of the workspace, and the cups are placed in fixed locations. The success metric is the robot stopping above the bowl while making no contact with any of the cups. The test set involves 10 random starting locations for the end effector that are fixed between assessments. The observation space is the 2D end effector position, and the strawberry is initialized inside the gripper. Our suboptimal dataset consists of 50 trajectories of 100 time steps on average where the robot is initialized in a random location, and the human moves the arm to a random different location, ignoring the positions of the cups or the bowl. In this setting, we investigated the following expert data, visualized in Figure 5 :• Few Trajectories: The expert data is drawn from a set where the expert is initialized in a random location, sometimes touching an obstacle, and must use the teleoperation system to avoid the obstacle and reach the goal. In this setting we used 15 expert trajectories.• Fixed start: The expert is initialized at the opposite corner of the workspace, and navigates to the goal location following different paths using teleoperation. In this setting we used 15 trajectories.• Few Uniform: Uses the same expert data as the Few Trajectories setting, but the dataset consists of randomly samples 300 transitions, where one trajectory is approximately 60 transitions of data.• Cross Embodiment Few/Fixed/Uniform: The expert is a person holding the strawberry in his/her hand, visualized in Figure 6 . They then move the strawberry tracked by the camera to the goal location while avoiding the obstacles, starting from random/fixed locations with 15 expert trajectories respectively or uniform with 300 transitions.Stationary Striking: This task involves moving the end effector to strike a stationary puck. The success metric is the robot touching the puck. The test set involves 10 initializations of the puck position across the length of the table. The ensure uniformity across evaluations, the set of initialization locations of the puck are fixed across methods. The end effector is initialized at 0.38m from the base in the center of the table, so a success strike does not require backward motion. The observation space is the 2D end effector position and the tracked position of the puck. Our suboptimal dataset consists of 50 trajectories of 75 time steps on average where the robot is initialized at the start position, and the human moves the arm in a random, vaguely striking pattern.In this setting we used an expert dataset of 400 trajectories where the expert uses mouse teleoperation to strike the puck. The expert efficiently strikes the puck in a single motion. We visualize the expert striking and the puck position in Figure 5 . We show the learned action vectors for all algorithms and tasks fixed start (Figure 9 ), Few Uniform (Figure 8 ) and Few trajectories (Figure 10 ).Dynamic Hitting: This task involves hitting a puck dropped from the top of the table. Because the table is set at an angle, this will cause the puck to fall with increasing acceleration towards the opposite side. The setup is visualized in Figure 5 . The test set involves 10 initializations of the puck position dropped from positions across the length of the top of the table. The locations of the 10 puck drops are fixed using indicators across methods to give fair evaluation, and the arm is initialized in the center of the table, 0.68m from the base. The observation space is the 2D end effector position and 2D end effector velocity and the history of the last 5 tracked positions of the puck relative to the position of the end effector. Our suboptimal dataset consists of 50 trajectories of 200 time steps on average where the robot is initialized at the start position, and the human moves the arm around the puck without striking it.We utilize two success metrics for this task: 1) touching: a trajectory is considered successful if the agent touches the puck. 2) hitting: the puck must have velocity in the opposite direction that it was dropped. This task is especially challenging for existing methods because of the long sequence of actions necessary to position the paddle properly, and the high level of both precision and timing: even a few millimeters of error or a movement at the wrong time will result in a failure, especially for hitting. Previous work has observed that this task is challenging even for humans, who often require several tries of practice, and many dataset trajectories consist of many inaccurate hits. In this domain, Behavior cloning only achieves 30% success at touching the puck, and Implicit Q-learning, a popular offline RL method, can only achieve 60% success, even though it employs a hand-designed reward function.We used the implementation details from the proprioception task with the difference that in all the real-robot tasks we tune the following parameters across different methods: For BCO, we tune the inverse dynamics model learning epochs between [1, 5, 10] . For SMODICE, we tuned discriminator learning epochs between [1, 5] , and gradient penalty between [1, 5, 10, 20] . For DILO we tune the conservatism parameter from previous section between [0.5,0.6,0.7,0.8].In this domain, these challenges appear to be empirically validated in the performance of the baseline methods. We hypothesize that the accumulation of error over long horizons in other learning from observation methods results in poor performance, as visualized in Table 3 and Figure 4 . For methods like BCO, learning the necessary inverse dynamics to interpolate the long sequence of actions for a successful strike is impractical, resulting in behavior that appears listless. On the other hand, while the SMODICE discriminator rewards are able to occasionally match the visitation distribution of the expert, there is an exponential explosion of possible combinations of puck history and paddle positions, resulting in poor generalization: on the left half of the table, SMODICE is unable to hit the puck.The hitting scenario involves two settings, both using mouse teleoperation to control the puck: one where the human strikes the puck only once and then the trajectory ends, which utilizes a dataset of 50 trajectories, and an expert dataset of 850 trajectories where the human keeps hitting the puck repeatedly for up to 2000 timesteps. The task is challenging for human, so trajectories average only 500 timesteps and cleaned so that human mistakes are removed from the expert dataset. We visualize the expert hitting and the puck position in  


Decision: **Before Rebuttal**

Strengths.
This paper studies learning from observations with action-free videos, an important problem to solve given the abundance of such data. The proposed method is novel, with strong theoretical motivations. Experimental results in both simulation and reality show the benefit of the method over alternative approaches. As well as comparing to alternative methods for learning from observation without action data, comparisons to methods which additionally have access to action data offer interesting insights.

Weaknesses. 
In some places, the paper is difficult to follow and understand due to low quality of the writing. There is insufficient information in the method and experiments to appreciate the intended contributions, and to understand what the key conclusions are from the results. The real-world experiments assume that low-dimensional state representations are available, which are challenging to obtain in practice, and so there is no evidence that the method performs well in the real world directly from visual observations.

---

**After Rebuttal**

Following the reviews, authors provided a revised paper which has now improved in clarity and addressed some of the concerns raised by the reviewers. Although the real-world experiments are limited due to the need for low-dimensional states, overall this is only a minor issue given the strong theoretical contribution of the paper. The AC and reviewers discussed the paper together, after which there is now unanimous agreement amongst reviewers that the paper should be accepted.

Reviews: review: title: title: The paper proposes a novel learning-from-observations method with substantial technical contributions. However, presentation quality needs to be improved.
paper_summary: paper_summary: The paper proposes a novel method for learning policies from expert observation trajectories without action information and from an action-labeled dataset of arbitrary quality. An action-free distribution matching objective is used to learn a utility function that quantifies the utility of transitioning between states based on the visitation distribution of an expert. The policy is extracted from the learned utility function using value-weighted regression on the action-labeled dataset. Experiments show improved performance over alternative methods that address the problem of learning from observations in a variety of settings.
main_review: main_review: **Strengths**:

* Learning from observations and suboptimal action-labelled data is an interesting and important problem, which is well motivated in the paper.
* Learning a multi-step utility function instead of an inverse dynamics model or a discriminator is a nice idea and is motivated well throughout the paper.
* An action-free objective that allows learning the utility of transitioning between states based on the visitation distribution of an expert is a valuable contribution.
* Derivations provided in the appendix are useful for understanding the method.
* Experiments in various settings effectively demonstrate the advantages of DILO compared to alternative methods for learning from observations.		

**Weaknesses**:

* The paper is quite convoluted with many ideas repeated multiple times without a clear and concise overview, making it hard to follow.
* The paper is poorly written overall, with numerous grammatical errors and incomplete sentences.
* It is not discussed what "suboptimal" or "arbitrary" data actually means. How different can the distribution of this suboptimal data be from the expert data distribution?
* The experimental setting is not well described. It is not clear what different Suboptimal Datasets in Table 1 represent and how big they are. Adding exact numbers of action-free and action-labeled data points in the first column of Table 1 would really help.
* A more detailed discussion about the results presented in Table 1 would be helpful in understanding settings where DILO really shines (e.g. dimensionality of state representations, dataset size and quality).
* A more detailed discussion about using image observations and how the performance could be increased would be helpful.
questions: questions: * How different can the distribution of suboptimal data be from the expert data distribution while still being able to extract a policy from the learned utility function?
* How well does DILO scale with increasing dataset size?
* Could DILO be extended to a multi-task (or goal-conditioned) setting?
* How can the performance of DILO with image observations be improved? What properties should a representation learned from visual inputs have to be easily applicable to DILO?
* What theoretical properties of DILO you are referring to on lines 283-284?
limitations: limitations: 2
review_summary: review_summary: After rebuttals, I would recommend accepting this paper for publication.
title: title: Initial review
paper_summary: paper_summary: The paper works on the problem of learning robot manipulation skills from actionless expert demonstration videos. The proposed method first learns a utility function over the state transitions from the offline dataset, and then extracts the policy through maximizing the expected returns of the learned utility function. Experiments are performed in simulation and real world.
main_review: main_review: Strengths:

The problem of learning from action-free videos is both highly relevant and significant to the field of robotics research. The proposed method addresses this problem in a novel and reasonable manner.

The paper is well-structured and thoroughly written, providing sufficient technical details to clearly convey the overall idea and methodology. The inclusion of mathematical formulas and proofs adds rigor and credibility to the proposed approach.

The method is validated through real-world hardware experiments, demonstrating practical applicability.

Weaknesses:

The effectiveness of the proposed approach is contingent on a substantial amount of training data that adequately covers the state transition space, which may be a limitation in terms of data requirements.

The performance of the proposed method appears to be less effective compared to simpler behavior cloning approaches, potentially indicating room for improvement.

The real-world implementation relies on object state representations that depend on robust detection algorithms. This dependency could pose challenges when applying the method to more complex tasks or environments.
questions: questions: Would the proposed method be sensitive to the quality of demonstrations? It would be valuable to analyze how policies learned from demonstrations of varying quality (e.g., using datasets from Robomimic[1]) perform on the tasks. 

The performance for Robomimic benchmark seems to be not ideal, could authors give details analysis and explanations for this?  

Besides the proposed tasks, demonstrating the effectiveness of the approach on other benchmarks (e.g., RLBench) would further strengthen the paper. 

[1] What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL’21. 

[2] RLBench: Robot Learning Benchmark, RA-L'20.
limitations: limitations: 2
review_summary: review_summary: My rating is based on insufficient evaluation and weak performance
title: title: Review of Submission 200
paper_summary: paper_summary: This paper presents a novel approach for learning from observation (LfO) without access to a simulator. The authors start by motivating their approach, noting that many prior works fail in the offline LfO space. The authors then present their approach (Dual Imitation Learning from Observations or DILO), which directly learns a utility function that quantifies how state transitions impact the agent's divergence from the expert's visitation distribution. Importantly, the method is backed with several theoretical results, derivation, and ample citations, and explained both intuitively and in-depth. The authors conclude by displaying an abundance of results, both in synthetic domains and real-world, displaying that DILO presents a novel high-performing algorithm for learning robot behavior.
main_review: main_review: Overall, this paper contains a solid contribution to the field of imitation learning.


Strengths
+ This paper focuses on one of the most challenging settings in imitation learning, offline imitation learning from observations, where the agent has access to an offline dataset of its own trajectories of arbitrary quality and a few task-relevant expert trajectories only containing observations. Given this focus, the results are very impressive.
+ The paper is well-written. There are ample citations justifying the approach and its design decisions, detail regarding the formulation of DILO, techniques to improve training practicality, and intuitive explanations.  
+ The evaluation is very extensive and displays the success of the proposed technique. It is clear that the authors were thorough in their evaluation by including results in typical Mujoco-like benchmarks, LfO from images, and finally from real-world human demos.



Weaknesses
- Table 1 displays an abundance of results and DILO is outperforming baselines on average. However, as SMODICE is a close competitor, it would be beneficial to add additional description on why SMODICE is able to outperform DILO in some instances (e.g., random + expert) and not in others (e.g., cloned + expert). Also, could the authors comment on why the deviations between DILO and SMODICE are very large in domains like pen, door, and hammer?
questions: questions: - Could you please address the weakness noted above?
-  DILO achieves similar results for Safe Object Manipulation across the teleop and cross-embodiment demonstration. I would expect it would be easier to learn from the teleop demonstrations. Could you comment on why the performance is similar and if this was expected/designed for?
limitations: limitations: 3
review_summary: review_summary: The paper presents a clear advancement for offline learning from observation, contains a strong mathematical basis behind the algorithm, and has ample results across synthetic and real-world domains.