Learning Control by Iterative Inversion
Authors: Gal Leibovich
Guy Jacob
Or Avner
Gal Novik
Aviv Tamar
Venue: Submitted to ICLR 2023 (2023)
Keywords: RL
IRL

Abstract: We formulate learning for control as an inverse problem - inverting a dynamical system to give the actions which yield desired behavior. The key challenge in this formulation is a distribution shift in the inputs to the function to be inverted - the learning agent can only observe the forward mapping (its actions' consequences) on trajectories that it can execute, yet must learn the inverse mapping for inputs-outputs that correspond to a different, desired behavior. We propose a general recipe for inverse problems with a distribution shift that we term $\textit{iterative inversion}$ - learn the inverse mapping under the current input distribution (policy), then use it on the desired output samples to obtain a new input distribution, and repeat.
As we show, iterative inversion can converge to the desired inverse mapping, but under rather strict conditions on the mapping itself.
We next apply iterative inversion to learn control. Our input is a set of demonstrations of desired behavior, given as video embeddings of trajectories (without actions), and our method iteratively learns to imitate trajectories generated by the current policy, perturbed by random exploration noise. We find that constantly adding the demonstrated trajectory embeddings as input to the policy when generating trajectories to imitate, a-la iterative inversion, we effectively steer the learning towards the desired trajectory distribution. To the best of our knowledge, this is the first exploration of learning control from the viewpoint of inverse problems, and the main advantage of our approach is simplicity - it does not require rewards, and only employs supervised learning, which can be easily scaled to use state-of-the-art trajectory embedding techniques and policy representations. Indeed, with a VQ-VAE embedding, and a transformer-based policy, we demonstrate non-trivial continuous control on several tasks. Further, we report an improved performance on imitating diverse behaviors compared to reward based methods. 

LEARNING CONTROL BY ITERATIVE INVERSION
Anonymous authors Paper under double-blind review
ABSTRACT
We formulate learning for control as an inverse problem - inverting dynamical system to give the actions which yield desired behavior. The key challenge in this formulation is a distribution shift in the inputs to the function to be inverted - the learning agent only observe the forward mapping (its actions' consequences) on trajectories that it can execute, yet must learn the inverse mapping for inputs- different, desired behavior. We propose general recipe for inverse problems with a distribution shift that we term iterative inversion learn the inverse mapping under the current input distribution (policy), then use it on the desired output samples to obtain a new input distribution, and repeat. As we show, iterative inversion can converge to the desired inverse mapping, but under rather strict conditions on the mapping itself.
outputs that correspond
We next apply iterative inversion to learn control. Our input set of demon- strations of desired behavior, given as video embeddings of trajectories (without actions), and our method iteratively learns to imitate trajectories generated by the current policy, perturbed by random exploration noise. We find that constantly adding the demonstrated trajectory embeddings as input to the policy when gener- ating trajectories to imitate, a-la iterative inversion, effectively steer the learning towards the desired trajectory distribution. To the best of our knowledge, this is the first exploration of learning control from the viewpoint of inverse problems, and the main advantage of our approach is simplicity - it does not require rewards, and only employs supervised learning, which can be easily scaled use state-of- the-art trajectory embedding techniques and policy representations. Indeed, with a VQ- VAE embedding, and a transformer- -aasee policy, we demonstrate non-trivial continuous control on several tasks. Further, we report an improved performance on imitating diverse behaviors compared reward based methods.
INTRODUCTION
The control of dynamical systems is fundamental to various disciplines, such as robotics and automa- tion. Consider the following trajectory tracking problem. Given some deterministic but unknown actuated dynamical system,
8++1 [st, ax),
(1)
where is the state, and is actuation, and some reference trajectory, so...., ST, we seek actions that drive the system in a similar trajectory to the reference.
For system that are 'simple' enough, e.g., linear, or low dimensional, classical control theory (Bert- sekas 1995) offers principled and well-established system identification and control solutions. How- ever, for several decades, this problem has captured the interest of the machine learning community, where the prospect is scaling up high-dimensional systems with complex dynamics by exploiting patterns the system (Mnih et al. 2015 Lillicrap et al.) 2015 Bellemare et al.] 2020).
In reinforcement learning (RL), learning is driven by manually specified reward signal r(s, a). While this paradigm has recently yielded impressive results, defining a reward signal can be difficult for certain tasks, especially when high-dimensional observations such as images are involved. An alternative to RL inverse RL (IRL), where a reward is not manually specified. Instead, IRL algorithms learn implicit reward function that, when plugged into RL algorithm in inner loop, yields a trajectory similar to the reference. The signal driving IRL algorithms a similarity metric between trajectories, which can be manually defined, learned (Ho & Ermon 2016).

---

We propose different approach to learning control, which does not require explicit nor implicit reward functions, and also does not require a similarity metric between trajectories. Our main idea is that Equation prescribes mapping F from sequence of actions to a sequence of states,
F (aa.....,rr-1
(2)
The control learning problem can therefore be framed as finding the inverse function, F- without knowing F, but with the possibility of evaluating F on particular action sequences (a.k.a. roll-outs).
Learning the inverse function F-1 using regression can be easy one has samples of action sequences and corresponding state sequences, and a distance measure over actions. However, in our setting, we do not know the action sequences that correspond to the desired reference trajectories. Interestingly, for some mappings F, an iterative regression technique can be used find FF In this scheme, which we term Iterative Inversion (IT-IN), we start from arbitrary action sequences, collect their corresponding state trajectories, and regress to learn an inverse. We then apply this inverse on the reference trajectories to obtain new action sequences, and repeat. We show that with linear regression, iterative inversion will converge under quite restrictive criteria on F, such as being strictly monotone and with a bounded ratio of derivatives. Nevertheless, our result shows that for some systems, controller can be found without a reward function, nor a distance measure on states.
We then apply iterative inversion to several continuous control problems. In our setting, the desired behavior is expressed through a video embedding of a desired trajectory, using a VQ-VAE (Van Den Oord et al. 2017). and deep network policy maps this embedding and state history to the next action. The agent generates trajectories from the system using its current policy, given the desired embeddings as inpuf, and subsequently learns to imitate its own trajectories, conditioned on their own embeddings. Interestingly, we find that when iterating this procedure, the input of the desired trajectories' embeddings steers the learning towards the desired behavior, as in iterative inversion.
Given the strict conditions for convergence of iterative inversion, there is no a-priori reason to expect that our method will work for complex non-linear systems and expressive policies. Curiously, however, we report convergence on all the scenarios we tested, and furthermore, the resulting policy generalized well to imitating trajectories that were not seen in its 'steering' training set. This surprising observation suggests that IT-IN may offer simple supervised learning-base alternative to methods such as RI. and IRI., with several potential benefits such as a reward-less formulation, and the simplicity and stability of the (iterated) supervised learning loss function. Furthermore, on experiments where the desired behaviors are abundant and diverse, we report that IT-IN outperforms reward-based methods, even with an accurate state-based reward.
2 RELATED WORK
In learning from demonstration (Argall et al.) 2009), it is typically assumed that the demonstration contain both the states and actions, and therefore supervised learning can be directly applied, either by behavioral cloning (Pomerlcau) 1988 interactive methods such DAgger (Ross et al. 201I). In our work, we assume that only states are observed in the demonstrations, precluding straightforward supervised learning. Inverse RL is similar problem to ours, and methods such as apprenticeship learning (Abbeel & Ng (004)) or generative adversarial imitation learning (Ho & Ermon, (0066) simultaneously train critic that discriminates between the data trajectories and the policy trajectories (a classification problem), and a policy that confuses the critic as best as possible (an RL problem). It shown that this procedure will converge
policy that visits the same states as the data. While coal-conditioned IRL setting, we are
works such as (uu et al.) 2019: Ding et al., 2019) considered not aware of IRL methods that can conditioned on a more expressive description than target goal state, such as a complete trajectory embedding, as we explore here. In addition, our approach avoids the need of training a critic, as (Ding et al. 2009). or training an RI. agent in an inner loop.
Most related to our work, Ghosh et al. 2019) proposed goal conditioned supervised learning (GCSL). In GCSL, the agent iteratively executes random trajectories, and uses them as direct supervision for training goal-conditioned policy, where observed states in the trajectory are substituted as goals. The desired goals are also input to the policy when generating the random trajectories. In comparison to GCSL, do not consider tasks of only reaching goal states, but tasks where the whole trajectory is important. This significantly increases the diversity of possible tasks, and thereby increases the difficulty of the problem. In addition, the theoretical analysis of Ghosh et al. 2019
2

---

Figure 1: Learning an inverse function under distribution shift. We wish to learn the inverse function over outputs y1..... Mar, using linear least squares, having matching inputs-outputs for I1 : Z.M
y
F
31 32
23
2
X
showed convergence under an assumption that all goal states have probability of being visited under the initial policy. The analysis we show for iterated inversion, and the fact that this assumption almost never holds in practice (except, e.g., in offline RI, where the data is already 'explorative" enough Emmons al. 202T). suggest that the practical success of GCSL, less obvious than as the theory of Ghosh et al. (2019) predicts.
In self-supervised RL, the agent is not given reward, and uses its own experience to explore the environment, typically by training goal-conditioned policy, and proposing to it goals that are novel in some measure (Ecoffet et al., 2019 Hazan et al., 2209 Sekar et al. 2020) Endrawis et al. 2021 Mendonca et al. 2021). The space of all trajectories is much larger than the space of all states, and we are not aware of methods that demonstrably explore such a space. For this reason, in our approach steer the exploration towards set of desired trajectories.
Very recently, in their work on video pretraining, Baker et al. (2022) also used a transformer to learn video. Importantly, Baker al. (2022) collected human-labelled
inverse model conditioned data to train their inverse model on desired behavior trajectories. The main point in our work is self-supervised learning paradigm that automatically steers data collection to the desired behavior.
Finally, we mention the extensive literature on deep learning based solutions for inverse prob- lems (Lucas et al. 20188] Kamyab al., [021]. many studies, the forward mapping known, and differentiated to iteratively optimize reconstruction (Xia et al., (022). In contrast, blind inversion methods learn the inverse mapping directly from data (Kulkarni al., 2016). To the best of our knowledge, the formulation of learning control as an inverse problem, and iterative inversion as solution for the resulting distribution shift problem, are novel.
3 ITERATIVE INVERSION
In this section we describe general problem of learning an inverse function under distribution shift, and present the iterative inversion algorithm. We then analyse the convergence of iterative inversion in several simplified settings. In the proceeding, we will apply iterative inversion to learning control.
Let F. and an arbitrary set of M initial inputs 21....: E X. We assume that F is not known, but we are allowed observe F(x) (r) for any x E X that we choose during our calculations. Our goal is to find function G y + X such that for any desired output yi. we have ((yx) = =F-1(y)).
y be a bijective function. We are given a set of M desired outputs y1..... : VM V,
More specifically, we will adopt a parametric setting, and search for a parametric function Go, where 0 E θ is a parameter vector, that minimizes the average loss:
M Σ
1 M
min BEe
(3)
For example, Go could represent the space of linear functions Go(y) 0t y 0o. and L could be the squared error between inputs, L(x, x²) (x - x²)2. This example, which depicted in Figure I for the 1 I-dimensional case & y R, corresponds to a linear least squares fit of the inverse function. As can seen, the challenge in this problem arises from the mismatch between the distributions of the desired outputs and initial inputs.
3

---

The iterative inversion algorithm, proposed in Algorithm |
seeks solve problem iteratively.
Algorithm 1 Iterative Inversion
Require: Desired outputs y E 3; loss function L: X x X' - R, initial parameter tg-
I: for n n= 0, 2,..... do
2: Calculate current inputs: æ/,.... FM = Go. (y1)
Go. (yn) 3: Calculate current outputs: :....., :VM = F(xt)) ([.t),...... F
4: Regression
M ((90(v)))
0.+1 arg min DEE
M
5:
end for
We next investigate when, and why, should iterative inversion produce an effective solution for (3) We restrict ourselves linear function class for Go, and the squared loss. We report convergence results for different classes of functions F.
Denote X" = (z (.",,..,, "²r)"" R 12 F (Fla))..., F(xM))) E RM as the input and output matrices, X EM /M E Rdim(X) and Y ΣM V//M, F(X)) = E F(x")/M E dd)i as the inputs, desired outputs, and current outputs means, (-) the Moore-Penrose pseudoinverse operator, and F the ground-truth inverse function.
We start with the simple case of linear F. As is clear from Figure I the distribution shift is not problem in this case, and iterative inversion converges in a single iteration.
Theorem 1. If F is linear function and rank(F(X") - F(Xo)) = dim()) then Algorithm 1 converges in one iteration, i.e., vi...., Vi = ......... yMM-
Iterative Inversion can be interpreted as variant of the classic Newton's 's method (Ortega & Rheinboldt 2000), where we replace the unknown Jacobian of F with a linear approximation using the current input-output pairs, and the evaluation of F with the mean of the current outputs. Recall that Newton's method seeks to find the root x² of a function r(x) = F(x) - using the iterative update rule the Jacobian inverse of F at x", Iterative
x=+1 x² + (y- - [[=")))[(<")) Inversion, similarly, applies the following updating rule, as proved in Appendix A.l
where [J(=")]
x (Y -(-((x)) J,-1
) is the Jacobian of Go... the linear regressor plane from F(x) to at 2?. which can be considered to be an approximation of [J(X)) When the approximations J/ [J(X)]- and F(X") Re F(X*) accurate, Iterative Inversion coincides with Newton's method, and enjoys similar convergence properties, as we establish next. Assumption 1. F: RK - R bijective, and F and F-1 are both continuously differentiable.
where J=
Denote J(x) the Jacobian matrix of F at x E X and J (x) [J(z)]- the inverse matrix of (xx) and the Jacobian of F at F(x) E Vi under Assumption
Also denote || || to be any induced
norm (Horn& & Johnson 2012). We assume that the derivatives of J and F-1 are bounded, as follows.
Assumption 2.
[((1)) [x22]| S 7 [](x)] < Ç and [x)]l $ 3 Væ,, 22, x E E
Further assume that at every iteration the approximations J, and F(X*) are accurate enough. and J J (Xn ( A.). |||.|| 5 < 1/C8.
Assumption 3. Vn: [[[(X") F(X
Assumption 3 may hold, for example, when the inputs 21 ..... Thi are distributed densely, relative to the curvature of F. and evenly, such that the regression problem Algorithm is well-conditioned. The requirement& 1/CB is set ensure that J,- 1 is non-singular.
Theorem 2. Suppose Assumptions and 3 hold. Let 82 and assume ((1++)(+++))) <.
T-ç988
(x+()) Then for every 0 there exists k < such that -
Let p 1=8(1+0)(++))*
4

---

The proof for Theorem builds on the analysis of Newton's method to show that IT-IN is iterated contraction, and is reported in Section AA. of the supplementary material. In Theorem 2 the term p can be interpreted as the radius of the ball centered at Y that the sequence convergences to. To get some intuition about Theorem 2 consider the 1-dimensional case F R + IR, where the approximations in Assumption
/ p O. Then, the condition for 3 are perfect, i.e., X =
max (w) < 2, which can be interpreted as a 'close to linear' FF The conditions in Theorem can therefore be intuitively interpreted as F being 'close to linear' globally, and the linear approximation being accurate locally.
convergence is B(1 + 0) + p1) < <1
min
In Appendix A.4. we provide additional convergence results that use a different analysis technique for the simple case F:: R
- IR. These results do not require Assumption 3 but still require a condition similar to max (x)]
min [x] < 2, and show linear convergence rate. We further remark that quadratic
convergence rate is known for Newton's 's method when the initial iterate is close to optimal; we believe that similar results can be shown for IT-IN as well. Here, however, we focused on the case of an arbitrary initial iterate, similarly to the experiments we shall describe in the sequel.
ITERATIVE INVERSION FOR LEARNING CONTROL
In this section, we apply iterative inversion for learning control. We first present our problem formulation, and then propose an IT algorithm.
We follow standard RL formulation. Let denote the state space, A denote the action space, and con- sider the dynamical system in Equation | We assume, for simplicity, that the initial state is fixed, and that the time horizon is T Given a state-action trajectory = 8p, ag ... ST, ar- 8T E $2, where ? denotes the 7-step trajectory space, we denote by т, E Ω2 its state component and by To E Sa its action component, i.c., Ta = 80...., ST, Ta = ao...... aT-1, and Q = S2g x Qa. We will henceforth refer to Ts state trajectory and to To as an action trajectory. Let J denote the mapping from an action trajectory to the resulting state trajectory, as given by Equation 2
For presenting our control learning problem, we will assume that F is bijective, and therefore F-1 is well defined, We emphasize, however, that our algorithum makes no explicit use of FF and our empirical results are demonstrated on problems where this assumption does not hold.
We represent a state trajectory using an embedding function 2 = Z(T,) E Z, and we term 2 the intent. Note that 2, by definition, can contain partial information about r., such the goal state (Ghosh et al., 2019). In all the experiments reported in the sequel, we generated intents by feeding a rendered video of the state trajectory into a VQ-VAE encoder, which we found to be simple and well performing.
Consider a state-action trajectory т, with a corresponding intent (Tx)). We would like to learn a policy that reconstructs the intent into its corresponding action trajectory Ta Let H, denote the space 2 x H,, + A. With a slight abuse of notation, we
of t-length state-action histories, and a policy denote by 7 <2)) E D2 the action trajectory that obtained when applying मे sequentially for T time steps (i.e., rollout). Similarly to the problem in Section 3 our goal is to learn a policy such that (Z(T.) = - (T.). More specifically, let L Q2 x Ω + be a loss function between action trajectories, and let P(T.) denote distribution over desired state trajectories, we seek policy parametrized by 0 E θ that minimizes the average loss:
min E.,~P [TT(((()))) BEO
In our approach assume that P(Ta) is not known, but we are given set Dateer of M intents, 21...., Zas. where 2; = Z(r)), and Tg are drawn i.i.d. from P(Tz). ). Henceforth, we will refer to Dateer the steering dataset, it should act to steer the learning of the inverse mapping towards the desired trajectory distribution P.T.).
It is worth relating Problem as the distribution shift problem. Initially, the policy is not expected to be able to produce state-action trajectories that match the state trajectories in Dateer. but only trajectories that are output by the initial (typically random) policy. While these initial trajectories could be used for imitation learning.
the general inverse problem in Section and what we referred to
A varying time horizon be handled as an additional input
5

---

yielding an intent- policy, there is no reason expect that this policy will be any good for intents in Dtteer, which are distrribbitittto with respect to the policy's 's training data.
We propose a method for solving Problem (5) based iterative inversion, detailed in Algorithm There are four notable differences from the iterative inversion method Algorithm 1 First, operate batches of size N instead of the whole steering data (of size M), for computational efficiency. Second, we sample a batch of intents from a mixture of the steering dataset and the intents calculated for rollouts in the previous iteration. We found that this helps stabilize the algorithm. Third, we add random exploration noise to the policy when performing the rollouts, which we found be necessary (see Sec. 55. Fourth, we used a replay buffer for the supervised learning part of the algorithm, also for improved stability. For L, we used the MSE between action trajectories, and for the optimization in line we perform several epochs of gradient- Adam (Kingma & Ba 2014), keeping the state history input to T= fixed as when computing the gradient. The size of the replay buffer was set K N.
optimization using
Algorithm 2 Iterative Inversion for Learning Control
Require: Steering data Dsteer exploration noise parameter 17, steering ratio a E [0, 1]. batch size N 1: Initialize D Dprey- Dsteer, 0g arbitrary
2: for n n==0 0, 1, 2... do
3: Sample a intents from Dsteer and (1 - a intents from Drrev yielding 27 3.... NN N
4: Perform N rollouts 77 ,-..., N using policy mo,, with input intents 2 ,-,..,2" adding
exploration noise
5: Compute intents for the rollouts & 2(r)),
N
6: Add intents and trajectories J'} to Replay Buffer
7: Train To by supervised learning: 0n+1 arg mingge EERRppy Butfer
[C (TT(
8: Set D Dprev = {2 1=
9:
end for
Note the simplicity of the IT-IN algorithm only involves exploration and supervised learning: there are no rewards, and the loss function routine. In the following, we provide empirical evidence that, perhaps surprisingly given the strict conditions for convergence of iterative inversion IT-IN yields well-performing policies on several nontrivial tasks.
EXPERIMENTS
In this section, we evaluate IT-IN on several domains. Our investigation is aimed at studying the unique features of IT-IN and especially, the steering behavior that we expect observe. We start by describing our evaluation domains, and implementation details that are common to all our experiments. We then present a series of experiments aimed at answering specific questions about IT-IN. To appreciate the learned behavior, encourage the reader view our supporting video results at: htttps://sites www.le.ooooo.o.oooooooooooo www./www/.../.....ooo..ooo.....e wtwp://eeriiveeeee inver
COMMON SETTINGS:
VQ-VAE Intents:
For all our experiments, we generate intents using VQ-VAE embedding of rendered video of the trajectory. Rendering settings are provided next for each environment. We use VideoGPT's VQ- VQ-VAE implementation (Yan al. 202T An input video of size 64 x 64 x T (w. h, t) is encoded into 16 x 16 x T/4 integer intent =² given codebook of size 50. Each integer represents a float vector of length 4. The training of the VQ-VAR VAE is not the focus this work, and we detail the training data for each VQ-VAE separately for each domain in the supplementary material. We remark that by visually inspecting the reconstruction quality, we found that our VQ-VAEs generalized well to the trajectories seen during learning.
GPT-based policies and exploration noise The policy architecture is adapted from VideoGPT Yan et al.) 2021), and consists of 8 layers, 4 heads and hidden dimension of size 64. The model is conditioned on the intent via cross-attention. the supplementary material, we report similar results with GRU-based policy. Our exploration noise adds Gaussian noise of scale v) to the action output.

---

Evaluation Protocol: While our algorithm only uses loss actions, loss on the resulting trajectories is often easier to interpret for measuring performance. We measure the sum of Euclidean distances between agent state variables, accumulated over time, proxy for trajectory similarity; in our results, this measure is denoted as MSE. Except when explicitly noted otherwise, all our results evaluated on test trajectories (and corresponding intents) that were not in the steering data, but were generated from the same trajectory distribution. None of the trajectories we plot or our video results are cherry picked.
DOMAINS
2D Particle: A particle robot is moved on friction-less 2D plane, by applying a force F = [Fx, Fy] for a duration of At. The observation space includes the positions and velocities of the particle S Y,, Vy], and motion videos are rendered using Matplotlib Animation (Hunter 2007).
While relatively simple for control, this environment allows for distinct and diverse behaviors that are easy to visualize. In particular, we experiment with 2 behavior classes, for which we procedurally created training trajectories: (1) Sp 1 line motion, and (2) Dece poceleraa lerat i ion motion. Both require highly coordinated actions, and are very different from the motion that randomly initialized policy induces. Full details about the datasets are described in Appendix B.1.1
Reacher: A 2-DoF robotic arm from OpenAI Gym's Mujoco Reacher-v2 environment (Brockman iet al. 2016). While usually in Reacher-v2 the agent is rewarded for reaching a randomly generated target, the goal in our setting is for the policy reconstruct the whole motion, given by the intent, which is encoded from a video of the motion rendered using Mujoco (Todorov et al. 2012). We handcrafted trajectory dataset, termed F FixedJoin nt, which is fully described in Appendix B
Hopper: From OpenAl Gym's 's Mujoco Hopper-v2 environment (Brockman et al. 2016). The dataset from D4RL's hopper-mede v2 (Fu et al. 2020), and consists of mostly forward hopping behaviors. There are several challenges in this domain: (1) the dynamics are non-linear, and include non non-smooth contact with the ground: (2) the desired behavior (hopping) is very different from the behavior of an untrained policy (falling), and requires applying very specific force exactly when making contact with the ground (a 'bottleneck the state space); and (3) the camera fixed on the agent, and forward movement can only be inferred from the movement of the background.
STEERING EVALUATION
The first question we investigate is whether IT-IN indeed steers learning towards the desired behavior. To answer this, we consider domains where the desired behavior is very different from the behavior of the initial random policy the Spline and Decelerat ion motions for the particle, and the hopping behavior for Hopper- Ho)per-v2), IT-IN produces a policy that can track the desired behavior with high accuracy. We further show, in Figure 5 and Figure 6 in the supplementary material, that IT-IN works well for different trajectory lengths T.
As show in Figure 2 (for particle), and Figure (for
Another question is whether IT-IN really steers the policy towards the desired trajectories, or perhaps improves some general properties of the policy, allowing a generally better reconstruction. We explore this question by cross-evaluation evaluating the performance of a policy trained with steering intents from P Partion icle: splismin pplines on test intents from Particle Decele lecelelt eereeeeratit ion, which we will refer as out-of-distribution intents, and vice versa. Interestingly, as Table [ shows, performance out-of-distribution intents is significantly worse than the performance that would have been obtained by training the policy with these intents as the steering dataset, and even worse or comparable to training with no steering all (cf. Table
We also evaluated the importance of the exploration noise. We tested Splines with 64 and Hopper-ve v2 with T 128 with and without exploration noise, and large Dateer (2180 for Hopper-v2, 500 for Particle). As the results in Table the supplementary material show, exploration noise 17 is crucial for the training procedure converge towards the desired behavior. Relating this observation to our theoretical analysis, we believe that exploration improves the conditioning of the supervised learning problem.
7

---

-23
00
15
2
2
00
-23
00
23
53
25
0
Reference Trpedorry
Policy Tajecto'y
Figure 2: Particle results on Splines (top) and Deceleration (bottom). Here T = 64 and [Dateel 500. All trajectories start (0, 0), marked by blue circle. In Decelerat ion, the particle quickly decelerates to a stop at / = 32 note the small overshoot at the end of each reconstructed trajectory, due to imperfect reconstruction of stopping in place.
Showing
Reference Wajectory
Tajactory
Trajectory reconstructions in Hopper-v2 with T 128 and Dstee 500. Additional rollouts are presented in Appendix C.5 and in the supporting video results.
Figure 3:
STEERING DATASET SIZE AND GENERALIZATION
We next evaluate the generalization performance of IT-IN to intents that were not seen in the data, but correspond to state trajectories drawn from P(T,). To investigate this, we consider domain where desired behavior is very diverse- the Spline motions for the particle. We also report results on domains where the behavior is less diverse, such Hopper-v2 and Deceleratin for particle. Naturally, we expect generalization to correlate with M, the size Dsteer- As our results in Table T show, additional steering data indeed improves generalization unseen trajectories, albeit with diminishing returns as the amount of steering data is increased. As expected, in the more diverse distribution there more gain reap from additional data (significant improvement up Dateer 50), compared with the less diverse domain (most of the improvement is achieved already with D steer 10). Trajectory visualizations for Splines with different sizes of D Dtter shown in Appendix
motions
COMPARISON WITH RL BASELINES
We compare IT-IN with reward-driven RL baselines. We consider the Particle: Sp1 ines environment, and two reward functions: (1) STATE MSE: MSE between desired position and current position, and (2) INTENT-MSE: a sparse reward that is the MSE between the intents of the desired trajectory and the executed trajectory, given at the end of the episode. STATE-MSE is privileged compared to IT-IN and arguably stronger than any IRL method in this task, as the reward is dense, and exactly captures the desired behavior. Any IRL method will run RL in inner loop, with reward that less precise. INTENT-MEN MSE, is motivated by the fact that IT-IN effectively learns some similarity measure intent space, and this reward captures this idea explicitly.
We used exactly the same policy architecture for all comparisons. We found that both RL methods did train well with the GPT-based policy architecture therefore we report results for the GRU
Difficulty with transformers was discussed PParisotto et al. 2020 Hausknecht Wagener 2022
Steering Dataset ([Dsteerl 500) Splines Deceleration
Table 1: Steering cross- evaluation. See Ap- pendix C.8 for corre- sponding trajectory vi- sualizations.
Splines Deceleration
69.2
210.9
Test Dataset
28.1
18.5
8

---

Table 2: Steering Dataset Size and Generalization. Here T 64, and we show MSE averaged over random seeds. Note that [ Dateer = 0 represents the case where no steering is used at all. In this case, we use trajectories sampled from random policy initialize prrev (see Algorithm 2 (*) For Hopper- the maximal D is 1740 due to limited amount of data in D4RL.
Daver 0
Duver 10
| 50
Deter 100
[Dner 500
[Daterl 2000
Particla:8p inee Particle:Deeelsratt.e
199.7
105.4
75.8
72.7
69.2
66.9
30.0
20.2
17.9
18.6
Hopper-v2
173.3
68.2
67.2
64.9
67.0
63.0
Figure 4: Comparison of IT-IN with
RL Baselines. All results are MSE (lower is better) averaged over 3 ran- dom seeds. Note that IT-IN outper- forms RL on test trajectories (dashed lines) for all Dste sizes.
PPO teet
NTttannn
ITttest
107 107
107
10
Shewning Sufter Size
policy, which is described detail in the supplementary material. We used PPO (Schulman et al. 2017 for RL training, based on the implementation of Kostrikov (2018).
In Figure 4 we report results both on held-out test set of trajectories, and on the training trajecto- ries. As expected, for small Dsteer, STATE MSE obtains near perfect reconstruction of training trajectories, yet high error on test trajectories, as the precise reward makes it easy for PPO to over- fit. Interestingly, however, when increasing the size of Dsteer becomes more difficult overfit with PPO, even with the STATE-MS? reward. Note that for |ateerl 2000, the performance of STATE-MSE on training is worse than the performance of IT-IN on test! We are not aware of studies that investigated RI. training of policies conditioned on very diverse contexts, and our results suggest that vanilla PPO is not well suited to this task. Importantly, on test data, IT-IN significantly outperforms both RL methods for all Dsteer sizes. We attribute this finding the combination of stable supervised learning updates, and not relying on a reward. Finally, our results for INTENT-MSE do not come close to IT-IN, which attribute the more difficult learning from sparse reward.
DISCUSSION
We presented a new formulation for learning control, based on an inverse problem approach, and demonstrated its application learning deep neural network policies that can reconstruct diverse behaviors, given an embedding of the desired trajectory. We developed the fundamental theory underlying iterative inversion, and demonstrated promising results on several simple tasks.
We only considered a particular trajectory embedding based on an off-the-shelf VQ-VAE, which we believe be general and practical. Important questions for future work include characterizing the effect of the embedding on performance, and training an embedding jointly with the policy. Additionally, the exploration noise, which we found to be important, can potentially be replaced with more advanced exploration strategies from the RL literature.
Another interesting question is how to generate intents from partial description of a trajectory, such natural language description. Diffusion models, which have recently gained popularity for learning distributions over latent variables (Rombach et al. 2021). are one potential approach for this.
One open question that remains is the gap between the strict conditions for convergence under linear approximation in our theory, and the generally stable performance we observed in practice with expressive policies and non-linear dynamics. Another open question is whether iterative inversion can be extended to non-deterministic systems. We believe that our work provides the fundamentals for further investigating these important questions.

---

REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Brenna Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (ypt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11995, 2022.
Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Sub- hodeep Moitra, Sameera Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82, 2020.
Dimitri Bertsekas. Dynamic programming and optimal control, volume 1. Athena Scientifie, 1995.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
Shadi Endrawis, Gal Leibovich, Guy Jacob, Gal Novik, and Aviv Tamar. Efficient self-supervised data collection for offline robot learning. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 4650-4656. IEEE, 2021.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv:1902.07742, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXfv:1912.06088, 2019.
Matthew Hausknecht and Nolan Wagener. Consistent dropout for policy gradient reinforcement learning. arXiv preprint 220.: 2022.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pp. 2681-2691 PMLR, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007. doi: 10. MC..009...EE200..555.
Shima Kamyab. Zihreh Azimifar, Rasool Sabzi, and Paul Fieguth. Survey of deep learning methods for inverse problems. arXiv preprint arXiv:2111.0473/ 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 422266900 2014.
10

---

Ilya Kostrikov. Pytorch implementatioons of reinforcement learning algorithms. https://githu how/ http://siiioo//ptrrcc------eaeee
acktr-qaii 2018.
Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. Reconnet: Non- iterative reconstruction of images from compressively sensed measurements. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pp. 449-458, 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint urXiv:1509.02991 2015.
Alice Lucas, Michael Iiiddis, Rafael Molina, and Aggelos K Katsaggelos. Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Processing Magazine, 35(1):20-366 36, 2018.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. arXiv preprint arXiv:2110.09514, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529-55 -33399-333333 2015.
J. M. Ortega and W. C. Rheinboldt. Iterative Solution of Nonlinear Equations in Several Variables. Society for Industrial and Applied Mathematics, 2000. doi: 0137/1.97808877199
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. Advances in neural information processing systems, 1, 1988.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-6676655 633.. JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00..vv777.664477 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.05960) 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model -asee control. In 2012 IEFE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va-vae and transformers, 2021.
11

---

A PROOFS
A.1 PROOF OF EQUATION
Throughout this and the rest of the theoretical proofs, with slight abuse of notation, when vector
E IN is added to matrix A E RMN,, the addition is row-wise: A + 1u (u,...,,) MXN.
= A + 1u where
Denote Y (y1....., MM) xMxxmmm)))) The approximated linear function is Go. = YO + where θ (gdm(())dm(())) and [x (note the added parameter b to account for the bias). At iteration + 1:
Ont1, bnt1 = arg min [F(X") | + +
0,6
This is ordinary linear least squares problem with the solution
Onti= (F(X") (XXx)
bp+1 X -
(6)
Then
X Sem+1d044 (Y) y0 ba+1 X +Y (X-F(Xx))0mt
and averaging over the points obtains the result:
X X" + ( (+--FFFF O(X
A.2 PROOF OF THEOREM
Using the notation defined in Appendix A.
Assuming F(X) XF + h is a linear function with F E (ddm((')xdim()) and h E R I (xdim()))
Assuming rank(F(Xx)
F(X"))
dim(Y) then (F(X") (F(X) - -(X")) is
invertible and O, defined on Equation 6 is well defined.
Then using the fact that F(X") F(X") = (X" - F
θ = (F(X)) F(X")) (x" -X") = ((x- -x0))))))) (x"
and it satisfies
1==
where / is the identity matrix. The bias term, according to Equation 6 is
bi = X F(X")o, X FX + h)
At the end of iteration 1, X- = -Yi + +b, and its matching outputs equal to the desired outputs:
F(X*) F
h
= -X - FX + F +h
+++X00F XUF +h =
A.3 PROOF OF THEOREM 2
For clarity in the representation, we will use the following notation: J J- ( Jn J(X"). F. F(X") and F,, = F(X"). We also define H,, = F, and Hn F Y
First we show that J, is non-singular Since d < 1 then p( < || - SC$ < where ((A) denotes the spectral radius of A. Therefore (I + J, J, is non-singular and j-1= = (T J,1) is non-singular multiplication of non-singular matrices.
ç3
12

---

We denote by Jn (mm
its inverted matrix, and obtain the following bounds:
H - f1,11 11 -M
|| = (I + A >
II & 8(1 60)
(8)
11 Jw-l (1)
| --''l
5288
- 1- 1| J
6 1 C8B
(9)
1- | An J
|u-ll - || - Jall + || 5
(10)
1 = ||ml - - | - | - S (m -1 (11)
Inequality (1) developed in Horn & Johnson 2011 p. 381). Also note that the rest of the inequalities in 9 are well defined since ò << 1/Ç3.
The proof now continues similarly to the proof of Ortega & Rheinboldt (2000 12.3.3). We set Gn = X H= Xn+I, and show that G2 is an Iterated Contraction:
Xnt2 - X
| Hn+1
X n+1 || 3 B(1 8(1+6))11 +
1 B(1+ +6)|+|Hn++1111 Hn - -x"
(3)
3 3(1 + +8]] H)| Hn - (X++T - X" Xn)Jall + +(1 8)115, -Ja -1 X
(4)
S 3(1 6) (2x+ |1a+1 H
B11 80t1|
(6)
2 3(1 + 6)
(2x |1+ 11++xxx+--xxi] + + + 6)
2x
$ 00)
|x++I X + + 14 || X"1
(6)
(2 +
S 3(1 + 8)
||,-- -Vll + | X
=(1
-I)]|x+tT X
where inequality (2) holds because of Bound g
(3) is the triangle inequality. (4) is due to the Bounds
7 and 9 and the triangle incquality. Inequality (5) is proven in Ortega & Rheinboldi (2000 3.2.5) and inequality (6) is from Bound 11
Assuming 3(1 + 5) + 14) <
2x8 80 =p 1- 3(1 + d) + +7)
A( - FII =
1 Y
g 2 1 only when F,, is close to Y. g is
function of ||,F, - -YI. thus if Fr Y 2
P + c for some c > 0 then g(I Y() v a < 1 where a is independent of ||F - YII.
The convergence of {X") follows from Ortega & Rheinboldt 2000 12.3.2), as long as g S a < 1. Since the following holds
| YI1 (N X X S ... S (n (
then for every > 0 there exists k such that at < (1+0|XX-X*) and the convergence is
towards the ball | - Y S p. Inequality (7) is from Bound
11 For uniqueness, see the end of the
proof Ortega & Rheinboldi 2000 12.3.3).
13

---

A.4 CONVERGENCE RESULTS FOR 1- 1-DIMENSIONAL F
We restrict ourselves to the 1-dimensional case, where X' = ) = R. and assume the function F is strictly monotone and its maximum and minimum slopes are not too different, thus the function is "close to" linear. We then show convergence at a linear rate.
Let S (Z1, x2) (F(z1) (x2)))/((1 - x2) denote the slope of F between Iı and z2, and max [S = m2xX (8 (21, 02)l denote the maximum absolute slope of F and similarly min | = minay,aze 1S (x1, x2) the minimum absolute slope.
max 15 min [ST] min
Assumption 4. F is continuous and strictly monotone, and
2 - for some 0 < <<< 1.
Theorem 3. Assume X' y R. that Assumption f holds, and that there only two desired outputs M- = 2. Then for any i 21 and any iteration vı; [F(x] - vil S (1 F(x)) - M/ll
When the number of desired outputs is greater than 2, then convergence for each output is generally not guaranteed.
Theorem Assume X' =y- R. that Assumption 4 holds and that at iteration n, Vi x (Y)
Or Vi xh FV) Then X - S 111-0) X (Y)
Theorem guarantees that after finite iterations, the output segment intersects with the desired output segment. Note that Theorems 3 andd4dd not require any kind of approximations as in Assumption
nor for F to be differentiable.
A.4.1 PROOF OF THEOREM 3
Denote SJ mar max 22 SJ (I1, and similarly Siin ming, 22 S ((1112))
Assuming X = y = R. Then the approximated linear function is Go = y0 + where 8, E R scalars.
At iteration n + and for i E [1,
x z++ bent (3:)) y:0n+1 + bx+1
(12)
0nt1, but1 arg min 8,6
(OF(x,)) + x2?)²
Lemma 5. if = y = IR then Vn: Syae 3
S On+1 S
8yan F is strictly increasing and
Snt1 < zt if is strictly decreasing.
Proof. We will prove for strictly increasing F. The proof for strictly decreasing F is symmetrical.
W.L.O.G assuming that X is sorted: Vi: x. S 2f+1- Let h > i then:
F(uF) + S (1 - = < F F(x"))
S (2/ xx²"
1 - F(x?))
(F(x)) - F(x?)) S xx x $
(F(x))-F(x))
F maz
SF
min
M 11=1 (x)) - _xo) [[())) F(X*)
1 M
an+1
M E (F F(X.)) 2
- 1 ΣM-1 EM (1f - xf) ( F(
M
: Diel Dksitl (F(x))- - F(x?)))
M -1 MM
M
-M- MM
(F(xz) 2
M
Ek=it1 S
1
A/2 Σ
1 M-1 M
1 (F(xt) F(a;)))
SF
14

---

and
1 D M
2k=it1 Soae (F(ct) - F(()))
Out1
Σ M-1 i=l
Dimi (F(z))
- F(x?)))
S
M
When M = 2, the regression line passes exactly at the points (F(xT), and (F(x)), and ba+1 also takes the following forms:
bat1 = x" 1n+1F F(+)) = =x2 F
Then placing bp+1 in Equation
12
we get for every E [1,
z xt 0x+1 F(x
Denote the slope of F between i; =+1 and xp: S (xt xx?
F(a )-Fa")
Then the following equations hold:
F(a = F(x)) + 1n+1S' St (x 2x - [(x?))
y: F(u (1 (1 On+18 " (yi F(xx))
(13)
and Using Lemme and since F always increasing always decreasing, then 0n+18 (x++),;))) and
min ST
- max ST
I Ont1S (0 ,,,), min |
2 -
2 max |
1 - 0 1nt1St S (17tt1 x.ai 5 max { ||
1 -
2 1, -El
= =1-6 El
(14)
Then, placing into Equation 13
|/ F(+,+))| = 0 S* (5 ,
- - ( -) 14 ((1-))
F(x?))
Note the convergence in one iteration for the linear case when 1.
A.4.2 PROOF OF THEORFM
Denote Ln:
In Y- F-X(")
I ΣN Y - f(x7)
M
F- - x Ehm FY) xx
Y Y- -((()
M 11
Σ
F-1(V) X"
2 k
F-1 - mm1
F- x
NI
Y- fx)()
Σ Wn.d FV) - x
Σ
Wn,i |(FFFFF
OF (Y): x?!')
Where Wn,s Ems (M))
f -8)
M Wnii 1 and, since assumed Vi x. FY)
that Vi r > SF.
(Y). then Vi Wn,i > O. Therefore Ln is a weighted- -man of the slopes and
min L, < Staz.
From Equation the following holds:
Xn+I X O OntilV ( = 1n ( - X"
F Xa+l (1 On+1 (;)
X
(15)
Using Lemma and the inequalities S.Cm S Ln SF Smann Inequality (14 from Appendix A.4.1 also applies for L,, and we obtain:
11 p Lnd < -
F- - X $ (1 (1-c) X"
15

---

B EXPERIMENTAL DETAILS
Table 3 contains a list of common hyperparameter values that we have used for all the experiments. Table contains Parti cle and Reacher-v2 specific hyperparameters, while Table listing Hopper r- specific hyperparameters. We note that the minor difference in hyperparameter values between the domains evaluated is purposed only at achieving slightly better MSE results per domain. We observed that the steering behavior was relatively robust to hyperparameter values.
B. PARTICLE ROBOT
The 2D plane in which the robot is allowed to move is finite, with the maximum coordinates increasing for longer horizons. When rendering the videos we include the entire 2D plane, up the maximum coordinates. When evaluating policies, a validation set of 2,000 trajectories was used, which were unseen during training the policies.
B.1.1 DATASETS
Splines Trajectories follow the function B-spline curve The curves are of degree with control points, which uniformly sampled between 0-1 in a 2- 2-dimensional space.
Deceleration Random X and Y forces for the first face trajectory steps, and then T - face steps of deceleration, where is the time horizon. Deceleration at step j face done by setting Py vg- Ar (assuming the mass of the particle is 1). V and are defined in
V2-1
F
Section
0.1
B.2 REACHER-V2
B.2.1 DATASETS
Fixed Joint Trajectories were collected represent scenario where of the two robot joints is malfunctioning and is force fixed in place. The policy can only control the other robotic arm joint. When evaluating policies, validation of 2,000 trajectories used, which unseen during training of the policies.
B.3 HOPPER-v2
B.3.1 DATASETS
Hopping The datasets of size 2180 trajectories used for sequence-lengths 64 and 128 were extracted from D4RL's hopper-mesn medii um um-v2, and consist mostly forward hopping behaviors. When evaluating policies, validation set of 436 trajectories was used, which were unseen during training of the policies. Unlike in the other evaluated domains, where trajectories sampled from a random policy used train the VQ-VAE, pooper-medd um-v2 2 the reason that using the initial random policy, the trajectories terminated (hopper fell down) before reaching the desired T. For IT-IN training, we have modified Hopper-v2 slightly so that the episode will not terminate when the Hopper falls, thus allowing it to reach T steps.
Hopper-v/ we have used input videos from D4RL's
B.4 GPT-BASED ARCHITECTURE
The model is conditioned on the intent via cross-attention. The actor network is comprised of Linear layers size 64, with tanh activation.
B.5 GRU-BASED ARCHITECTURE
The single-layer GRU's hidden state size is set match the flattened intent size of 4096. The actor network is comprised of 2 hidden Linear layers of size 4096 and a tanh activation is used.
16

---

Table 3: Common hyperparameters for all experiments.
Hyperparameter
Value
Learning rate
5e-4
Sampled rollouts per epoch (N)
200 Minibatch size
Training iterations
2,000
Training buffer size [rollouts] (K x N)
40 x N
Steering buffer Dsteer size [rollouts]
500
Ratio of stecring intents in minibatch (a)
0.3
Gradient norm clipping
0.5
GPT: # layers
8
GPT: # heads
4
GPT: hidden layer size
64
GPT: dropout
0.2
GPT: attention dropout
0.3
Table 4: Particle & Reacher-v2 hyperparameters.
Hyperparameter
Value
Minibatch size (rollouts)
Noise scale (7)
4.0
Total epochs (n)
160
C ADDITIONAL EXPERIMENTAL RESULTS
C.I PARTICLEISPLINES EFFECT TRAJECTORY LENGTH
We tested IT-IN multiple horizons I' in the Splines domain, and found work well across horizons of 32, 64 and 128. We present sample visualizations with different ''' values in Figure 5 (showing the final reconstructed trajectories) and in Figure 6 (showing trajectory progression during an epsiode).
C.2 PARTICLE:SPLINES EFFECT OF STEERING DATASET SIZE
In Figure we present trajectory visualizations showcasing the effect of the size of the steering buffer Dateer (cf. Table 2).
C.3 PARTICLEIDECIIRRATONN EFFECT STEERING DATASET SIZE
Similarly Section C.2 in Figure showcase the effect of the size of the steering buffer Dsteer (cf. Table 2 in the Particle wew.:iill:eeeeeeeeeee domain.
C.4 RRACHER-V2
We present sample reconstruction visualizations for random-action trajectories from Reacher-v2 on 16-step sequences in Figure I0 Sample videos for 64-step FixedJoint sequences (trained with a GPT-based policy) can be found in the project's website:
view/iter-inver
Table 5: Mujoco Hopper-v2 hyperparameters.
Hyperparameter
Value
Minibatch size (rollouts)
6
Noise scale (7))
1.0
Total epochs (n)
130
17

---

Length e 52
NSE 18.05
Langth Mean
-
Overall Moen
Reforsnce Eijecilly
Pulicy Enjectery
Figure 5: Example results on the Splines dataset, for different sequence lengths. In all cases shown here [Satear] = 500. To the left of each 3 policies trained with different seeds. Note the increasing scale of the plots as the sequence length increases. Also note that trajectories start (0, 0), marked by the blue circle in each plot.
state the average MSE evaluation set of
Step
Qep
Step
Seep 16
Qhep 20
Ctep 24
Oep 28
Owe
Shawing
Every
-2.
-23
Drgy
Ong
Benan
Length
Every
Org
Ong.
Recan.1
Recon.2
Step
Length B4
Every Sheps
Recos
Step
Suep
Stap
Stap 32
Step 40
Stop 48
Step 56
Stog 54
Length
Showing
steps
Ofly
Ofg.2
Shep
Step 24
Step
Rac
Step
Shep 128
Longth
shawing Every
<
Ong.1 1
g.g.2
-
Bacon
Bacon
Length
Every 16
Stego
0.
Facon
Bacon
Figure 6: Visualization of trajectories progression in the Sp1 i nne domain for different horizons 7. Note the increasing scale of the plots as the sequence length increases. D 500.
18

---

R
[D_cteer]
R
[P_steer]
[D_stesr]
DD. stwerl
<da
R
2
(D_teerr
-21
Refererce
Figure 7: Example results the Parti particle:ss sizes of Dstcer- Each row corresponds to a different size. Each column corresponds to a specific reference trajectory from the dataset, the intent of which used input to the policies. T 64 was used in all experiments.
dataset for policies trained with different
19

---

DD_steer
00. stwer
10_steer
10 D0_steer
10. st_steer//
10 ststter
Figure 8: Example results on the Particle:Decelersse dataset for policies trained with different sizes of D D T = 64 was used in all experiments. Figure structure same as in Figure 7
20

---

Length Every
Lengsh Showing Every Steps
Lang Showing Every
Sefteetce Tojectmry
Pilry Teijectrry
Figure 9: Examples of trajectory reconstructions in the Hopper v2 domain, with T 128 and D 500.
Figure 10: Examples of trajectory reconstructions in the Reacher-v2 domain. each plot, the red row is the reference trajectory and the blue row is the policy reconstruction. These are based GRU policy. For ease of viewing, we modified the dark colors of the original rendered images.
C.5 HOFPER-V2
We show additional examples of rollouts for the Hopper-v2 domain on 128-step sequences in Figure
C.6 EXPLORATION
Table 6 is showcasing Splines and Hopper-vz reconstruction MSEs when trained with and without exploration noise 7).
21

---

Table 6: Evaluation of policies trained with and without exploration. We show average MSE for policies; due to different domains, MSEs are comparable only within each row.
Exploration Noise 7)
No Exploration
Particle: T = 64, Dtserr = 500
69.2
454.7
Hopper-v2, T 128, Dateer 1740
483.3
920.6
Table 7: RL hyperparameters
Hyperparameter
Value
PPO Clip Ratio
0,2
GAE À
0,95
Discount rate Y
0.99
Learning rate
Ic-4
Value loss coefficient
0.5
# epochs
4
# rollouts sampled per policy update
128
Total iterations
5000
C.7 RL BASELINE COMPARISON
Table is summarizing the hyperparameters used for training RL policies with PPO Schulman et al.) (0117) and GRU policy.
C.8 STEERING CROSS EVALUATION
In Figure II we show example rollouts from the experiments described in Section
C.9 GRU-BASED POLICY EXPERIMENTS
We report similar results with GRU-based policy to the the results shown in Table (analyzing the effect of steering dataset size) in Table & and similar results to Table [ (steering cross-evaluation) in Table
Table 8: Evaluation IT-IN with GRU policy on variable Steering Dataset size. T 16. Note that Deteer = 0 represents the case where steering is used all. In this case, trajectories sampled from a random policy to initialize Drrev (see A Algorithm
D
Dator
10
Dater
50
Datcee
100
500
Dater
5000
Particle:Splines Masticle:Deeelle Reacher-v2. v2nher--2:11 FixeJoinnn
5.48
5.18
3.86
3.61
3.02
2.89
0.89
0.75
0.73
2.49
2.05
1.68
1.64
1.58
1.61
22

---

Stearing:
bett:
oa
Decelaration
splines
Rafauuss Tejuclary
Tokk Tejectorr
(a) Testing trajectories from Particle:
Splinea
Deceleration
Steering:
Deceleration
Deceleratio
Refewenc Tijecily
Palay Vajecter
(b) Testing trajectories from
Figure 11: Examples comparing how policies trained with steering intents from either Decee aarticle:Deceeeatt:
Particle: Splines P Particcle either datasets. We can see that when a policy is trained with steering intents from one dataset, it performs well that dataset and performs poorly on the other. In each column the reference trajectory is the same.
perform when tested on trajectories from
Table 9: Steering cross-evaluation for GRU-policy. 16.
Steering Dataset D 500)
Splines Receleratinn
Splines Deceleration
2.79
5.4
Test Dataset
1.46
0.72
23


Reviews:
[{"review_id": "LsPGBIpRR8", "reviewer": "Reviewer_2R3c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents iterative inversion, an approach for learning control from video demonstrations (without actions) and online interaction. The approach works as follows:\n- Start with a random exploration policy\n- Train an inverse dynamics model on the online data mapping a sequence of states to a sequence of actions\n- Apply the inverse dynamics model to the demo trajectory to get the reference action sequence\n- Update the policy to match the reference action sequence\n- Repeat while the policy improves+explores until it matches the reference video demo.\n\nThere are a few additional tricks to stabilize thing but that is the essence of the method. The inverse dynamics modeling is done in a seq-to-seq fashion using a VQ-VAE image encoder and a transformer sequence model. \n\nIn experiments, they first confirm that the method indeed steers behavior toward the demos effectively in some toy tasks, then compares to some RL methods on image-based Hopper/Reacher.\n", "summary_of_the_review": "Overall this is an interesting paper tackling an interesting problem. However, the work needs to better position its contribution w.r.t the many similar works in the field, and expand its experiments to more challenging domains and consider more relevant baselines. \n\n--- Post rebuttal\nThe rebuttal has addressed most of my concerns.", "strength_and_weaknesses": "*Strengths*\n\n- This is an interesting approach/problem. Learning to match a video demonstration through online RL is indeed an important and well studied problem, and the paper presents an interesting take on it.\n\n*Weaknesses*\n\n- The first major limitation of the work is the novelty of the proposed method. There is a vast literature of works that learn inverse dynamics models for learning to generate actions from observations, from Schaal et al. to more recent works like Ghosh et al. and Zheng et al. So fundamentally the idea of supervised learning of an inverse model to match demonstrations is not new, nor is the idea of using such an inverse model iteratively online during agent interaction (while GCSL trains a goal conditioned policy, extending the policy to follow the sequence of goals in a demonstration is a trivial extension). The main thing that appears to be new about the proposed approach is the method of training the inverse model, specifically through a VQ-VAE and a transformer that maps the full sequence of states to a sequence of actions directly (rather than operating on single s_t, s_t+1 pairs). I actually think this part of the contribution is quite interesting and novel, and could make sense as an improvement over prior works. I would advise the authors to emphasize this as the main contribution of the work. \n\n- The second issue is with the experiments. While the toy experiment showing how the method works is nice, there really isn't a comprehensive comparison to the relevant baselines. Even the RL baselines used are only tangentially relevant to the proposed method. Fundamentally the proposed method is an inverse RL method (demonstrations --> online interaction to produce a policy that matches the demos). Therefore the work should compare to the state-of-the-art inverse RL methods like GAIL from observation or the various extensions like Berseth et al. and Reddy et al, as well as methods like GCSL (Ghosh et al.) on more challenging benchmarks like vision-based robotic manipulation. \n\nBerseth et al. Towards Learning to Imitate from a Single Video Demonstration. 2019. \nReddy et al. SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards. ICLR 2020.\nSchaal et al. Real-Time Robot Learning With Locally Weighted Statistical Learning. ICRA 2000. \nGhosh et al. Learning to Reach Goals via Iterated Supervised Learning. ICLR 2021.\nZheng et al. Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3AIdpZWMLn", "reviewer": "Reviewer_9eeh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors try to solve the problem of inverting a deterministic dynamics model when query access to the said model (being able to query the next state given the current state and the current action). The performance of the trained inverse model is evaluated on a distribution of reference state trajectories.\n\nThe core idea proposed is to assume that the dynamics is invertible, and we try to find the inverse of the dynamics which we can apply to the observed state trajectory to find its actions. Specifically the mapping from a sequence of actions to a sequence of states $F : A^n \\rightarrow S^{n-1}$ is invertible so $F^{-1}$ is well defined. Then we learn $F^{-1}$ with a parametric model $G_\\theta$ such that $G_\\theta \\circ F \\approx F^{-1} \\circ F = \\text{id}$. Crucially we assume access to a \"reasonable\" loss over the action sequences which we use to train $G_\\theta$. The trajectories on which we optimize $G_\\theta$ is a mixture of samples from the distribution of reference trajectories and trajectories from randomly interacting with $F$.", "summary_of_the_review": "I like aspects of the proposed approach and the problem of learning inverse dynamics model. But the presentation could be made clearer. In particular, the problem should be formulated more clearly. ", "strength_and_weaknesses": "Strengths:\n1. The basic approach (algorithm) seems reasonable and novel. Instead of access to a loss measuring the similarity of state trajectories, it assumes access to a loss measuring the similarity of action sequences during training. However, it fails to explicitly discuss the connection and assumption between the two distances (over states and over actions). (see Weakness1)\n\nWeaknesses:\n1. The formal problem formulation and the exposition has gaps. If the ultimate goal is to reproduce the reference state trajectory (as suggested by the evaluation protocol), then we are assuming that low action-action loss implies low state-state loss, correct? I imagine that differences in early actions in a sequence would have a larger impact than the differences in later ones. How do MSE over states and over actions relate to each other in your experiments?\n1. The proposed problem setting does not seem adequately motivated. Why do we want to learn an inverse model that is accurate over a small (in comparison to all possible trajectories) set of state trajectories? In learning dynamics via interactions, a key challenge is the difficulty to access many states (unlike in IID setting) with a simple policy, e.g., uniformly random policy. Are the reference trajectories helping us access such region of the state space?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0zeIg21Ujx", "reviewer": "Reviewer_VhZe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper solves the problem of  learning for control by learning a inverse mapping between action sequence and states sequence.  To handle the distribution shift, the authors proposed  a general algorithm called interactive inversion which interactively  learn the inverse mapping under the current policy, then apply it to the desired output to obtain a new policy for it. Under certain assumptions, the proposed algorithm is proved to converge to the desired mapping theoretically.  By applying the iterative inversion, the authors further present a IT-IN algorithm for learning actions given any desired state sequence. Through experiments, the authors show the proposed algorithm has an improved performance on imitating diverse behaviors compared to reward based RL methods.", "summary_of_the_review": "The proposed method is shown to perform significantly better than other works. Sufficient proofs are provided to support the statement in this paper. ", "strength_and_weaknesses": "This paper presents a new viewpoint of the problem of learning control which directly mapping state sequence to action sequence. \nCompared to the RL methods, the proposed algorithm does not need a pre-defined reward function which could be hard for many real applications. The analysis looks valid to me. And enough experiments are presented to support the statement in this paper. \n\nThe assumptions raised in this paper are pretty strict to satisfy in real applications. Like stated in section 3, the conditions can be interpreted as F can be well approximated by a linear function. This means the mapping from state sequence to the action sequence should be close enough to a linear function in order to have convergence guarantee.  Do these assumption hold in the examples under the experiment section? \n\nI would also question the size of the steering dataset for more complex tasks with more intents. In order to learn the mapping function, it is necessary  to have the steering dataset sufficient enough to capture all the sequences needed for accomplishing tasks. Could you give more explanation about how to construct such dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject