Watch and Match: Supercharging Imitation with Regularized Optimal Transport
Authors: Siddhant Haldar
Vaibhav Mathur
Denis Yarats
Lerrel Pinto
Venue: CoRL 2022 Oral (2022)
Keywords: Imitation Learning
Manipulation
Robotics

Abstract: Imitation learning holds tremendous promise in learning policies efficiently for complex decision making problems. Current state-of-the-art algorithms often use inverse reinforcement learning (IRL), where given a set of expert demonstrations, an agent alternatively infers a reward function and the associated optimal policy. However, such IRL approaches often require substantial online interactions for complex control problems. In this work, we present Regularized Optimal Transport (ROT), a new imitation learning algorithm that builds on recent advances in optimal transport based trajectory-matching. Our key technical insight is that adaptively combining trajectory-matching rewards with behavior cloning can significantly accelerate imitation even with only a few demonstrations. Our experiments on 20 visual control tasks across the DeepMind Control Suite, the OpenAI Robotics Suite, and the Meta-World Benchmark demonstrate an average of 7.8x faster imitation to reach 90% of expert performance compared to prior state-of-the-art methods. On real-world robotic manipulation, with just one demonstration and an hour of online training, ROT achieves an average success rate of 90.1% across 14 tasks.

Watch and Match: Supercharging Imitation with Regularized Optimal Transport
Siddhant Haldar
Vaibhav Mathur
Denis Yarats
Lerrel Pinto
New York University
rot-robot.gii citesgiigtuubii
Abstract:
Imitation learning holds tremendous promise in learning policies efficiently for complex decision making problems. Current state-of-the-an algorithms often use inverse reinforcement learning (IRL., where given set of expert demonstrations, an agent alternatively infers a reward function and the associated optimal policy. However, such IRL approaches often require substantial online interactions for complex control problems. In this work, we present Regularized Optimal Transport (ROT), new imitation learning algorithm that builds on recent advances in optimal transport based combining trajectory-mand matching rewards with behavior cloning can significantly accelerate imitation even with only few demonstrations. Our experiments on 20 visual control tasks across DeepMind Control Suite, OpenAl Robotics Suite, and the Meta-World Benchmark demonstrate an average of 7.8x faster imitation to reach 90% of expert performance compared prior nate-of-the-ar methods. On real-worls robotic manipulation, with just one demonstration and an hour of online training, ROT achieves an average success rate of 90.19 across 14 tasks.
matttiin... Our key technical insight is that adaptively
Keywords: Imitation Learning, Manipulation, Robotics
Adaptive Regularization
Environments
Expert
OT Computation
ROT
Environment Interactions
Agent
Rewards
Opening a box
Hanging tote bag
Placing peg box
Pouring almonds
Figure 1: (Top) Regularized Optimal Transport (ROT) adaptively combines offline behavior cloning with online trajectory-matchin based rewards. This enables significantly faster imitation across variety of simulated and real robotics tasks. while being compatible with sigh-dimensional visual observation. (Bottom) On our xArm robot, ROT can learn visual policies with only single human demonstration and under an hour of online training.
imitation learning algorithm that
'Corespondence to: siddhanthaldar nyu.edu
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Introduction
Imitation Learning (IL) 21, 2, has a rich history that can be categorized across two broad paradigms, Behavior Cloning (BC) [1] and Inverse Reinforcement Learning (IRL) [4]. BC uses supervised learning to obtain a policy that maximizes the likelihood of taking the demonstrated action given an observation in the demonstration. While this allows for training without online interactions, it suffers from distributional mismatch during online rollouts [5]. IRI. on the other hand, infers the underlying reward function from the demonstrated trajectories before employing RL to optimize policy through online environment rollouts. This results in a policy that can robustly solve demonstrated tasks even in the absence of task-specific rewards [6, 7].
Although powerful, IRL, methods suffer from a significant drawback - they require numerous expensive online interactions with the environment. There three reasons for this: (a) the inferred reward function is often highly non-stationary, which compromises the learning of the associated behavior policy [7]: (b) even when the rewards stationary, policy learning still requires effective exploration to maximize rewards [8]; and (c) when strong priors such as pretraining with BC are applied accelerate policy learning, ensuing updates to the policy cause a distribution shift that destabilizes training [9, 10]. Combined, these issues manifest themselves on empirical benchmarks, where IRL methods have poor efficiency compared vanilla RL methods on hard control tasks [11].
In this work, we present Regularized Optimal Transport (ROT) for imitation learning, a new method that is conceptually simple, compatible with -igh-simensiona observations, and requires minimal additional hyperparameters compared to standard IRL approaches. In order to address the challenge of reward non- non-stationarity in IRL, ROT builds upon recent advances in using Optimal Transport (OT) [12, 13, ]1] for reward computation that use non-parametric sajjector--macching functions. To alleviate the challenge of exploration, pretrain the IRI. behavior policy using BC on the expert demonstrations. This reduces the need for our imitation agent to explore from scratch.
However, with OT-based reward computation and pretrained policies, we only obtain marginal gains empirical performance. The reason for this is that the high-variance of IRL policy gradi- ents [14, 15] often wipe away the progress made by the offline BC pretraining. This phenomenon has been observed in both online RL [16] and offline RL [9] methods. Inspired by solutions presented in these works, stabilize the online learning process by regularizing the IRL policy stay close to the pretrained BC policy. To enable this, we develop a new adaptive weighing scheme called soft Q-filtering that automatically sets the regularization prioritizing staying close the BC policy in the beginning of training and prioritizing exploration later on. In contrast to prior policy regularization schemes [16, 17), soft Q-filtering does not require hand-specifiiation of decay schedules.
To demonstrate the effectiveness of ROT, we run extensive experiments on 20 simulated tasks across DM Control [18], OpenAl Robotics [19], and Meta-world [20], and 14 robotic manipulation tasks on an xArm (see Fig. 1). Our main findings are summarized below.
1. ROT outperforms prior tatee-oftthee art imitation methods, reaching 90% of expert performance 7.8 x faster than our strongest baselines on simulated visual control benchmarks.
On real-world tasks, with a single human demonstration and an hour training, ROT achieves an average success rate of 90.1% 1% with randomized robot initialization and image observations. This significantly higher than behavior cloning (36.1%% 1%) and adversarial IRL (14.6%).
3. ROT exceeds the performance of state-of-the-an RL trained with rewards, while coming close to methods that augment RL with demonstrations (Section 4.5 & Appendix H.3). Unlike standard RL methods, ROT does not require hand-specification of the reward function.
4. Ablation studies demonstrate the importance of every component in ROT, particularly the role that soft Q-filtering plays in stabilizing training and the need for OT-based rewards during online learning (Section 4.4 & Appendix H.4).
Open- code and demonstration data has been publicly released on our project website. Videos of our trained policies can be seen here: rot-robot github.in io.

---

(a) Task: Particle Reach
(b) IRI. Finetune w/o Reg.
(c) ROT
(d) ROT + random init.
100k
timesiey
Expert trajectory
BC trajectory
Start location
Goal location
Figure 2: Given a single demonstration to avoid the grey obstacle and reach the goal location, BC is unable to solve the task (a). Finetuning from this BC policy with OT-based reward also fails solve the task (b). ROT, with adaptive regularization of OT-based IRL with BC successfully solves the task (c). Even when the ROT agent initialized randomly, it is able solve the task (d).
Background
Before describing our method in detail, provide brief background to imitation learning with optimal transport, which serves as the backbone of our method. Formalism related to RI. follows the convention prior work [8, 11] and is described in Appendix A.
Imitation Learning with Optimal Transport (OT) The goal of imitation learning is to learn behavior policy Th given access either the expert policy л² or trajectories derived from the expert policy T While there are multitude of settings with differing levels of access to the expert [21], our work operates in the setting where the agent only has access to observation-basee trajectories, i.e. T = ((r,, { Here N and 7 denotes the number of trajectory rollouts and episode timesteps respectively. Inverse Reinforcement Learning (IRL.) [4, 22] tackles the II. problem by inferring the reward function r based expert trajectories T Then given the inferred reward r", policy optimization used to derive the behavior policy T². To compute r", new line of OT- based approaches for IL [12, 13, 11] have been proposed. Intuitively, the closeness between expert trajectories T and behavior trajectories T can be computed by measuring the optimal transport of probability mass from T - T Thus, given cost matrix C..' c(o;: of: and the optimal alignment μ² between a behavior trajectory o² and and expert trajectory of, reward signal for each observation can be computed using the equation:
OT =
- Cee
detailed account of the OT formulation has been provided in Appendix A.
Actor-Critic based reward maximization Given rewards obtained through OT computation, effi- cient maximization of the reward can be achieved through off-policy learning [7]. this work, we use Deep Deterministic Policy Gradient (DDPG) [23] as our base RL optimizer which is an actor-critic algorithm that concurrently learns deterministic policy Ts and a Q-function Qs. However, instead of minimizing a one step Bellman residual in vanilla DDPG, we use the recent n-step version of DDPG from Yarats et [8] that achieves high performance on visual control problems.
Regularized Optimal Transport
fundamental challenge imitation learning is to balance the ability mimic demonstrated actions along with the ability to recover from states outside the distribution of demonstrated states. Behavior Cloning (BC) specializes mimicking demonstrated actions through supervised learning, while Inverse Reinforcement Learning (IRL.) specializes in obtaining policies that can recover from arbitrary states. Regularized Optimal Transport (ROT) combines the best of both worlds by adaptively combining the two objectives. The challenges in online finetuning from pretrained policy have been

---

described in Fig. 2, with more details provided in Appendix B.1. ROT operates in two phases. In the first phase, a randomly initialized policy is trained using the BC objective on expert demonstrated data. This "BC-pretrained" policy then serves an initialization for the second phase. In the second phase, the policy is allowed access to the environment where can train using an IRL. objective. To accelerate the IRI. training, the BC loss is added to the objective with an adaptive weight. Details of each component are described below, with additional algorithmic details Appendix C.
3.1 Phase 1: BC Pretraining
BC corresponds to solving the maximum likelihood problem shown in Eq. 2. Here T refers to expert demonstrations. When parameterized by a normal distribution with fixed variance, the objective can be framed as regression problem where, given inputs sf, BC needs to output a².
C BC B(stla")~==llaa
After training, it enables n BC to mimic the actions corresponding to the observations seen in the demonstrations. However, during rollouts in environment, small errors in action prediction can lead to the agent visiting states not seen in the demonstrations [5]. This distributional mismatch often causes n BC to fail empirical benchmarks [16, 11] (see Fig. ?? (a) in Appendix B).
3.2 Phase 2: Online Finetuning with IRL
Given a pretrained BC model, we now begin online 'finetuning' the policy ='= ROT in the environment. Since we operating without explicit task rewards, we use rewards obtained through OT-based trajectory matching, which described Section 2. These OT-based rewards O' enable the use of standard RL optimizers to maximize cumulative reward from T² π ROT, In this work use n-step DDPG [23], a deterministic actor-critic based method that provides perforerformancc in continuous control [8].
Finetuning with Regularization x BC is susceptible to distribution shift due to accumulation of errors during online rollouts [5] and directly finetuning BC also leads to subpar performance (refer to Fig. 2). To address this, we build upon prior work in guided RL [16] and offline RL [9], and regularize the training of n ROT by combining with BC loss as seen in Eq.
ROTT argmax [(1 π
(Q(s, ad( Bor [+*)
Here, Q(s, a) represents the Q-value from the critic which is optimized using OT-based rewards during the actor-critic policy optimization. a is a fixed weight, while X(T) is a policy-depende adaptive weight that controls the contributions of the two loss terms. Dg refers to the replay buffer for online rollouts.
Adaptive Regularization with Soft Q-filtering While prior work [16, 17] use hand-tuned sched- ules for A(T), we propose a new adaptive scheme that removes the need for tuning. This is done by comparing the performance of the current policy n ROT and the pretrained policy xt BC batch of data sampled from the replay buffer for online rollouts Dz. More precisely, given a behavior policy n BC (s), the current policy 7 ROT
(s), the Q-function Q(s, a) and the replay buffer De, we set as:
X(T ROT = E(s,y~Dy [ (s)),
(4)
The strength of the BC regularization hence depends on the performance of the current policy with respect to the behavior policy. This filtering strategy is inspired by Nair et al. [24]. where instead of binary hard assignment we use a soft continuous weight. Experimental comparisons with hand-tuned decay strategies are presented in Section 4.4.
Considerations for image-based observations Since we are interested in using ROT with high- dimensional visual observations, additional machinery is required to ensure compatibility. Following prior work in image-based RI and imitation [8, 11], we perform data augmentations on visual observations and then feed it into CNN encoder. Similar to Cohen et al. [11], we use target

---

encoder with Polyak averaging to obtain representations for OT reward computation. This is necessary to reduce the -ont-ottion caused by learning the encoder alongside the ROT imitation process. Further implementation details and the training procedure can be found in Appendix C.
4 Experiments
Our experiments are designed to answer the following questions: (a) How efficient is ROT for imitation learning? (b) How does ROT perform on
tasks? (c) How important is the choice
of IRL method ROT? (d) Does soft Q-filtering improve imitation? (c) How does ROT compare to standard reward-based RL? Additional results and analysis have been provided in Appendix H.
Simulated tasks We experiment with 10 tasks from the DeepMind Control suite [18, 25). 3 tasks from the OpenAI Robotics suite [26]. and 7 tasks from the Meta-world suite [27]. For DeepMind Control tasks, we train expert policies using DrQ-v2 [8] and collect 10 demonstrations for each task using this policy. For OpenAI Robotics tasks, train a state-based DrQ-v2 with hindsight experience replay [28] and collect 50 demonstrations for each task. For Meta-world tasks, we single hard-coded expert demonstration from their open-source implementation [27]. Full environment details can be found in Appendix D and details about the variations in demonstrations and initialization conditions can be found in Appendix E.
Robot tasks Our real world setup for each of the 14 manipulation tasks can be seen in Fig. 4. We use Ufactory xArm 7 robot with xArm Gripper the robot platform for our real world experiments. However, our method is agnostic to the specific robot hardware. The observations are RGB images from a fixed camera. In this setup, we only use a single expert demonstration collected by human operator with a joystick and limit the online training to a fixed period of I hour. Descriptions of each task and the evaluation procedure is in Appendix F.
Primary baselines We compare ROT with baselines against several prominent imitation learning methods. While full description of our baselines are in Appendix G, brief description of the two strongest ones are as follows:
1. Adversarial IRL (DAC): Discriminator Actor Critic [7] is a state-of-the-arr adversarial imitation learning method [6, 29, 7] DAC outperforms prior work such GAIL [6] and AIRL [30), and thus it serves as our primary adversarial imitation baseline.
Trajectory-matching IRL (OT): Sinkhorn Imitation Learning [12, 13] a state-of-the-an trajectory-matchie imitation learning method [31] that approximates OT matching through the Sinkhorn Knopp algorithm [32, 33J. Since ROT is derived from similar OT-based foundations, we use SIL as our primary state-matching imitation baseline.
4.1 How efficient is ROT for imitation learning?
Performance of ROT for image- based imitation depicted on select environments in Fig. 3. On all but one task, ROT trains significantly faster than prior work. To reach 90% of expert performance, ROT on average 8.7x faster on Deep Control tasks, 2.1 x faster on OpenAl Robotics tasks, and 8.9x faster on Meta-world tasks. We also find that the improvements of ROT are most apparent on the harder tasks, which are in rightmost column of Fig. 3. Appendix H.1 shows results on all 20 simulated tasks, along with experiments that exhibit similar improvements in state-based settings.
4.2 How does ROT perform on real-world tasks?
We devise set of 14 manipulation tasks on our xArm robot to compare the performance of ROT with BC and our strongest baseline RDAC, an adversarial IRL method that combines DAC [7] with our pretraining and regularization scheme. The BC policy is trained using supervised learning on a single expert demonstration collected by a human operator. ROT and RDAC finetune the pretrained BC policy through hour of online training, which amounts to ~ 6k environment steps. Since there is just one demonstration, our tasks are designed to have random initializations but fixed goals. Note that single demonstration only demonstrates solving the tasks from one initial condition. Evaluation results across 20 different initial conditions can be seen in Fig. 4. We observe that ROT
5

---

dmc chwp_ehhhehhhhheee
dmc hopper hop
dmc walker
300
spóode, rewerdd
200
epsode
ep3ode
400
200
200
0.0
0.5
1.0
1.5
2,0
0.0
3.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Treimhe
le5
Tramme fetch_push
ls5
frame
leb
fetch fetch_reach prarchhchhhh
fetch_pick aad_anndddpp pplace
1.0
success rate
0.8
success rale
0.8
surcess rale
A Mman
0.6
0.5
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
fraime
frame
frame
metaworid wetpddddwwaaerrreee close
metaworid hammer
metaworid door _oor_open
1.0
1.0
0.8
Success rate
0.8
success rate
MM
success
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.8
0.5
0.8
1.0
frame
les
frame
1e6
frame
1e6
Expert
BC
OT
DAC
ROT (Ours)
Figure Pixel-based continuous control learning on selected environments. Shaded region represents +1 standard deviation across 5 seeds. We notice that ROT is significantly more sample efficient compared to prior work.
doer
Hong hangor
Eraving board
Flnning mug
Hanging tose beg
a
Stacking cups
Pressing swith
Peg in box (Cagg)
Peg in box (Med)
Peg hox (Tland)
Opening a box
Peuring almonds
1.0
0.8
Successs rafe
0.4
0.2
Close
Hang
Frase
Reach
Hang
Hang
Tumn
Stuck
Press
Peg (Easy)
Peg (Med)
Peg (Hard)
Open Box
Pour
Hanger
Mug
Bag
Knob
Cups
BC
RDAC
ROT
(Ours)
Figure 4: (Top) ROT evaluated on a set of 14 robotic manipulation tasks. (Bottom) Success rates for each task is computed by running 20 trajectories from varying initial conditions the robot.

---

dmc_ walker_ run
fetch_pice Bai___pccc_aa__aa ann__
metaworld hammer
1000
sploote rewerrd
600
rato
0.0
0.6
0.6
0.6
400
0.6
02
0.2
0.0
0.0
0.0
0.5
1.0 frame
1.5
2.0
0.00
0.25
0.50
0.75 frame
1.00
1.25
1.50
0.0
0.2
0.4
0.8
0.8
1.0
1w5
Trame
1ue
Expert
BC
DAC
OT
DrQ-v2 (RL)
RDAC
ROT (Ours)
Figure 5: Ablation analysis on the choice base IRL method. We find that although adversarial methods benefit from regularized BC, the gains seen smaller compared to ROT. Here, also that ROT can outperform plain RL that requires explicit task-rewards.
has an average success rate of 90.15 1. over 20 evaluation trajectories across all tasks as compared to 36.1% 1 for BC and 14.6% for RDAC. The poor performance of BC can be attributed distributional mismatch due to accumulation of error in online rollouts and different initial conditions. The poor performance of RDAC can be attributed slow learning during the initial phase of training. More detailed evaluations of RDAC on simulated environments is present in Sec. 4.3.
4.3
How important is the choice of IRL method in ROT?
In ROT, we build on OT-based IRL instead of adversarial IRL. This is because adversarial IRL methods require iterative reward learning, which produces a highly non-stationary reward function for policy optimization. In Fig. 5, we compare ROT with adversarial IRL methods that use our pretraining and adaptive BC regularization technique (RDAC). We find that our soft Q-filtering method does improve prior state-of-the adversarial IRI. (RDAC vs. DAC in Fig. 5). However, our OT-baaed approach (ROT) is more stable and on average leads to more efficient learning.
4.4 Does soft Q-filtering improve imitation?
dmc hoophopperrrccm hop
metaworld hammer
To understand the importance of soft Q-fittring, we compare ROT against two variants of our proposed regular- ization scheme: (a) A tuned fixed BC regularization weight (ignoring A(=)
Eploado reward
300
1.0
0.8
200
100
0.4
0.2
in Eq. 3); (b) A carefully designed linear-decay schedule for A(T), where it varies from 1.0 to 0.0 in the first 20k
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.8
0.8
frame
frame
teB
Finetune with fixed weight Finetune with fixed schedule
environment steps [16]. As demon- strated in Fig. 6 (and Appendix H.2).
ROT (Ours)
ROT is on par and in some cases ex- ceeds the efficiency of a hand-tuned decay schedule, while not having to hand-tune its regularization weights. We hypothesize this improvement is primarily due to the better stability of adaptive weighing as seen in the significantly smaller standard deviation on the Meta-world tasks.
Figure 6: Effect of various BC regularization schemes com-
pared with our adaptive soft-Q filtering regularization.
4.5 How does ROT compare to standard reward-based RL?
We compare the performance of ROT against DrQ-v2 [8], a state-of-the-an algorithm for image-based RL. As opposed to the reward- free setting ROT operates in, DrQ-v2 has access to environments rewards. The results in Fig. show that ROT handily outperforms DrQ-v2. This clearly demonstrates the usefulness of imitation learning in domains where expert demonstrations are available over reward-based RL. We also compare against demo-assisted variant of DrQ-v2 agent using the same pretraining and regularization scheme as ROT (refer to Appendix H.3). Interestingly, we find that our soft Q-filterigg based regularization can accelerate learning of RL with task rewards, which can be seen in the high performance of the demo-assisted variant of DrQ-v2.
7

---

5 Related Work
Imitation Learning (IL) IL [34] refers to the setting where agents learn either from an expert policy or from demonstrations derived from an expert policy without access to environment rewards. IL can be broadly categorized into Behavior Cloning (BC) [1, 21] and Inverse Reinforcement Learning (IRL) [4. 22]. BC solely learns from offline demonstrations but suffers on outtof-distrisutions samples [5] whereas IRI. focuses learning a robust reward function through online interactions but suffers from sample inefficiency [7]. Deep IRL. methods can be further divided into two categories: (1) adversarial learning [35] based methods, and (2) Gate-matching [36, 37] based methods. GAIL [6] is an adversarial learning based formulation inspired by maximum entropy IRL [38] and GANs [35]. There has been a significant body of work built up on GAIL proposing alternative losses [30, 39, 29], and enhancing its sample efficiency by porting it to an off-policy setting [7]. There have also been visual extensions of these adversarial learning approaches [40, 41, 42]. However, although adversarial methods produce competent policies, they are inefficient due to the non -ta-stationarity associated with iterative reward inference. To alleviate the non non-stationary reward problem with adversarial IRL frameworks, a new line of OT-based state-ma matteeintchingg approaches have recently been proposed [12, 13, 11].
Optimal Transport (OT) OT [36, 37] is a tool for comparing probability measures while including the geometry of the space. In the context of IL, OT computes an alignment between a set of agent and expert observations using distance metrics such Sinkhorn [33]. Gromov- Wassersttnn [43], GDTW [44], CO-OT [45] and Soft- [46]. For many of these distance measures, there is an associated IL algorithm, with SIL [12] using Sinkhorn, PWIL [13] using greedy Wasserstein, GDTW-IL. [44] using GDTW, and GWIL [47] using Gromov-Wassersterr Wasserstein. Recent work from Cohen al. [11] demonstrates that the Sinkhorn distance [12] produces the most efficient learning among the discussed metrics. They further show that SII is compatible with high-dimensional visual observations and encoded representations. Inspired by this, ROT adopts the Sinkhorn metric for its OT reward computation, and improves upon SIL through adaptive behavior regularization.
Behavior Regularized Control Behavior regularization is a widely used technique in offline RL [48] where explicit constraints are added to the policy improvement update to avoid bootstrapping on out-of-distribution actions [49, 50, 51, 52, 53, 54J. an online setting with access to environment rewards, prior work [16. 10] has shown that behavior regularization can be used to boost sample efficiency by finetuning pretrained policy via online interactions. For instance, Jena et al. [17] demonstrates the effectiveness of behavior regularization to enhance sample efficiency in the context of adversarial IL. ROT builds upon this idea by extending to visual observations, OT-based IL, and adaptive regularization, which leads to improved performance (see Appendix H.4). We also note that the idea of using adaptive regularization has been previously explored in RL [24]. However, ROT uses soft, continuous adaptive scheme, which initial experiments provided significantly faster learning compared to hard assignments.
6 Conclusion and Limitations
In this work, we have proposed new imitation learning algorithm, ROT, that demonstrates improved performance compared to prior state-of-the-art work on a variety of simulated and robotic domains. However, we recognize few limitations in this work: (a) Since our OT-based approach aligns agents with demonstrations without task-specific rewards, it relies on the demonstrator being an 'expert'. Extending ROT to suboptimal, noisy and multimodal demonstrations would be an exciting problem to tackle. (b) Performing BC pretraining and BC-based regularization requires access to expert actions, which may not be present in some real-worl, scenarios particularly when learning from humans. Recent work on using inverse models to infer actions given observational data could alleviate this challenge [55]. (c) On robotic tasks such as Peg box (hard) and Pressing a switch from Fig. 4, we find that ROT's performance drops substantially compared to other tasks. This might be due to the lack of visual features corresponding to the task success. For example, in the 'Peg' task, it visually difficult to discriminate if the peg is in the box or behind the box. Similarly for the 'Switch' task, it is difficult to discern if the button was pressed or not. This limitation can be addressed by integrating more sensory modalities such as additional cameras, and tactile sensors in the observation space.
8

---

Acknowledgments
We thank Ben Evans, Anthony Chen, Ulyana Piterbarg and Abitha Thankaraj for valuable feedback and discussions. This work was supported by grants from Honda, Amazon, and ONR awards N00014-21-1-2404 and N00014-21-1-27558.
References
[1] D. Pomerleau. An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems, 1, 1998.
[2] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20, 2020.
[3] P. N. Kolm and G. Ritter. Modern perspectives reinforcement learning in finance. Modern Perspectives on Reinforcement Learning in Finance (September 6, 2019). The Journal of Machine Learning in Finance, 1(1,, 2020.
[4] A. Y. Ng, Russell, al. Algorithms for inverse reinforcement learning. In Ieml, volume page 2, 2000.
[5] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings of the fourteenth international conference on artifi- cial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[6] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[7] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson, Discriminator-an act.atoooorttoottiiiiiiio. Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint 2018.
[8] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved laga-nue reinforcement learning. arXiv preprint arXiv:2107.00645 2021.
[9] A. Nair, A. Gupta, M. Dalal, and Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 20KIv:2000.09359) 2020.
[10] Uchendu, Xiao, Y. Lu, Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, et al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02337,, 2022.
[11] S. Cohen, B. Amos, M. P. Deisenroth, M. Henaff, E. Vinitsky, and D. Yarats. Imitation learning from pixel observations for continuous control, 2022. URL /ttper///ppnreviee net/ forum?Pdd-JIbXXHHH..C....
[12] G. Papagiannis and Y. Li. Imitation learning with sinkhorn distances. arXiv preprint arXiv:2008.0916 2020.
[13] R. Dadashi, L. Hussenot, M. Geist, and O. Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04678, 2020.
[14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1000000077333 2017.
[15] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.
[16] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. I Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1oXi:1700.110000 2017. 2017.

---

[17] R. Jena, C. Liu, and K. Sycara. Augmenting gail with be for sample efficient imitation learning.
arXiv preprint (xXiv:2001.07990 2020.
[18] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1801.00800000900 2018.
[19] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, Janng, and W. Zaremba. Openai gym. arXiv preprint arXiv: 160:::666.000 01540, 2016.
[20] T. Yu, D. Quillen, Z. He. R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100. PMLR, 2020.
[21] F. Torabi, G. Warnell, and Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv: 13566, 2019.
[22] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-fire international conference on Machine learning, page 1, 2004.
[23] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Frez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: motp:/099/991111 2015.
[24] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018
[25] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
[26] M. Plappert, M. Andrychowicz, Ray, B. McGrew, B. Baker, Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: I 1802.09464, 2018
[27] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019. URL https: / /ttp///axvvv////////////////.av/
10897.
[28] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[29] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807.007706666 2018.
[30] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprini arXiv: 1248, 2017.
[31] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pages 1259-1277, PMLR, 2020.
[32] R. Sinkhorn and Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-348, 1967.
[33] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[34] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[35] I. Goodfellow, J. Pouget-Abaale Abadie, M. Mirza, B. Xu, D. Warde-Farles S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
[36] C. Villani. Optimal transport: old and new, volume 338. Springer, 2009.
10

---

[37] G. Peyréé M. Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-60) 2019.
[38] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning, In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[39] H. Xiao, M. Herman, J. Wagner, S. Ziesche, J. Etesami, and T. H. Linh. Wasserstein adversarial imitation learning. arXiv preprint /rXiv: 13, 2019.
[40] E. Cetin and O. Celiktutan. Domain-robust visual imitation learning with mutual information constraints. arXiv preprint arXiv:2203.0505:: 2021.
[41] S. Toyer, R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. Advances in Neural Information Processing Systems, 33:18284-18299 2020.
[42] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. Advances Neural Information Processing Systems, 34, 2021.
[43] G. Peyré, M. Cuturi, and J. Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning, pages 2664-2672. PMLR, 2016.
[44] S. Cohen, G. Luise, A. Terenin, B. Amos, and M. Deisenroth. Aligning time series on incom- parable spaces. In International Conference on Artificial Intelligence and Statistics, pages 1036-1044. PMLR, 2021.
[45] I. Redko, T. Vayer, R. Flamary, and N. Courty. Co-optimal transport. arXiv preprint arXiv:2002.03731, 2020.
[46] M. Cuturi and M. Blondel. Soft-dtw: a differentiable loss function for time-series. In Interna- tional conference on machine learning, pages 894-903 PMLR, 2017.
[47] A. Fickinger, S. Cohen, S. Russell, and B. Amos. Cross-domain imitation learning via optimal transport. arXiv preprint wxXiv:21100036844 2021.
[48] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiw:2005.01643, 2020.
[49] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[50] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911 1. 1361, 2019.
[51] A. Ajay. A. Kumar, P. Agrawal, S. Levine, and O. Nachum. {OPAL]: Offline primitive discovery for accelerating offline reinforcement learning. International Conference on Learning Representations, 2021. URL https://
[52] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off- q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
[53] N. Y. Siegel. J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
[54] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[55] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7875-7871. IEEE, 2020.
[56] R. Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[57] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018
11

---

[58] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare. A. Graves, M. Riedmiller, A. K. Fidjeland, Ostrovski, al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-533 -333, 2015.
[59] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. arXiv preprint arXi::2012.0997,, 2020.
[60] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, Abbeel, and L. Pinto. Visual imitation made easy. arXiv preprint arXiv:2008.04899 2020.
[61] A. Knight. The sinkhorn- algorithm: convergence and applications. SIAM Journal on Matrix Analysis and Applications,300 30(1):261-275, 2008.
12


Reviews:
[{"review_id": "fpeNW2Bt7FG", "reviewer": "Reviewer_Ldfy", "summary": "The paper proposes a method for robustifying a policy obtained via behavior cloning by collecting additional experiences in the environment and minimizing an optimal-transport-based (OT-based) trajectory matching reward between expert demonstrations and trajectories generated by the behavior policy (Algorithm 1 in Appendix C). Additionally, the paper proposes a particular adaptive scheme (Eq. (4)) for tuning the weight between the objectives for behavior cloning and for maximizing the OT-based returrn. Experiments on vision-based control tasks from DeepMind Control suite, OpenAI Robotics suite, and Meta-World suite, as well as real-world experriments on an xArm robot arm demonstrate that the proposed method ROT compares favorably to Discriminator Actor Critic (DAC) [7], even when DAC is augmented with the proposed prertreaining and regularization scheme, and furthermore ROT outperforms a state of the art OT-based Sinkhorn Imitation Learning (SIL) algorithm [12].", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper provides a strong empirical evaluation and convincing results for the proposed ROT method\n- The presentation is clear, the questions addressed by the experiments are explicit and easy to follow\n- The method combines several ideas from prior works (as well explained in Sec. 5) but is sufficiently novel and may become a new baseline for BC+finetuning algorithms, plus further extensions are possible\n\nWeaknesses\n- A few important details about the algorithm can only be found in the appendix (e.g., what networks exactly are trained, feature preprocessing for OT, freezing of the feature processor, etc.). However this is OK due to the space limitations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel imitation learning algorithm that adaptively combines offline behavior cloning with online trajectory-matching based rewards. The method is thoroughly evaluated and shown to outperform the baselines, including in experiments on real hardware. There are no concerns with regards to quality, significance, clarity, etc.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "None", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "WriKZPNi7dcT", "reviewer": "Reviewer_xvuE", "summary": "Summary: This paper proposes Regularized Optimal Transport, a new imitation learning algorithm that builds on top of optimal transport-based trajectory matching. The algorithm can learn from a few expert demonstrations and an online reinforcement learning phase. The algorithm is benchmarked in simulation and on real robots and achieved a high success rate.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. This is a great paper with solid work put in. It proposes an effective imitation learning algorithm that has strong performance. The proposed method works well on both simulated benchmarks and on real robots. On simulated benchmarks, it is several times faster in sample efficiency. And on real robotics tasks, the proposed method can achieve 90% success rate with 1 demonstration and 1 hour of online learning.\n2. The authors provide a detailed analysis of the design choices and contribution of each component, providing insight of why soft Q-filtering and OT-based rewards are needed to stabilize the training.\n3. Despite not in the main text, Figure 6 is very helpful to understand the intuition of the proposed method and why it is promising. \n\n### Weaknesses\n\n1. The authors showed that the algorithm is somewhat robust to the out-of-distribution initial state. I am wondering how robust the proposed method is to other types of out-of-distribution states, such as adding a task-irrelevant distractor object in the scene during the second stage of training.\n2. What action space does the real robot experiment use and does the proposed method work with other types of action spaces, e.g. impedance control, for contact-rich tasks, such as wiping a board?\n3. What is the extent of initial condition deviation the proposed method can handle? When the deviation is too big, would there be a scheme where not initializing the policy with $\\pi_{BC}$ be more beneficial?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given the strong results and the thoroughness of the experiments, answers to scientific questions proposed in the experiment section, and the clarity in writing, I recommend a strong acceptance for this paper. I believe this is a great paper to have for the community.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No major issues but hope to get clarification of the questions mentioned in the review.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "KeqWyCtbzg6", "reviewer": "Reviewer_qDWt", "summary": "This paper attempts to accelerate the imitation with a few demonstrations. It adaptively combines offline behavior cloning and online trajectory-matching based rewards. Specifically, the paper contributes a method of adaptive regularization with soft Q-filtering, which regularizes the IRL policy to stay close to the BC pretrained policy. The paper shows in various tasks that the proposed method can accelerate the imitation while at the same time reach high success rate. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tThe paper is well motivated in such that it proposes improving the efficiency of imitation learning both by decreasing the training time and cost of demonstrations. \n\n•\tThe experimental evaluation is extensive. The authors experimented on 20 simulation tasks and 14 real robot tasks and provides convincing results that the proposed method generally outperforms baselines in these tasks. \n\n•\tThe figures in the paper are well designed and composed. The ablation experiments are detailed and clear.\n\nWeaknesses:\n\n•\tAs the paper admits, on tasks such as Peg in box (hard) and Pressing a switch, ROT’s performance drops substantially. Also in some other tasks, the performance is not stable.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes to accelerate the imitation with a few demonstrations. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are extensive and convincing.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1.\tAs mentioned in “Weaknesses”, the paper mentions that this performance drop might be due to the lack of visual features, but for fetch_push, fetch_reach, fetch_pick_and_place, the performance is also unstable and worse than other baselines. Since tasks like reach and push do not need much visual features, what could be reason for that?\n\n2.\tAs mentioned in the paper, the expert policy is trained using DrQ-v2 for DeepMind Control tasks and OpenAI Robotics tasks. While in section 4.5, the proposed ROT is compared and outperforms DrQ-v2. Could you clarify what’s the difference between the expert policy DrQ-v2 and the one is compared?\n\n3.\tIn figure4, the metaworld_hammer plot is missing the line for expert.\n"}}, {"review_id": "ADJyBmXWA09", "reviewer": "Reviewer_fHhv", "summary": "This paper addresses jointly the issues of (1) sample inefficiency of inverse reinforcement learning (IRL) and (2) brittleness of behavior cloning (BC). To do so, this paper introduces a new imitation learning algorithm, \"Regularized Optimal Transport\" (ROT). The particular novelty of ROT as I understand it is to initialize a policy via BC, then refine that policy using IRL, with rewards computed using optimal transport (OT). A key insight seems to be to take care during the fine tuning step to (a) ensure that the fine tuning step doesn't take the refined policy \"too far\" away from the BC policy and (b) to control the degree of regularization with a simple heuristic based on the relative performance of the BC policy compared to the fine-tuned policy at a given iteration. The approach is shown to give good performance across a variety of tasks both in benchmark simulations and on a real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- I found the paper to be very clear and well-written. I am not an expert in IRL, but despite that I had no trouble following the technical approach. I believe this paper would be very accessible to a reader.\n\n- The overall idea is simple and intuitive; or at least, from the presentation it seems to follow quite naturally.\n\n- The approach seems technically sound.\n\n- The benchmark simulation + real robot experiments were well-presented and thorough overall, but see Weaknesses for a minor comment on this.\n\n- The approach was demonstrated on a real robot.\n\n# Weaknesses\n\n- The present experimental evaluation compares the soft Q-filtering approach to hand-tuned regularization. However, given that [24] uses essentially the same idea, but with a hard assignment, it seems like the relevant experimental comparison would be to the hard assignment form of the regularization in eq (4) of this paper. It is mentioned briefly that on some experiments the soft assignment scheme improved speed of policy learning, though no such demonstration is presented (unless I missed something).\n\n- The novelty with respect to prior work might be clarified a bit. In particular, as I understand it, the novelty is in both the use of the optimal transport reward function and the soft Q-filter regularization. In the context of prior work using similar regularization approaches (i.e. the hand-tuned approaches or hard assignment) then, I think this method differs in both the formulation of the reward *and* in the regularization. Is that correct? I wasn't 100% certain from the paper. I'll try to provide a little more detail in Issues.\n\n## Update after rebuttal\n\nAll of my comments have been addressed and I have upgraded my rating from \"weak accept\" to \"strong accept.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I felt this was a very well-written and executed paper. The technical ideas and approach were clear and well-motivated. The main two technical contributions seem to be in the use of optimal transport-based rewards and the \"soft Q-filter\" regularization when fine-tuning models trained using behavior cloning. The experimental evaluation is thorough and convincing. My main suggestions to improve the work are minor: i.e. to add a comparison to the hard-assignment regularization approach (for completeness), and to perhaps clarify the contribution with respect to existing work.\n\n## Update after rebuttal\n\nI have upgraded my recommendation to \"strong accept\" as my comments have been addressed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Recapping from \"Weaknesses\" above, and giving a little more detail:\n\n- The present experimental evaluation compares the soft Q-filtering approach to hand-tuned regularization. However, given that [24] uses essentially the same idea, but with a hard assignment, it seems like the relevant experimental comparison would be to the hard assignment form of the regularization in eq (4) of this paper. It is mentioned briefly that on some experiments the soft assignment scheme improved speed of policy learning, though no such demonstration is presented (unless I missed something).\n\nAn interesting addition to this paper would be an explicit comparison with the hard assignment approach. Given that (as I understand it) this paper is proposing both OT based rewards and an additional soft regularization, I'm not sure this is strictly \"necessary\" but such a comparison would be interesting and perhaps support the claimed improvements in speed versus the hard assignment approach.\n\n- The novelty with respect to prior work might be clarified a bit. In particular, as I understand it, the novelty is in both the use of the optimal transport reward function and the soft Q-filter regularization. In the context of prior work using similar regularization approaches (i.e. the hand-tuned approaches or hard assignment) then, I think this method differs in both the formulation of the reward *and* in the regularization. Is that correct? I wasn't 100% certain from the paper. I'll try to provide a little more detail in Issues.\n\nHere I can try to be more precise. In Section 3.2, the OT rewards are mentioned, but they don't appear in the actual optimization objective for the policy in eq (3). As I understand it, the Q function is building up an approximation of the value function with rewards computed via optimal transport. Is that correct? If so, it may be worth mentioning this under eq (3). If not, this should definitely be clarified."}}]

Author rebuttal:

Decision: Accept (oral)