Dual RL: New Methods for
Reinforcement and Imitation
Learning
Amy Zhang
Assistant Professor, UT Austin

A Refresher on Reinforcement Learning

State-ac;on/ State visita;on distribu;on:Regularized op;miza;on objec;ve:

max 𝔼" ! ($,&) 𝑟 𝑠, 𝑎!

𝑑 ! : offline data visitation− 𝛼𝐷( (𝑑 ! 𝑠, 𝑎 || 𝑑 ) (𝑠, 𝑎))

A Refresher on Reinforcement Learning

Reinforcement Learning as a Linear Program
We can rewrite the problem as a convex op;miza;on problem (CoP):
Primal-Q: max 𝔼& ),* 𝑟 𝑠, 𝑎$,&'(− 𝐷+ (𝑑 𝑠, 𝑎 ||𝑑 , (𝑠, 𝑎))s.t. 𝑑 𝑠, 𝑎 = 1 − 𝛾 𝑑( 𝑠 𝜋 𝑎 𝑠 + 𝛾 ∑)-,*- 𝑑 𝑠 - , 𝑎- 𝑝 𝑠 𝑠 - , 𝑎- 𝜋(𝑎|𝑠)Constrains the visita-on distribu-on d to be valid
Called Bellman Flow constraint

Manne, 1960

Lagrangian Dual without Constraints• Deﬁne operator
𝑇 $! 𝑄 𝑠. , 𝑎. = 𝑟 𝑠. , 𝑎. + 𝔼)"#$ ∼0,*"#$ ∼$! [𝛾𝑄(𝑠.12 , 𝑎.12 )]Dual-Q: max min 1 − 𝛾 𝔼&% ) ,$(*|)) 𝑄 𝑠, 𝑎$3

where 𝑓 ∗ is the convex conjugate of 𝑓+ 𝛼𝔼),*∼&& [𝑓 ∗ ([𝑇 $ 𝑄 𝑠, 𝑎 − 𝑄 𝑠, 𝑎 ]/𝛼)]Dual-Q is overconstrained
Primal-V : max 𝔼& ),* 𝑟 𝑠, 𝑎&'(− 𝛼𝐷+ (𝑑 𝑠, 𝑎 ||𝑑 , (𝑠, 𝑎))s.t. ∑*∈𝒜 𝑑(𝑠, 𝑎) = 1 − 𝛾 𝑑( 𝑠 + 𝛾 ∑)-,*- 𝑑 𝑠 - , 𝑎- 𝑝 𝑠 𝑠 - , 𝑎-

• Deﬁne operator:
𝑇𝑉 𝑠. , 𝑎. = 𝑟 𝑠. , 𝑎. + 𝔼)"#$ ∼0()" ,*" ) [𝑉(𝑠.12 )]dual-V: min 1 − 𝛾 𝔼&% ) 𝑉(𝑠) + 𝛼𝔼),*∼&& [𝑓 ∗ ([𝑇𝑉 𝑠, 𝑎 − 𝑉 𝑠 ]/𝛼)]:Single-player non-adversarial optimization

Gives rise to RElaxed Coverage for Off-policy Imitation Learning(ReCOIL): Imitation from arbitrary experience●Consider the f-divergence between the mixture distribu;ons:𝐷( (𝛽𝑑(𝑠, 𝑎) + 1 − 𝛽 𝑑 ' 𝑠, 𝑎 || 𝛽𝑑 % 𝑠, 𝑎 + 1 − 𝛽 𝑑 ' 𝑠, 𝑎 )
'𝑑"#$

●

●%,'𝑑"#$Is a valid imita;on learning objec;ve: shares the same global minima as tradi;onalobjec;ve (𝑑 = 𝑑 ! )" ' #,%
Avoids es;ma;ng (which is ill-deﬁned in state-ac;on space with zero expert" (#,%)

support.

𝑑 % : expert data visita6on, 𝑑 ' subop6mal data visita6on

7ReCOIL: Imitation from arbitrary experience●Primal-V with the mixture distribu;ons:
Primal-V:%,''max −𝐷( (𝑑"#$𝑠, 𝑎 ||𝑑"#$(𝑠, 𝑎)))*+s.t. ∑,∈𝒜 𝑑(𝑠, 𝑎) = 1 − 𝛾 𝑑+ 𝑠 + 𝛾 ∑/0,,0 𝑑 𝑠 0 , 𝑎 0 𝑝 𝑠 𝑠 0 , 𝑎 0●Dual for Primal-V with mixture distribu;ons:
ReCOIL-V
min 𝛽 1 − 𝛾 𝔼)! / 𝑉(𝑠) + 𝛼𝔼/,,∼) %,' 𝑓 ∗ 𝑇𝑉 𝑠, 𝑎 − 𝑉(𝑠)1"#$− (1 − 𝛽) 𝔼/,,∼) ' [𝑇𝑉 𝑠, 𝑎 − 𝑉(𝑠)]8What is ReCOIL doing behind the scenes?| Intuition
ReCOIL is just a Bellman-Consistent EBM.

ReCOIL: Imita>on from arbitrary experience
ReCOIL-Q

ReCOIL-V

Key features:✔ Non-adversarial✔ ReCOIL-V is a single player op;miza;on instead of a game.
✔ Does not require learning a discriminator.
✔ Relaxes the coverage assump;on✔ Works for arbitrary f-divergence10Manipula-on

Locomotion

Oﬄine IL experiments: ReCOIL
Methods based on coverage assump6on
fail when coverage is low (few expert
trajectories in dataset),
andin high dimensional tasks where theDiscriminator easily overﬁts.

ReCOIL outperforms baselines by a largemargin!

11

Methods based on coverage assump;on ‘almost’ learn to imitate…but fail to recover from mistakes

Our methodhammerhuman-v0

relocatehuman-v0

RCE

ORIL

SMODICEReCOIL

Methods based on coverage assump;on ‘almost’ learn to imitate…but fail to recover from mistakes

Our method

Environment: Kitchen-partial-v0

RCE

ORIL

SMODICEReCOILDual Formulation for SelfSupervised Pre-training

Key Idea: Learning from Human Videos as a
BIG Oﬄine Goal-Condi0oned RL Problem
Oﬄine Dataset:
Diverse Human Videos•••

Mathematically SoundWhat are human actions?
Can’t be optimized in practice

Human videos are rich sources of goal-directed behavior!

Offline Value Learning on Human Videos
Oﬄine Dataset:
Diverse Human Videos

Dual Problem depends only on
oﬄine data! No dependence onac-ons!
goal frame

ini7al frame

middle frame

VIP: Towards Universal Visual Reward andRepresentation Via Value-Implicit Pre-Training
Self-Supervised
Goal-Condi-oned Value Func-on Training

Diverse Visuomotor Control:Imita)on, Trajectory Op)miza)on, Online RL,Few-Shot Real-World Oﬄine RL

Attract

VIP Embedding
Positive1“Negatives”

AnchorRewardVIP NetworkRepresenta)on

InitialMiddle

Goal

Diverse, In-the-Wild Unlabeled Human Videos

CloseDrawer & PushBottle
VIP-RWR (100%)

VIP-BC (50%)

R3M-RWR (90%)

R3M-BC (10%)

VIP-RWR (90%)

VIP-BC (50%)

R3M-RWR (70%)

R3M-BC (50%)

PickPlaceMelon & FoldTowel
VIP-RWR (100%)

VIP-BC (50%)

R3M-RWR (90%)

R3M-BC (10%)

VIP-RWR (90%)

VIP-BC (50%)

R3M-RWR (70%)

R3M-BC (50%)

Extended to MultimodalSettings

This Work : Representa1ons as Mul1-Modal Value Func1ons
Image 𝑜Language 𝑔“CloseMicrowave”

𝜙(⋅)

𝜓(⋅)

𝜙(𝑜)

𝜓(𝑔)

Shaped Dense Reward:
𝑅 𝑜!"# , 𝑜! ; 𝑔 ≔ 𝑉 ∗ 𝜙 𝑜!"# ; 𝜓 𝑔− 𝑉 ∗ (𝜙 𝑜! ; 𝜓 𝑔 )𝑉 ∗ (⋅; 𝑔)𝑉 ∗ (𝑜; 𝑔)Embedding Distance

0.6

Q: How can we eﬀec;vely learn mul;modal implicit value representa;ons?

Language-Image Value Learning (LIV)•Theory: Combining VIP and CLIP objec;ves amounts to learning a mul;-modal value
representa;on compa;ble with image and language goals
LIV Objec;ve“put down glass”VIPCLIP

Language-Image Value Learning (LIV) Applications•SOTA results on pre-training, fine-tuning, and reward learning for language-conditioned
robotic manipulation
LIV CapabiliGes
Zero-shot Mul)-Modal Reward

Language-Reward MPC

Language-Condi)oned Imita)on Learning
short, atomic tasks

unseen, composite tasks

LIV performs best on all RealRobot tasks

Closing: Unifying exis/ng work with dual RL●Most successful techniques for oﬄine RL:

Policy Constraint

Pessimistic Value function

Implicit Maximiza6on
𝑄 𝑠, 𝑎 ≔ 𝑟 𝑠, 𝑎 + 𝛾max
𝑄(𝑠 " , 𝑎′)(!

Ensure policy during training
stays close to replay buffer policy

Ensure underestimation of Q for
OOD actions in dataset

Ensure that the maximum istaken over in-distribu6on ac6ons

We show all these classes of prior methods come from a uniﬁed dual perspec<ve!

25

Closing: Unifying exis/ng work with dual RL
We show a number of prior methods in IL and RL to be dual RL methods! Some surprising ones are
CQL, Implicit Behavior Cloning, XQL, IQLearn.

To Summarize
●●
●

1.
2.
3.

Dual RL provides a unifying perspective on imitation learning andregularized reinforcement learning
Gives rise to new IL and RL algorithms
Can also be leveraged for pre-training representations and reward
functions in vision and multimodal settings
Dual RL: Unification and New Methods for Reinforcement andImitation Learning. Harshit Sikchi, Qinqing Zheng, AZ, Scott
Niekum. In submission.
VIP: Towards Universal Visual Reward and Representation via
Value-Implicit Pre-Training, Yecheng Jason Ma, Shagun Sodhani,
Dinesh Jayaraman, Osbert Bastani, Vikash Kumar*, AZ*. ICLR 2023.
LIV: Language-Image Representations and Rewards for RoboticControl. Yecheng Jason Ma · Vikash Kumar · AZ· Osbert Bastani ·Dinesh Jayaraman. ICML 2023.