Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
Authors: Jiayu Chen
Bhargav Ganguly
Yang Xu
Yongsheng Mei
Tian Lan
Vaneet Aggarwal
Venue: Accepted by TMLR ()

Abstract: Deep generative models (DGMs) have demonstrated great success across various domains, particularly in generating texts and images using models trained from offline data. Similarly, data-driven decision-making also necessitates learning a generator function from the offline data to serve as the policy. Applying DGMs in offline policy learning exhibits great potential, and numerous studies have explored in this direction. However, this field still lacks a comprehensive review and so developments of different branches are relatively independent. In this paper, we provide the first systematic review on the applications of DGMs for offline policy learning. We cover five mainstream DGMs, including Variational Auto-Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, and their applications in both offline reinforcement learning (offline RL) and imitation learning (IL). Offline RL and IL are two main branches of offline policy learning and are widely-adopted techniques for sequential decision-making. Notably, for each type of DGM-based offline policy learning, we distill its fundamental scheme, categorize related works based on the usage of the DGM, and sort out the development process of algorithms in that field. In addition, we provide in-depth discussions on DGMs and offline policy learning as a summary, based on which we present our perspectives on future research directions. This work offers a hands-on reference for the research progress in DGMs for offline policy learning, and aims to inspire improved DGM-based offline RL or IL algorithms.  For convenience, we maintain a paper list on https://github.com/LucasCJYSDL/DGMs-for-Offline-Policy-Learning.

Published in Transactions on Machine Learning Research (08/2024)
Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
Jiayu Chen
chen366668purdee.cm
Purdue University West Lafayelle, IN 47907
Bhargav Ganguly
Sgunguky@purrdac edu
Purdue University West Lafayette, IN 47907
Yang Xu
mu1780@purdue. edu
Purdue University West Lafagette, IN 47907
Yongsheng Mei
ysmei@gun. edu
The George Woshington University Washinglon, DC 20052
Tian Lan
tlan Reguu. edu
The George Washington University Woshington, DC 20052
Vaneet Aggarwal
v edu
Purdue Universily
West Tafayette, IN 47907
Reviewed on OpenReview: https: openreview. net/ forum? id= Mm2cMD11p
Abstract
Deep generative models (DGMs) have demonstrated great success across various domains, particularly in generating texts and images using models trained from offline data. Similarly, data-driven decision- decision-making also necessitates learning generator function from the offline data to serve the policy. Applying DGMs in offline policy learning exhibits great potential, and numerous studies have explored in this direction. However, this field still lacks compre- hensive review and so developments of different branches are relatively independent. In this paper, we provide the first systematic review on the applications of DGMs for offline policy learning. We cover five mainstream DGMs, including Variational Auto-Encoders, Gener- ative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, and their applications in both offline reinforcement learning (offline RL) and imitation learning (IL). Offline RL and IL are two main branches of offline policy learning and are widely- adopted techniques for sequential decision making. Notably, for each type of DGM-based offline policy learning, we distill its fundamental scheme, categorize related works based on the usage of the DGM, and sort out the development process algorithms in that field. In addition, we provide in-depth discussions on DGMs and offline policy learning as a sumunar, based on which we present our perspectives on future research directions. This work offers hands-on reference for the research progress in DGMs for offline policy learning, and aims to inspire improved DGM-based offline RL or IL algorithms. For convenience, maintain paper list on

---

Published in Transactions on Machine Learning Research (08/2024)
Contents
Introduction
5
Background on Offline Policy Learning
6
2.1
Offline Reinforcement Learning
2.1.1
Dynanic-Progranminn---ae Oflline Reinforcement Learning
2.1.2
Model-based Offline Reinforcement Learning
8
2.1.3
Wnajetory-Oppimizaionn. based Offline Reinforcement Learning
2.2
mitaaiio Learning
9
2.2.1
Behavioral Cloning
2.2,2
Inverse Reinforcement Learning
2.3
Challenges of Offline Policy Learning
11
Variational AutodEnssders in Offline Policy Learning
3.1
Background Variational Auto-Eneoders)
................................
13
3.2
Variational Auto- Encoders in Offline Reinforcement Learning
16
3.2.1
An Overview
16
3,2.2
Addressing the Issue of Out-of-Distribnne Distribution Actions
17
3.2.3
Data Augmentation and Transtormation
18
3.2.4
VAF-basod Multi-task/ Hierarchical Offline RI.
3.3
Variational Auto-Eneoders in Imitation Learning
21
3.3.1
Core Schemes
21
3.3.2
Improving Data Efficiency
23
3.3.3
Managing Multi-Modal Inputs
24
3.3.4
VaE-b Skill Acquisition and Hicrarchical Imitation
................................
25
3.3.5
Tackling Causal Confusion
Generative Adversarial Networks in Offline Policy Learning
28
Background on Generative Adversarial Networks
28
4.2
Generative Adversarial Networks in Imitation Learning
30
4.2.1
Fundamental GAN-hased Imitation Learning Al Algorithms: GAIL. and AIRL.
30
12.2
Extensions of GAIL
32
42233
Extensions of AIRL
32
4.2.4
Integrating VAEs and GANs for Imitation Learning: A Spotlight
34
4.3
Generative Adversarial Networks in Offline Reinforcement Learning
35
4.3.1
Policy Approximation using GANs
35
4.3.2
World Model Approximation using GANs
37
2

---

Published in Transactions on Machine Learning Research (08/2024)
5
Normalizing Flows in Offline Policy Learning
37
5.1
Background on Normalizing Flows
37
5.2
Normalizing Flows in Imitation Learning
39
5.2.1
Exact Density Estimation with Normalizing F Flows
40
5.2.2
Policy Approximation using Normalizing Flows
41
5.3
Normalizing Flows for Reinforcement Learning with Offline Data
42
5.3.1
Adopting Normalizing Flows in Offline Reinforcement Learning
42
5.3.2
Adopting Normalizing Flows in Online Reinforcement Learning with Offline Data
43
6
Transformers in Offline Policy Learning
45
6.1
Background Translormers
45
6.2
Transformers in Offline Reinforcement Learning
47
6.2.1
Representative Works
48
6.2.2
Balancing Model Capacity with Training Data
49
6.2.3
Mitigating Impacts from Environmental Stochasticity
50
6.2.4
Transformers in Extended Offline Reinforcement Learning Setups
50
6.2.5
Rellections and Future Works on Transformer-based Oflline Reinlorcement Learning
6.3
Transtormers in Imitation Learning
54
6.3.1
A Paradign of Transformer-base Imitation Learning
54
6.3.2
Policy Approximation using Transformers
54
6.3.3
Transformer based Generalist Imitation Learning Agents
55
Diffusion Models in Offline Policy Learning
57
Background on Diffusion Models
57
7.2
Dillusion Models in Imitalion Learning
7.2.1
Policy Approximation using Diffusion Models
59
7.2.2
Addressing Common Issues of Imitation Learning with Diffusion Models
60
7.3
Diffusion Models in Offline Reinforcement Learning
62
7.3.1
Diffusion Models Policies
62
3.2
Diffusion Models as Planners
63
7
3.3
Diffusion Models as Data hesizess
65
7.3.4
Diffusion Models in Extended Offline Reinforcement Learning Setups
65
Discussions and Open Problems
67
Discussions on Deep Generative Models and Offline Policy Learning
67
8.2
Perspectives on Future Research Directions
72
8.2.1
Future Works Data-centric Research
72
3

---

Published in Transactions on Machine Learning Research (08/2024)
8.2.2
Future Works on Benchmarking
73
8.2.3
Future Works on Theories
74
8.2.4
Future Works on Algorithm Designs
75
Conclusion
77

---

Published in Transactions on Machine Learning Research (08/2024)
1 Introduction
Offline Policy Learning is machine learning discipline that leverages pre-existing, static datasets learn effective policies for sequential decision-making control. This paper covers its two primary branches: Offiine Reinforcement Learning (Offline RL) and Imitation Learning (IL). Offline RL utilizes a pre-compiled batch of experience data collected from other policies or human operators, typically comprising a series of aatt--atioo-rrovadd-nets state tuples. The objective of offline RL is to develop a policy that can maximize the expected cumulative rewards, which may necessitate deviating from the behavior palterns observed in the training data. On the other hand, IL trains a policy by mimicking an expert's behaviors. The data used for IL should be demonstrated trajectories from experts. These trajectories, containing experts' responses to various scenarios, usually consist of sequence of state-action pairs (Learning from Demonstrations, LiD) or state-next state pairs (Learning from Observations, L.O)..
Generative models are a class of statistical models that are capable of generating new data instances. They learn the underlying distribution of a dataset and can generate new data points with similar characteristics. generative models (DGMs) are a subset of generative models that leverage the powerful representational capabilities of deep neural networks to model complex distributions and generate high-quality, realistic samples. Notably, the application of DGMs in computer vision (CV) and natural language processing (NLP) has been highly successful. Examples include text-to-image generation using Diffusion Models (Stable Diffusion (Rombach ct al. (2022)), )), text-to-video generation with Diffusion Models and Transformers (Sora [Brooks et al. (2024h)), large language models based on Transformers (ChalGPT (OpenAl (2023)) )), and so on Offine policy learning bears similarities to CV and NLP, as all these domains involve learning from a set of offline data. While CV or NLP models generate images or texts based on given contexts, offline RI,/II. models produce actions or trajectories conditioned on task scenarios. This similarity suggests that applying DGMs to offline policy learning could potentially replicate the success witnessed in CV and NL.P, leveraging their capability to model and generate complex data patterns.
Great advances have been made in both DGMs and offline policy learning, and there are numerous works on applying state-of-the-art DGMs to aid the development of offline policy learning. This paper provides a comprehensive review of DGM-based offline policy learning. Specifically, the main content is categorized by the type of DGMs, and we cover nearly all mainstream DGMs in this paper, including Variational Auto- Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, where Transformers and Diffusion Models are representatives of autoregressive and scoro-based generative models, respectively. For each category, present related works in both offline RL and IL as two subsections. In each subsection (e.g., VAE-based IL in Section 3.3), we abstract the core schemes of applying that DGM in IL or offline RL, categorize related works by how the DGM is utilized, and provide a summary table outlining the representative works with their key noveltics and evaluation benchmarks. It is noteworthy that our paper is more than a survey: (1) For the introduction of each work, we include its key insights and objective design, to help readers get the knowledge without referring to the original papers. (2) For each group of works (c.g., Section 3.2.4| 7.3.2), algorithms are introduced in different levels of details. The most representative ones are highlighted as a tutorial on that specific usage of DGMs for offline policy learning, while subsequent extension works are discussed more briefly to provide a comprehensive overview. (3) We present research works within the same group using unified notations, aiming to clarify the relationships between their objectives and the evolution of algorithm designs.
The main content begins with background on offline policy learning, highlighting existing challenges that can be addressed by DGMs. Notably, different DGMs could adopt distinct base offline RL/IL algorithms. For instance, three main branches of offline RL: jymaamic-prgrammmnn optimization-based offline RL, are introduced and they are mainly used in Section 2.1.1 2.1.2 6.2.1 re- spectively. Then, in the order of Variational Auto-Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, we present the background on each DGM and its application For the background of each DGM, we introduce its mathematical
based, mased- and trajectory-
for offline policy learning, in Section 3 7 basics and specific model variants that are utilized offline policy learning, such that necessary knowledge can be conveniently looked up. Following the main content, in Section g we present in-depth discussions on the main topic of this paper and provide perspectives on future research directions. For discussions (Section
5

---

Published in Transactions on Machine Learning Research (08/2024)
8.1 we analyze both the common and unique usages of different DGMs in offiine policy learning, summa- rize the seminal works in each category and the issues of offline policy learning that have been targeted by DGM-based algorithms. For the future works (Section (.2). we offer our perspectives on potential research directions across four aspects: data, benchmarking, theories, and algorithm designs.
The main contributions of this paper are as follows: (1) This the first review paper on Deep Genera- tive Models for Offline Policy Learning. (2) This paper covers wide array of topics, including five main- stream Deep Generative Models and their applications in both Offine Reinforcement Learning and Imitation Learning. (3) Throughout this paper, we distill key algorithmic schemes/
schemes/paradigna paradigms and selectively highlight
seminal research works, as tutorials on respective topics. (4) Our work showcases the evolution of DGM- based offline policy learning in parallel with the progress of generative models themselves.
(5) The summary provided at the end of this paper offers valuable insights and directions for future developments in this field.
2 Background on Offline Policy Learning
In this section, we introduce two main branches of offline policy learning: offline reinforcement learning (RT.) and imitation learning (IL). The purpose of this part is not to thoroughly review these two areas but provide necessary background for the applications of DGMs in offline policy learning. At the end of this section, we discuss challenges in offline policy learning that can potentially be addressed by DGMs.
All content is grounded in the fundamental framework of Markov Decision Process (Puterman 2014) denoted as M = T,,, Po 7). S is the state space, A is the action space, T T:S. x x AAx
0 is
the transition function, Po S + [o, 1] is the distribution the initial state, r S:S x A R the reward function, and 7 E [0, 1] is the discount factor.
2.1 Offline Reinforcement Learning
Here, we discuss several categorics of offline RL algorithms, including based, and reajector-opplinizzton--eee been broadly combined with VA Normalizing Flows, and Diffusion Models for improvements; Model-based offline RL is mainly used with GANs; Trajectory Optimization can be implemented using Transformers Diffusion Models.
baagrramming-beddd,,.. model-
ones. In particular,
based offline RL has
2.1.1 Dynamic-Programminn based Offline Reinforcement Learning
Dynamic-proge programming- based offline RL is main branch of model-free offline RL as detailed in (Levine et al. (2020)) Given an offline dataset D2 collected by the behavior policy p(als), synamic-progaamming bassdd offline RL usually would adopt constrained policy iteration process as below: (k denotes the index of the learning iteration.)
Q = arg min E (eaand))D [Q(s, - (r 7 ( Q
(1)
Tk+1 = arg max EewD, [Ea (s) 1 as) s.t. E.ED, (D(T(1ss) S
*
Here, the first and second equations are referred to as the policy evaluation and policy improvement steps, respectively. When updating the Q function, (s, r,,, are sampled from Dp but the target action a' is sampled by the being-learned policy Zg. If (a'ls')) differs substantially from u(a'ls'); out-of-distribution (OOD) actions, which have not been explored by ,4 can be sampled. Further, the Q-function trained on Dus i.., Q may erroneously produce over-optimistic values for these OOD actions, leading the policy Th+1 to generate unpredictable OOD behaviors. In online RL, such issues are naturally corrected when the agent interacts with the environment, attempting the actions it (erroncously) believes to be good and observing that in fact they are not. In oflline RL, interactions with the enviromment are not accessible, but, alternatively, the over-optimism can be controlled by limiting the discrepancy between क and μ, as shown in Eq. (1). The discrepancy measure D(-II.) has multiple candidates. For a comprehensive review, please refer to (Levine et al. (2020)). Built upon this basic paradigm (i.c., Eq. (1), , we introduce some practical and representative offline RT algorithms that focus addressing the issue of OOD actions.
6

---

Published in Transactions on Machine Learning Research (08/2024)
Policy constraint methods, such as AWR (Peng et al. (001bb)) and AWAC (Nair et al. (2020)
choose
the KT.-divergence as the discrepancy measure, for which the policy improvement step in Eg. closed-form solution:
has a
Tkt1 arg max D, ( c)
(2)
T (als) = u(aas) exp (s, (z))
In this expression, 1/2(s) serves the normalization factor across all possible action choices at state s; A is the Lagrangian multiplier to convert the original constrained optimization problem to an unconstrained one, typically set as constant rather than being optimized. In practice, such an policy can be acquired through weighted supervised learning from D, Wang et al. (020 ), where exp (QT+1 a)) serves as the weight for (s, a), that is: Tk+1 = arg max R (a,,) [log (als) exp (QR+1(s, (s))))/)))))))))))))) /Z(s)]
(3)
a
Policy penalty methods, such as BRAC Wu et al. 2019a)), utilize approximated KL-divergence (x((((()s)l((((ss)) = E. (a [log (als) - log ([ass),, where á is learned via Behavioral Cloning (intro- ducod in Section 2.2.1) from D₁ to estimate μ and E denotes the estimated expoctation via Monte Carlo sampling, and modify the reward function as F(s, = r(s,a) the deviation from is implemented as a penalty term in the reward function for the policy learning to avoid deviating from μ not just in the current step but also in future steps. This method is usually implemented in its equivalent form (Wu al. (2019a)) as below:
(= (A > 0). In this way,
- a) (r (Barmaulo)) ( (Dx2(((16)))(()))
Qu+ = arg min D(
(4)
Tk+1 = arg max D, (Rawar-10) Q (8, -
One significant disadvantage of this approach that it requires explicit estimation of the behavior policy, i.e. f. and the estimation error could hurt the overall performance of this approach. Instead, recent work (Park lt al. (2024)) proposes that E,-D, [>D ((((s)))l(((ss)) can be replaced by ((,,a)-D, [Alog [[]]s)], which implicitly constraints π to remain close to the behavior policy. They further show through intensive empirical results that this objective design consistently outperforms aforementioned policy constraint methods (e.g..
AWR). Notably, the application of Diffusion Models in offline RL (i.e., Section 7.3.1 predominantly adheres to these policy constraint methods, suggesting potential future improvement.
Support constraint methods, such BCQ (Fujimoto et al. (201 and BEAR (Kunar et al. (0019) propose to confine the support of the learned policy within that of the behavior policy to avoid OOD actions, because constraining the learned policy to remain close in distribution to the behavior policy, in the previous two methods, can negatively impact the policy performance, especially when the behavior policy is substantially suboptimal. As an example, BEAR proposes to replace the discrepancy constraint in Eq. [ with a support constraint: π E { : S x A + [0, 1] T'(als) = whenever μιaas)) < c] 3 With such a support constraint, the target actions a' used in policy ovaluation (i.c., the first equation in Eq. (1). would all satisfy μ(als)) and so are in-distribution actions, that is:
U arg min E ((san))), [Q(s, a) (r + 77B an p(al))z
(5)
Pessimistic value methods regularize Q-function directly to avoid overly optimistic values for OOD ac- tions, as an alternative to imposing (support or distributional) constraints on the policy. As representative, CQL [Kumar et al. (2220) removes the policy constraint in Eq. (I) and modifies the policy evaluation pro- cess as below:
2i+ = arg
min min ((g,,n)) [Q(s, a) - (r + (T (((''''))) a'))]²+
Q
X Egn (11)) Q(s,)) -( D. (is,a) a)]
In practice, they use sampled maximum mean discrepancy (MMD, Gretton 2012 between the being-learned policy and behavior policy implement such a support constraint, of which the effectiveness empirically justified.
7

---

Published in Transactions on Machine Learning Research (08/2024)
Here, the second term (with À > 0) can be viewed regularizer for the policy evaluation process, which minimizes Q-values under the being-learned policy distribution, Le., mT(-(s), and maximizes the Q- values for (s,a) within Dy. Intuitively, this ensures that high Q-values are only assigned to in-distribution actions. Suppose Q lom QK and define Q" as the true Q-function for x, they theoretically prove that Ba-n(-1s)) (s..a I (a--x(1s)) (s), Vs Duy with high probability, when À is large enough. Thus, this algorithm mitigates the overestimation issue with a theoretical guarantee. However, it tends to learn an overly conservative Q function
Notably, DM-basod and NF-based offline RL algorithms usually adopt policy constraint and support con- straint. methods, respectively, as detailed in Section 7.3.1 and 5.3.1 While, VAB-based offline RL (i.e., Section 3.2.2) has applied all the aforementioned methods except for policy constraint ones. The motivation to involve DGMs in these methods is either improving expressiveness of the policy 9, providing better estimations of the policy or support constraints to avoid OOD actions.
2.1.2 Model-based Offline Reinforcement Learning
In model-based offline RL (Kidambi et al. 2020 Yu al. (2020a)) ), parametric dynamic model (s') a) and reward model f(s, a) need to be learned through supervised learning:
max )o [log (sslss, min ((.a)),, [(((s r f
(7)
The approximated MDP M AS, (u: 7), where Po is the empirical distribution of initial states in Das can then be used as proxies of the real enviromment, and (short-horizon) trajectories can be collected by interacting with M using the being-learned policy ma. to form another dataset Dx.
To mitigate the impact from the model bias of M, an estimation of the model uncertainty, i... u(s, a), is utilized penalty term to discourage the agent to visit uncertain regions (where u is high) in the state- a). (x, > 0 is hyperparameter.) This
action space, by setting the reward at a) as f(s, a) ()((sa)-\)pa) conservative way to avoid OOD behaviors resembles the policy penalty method in dynamic-programming based offline RT. The uncertainty measure can be acquired through Bayesian Neural Networks (Chua et al. (2118), or modeled as disagrcement among ostimations from an ensemble of Q-funetions or approximated environment models (Lu et al. (2022b)). Specifically, Dr can be generaled as follows: 80 ~ po(-), dn ~ m-(-ss)), Fo = f(so, an) ( ao), $1 ~ (-|sso, ao), and so on. Finally, offline/off- be applied to train the policy mo on an augmented dataset Daug = AnDP + (1 - AD) Here, AD E [0, 1] typical (off-policy) actor-critic framework for this process as below (Levine) let al. 20200): (Q, is the target Q-function.)
RL algorithms can
is another hyperparameter.
max B Dan ao (11s) (s, min B(s,aF [(Qg(s,a) - (F(s, + (1a)
(8)
As detailed in Section 4.3 GANs have been utilized for model-based oflline RL to approximate the policy or environment models.
In fact, with the approximated model M, planning algorithms can be directly applied to find the optimal action sequence as below:
T
Go. arg max RO.T
r() Xpu(se, ai) s.t. s1+1 -(ss
(9)
The equation above is a planning objective for deterministic environments, which can be efficiently solved with algorithms like Model Predictive Control (Holkar & Waghmare (2010)), ), similarly for stochastic environments.
2 "According to Theorem 3.2 in (Kumar et al. (200 ) the minimum value of À is linked to max.g P,
[=], where [>)())]
denotes the frequency of state Dp- Given that D 's coverage could be limited and eertain states would have low visitation frequencies, this can result. in high value of A. However, as shown Eq. 0 this same value used the Q-update for every state, potentially leading to underestimation for other states Da-
8

---

Published in Transactions on Machine Learning Research (08/2024)
Eq. (8 and 9 represent obtaining the policy through learning and planning, respectively. Notably, there
is third line of research that unifies planning and learning for efficient model-based offline RI, exemplified by MuZero and its variants (Schrittwieser et al. (2020g; Hubert et al. ] Schrittwieser et al. (2021);
Antonogiou et al. (2022)). The applications DGMs have not boen extensively explored in the last two
model-based oflline RI. manners, suggesting promising future research direction.
2.1.3
Offline Reinforcement Learning
Trajectory optimization casts oflline RL. as sequence modelling (Prudencio et al. 2023 )). One approach
in this category is to model the distribution of augmented trajectories (To:r (8o. 0n, Ro. ... ,8, or, Rr)}
induced by the behavior policy μa, where R, =i= denotes the return-to-go (RTG). Each trajectory To:T: instead of being treated as single data point, can be modeled /generated using autoregressive models like Transformers, and the objective may take the form:
max E To Dr
log (T Tou-1), log (TelTooo11) log (8t, Re RolTo:c1) + log n² (allTo:-1;; Re) (10)
The implementation of π² and x² can vary in different algorithms. After training, R can be used as the policy to generale trajectories of a target relurn Ro in auloregressive manner: po(+), ag m- Ra); The rationale behind this is that by learning from a
81 ~ ((-ss,, 0o), ro = r(sp, ao), R1 = Ro distribution of return and trajectory pairs, the RTG-based policy x² expected to induce trajectorics that correspond to the conditioned RTG.
Such sequence modelling objective makes trajectory- segfeettrryoppiiizzatoooo-esssse offline RL less prone selecting OOD actions, because multiple state and action anchors throughout the trajectory prevent the learned policy from deviating too far from the behavior policy μ. As shown above, the training is simply through supervised learning, which gets rid of the difficulty of dynamic programming or policy gradient optimization as in previous two categories of offline RL. Also, trajectooy-opuimizztionn--- sparse reward settings, where temporal-difference methods typically fail, since they rely on dense reward estimates to effectively propagate Q-values over long horizons. Among the five DGMs that we cover in this paper, only Transformers belong autoregressive models. Thus, we introduce representative works of autoregressive trajectory modelling and their extensions in Section 6.2 We also provide detailed discussions the fundamental limitations of these methods in Section 6.2 For instance, to recover optimal policies with these methods, environment dynamics (i.c., T and in MM) should be nearly deterministic and the offline dataset D2 should provide good coverage of all possible return-trajectory pairs. These approaches
offline RL performs well in
assumptions are somewhat strong compared to those needed for Jynamic-programming--oe Brannfonnrrenr et al. )022)).
Alternatively, due to the capability of Diffusion Models to handle utilized to directly model/generate complete trajectories (7e:7 (Sp, un, 88, aT)) as data points. During execution, the target return Ro can serve conditioner to guide the trajectory generation. This approach relies heavily on a deep underslanding of Diffusson Models, and as such, the details are elaborated in the corresponding Section 7.3.2. Unlike the previous two categories of offline RL, trajettrr-optiiizztttn---sss offline RL significantly depends on the capabilities of DGMs, such as Transformers and Diffusion Models, establishing a closer connection with them.
data, they have been
2.2 Imitation Learning
Imitation Learning (IL, Zare et al. (2023) aims at learning a mapping between states and actions, from a set of demonstrations DE for a certain task. DE is usually composod of state-action pairs demonstratod by domain experts. Next, we introduce the two primary categories of IL: Behavioral Cloning and Inverse Reinforcement. Learning.
9

---

Published in Transactions on Machine Learning Research (08/2024)
2.2.1 Behavioral Cloning
Behavioral Cloning (BC, Pomerleau (1991g) casts IL as supervised learning which is well-studied and com- putationally efficient. To be specific, the objective of BC is to train a policy a with which the likelihood of the expert demonstrations can be maximized:
max (s [og (aass]
(11)
Despite its simplicity, BC has several significant drawbacks.
First, the BC agent is trained on states generated by the expert policy but tested on states induced by its own action. As a result, the distribution of states observed during testing can differ from that observed during training, and the agent could encounter unseen states, which is known as the covariate shift problem (Pomerleau| (1988)). BC agents do not have the ability to return to demonstrated regions when it encounters (OOD) states, so deviations from the demonstrated behavior tend to accumulate over the decision horizon, causing compounding errors. Here are some strategies for addressing this issue. Interactive II, such as DAgger (Ross et al. (001])), assumes access to an online expert which can relabel the data collected by the BC agent with correct actions, such that the agent can learn how to recover from mistakes. Alternatively, when the simulator is available, the BC agent can be trained to remain on the support of demonstrations through an additional RL process. For instance, Brantley et al.] (2020) propose training an ensemble of policies, using the variance of their action predictions as a cost. The variance outside of the expert's s support would be higher, because the ensemble policies are more likely to disagree on states not encountered in the demonstrations. An RL algorithm conjunction with supervised learning using BC. The RL process helps guide the agent back to the expert distributions while BC ensures that the agent closely mimics the expert within that distribution.
then be applied to minimize this cost, in
Second, it is difficult for a supervised learning approach to identify the underlying causes of expert actions, leading to problem known as causal misidentification (De Haan et al. (2019)). Consequently, the BC agent may fail to differentiate between nuisance correlates in the states and the actual causes of expert actions, making it challenging for the learned policy to generalize to OOD states. Strategies addressing this issue either incorporate the causal structure of the interaction between the expert and environment into policy learning, or identify and eliminate muisance correlates in observations. We have seen a series of studies that apply VAFs to tackle causal misidentification, as detailed in Section 3.3.5)
Notably, Florence et al. 20021 investigate a fundamental design decision of BC that has largely been over- looked: the form of the policy. Empirically, they show that using explicit continuous ferd-do networks as policies, i.e,, a = (s),,, struggles to model discontinuities, making policies incapable of switching deci- sively between different behaviors. Instead, they propose Implicit BC, where an energy model Eg is utilized to represent the policy mo. The policy definition and training method are as below:
N
}]
a = arg min (s, a), max maxE De.) IO log e Fo(s.n)/ / Fo(s.a) Σ
- Fo Fo(maapne)
(12)
à
N {a, TE are generated negative examples (i.e., non-expert actions) corresponding to (s, a). Such an InfoNCE-style objective (van den Oord et al. (2018) is to approximately maximize the expectation of log (e ---ols.n)/ / e Fo(a))). As a result, the expert action a would correspond to the lowest energy Eg a). During inference, arg mina (s a) can be acquired through techniques like stochastic optimization, which brings higher computation cost. but significantly better empirical performance in both simulated and real- world tasks. We have seen lot of works that apply DGMs as the policy model improve the policy expressiveness handle complex input/output data, which are introduced in later sections, but whether such an implicit design could improve DGM-based policies remains to be explored.
2.2.2 Inverse Reinforcement Learning
Inverse Reinforcement Learning (IRL, Russell (9988) involves an apprentice agent that aims to infer the reward function underlying expert demonstrations. Subsequently, policy agent can be trained through
10

---

Published in Transactions on Machine Learning Research (08/2024)
RL based on this inferred reward function, attempting replicate the expert's behavior. Unlike BC, most IRL algorithms require not only offline demonstrations but also access to simulator. By interacting with the simulator, the IRL agent can observe long-term consequences of its behavior and learn to recover from mistakes through RL, making them less sensitive to covariate shift than BC. Also, reward functions provide explanation for expert behaviors, which can help eliminate causal confusion. Traditional IRI. approaches (Ng & Russell (2000); Abboel & Ng (2004)) usually restrict the form of reward functions to be, e.g., a lincar combination of manually-designeed features, which may be overly simplistic for real-world tasks. Additionally, these approaches follow an iterative process that alternates between reward updating and policy training. At each iteration, the policy is trained to be near-optimal under the current reward function, so as to evaluate the utility of the reward and inform its next-iteration update, which is computationally expensive. Further, a policy can be optimal with respect to an infinite number of reward functions ions (Kim el al. (2021b)) ). This inherent ambiguity in the relationship between the policy and the reward function can pose significant challenges for reward learning.
DGMs have played significant role in addressing aforementioned issues of IRL. (1) Within GAN-like framework, GAIL (Ho & Ermon (2016)) and AIRL (Fu et al. (2017)) implement IRL as adversarial learning process. They adopt a discriminator network as a pscudo reward function, which can take raw states as input and does not restrict the form of rewards. They also learn an actor network as the policy, which is concurrently trained alongside the discriminator. Intuitively, the training process involves a two-player game between the actor and discriminator. The discriminator attempts to distinguish agent trajectories from expert trajectories, while the actor endeavors to deceive its adversary by generating trajectories that closely resemble expert. ones. At each iteration, both the actor and the discriminator are trained for only several epochs, in GANs, which greatly reduces the computation cost compared to traditional IRL. Further, these algorithrs are based on MaxEntRR (Ziebart et al. (2010 which cleanly resolves the ambiguity wetween the reward function and policy using the principle of maximum entropy. We push the mathematical details of MaxEntIRI, GAIL, and AIRL and their extensions to Section 4.2 (i.c., GAN-based IL). (2) As another representative work, AVRIL (Chan & van der Schaar (2021) is based on the Bayesian IRL (Ranachandran & Amir (2007) framework and VAFs. In Bayesian IRI., a posterior distribution over candidate reward functions is derived from an assumed prior distribution over the rewards and a likelihood of the reward hypothesis. Classical Bayesian IRL algorithms are computationally intractable in complex environments, since generating each posterior sample requires solving an entire MDP with RL. As introduced in Section 3.1 VAEs (Kingıa & Welling (2014)) facilitate scalable estimation of Bayesian posteriors through variational inference, so AVRIL utilizes VAEs to implement scalable Bayesian IRL. Specifically, within a VAE, the encoder is used to approximate the posterior distribution over the rewards, and the decoder is employed to approximate the Q-value, which implicitly defines the policy. Compared to GATT, AVRIL also offers computational efficiency and does not impose restrictions on the forms of rewards. However, its algorithmic design lacks robust theoretical grounding, and its empirical performance tends to be inferior, indicating that more future work is required. We find that most research scalable IRL favor GAN-based approaches.
2.3 Challenges of Offline Policy Learning
In this section, we summarize the significant challenges faced by offline RL and IL, providing guidance for the application of DGMs in these areas. pynaaiccprogramming-baarr estimations of some key quantities, such as the approximated behavior policy f in policy penalty methods and the expert support in support constraint methods, to avoid OOD behaviors. Achieving these accu- rate estimations poses a significant challenge. On the other hand, the pessimistic value method addresses the overestimation issue common in other dynamic programming methods but can suffer from excessive pessimism, which may limit improvements beyond the behavior policy. Therefore, maintaining balance between conservatism and optimism is crucial. Model-based offline RL necessitates an accurate world model M and explicit uncertainty estimation. Modeling MDPs for environments characterized by complex dynamics, high-dimensional state spaces, and long decision horizons is particularly challenging. Further, uncortainty estimation, which serves to quantify the distributional shift, can significantly affect the perfor- mance of model- based offline RL. renjectory-optimizatonn based offline RL requires a large amount of training data that adequately covers possible return-trajectory pairs. Additionally, it remains uncertain
offline RL relies on accurate
11

---

Published in Transactions on Machine Learning Research (08/2024)
whether conditioning on the return-to-go ensures that a generator would produce trajectories achieving that specified return, highlighting some fundamental limitations of these methods.
BC can be improved by employing more expressive/ sxpressive/ddvanne /advanced policy models or by addressing its inherent issues, such as covariant shift and causal misidentification. Methods that effectively mitigate covariant shift or compounding errors; without requiring online experts or simulators (i.e., relying solely on offline data), could be particularly beneficial. IRL is a more demanding technique as it seeks to infer not only the expert policy but also the underlying reward function. Currently, there is no algorithm that perfectly balances computational efficiency with theoretical soundness in reward recovery. GAN-based IRL, such GAIL, often encounters challenges related to training stability and sample complexity. Most IRL. algorithms depend on simulators to train policies through RL. However, this survey focuses on algorithms that operate exclusively with offline data, and as such, we do not provide an extensive review of applying DGMs to IRL.
From a data perspective, a significant challenge offline RL effectively utilizing unlabeled data, which does not contain reward labels and is more readily available. As an example, some VAE-based unsupervised skill discovery algorithms, such as (Eysenbach et al. (2019b); Ajay et al. (2021]: Chen et al. (023c) )) attempt to extract skills from unlabeled data, which can then serve as policy segments for downstream RL, Similarly, a combination use of large amount of unlabeled data with a limited amount of high-quality labeled data represents a challenging yet promising direction for data-efficient offline RI. As for II., learning from imperfect demonstrations can be a fruitful research direction. Given the difficulty in obtaining large numbers of optimal demonstrations from human experts, an IL agent may have to learn from demonstratinns that vary in levels of optimality to ensure sufficient training data. As examples, there are two extensions of GATL for learning from imperfect demonstrations (Wu et al.] (2019b); Wang et al. (2021b)). They propose mechanisms to automatically weight demonstrations based on their quality and siguificance for training, and develop algorithms for learning from the weighted demonstrations.
Finally, several extended sctups for offline policy learning can be developed, including safety-critical, multi- agent, multi-task, and hierarchical settings. The safety-critical setting involves numerous safety constraints that the learned policy must. adhere to. Multi-lask learning aims to develop a policy that can either be directly applied to a variety of tasks or quickly adapted to them. Hierarchical learning usually requires constructing two-level policy, where the lower level consists of skills specific to subtasks and the high level manages the transition between these skills. This structure is particularly beneficial for complex, long- we introduce a
horizon tasks that can be decomposed into a sequence of subtasks. Throughout Section 3 7 series of works that apply DGMs to these extended setups. Additionally, for IL, learning from observations (LfO) is a more challenging setup, where the agent only has access to trajectories of expert states without knowledge of expert actions. Algorithms for this setup enable challenging tasks such as learning from videos and learning from agents with different embodiments (who share the same state space but have distinct action sets). GAIFO (Torabi et al. (2018)) adapts the GAIL objective for LfO by matching the expert and agent's is state-transition distributions instead, and there are several based algorithms designed for LfO as detailed in Section 3.3 However, the overall research applying DGMs to LfO remains sparse, suggesting further exploration in this direction.
In Section 8.1 we provide an in-depth discussion on offline policy learning and deep generative models (DGMs). In particular, we outline the base II/offfine RL frameworks to which cach type of DGM has been applied in Table [2] and the issues or extensions that each DGM has tried to resolve in Table [3 corresponding to the challenges introduced in this section. We encourage readers interested in solving specific problems of offline policy learning to look up these tables and refer directly to the corresponding sections.
3 Variational Auto-Encoders in Offline Policy Learning
This section begins with background on Variational Aulo-Encoders (VAEs), where we introduce their math- ematical foundations and model variants with a particular focus on those employed in offline policy learning. Subscquently, we present a comprehensive overview of the applications of VAEs offline RL (Section (.2) and IL (Section 3.3) as two subsections. Within each subsection, we categorize related research works based how VA utilized and summarize the algorithm design paradigm as tutorial. At the end of each
12

---

Published in Transactions on Machine Learning Research (08/2024)
subsection, we provide table to show the representative algorithms with their key novelties and evaluation tasks, serving as reference for future research in algorithm design and evaluation.
The sections focusing on other generative models, iee, Section 4 7 follow a similar structure with this section. Considering the extensive scope of content addressed in this work, notations utilized for different generative models are relatively independent. However, within the context of the same generative model, including its background and applications in offline policy learning, notations are consistent and can be cross-referenced. Consistently across these sections, we (x(-) to represent data points in the offiine dataset and (z(-) to represent their latent representations.
3.1 Background on Variational Ancodensoders
Variational Auto-Encoders (VAEs, Kingma & Welling 2014 2019 assume that the given data distribution
(x( can be generated with a deep lalent-variable model (z- = f f Po. (z) (x|z)dz,, i.e, sampling a continuous latent. variable z from the prior Pa- (2) and then generating x from the conditional (generative) comes from a parametric family of distributions infinite mixture of some base probability distri- butions Po(x)2) (e.g., the Gaussian distribution) across a range of possible latent variables 2, thus it can be quite flexible and powerful for modelling complex data distributions (x(x).
model Po- (z(2). It's also assumed that (z-(z)
Po(z) (Po(2)2)) 2)). Intuitively, such (z) can be seen
To learn the mapping from & to I, an extra model that can infer the latent variables 2 from the training data x is required, as z's are not provided with the offline data. However, according to the Bayes rule, the true posterior (inference) model Po(z|z) (all))
is intractable in most cases. For example, if Pa(æ])) is implemented as a neural network, we do not have the analytical form for Po(x) and so not for Po(2]z). Therefore, in VAEs, generative model P=(z]2) and corresponding inference model (x(x)) (for approximating the true posterior) are jointly learned. The most notable contributions of VAEs are two-fold: firstly, they propose a variational lower bound as a practical learning objective, and secondly, they introduce the reparameterization trick, which facilitates end-to-end training of both the generative and inference models.
The objective function of VAEs is a variational lower bound (a.k.a., evidence lower bound (ELBO)) of the log likelihood log Po(2), and its derivation process is widely adopted in algorithm designs related to latent-variabl models, so we show the process here:
Po(x, Po(2|z))
] EswPo(ts) log
Pola, 2) Po(2|1)) Po(2(xx) x)) Po(2|x)
log Pa(x)) =Egw Pa(lz) [log Po(x))] Pa(l2)))))))
log
2))
P2(2(x)
-Ea-Pa(1k) log
+ log
Co, + Dk(((P(cle)llP(((laa)) 2 (x)
(a(x))
(13)
log Pa(2(x:) = E2-Ps((a) log Po(x|z) log
[P( Po(2) (14)
(x) = EEm-P((a) [log Po 2)
=EzmPb((1)) [log Pa (x|2)] (Po
The inequality in Eq. 13) uses the fact that KL divergence (i.e., DKL(-Il-) is non-negative Csiszár & Shields (2004)). Po and Po are learned by maximizing Px (z), which in turns maximizes En- xx(( log Pr(r). When using neural networks for both the inference model Pa(e(r) and generative model P (=(=))), (x) trains a specific type of auto-encoder, where Pa(a)ar) and Po(x)2) function as the encoder and decoder respectively. Intuitively, this auto-encoder is trained by maximizing the data reconstruction accuracy (i.e., the first term in (z)( while regularizing the distribution of from the encoder to be close to the prior distribution of 2. This regularization is necessary, since is sampled from the encoder for data generation (i.e., z ~ (o(xx),, t- ~ Po(-)z) when training but, during evaluation, is sampled from the prior distribution (i.e., 2 ~~ Po(:), ~ Po(-1=))
Usually, VAES assume that Pa(=) = A((20, I) and P^(2|=) ((2; Φ), diag(o"(z; ())). Here, assuming E RRd, I is a d x d identity matrix, p(x; E Rd is mean vector, and diag(o? (z; E gaxd is a diagonal covariance matrix with (z) as the diagonal elements. In this case, the analytical form of
13

---

Published in Transactions on Machine Learning Research (08/2024)
DRe(P((c(e)]IP((e)) exists and we have:
1 d
Lo,o(z) = P2 Dlg Po(x|2)] + 2 [ (x; - - = La,a(x) + (z) (15)
Vol3(z) and VoL (2) can be casily estimated with gradient backpropagation. Howover, when calculating o the gradient V log (a((2) cannot, be backpropagated to i, since are samples from the encoder rather than a function of 6. One solution is the reparameterization trick. Specifically, can be reparame- terized as a deterministic function of á by externalizing the randomness from the output to the input of the encoder, i.e., z = Po(a,e) = ul () + (a; Φ) @ e, ~~ (N /). (e E R and @ denotes the element-wiss product.) In this way, the generative models and corresponding inference models can be jointly trained using straightforward learning techniques (e.g,, stochastic gradient descent (Amari (1993))).
Next, we introduce some representative variants of VAEs, which are utilized in Section 3.2 and
3-VAE: Assuming that the data Px(.) is generated by groundtruth simulation process with a number of independent generative factors, 3-VAE (Higgins et al. (2017)) is proposed to learn latent variables x that encode each generative factor in separate dimensions. For example, a simulator might sample independent factors corresponding to the object shape, colour, and size to generate an image of a small green apple; ideally, in the latent variable of the image, there should be separate dimensions for each of these factors. To learn such an encoder Po(2(z), the authors propose to constrain the KL divergence between (=(x()r and an isotropic unit Gaussian distribution Po(z) (zz;0,)) to encourage the dimensions of to be conditionally independent, resulting in the objective:
max Eg~Po(-lm) [log Po(x|z)] s.t. Dkr((a(sle)lIP((3) <e 0,0
(16)
Introducing a Lagrangian multiplier 3 > O, the equation above can be reformulated into an uncon- strained optimization problem: maxg,0.8 (x-p) [log - (Po((((()))))]PPPC - c]. They opt to fine-tune 33 as a hyperparameter instcad of learning it, leading to the final objective:
max Pe-Po())) (log Pa(x)=) 3D (P 0,6
(17)
When 3 = I, this recovers the original VAE formulation (i.e., Eq. 14 B controls the tradeoff between limiting the latent channel capacity (via the second term) and preserving the information for reconstructing the dala sample (via the first term). Empirically, they find that it's important to set 33 > 1 in order to learn the required disentangled latent variables.
Gated VAE: As mentioned in 8 V-E, disentangled latent variables can be viewed as data repre- sentations in which each element relates to an independent and semantically meaningful generator factor of the data. 3-VAE provides an unsupervised manner to learn such disentangled representa- tions (through constraining the KL divergence). However, consistent disentanglement has recently been demonstrated be impossible without inductive bias or subjective validation (Locatello et al.. (2019)). Thus, Gated VAE (Vowels et al. (2020)) is proposed to incorporate domain knowledge as weak supervision to encourage disentanglement. Specifically, as form of weak supervision, data can be clustered such that each cluster comprises data sharing certain generative factors. Then, the latent variable can be split into several partitions, each of which capture the generative factors of a specific cluster. Suppose that xı and z2 are from the same cluster which is assigned to the i-th partition of the latent variable, i.e., 2;, the encoder and decoder can then be trained with a modified ELBO: (which is a lower bound of log (x()))
La Lo = Ez-P((())) [log Po(2)1) - Dks(Po((\e:))IP((())
(18)
Note that the entire is utilized to generate 22 from Ø1, but only gradients corresponding to 2; are backpropagated to the inference model Pg. The rationale that 2g is intended to reconstruct only the shared generative factors with 21, and these factors should exclusively be captured by 2,. Gated VAE requires domain knowledge as supervision, which may limit its general applicability.
14

---

Published in Transactions on Machine Learning Research (08/2024)
CVAE: Conditional VAE (CVAE, Sohn et al. (2015) ) is model for conditional generation. Given a set of data and their corresponding conditional variables c. (e.g., classification labels or desired attributes of the data), CVAE learns a conditional prior (z(2le) and a conditional generative model ()(x),,)) so that a data sample x that satisfies a certain condition с can be generated via 2 ~ Po(-le), x ~ co( CVAE is particularly useful in lasks requiring controlled data generation, like scenarios where manipulating specific attributes of the generated data is necessary. As in VAEs, an inference model Pa(z|2r c) is introduced for the learning process, which gives the ELBO:
(a c) = -Exx,,(12,0) [log Po(z)2, c)] - DKz (
(19)
Compared with the original VAE formulation, a conditional variable c is introduced to cach function, i.e., the prior, inference, and generative model. Following a derivation process similar with Eq. (13 can be shown that Eq. (19 is a variational lower bound for log (z](c), i.e, the conditional generation likelihood. Further, they note that, during evaluation, latent variables are sampled from (z(e) rather than (= c), as only the conditions c are available. Thus, to enhance consistency of the data generation process during training and evaluation, they introduce an additional objective term to Eq. 19
chybrid (x,c) = aaa c) (1 az) (e [log Pa(xlz,c) L0.0
(20)
VQ-VAE: Unlike aforementioned VAE variants where 2 is contimuous, VQ- VAE (van den Oord et al. (2017) is an auto-encoder framework utilizing discrete latent variables. In particular, they assume the latent variable space is a codebook !ef.... " E R*xd where k is number of categories and d is the dimension of the latent vector for each category (i.e., ex). Such discrete representations are usually easier to model than continuous ones and can be particularly advantageous in some situations. For example, in a dataset of images with several categories, categorized latent representations are more appropriate, where each category is assigned a code from the codebook. In particular, the forward process of VQ-VAE is: 2g = Pa(x), 2D = 6 (i= = arg min, ||gg - -2fl2), 3 ~ Po(-1eD). Here, the latent variable from the encoder, i.e., ZE; is mapped to its nearest neighbour in the codebook, i.., 2D, which is then input to the decoder for data generation. The learnable parameters of VQ- VAE include 0, Φ; and eisk, for which the objective is as below: (ED and e; are defined above; sg denotes the stop-gradient operator.)
Lo.s.ena (I) log Po(x|>D) - ||sg [P(() e.1 -111((a) - sg [ei]
(21)
The generative model is trained to reconstruct the data x, as in other VAE variants, via the first term of Eq. 21p. The codebook parameters are trained to align with the encoded latent representations 2F (z(x) through the second term. As for Pa. it is trained by both the first and third terms. Specifically, they assume aoog P(()))( a log Pb(2(=)) such that the gradient of the first term on æD dsp can be backpropagated to Φ. The third lerm, on the other hand, is to make sure that the encoder commits to an embedding from the codebook. While empirical evidence demonstrates VQ-VAE's effectiveness, its objective design largely relies on intuition, so additional theoretical underpinning would be beneficial.
dsg
VRNN: Chung et al. (0015) introduce a recurrent version of the VAE for the purpose of model- ing sequences of data (i.e., ØST A ..r]). At each time step t, the historical information (Ien, z<1) is involved for sequential decision making, where 2<t are latent variables corresponding to Z<t. Instead of directly concatenating the sequence of variables, i.e., (zen, Zct), as the his- tory, recurrent neural network (RNN) is adopted to embed the historical information recursivcly: h = RNN, St, 11,, (1 (6= 0, ... ,) To involve historical information in decision-making, the prior, generative, and inference model are conditioned on the history embedding as: Po(zo(he-1), Pa 24, 1),,,, and Po (ra, 14-1), where h4-1 embeds the history (t<t, The overall objective
3 AAccrrdin to [Chung et al. 2013 ) additional embedding layers embx and embz can used extract features from the data ze and latent variables zt. respectively.
15

---

Published in Transactions on Machine Learning Research (08/2024)
(to maximize) for these three models and RNN, is below:
Σ (log Pa he-1) DKL. 1)l 1 ()))
(22)
where (2 S T|x T) IT- P& Pa h4-1) with he defined as above. This objoctive is a variational lower bound of the log likelihood of the data sequence, i.e., log (a as shown in Appendix A of (Chung et al. (2015)) The introduction of historical information and explicit use of the Hidden Markov Model (Rabiner & Juang (1986)) (e.g,, for defining Po. (z S T|x T))
significantly improve the representational capability of VRNN for sequential data.
3.2 Variational
in Offline Reinforcement Learning
Applications of VAEs in offiine RL are mainly based on the dynamic programming methods introduced in Based the given background, we explore specific applications of VAEs for offline RL in preceded by an overview in Scction 3.2.1.
Section 2.1.1
Scction 3.2.2
3.2.4
3.2.1 An Overview
A major use of VAEs for offline RL is to estimate the behavior policy from the offline data Dy- Specifically,
8 CVAE (see Section 3.1 is adopted for this estimation:
max B D, [E2s))(aas) [log Po s)] D 0.0
(23)
(19 and a here work as the condition variable e and data sample respectively. As introduced in Section 3.1 the objective above constitutes a lower bound for B(s,a)-D, [log (a(s))]. ie., the typical supervised learning objective. After training, actions at state 8 can be sampled from the estimated behavior policy ((-ls) as: 2 ~~ Po(-Is), à Po(-|s,2). In practice, the reconstruction term log (a 8) can be replaced with -Ila allž for continuous action spaces, and the prior Po(2)s) can simply be chosen (22 Following the 3-VAE (i.e., Eq. [17)), factor B 0 is usually introduced as the weight of the KT, term in Eq. (23) to balance the two objective terms.
Corresponding to Eq.
With the estimated behavior policy A the policy penalty and support constraint offline RL methods intro- duced in Section 2.1.1 can be naturally applied. As a representative of support constraint methods, BCQ (Fujimoto et al. 2019)) ) trains such fi and defines the policy to learn based on f as (-1s) = a+E(- @), where is a sample from p(-ls) and E(I|s, à) is a bounded and learnable residual term. π trained to maximize the approximated Q-values as in standard RI. + In this way, the action support of is constrriined to be close to the behavior policy's. Similarly, inspired by (Shamir (2018)), AQL (Wei et al. (2021)) proposes to improve the cstimation for the behavior policy μ using a residual generative model W1 (W) + a), where à ~ ji((ss) (as in BCQ) and W1,2, G(-) constitute residual network. They claim that such a residual structure could effectively reduce the estimation error (compared with using à) for the behavior policy μ.
Regarding the benefits of using a VAE as the policy network, compared to feedforward networks composed solely of fully-connected layers, VAEs excel at capturing the multiple modalities present in Dj,, which could be collected by a diverse set of policies, by utilizing the latent variable 2. Also, the action generation process, ~ NN(0, I) and (o(-ls,)), allows for stochastic action sampling. Compared to other deep generative models, VAEs may be less expressive but are more lightweight than Normalizing Flows and Diffusion Models and can provide more stable training than GANs.
Based on these background kuowledge, we provide a review of VAE-based offline RL algorithms in the following subsections, with a particular focus on works whose primary novelty lies in the use of VAFs. One category of such algorithms seeks to enhance mnaamic-prorramming based offline RL via the use of VAEs. They either modify the learning objective to further mitigate the issuc of OOD behaviors, apply
4 Forr BCQ, multiple Q functions are learned simultaneously, and the target value for policy evaluation specifically designed based on these Q functions, which though not our focus. Please refer (Fujimato al. (2019)) for the details.
16

---

Published in Transactions on Machine Learning Research (08/2024)
augmentatioon/converioo to the offline data for improved learning. The other category concentrates extended offline RT. setups, such as hierarchical or multi-task offline RI, leveraging the fact that the latent variable can be learned as embeddings for tasks or subgoals/subtaast within a task. Consequently, (a(als, z) can be interpreted as a task-conditioned policy (for multi-task RL) or a subtask-conditioned policy within a hierarchical policy structure (for hierarchical RI.).
3.2.2 Addressing the Issue of Out-of- of-Distribution Actions
As introduced in Section 2.1.1 there are four categories of algorithms for mitigating the issue of OOD actions. VAEs have been used lo improve three calegories among them, which are detailed as follows.
Applying support constraints: PLAS (Zhou et al. (2020)) proposes that the support constraint can be simply applied the latent space of a VAE. In particular, they first estimate the bchavior policy as a CVAE i with Eq. (23__ Ideally, after training (through maximizing the reconstruction accuracy of actions induced by the behavior policy), for latent variables z which have high probabilities under P& (a(ss,, the corresponding decoder Po(als, z) should output high-probability actions under the behavior policy distribution (i.e., in- distribution actions), since p(als) is estimated as f (o(2ls))o(alss,) de. In this case, they learn a latent space policy π<=[s) and use it in conjunction with the pretrained (and fixed) decoder Po (a(ass, 2) as the mapping from 8 to a. The output of π is constrained to [-o, Ø), ie., the high probability area of the prior distribution ((2; 0, I), to ensure that (P, o (alss outputs in-distribution actions. As introduced in Section
5.3 Na-eaaed offline RL algorithms adopt the same algorithm idea. SPOT (Wu al. (2022) suggests
that the constraint in BEAR (log (s)(s))s)= 2 c, Vs Dy) can be relaxed to Ey,D,, [log Ju (T [s) 2 c' for
practicality. Then, the constrained policy improvement step (i.e., the second equation in Eq. (1) can be
converted to: (We use π and Q to represent the policy and Q function from now on, for simplicity.)
max 12(s, T()) + A(log ((T))s) - e')l
(24)
Here, A > O is the Lagrangian multiplier. Again, they explicitly model j as a CVAE A = (Pa. Po) and propose to approximate log u(als) as below: (z(0 ~ P)) I= L.)
2o(als, ()(=(s,a)
1 I.
Polals, Po((")))
log ((als) = log Pa(a)s) = log E. Ba
z log
(25)
I
Pa(2(0|s,0)
1=1
As introduced in Section 5.1 Normalizing Flows enable exact density estimation, which can potentially climinate the need for sample- sample-based approximations as in Eq. (25).
Applying policy penalty: TD3-CVAE (Rezaeifar et al. (0022) proposes to replace the penalty term
Eq. (4) (i.e, ((())))))))) with a prediction error b(s a) = ||a - Pa o (s a)ll, where j = (Pa, Po) is the pretrained CVAE for estimating μ., Intuitively, if an action a from the being-learned policy x(-Is) corresponds to a high prediction error under A(-[s), a is probably an OOD action and so (s, a) should be assigned with a high penalty. Further, they theoretically show the equivalence (under certain conditions) a
of b(s.a) a) and KL-divergence penalty term DR (TT(((-11)) ((((1)))](m(((()s) where TT(-(s) SoftMax( -)/r) and T > O is a temperature parameter. Compared with (x (=(ss)))) (=)s)))((-sss), b(s, a) is casier to approximate and brings superior empirical performance. BRAC+ (Zhang et al. (2021a)) points out that estimating the penalty term (xz((=((ss)]ld( as (xz( (=(ss))l)((()) in BRAC requires generating a large number of samples to reduce the estimation variance. Therefore, they propose an upper bound for (()) which has an analytical form:
D DKI(T (1))lll)))) ss Ba (als) - log [[als)]
< E (+))) (log x (als)) Ea-a((a) (EewPs(H)nn)) [log Po(als, 2)] log Po(als, - Beax(()) (DxssPPd(lss.a)))PP/GM)))
(26)
= (e
Here, the incquality is based on the fact that a CVAE A (Po, Pb) is used to estimate μ and constitutes a variational lower bound for log ((als). shown in Eq. 19). Given that Pa(als, 2), Po(=)s), and P& (=|1,, a) all have Gaussian outputs and suppose
ExePo()saa) [log Pa(als, - DKL (P)(
17

---

Published in Transactions on Machine Learning Research (08/2024)
that the policy x(als) is also Gaussian, then both (als( - log Po(a)s,2) and DKz (Kz(PP (Po(ess a) Po((ss)
have analytical forms, which can reduce the sample variance. However, sampling for a and is still required.
Applying pessimistic values: CPQ (Xu et al. (202aa) ) extends the idea of CQL (introduced in Section 2.1.1 to safe RL, where the policy is trained to maximize its Q-values while minimizing the accumulative cost Qa(s,a) = c(s,a) +
(')]] In particular, they train the cost Q-function Qe to assign high costs to OOD actions, such that, by constraining Qe as in standard safe RL algorithms, OOD actions can be avoided at the same time. To realize this, the objective for Qn is designed below, which is similar in form with Eq. 61:
min (e.... [Qo(8,a) - (c+ (BB' "a (1)0) Qo(s', Qc
(27)
Here, j is defined based on f = (Po. Po). In particular, Ya ~ p(-1s), DKI D 2 d, where (z]ss) (22 0, and d is a predefined threshold. Intuitively, ((-ls) produces OOD actions a of which the corresponding posterior distribution Pa P=(2)s, a) deviates significantly with its training target (=(s),, and the expected cost Qe of OOD actions is trained to be high. MCQ (Lyu et al. (2022) explores mild but enough conservatism for offline RL to mitigate the underestimation issue of CQL. Their algorithm's I's design does not hinge on VAEs; instead, the VAE is employed solely for estimating as in previous works, so we do not provide details here. For similar reasons, we skip introductions of UAC (Guan et al. (2023) ) and O-RAAC (Urpí et al. (2021) ), both of which can be considered as extensions of BCQ, a support constraint method.
3.2.3 Data Augmentation and Transformation
Data augmentation: The provided offine data D, may have limited coverage of the state-action space or lack diversity in behavioral patterns. Consequently, constraint- or cosimissm-based algorithms may learn sub-optimal policies with limited gencralization capabilities in the entire environment. VAEs have been used for data sugmentation, aiming at improving the coverage diversity of the ollline data. (1) ROMI (Wang et al. 2021aa) is a model- based data augmentation strategy utilizing reverse rollouts. They first learn the backward dynamic model (s)s, a), reward model f(s,a), and reverse policy Prev(a)s') from Dμ via simple supervised learning, where s' denotes the next state. Specifically, (rew(als') is modeled with a CVAE (Po Po), and its training objective is the same as Eq. (23) but to replace s with s'. With a random sample 8141 from Dj,, a reverse rollout (of length h) can be generated as below:
(SI- (01-1)0---, St+1--) ae-i ~ Prev (144411)))) St-s ~
ar)) i=
(28)
Unlike the forward generation process, such a reverse manner prevents rollout trajectories that end in OOD states. Also, the CVAE makes possible for stochastic inference: 2 ~ (O, I), Pol-ls',2) 2), which improves the diversity. These generated rollouts are then combined with D2 for the use of offine RL. This work is closely related to model-based offline RL (Section 2.1.2 (2) To enable the learned policy to generalize to OOD states, SDC (Zhang et al. (2022) suggests training the policy on perturbed states and motivating it to revert to in-distribution states from any state deviations. In particular, a forward dynamic model 7(s')s, a) and a CVAE-based state transition model (('(ss) is learned from D, through supervised learning. At a state perturbed from 8, the policy и is trained to minimize MMD (115,, ((11))))()))) i.e., to produce actions that can lead it back to the next state s' in the original trajectory from the perturbation 5. MMD denotes the maximum mean discrepancy. (3) Han & Kim (0022) point out that the latent space of a VAE pretrained Dj can capture the data distribution in Dj.. Based on that, they suggest selectively augmenting the data region that is sparse in the original datasct through data generation with the VAE. However, that paper does not provide details on measuring sparsity through the latent space or data generation using the VAE. (4) KFC (Weissenbacher et al. (2022)) suggests inferring symmetries (Hambidge (1967)) of the underlying dynamics of an environment using a VAE forward prediction model, and applying such symmetry transformations to generate new data points as data augmentation. This work requires extensive knowledge of control theory, it will not be discussed in depth here.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Data conversion: VAEs have been used to transform the states actions in the offline dataset to simplify the learning process. Here, we present two notable works in this direction. When the state space is high- dimensional (e.g,, images), directly applying offline RL to the raw data could be challenging. Thus, Rafailov ct al. (2021) propose using VAEs to get compact representations 2 of high-dimensional states s to improve the learning efficiency. 2 effectively represents 8, since the VAE is trained to reconstruct. 8 from 2 while adhering to variational regulations (i..,, the KL term in Eq. (14)) However, this data conversion manner is not specific to VAEs. Other generative models with encoder-decoder structures, such as Normalizing Flows and Transformers, can also be employed for this purpose. On the other hand, SAQ (Luo et al (2023) proposes to convert continuous actions to discrete ones, to make it significantly simpler to implement. constraint-based offline RL methods. To realize this, they train a VQ-VAE (introduced in Section 3.1 on Djr: which can map a inuuus action a at a given state 8 to a discrete code (through its encoder) for training and map a given discrete code back to the original action space (with its decoder) for evaluation. This state-conditioned action discrctization scheme learned as below:
max (e, [lg Palaess - ||sg (P)(s. - -ssl - a) sg(e) [es 8,0,51
(29)
Here, Gi.k is the codebook and represents the & discretized actions; arg [IPo(s.a) is the index of the nearest latent code for the embedding as)s, Applying the pretrained VQ-VAE encoder Pa on D, leads to discretized actions, ie., (s,a) + (s) For discrete action spaces, the estimation of the constraint terms in offiine RL, such as the approximated behavior policy fu(āls) and KL divergence (xL(m(((s))lu((ls) in policy penalty methods, could be easier and more accurate.
3.2.4 bAE-based Multi-task/Hiiie taerraiaaaaiaaa Offline RL
Multi-task RL and hierarchical RL extend the basic RL setting. In particular, multi-task RL (odhani el al. (2021)) )) aims at learning a policy that can be directly applied to or quickly adapted to distribution of tasks. Hierarchical RL, on the other hand, learns a hierarchical (two-level) policy for complex tasks which can usually be decomposed into a sequence of subtasks. In this case, low-level policies can be used to accomplish each subtask, while the high-level policy coordinates the subtasks and the use of low-level policies. For example, in goal-achieving tasks, a goal-conditioned policy can be considered a multi-task policy, as it can be used to reach multiple goals in an enviromment by changing the goal condition. However, if the goal is distant, the entire path might be partitioned into several subgoals by the high-level policy, and to reach each subgoal, a corresponding low-level (subgoal-conditione)) policy can be employed.
Next, we formally introduce these two setups and explore how VAEs can be utilized to enhance them. By introducing the most notable research in each category, Le., BOReL for multi-task RL and OPAL & HiGoC for hicrarchical RL, our aim is to present the fundamental paradigms of these rescarch directions as a tutorial.
, where and are indexes for the trajectory and task respectively, multi-task offline RL aims at learning a multi-task policy that can be adapted to unseen tasks with zero- or few-shot training. These unseen tasks are required to be in the same distribution as the training ones. As a representative, BOReL (Dorfman et al. (2021)) is proposed for the case where the reward and dynamic function (ry,T) vary with the task. To train a multi-task policy, a straightforward manner is to condition that policy on the task information (r,, Tj). In particular, they adopt the latent variable 2 [ of a CVAE as a representation of the reward and dynamic function, and learns a latent-conditioned policy [alss,2 2) as the multi-task policy. To this end, the CVAE is trained as follows:
2. Multi-task RL: Given multi-task oflline dataset. DM [[7 (sb3 a .0
max
ELBO, ELBO, Be- (log
-0TT0 - DKL (ro((P((ero:)))IP((()))
0=
(30)
T-1
log -0 log Po (ool))
Σ
[lg 1o( 01, log (r
5 AActually they adopt the mean and variance of the latent variable (f.e., the output of the encoder) as the policy conditioner.
19

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
VAE Type
VAE Usage
Evaluation Task
BCQ
CVAE
Estimating μ (Support Constraint)
MuJoCo
AQL
CVAE
Estimating μ (Improved Estination for (i)
DARL (L)
D4RL (L, A, K),
PLAS
CVAE
Estimating μ (Support Constraint)
Real Robot
SPOT
CVAE
Estimating μ (Support Constraint)
DARL (L, M)
TD3-CVAE
CVAE
Providing prediction errors (Policy Penalty) Estimating and DxL (T([s))))l (Policy Ponalty)
DARL (L,
BRAC+
CVAE
D4RL (L)
Estimating lr and using the latent space for OOD detection (Pessimistic Value)
CPQ
8-CVAE
MuJoCo
MCQ
CVAE
Estimating μu (Pessimistic Value) Estimating ب (Support Constraint) Estimating l (Support Constraint)
D4RL (L)
UAC
CVAE
D4RL (L, M, A)
O-RAAC
3-CVAF
D4RL (T.
Estimating the reverse behavior policy
ROMI
CVAE
D4RL (L, M, M2d)
(rey(als') (Dala Augmentation)
Estimating the state transition
SDC
CVAE
GridWorld, D4RL (L)
model (s) of D,, (Data Augmentation) Modelling (s'(s) for inference of dynamic symmetries (Data Transformation)
DARL (L, M. A, K),
KFC
VAE
MetaWorld, RoboSuite
Discretizing the action space to simplify the learning (Dala Conversation) Embedding MDPs for Multi-task RT. Embedding skills for Hierarchical RL Embedding skills for Hierarchical RL
DARL (L, M, A, K),
SAQ
VQ-VAE
Robomimic Grid Meta-MuJoCo D4RL (M, K) CALVIN, Real Robot
BOReL.
CVAE (T)
OPAL
B-CAAE (T)
TACO-RL
CVAE (T)
HiGoC
CVAE
Generating subgoals (Hierarchical RL)
D4RL (M), CARLA Real Robot
FLAP
CVAE
Generating subgoals (Hicrarchical RL)
Table 1: Summary of VAE-based oflline RL algorithrs. In Column 1, we list representative (but not all) algorithms in this section. These algorithms are grouped by their categories. Regarding the VAE types, 8- CVAE refers to an integration of CVAE and 3-VAE, where a weight /3 is added to the KI, term CVAE. The annotation (T) means that the VAE is implemented on trajectories rather than individual state transitions. The evaluation tasks are listed in Column 4. Most works are evaluated on D4RL (Fu et al. (2020)), ) which provides offline datasets for various tasks, including Locomotion (L), AntMaze (M), Adroit (A), Kitchen (K),
Maze2d (M2d), etc. MuJoCo [Todorov et al. (2012) and CARLA (Dosovitskiy et al. (017 ) are commonly- used simulators for robotic and self-driving tasks, respectively. Mela-MuJoCo (Dorfman el al. (2021) is a multi-task version of MuJoCo. By Real Robot, we mean evaluations on real robotic platforms, which vary from one study to another. For other benchmarks, we provide their references here: GridWorld (Zintgraf et al. (2020)), MetaWorld (Yu et al. (2019c)), RoboSuite (Zhu et al. (2020)), Robomimic (Mandlekar et al. (2021a)), CALVIN (Mees et al. (2022b)).
In this context, To.T = (SO.T, ToTT-1, 0o:T- d0:T-1 and (S0.T, can be viewed as e and x in the CVAE framework. As shown in (Zintgraf et al.] (2020)), ), ELBO, constitutes a variational lower bound for Fq. (30p derived based on the Markov assumption (Puterman (2014)), and it can be observed from this equation that 2 is trained to embod information regarding the initial state distribution, reward and dynamic functions, so as to reconstruct the task-specific trajectory To:r. Subsequently, the pretrained Po(2|Toa) which is implemented as recurrent neural network as in VRNN (sec Section 3.1,, can be applied to D, to infer the task embedding 2 at each time step, leading to a dataset of transitions in the form ((st, 21), oe, rt). Note that (st, =x) can be viewed as an extended state 5t, and thus standard offline RL algorithms can then be directly applied to this dataset to learn a conditoond policy T(ar((s..
log Po(SO.T. To:T-1 ( The second equation
Hierarchical RL: This category of algorithms try to learn a hierarchical policy (Fiigh Flo (als, x)) from the offline dataset. Intuitively, the agent would segment the whole task into sequence of subtasks
20

---

Published in Transactions on Machine Learning Research (08/2024)
subgoals, each of which is denoted by (contimuous or discrete) variable and accomplished by a correspond- ing subpolicy/ (osicyy/skklli /skill [aow(ass (als, 2). This hierarchical scheme is especially beneficial for complex tasks with long horizons. OPAL (Ajay et al. (2021)) is a representive algorithm in this direction. They define the horizon of each skill to be and organize the offline data as a set of trajectory segments Dj = =[T' = [ 16:--1lli Then, they learn the low-level policy Flow for different sublasks z as the decoder of a CVAE:
max Bp Po 0.0
Σ log No (a:ls;;;;))
Nlow T
(31)
This objective is equivalent to a (3-) ELBO, where so and T work as the conditioner с and data z respectively [ However, unlike Eq. (23 here Th is trained on sets of trajectory segments rather than single- step transitions. This is because, as a skill policy, To is expected to extend temporally. Also, the prior Po(2|s) is not fixed but implemented as a neural network that takes 6o as input, to make sure the skill choice = is pre- dictable (by the high-level policy (z|s)) given only the initial state so. Applying the pretrained (2(2)) on Dj and introducing the reward signals, a new dataset Dhigh = (sh, x ~ Po(-Ir), Ei=0 rí, 1 can be obtained for training Thigh Thigh(2|s) with any offline RL methods. With this hierarchical policy (Tnigh: Tow), the decision horizon of offline RL is effectively shorten (by factor of h) and so OOD actions caused by the accumulated distribution shift can be mitigated. Rosete- Beas et al. (2022) propose TACO-RL, which is a very similar algorithm with OPAL but specifically tailored for goal-achieving lasks. HiGoC (Li et al.) (2022)) is also hierarchical framework for goal-achieving tasks, where the high-level part is model-based planner for generating the subgoal list and the low-level part is a goal-conditioned policy trained by offline RL to reach each subgoal sequentially. The subgoals are not labeled in the dataset, so the low-level policy is trained in an unsupervised manner. Specifically, for Thow |8t, Sty), St, (12 > f1) is the subgoal and randomly sampled from states after St, in the same trajectory. To be robust to possible OOD subgoals during evaluation, CVAE m( = (Pa(z]se, St-n), (o St-h) is pretrained on D2 for generating the subgoal 8t conditioned on the previous subgoal St-h; where h is predefined time interval for subgoal selections. With this CVAE, perturbed subgoal can be generated based on the sampled one (i.c., St2) Po (Pa(stz Stg-h) a)) e, 8t) (e is a noise vector), which can replace St, the subgoal of Tlow for robustness. In cases of high- digh-diiensional states, such as images, adding noise to a well-defined low-dimensiona embedding space (i.e., z in the VAE) is a more effective and reasonable approach. The pretrained decoder Po(se|2, 8t-k) can also be used for high-level planning. Specifically, a subgoal list can be generated by specifying a list of latent variables Zi:8: 8t, Po(-|z,,81, -t), E [1, [1..... k). Such a subgoal list can then be evaluated by task-relevant objectives. In this case, searching for an optimal list 2 is literally a model predictive planning problem and is more efficient than directly scarching the high-dimensional state space( i.c., St,a). ). FLAP (Tang et al. (2022) adopls quite similar protocol with HiGoC, where the CAVE m is referred to as the affordance model.
3.3 Variational
in Imitation Learning
In this section, we offer overview of VAE-based IL algorithms. First, we provide Lutorial-like overview of the four schemes of VAE-based IL. Based on this, we introduce a categorization of all related works based on how VAËs are utilized. The use of VAEs focuses on enhancing Behavioral Cloning, either from a data or algorithmic perspective.
3.3.1 Core Schemes
Imitation Learning (IL) aims at recovering the expert policy π T[als) from a set of demonstrations DE. Be- havioral Cloning (BC) is a straightforward and widely-used IL framework (Pomerleau (1991)), ), as introduced in Section 2.2.1 With its special encoder-decoder structure, the VAE has been utilized to improve BC from multiple perspectives. Here, we provide a summary of four such schemes.
6 For standard CVAE, the roconstruction term should be log Po(T(su, E log (aces))t, Ele log as) However, the transition function Please refer (Li al. 2022)) ) for further details.
trainable and so would influence gradient calculation.
21

---

Published in Transactions on Machine Learning Research (08/2024)
Scheme (1): Similar with the major use of VAEs for offinn RL, CVAE conditioned on the state can be directly used to model the expert policy:
max E (s.a)> (Z= log Po(als, DK 0,0
(32)
This objective constitutes variational lower bound for Ds log (a]s), i.c., the BC objective. After training, 2 Po(+Is), Po(-|s, 2), where Po(z]s) is a predefined prior and usually set as N2 0, 4), can be used the policy a r-Iss). With the latent variable, the VAF-based policy can model stochastic behaviors with diverse modes. Given that ([aas)) f (a Po s, (,,=) (z)sss),, rather than using fixed prior distribution (z(ss) Ren et al. (2020) propose an algorithm to fine-tune the prior online for specific tasks by optimizing a genera lization bound from the PAC-Bayes theory.
Scheme (2): The second scheme is based on representation learning (RepL), where the latent variable = is adopted as a representation of the state s and usually can be learned via state reconstructions:
max Dn 0,6
E [a--P(() Reco(s, (s, 2) Dx<((p(c)s))(P((e))] max
log (alz)
(33)
"
where the reconstruction objective Reco(s, 2) = log Po(s)2) or l|ss Po Note that the first objective in Eq. (33 is usually coupled with extra regularization terms to encourage the disentanglement of the Iearned representation, or enable the fusion of state inputs s from multiple modalities, etc. Based on the representation learning, a policy [aa2)) conditioned on the compact representation = can be learned with any IL algorithm. Compared with s, 2 is in a lower dimension and is less noisy for decision making. As a result, RepL can effectively reduce the need of IL for large amounts of training data, In Eq.
333 there are in total three functions (besides Po(z) which may not be learnable): P=(2))), (s]z), and x xa]=). One manner is to pretrain Po(z|s) and Pa(s)z) within the VAE framework and then train <[alz) with II as shown in the second term of Eq. (33). When training n, the parameters of Po can either be frozen or not. The other manner is to jointly train the three functions through an integrated objective, i.c., adding the two objectives in Eq. (33) logether with an adjustable weight A. In this case, the state reconstruction with a VAE can be viewed as an auxiliary task of IL. EIRLI (Chen et al. (2021a)) provides systematic empirical investigation of representation learning for imitation. They find that RepL using VAEs can effectively improve the performance of vision-based IL. However, they also mention that the relative impact of using representation learning tends to be lower than the impact of adding data augmentations. This may be because usual RepL techniques (for images) tend to capture the most visually salient axes of variation (e.g.. the color, background, or objects) but the action choices are often determined by more fino-grained, local cues in the environment, which calls for RepL that is more specific to decision making. Additional examples in this scheme include (Lee al. 2019);; Rahmatizadeh et al. (018)))
Scheme (3): VAEs can be applied to directly model expert trajectories T (sp, (s0,a0, a.. ...
,8)))
E EZ- [log Po(T|=)] - D max Ex>DE 0,
T-1
(34)
log Po(T|2) = log (o(l)) +
Σ
log Po 2) + log Po( ar, 2)]
The decomposition of Po(T(z) makes use of the MDP model, where the three terms correspond to the initial state distribution, policy distribution, and transition dynamic, respectively. (sol)) and Pa az, 2), which may be independent of 2, are usually defined within the simulator and not required to model, so log Po(T]z) can be replaced with EnD log (s 2) in the original objective. Note that Pa(T(a) has multiple alternative decomposition forms. For example, T-VAE (Lu et al., (2019)) proposes to model it as log (o (z) + log Po ( /2, Sp.T). Specifically, two RNNs are adopted to generate the state and action sequences respectively, leading new definition of log Po(T]z): (The state sequence šo. is generated before 3,6
the action sequence do:r-1; the generation of á is conditionod on / which embods the history $1.21 51:211($122 (/112111,1111111111111111 (.:
T
log Po(T|z)= = log Pa 1( +
Σ
Σ
log Pa (arl2,
(35)
t=0
22

---

Published in Transactions on Machine Learning Research (08/2024)
In this way, the entire trajectory can be predicted without interacting with the wwvironment/simmlatae which is necessary in certain scenarios. In this scheme, the embedding z is inferred from the entire trajectory T, necessitating that the encoder Po be implemented with specialized architectures, such as encoder-only transformers (sce Section 6.1 or RNNs (like VRNN in Section 3.1. Different from aforementioned two schemes, such trajectory-coondiiionee encoder can out put embeddings z that encapsulate information at the task or subtask level. Thus, the learned (a 2) can be viewed as a multi-task policy which varies with the task embedding z. As a side note, if replacing the decoder Po(T]=) with Pa(r'(z), where (T, ~ Dr are (state-only) expert trajectories and T' is future trajectory segment of T, Eq. (34) can be used for training trajectory forecasting model, as detailed in RC-VAE (Qi et al. (2020)) ). By using such a temporal target, i., the future segment, the representation 2 is forced to contain predictive information.
The last scheme is based on the Variational Information Bottleneck (VIB) framework Alemi ct al. 2017 ) of which the original objective function as below:
Scheme (4):
max 1(2 s.t. 1(X, f) max f:X
f)
(36)
F:X-Y
where I(.) denotes the mutual information, Ie is the information constraint, and > 0 is the Lagrangian multiplier. The goal of this framework is to learn an encoding Z that is maximally expressive about the target Y while being maximally compressive about the input X. Substituting X,Y with 8, a, the function to learn, i.c., f, is then the expert policy (a)s). VIB-based II. can be practically solved by the following objective, which is lower bound of Eq. 336 (see (Alemi et al. 2017))):
max R (e (P log Po(alz) - -300-330x (DKx( 6,8,00
(37)
This objective is similar with the VAE ELBO (i.c., Eq. 14 in form but has two differences. First, the decoder Po here can predict variables different from the input of the encoder, whereas in VAEs, it is typically trained to reconstruct the imput variable. Second, (z) is not an assumed prior ribbuiin of a as in VA but variational approximation of the marginal distribution of 2, i.e., Pz(=) f (=)l and it is usually learned as neural network. Both Scheme (2) and Scheme (4) can be regarded as forms of imitation learning that incorporate effective representation learning. After training, to sample an action a ~ (-1))), the policy encoder first compresses the state to a representation, i.e,, 2 ~ Po(((s), and then its decoder predicts the action based on 2, i.e,, a ~ Po(-)z). When dealing with high-dimensional and noisy observations s, using an information bottleneck can filter out redundant information while retaining essential knowledge for action predictions in the representation 2. A higher compression rate on the input often results in better generalization of the learned policy across tasks.
Next, we provide an overview of the applications of VAEs in IL. Most works in this direction focus on extending BC. From a data perspective, VAFS can increase BC's data efficiency in scenarios where original, task-specific data is limited, or enable the use of multi-modal input data (e.g., from various sensor types). These applications can mitigate the issue of insufficient demonstrations for II, as noted in Section 2.3
Regarding algorithmic advancements, VAEs have been utilized for skill discovery to enable hierarchical IL, and to address the causal misidentification issuc of BC.
3.3.2 Improving Data Efficiency
II., especially BC, requires a large amount of training data to cover possible task scenarios for robust per- formance. However, expert-level demonstrations for a specific task, i.c., DE, are usually costly to acquire. Therefore, elficient data usage in IL is crucial. This can be achieved by leveraging the inherent structures within the data, or by gathering and processing task-related data from alternative sources for training. The capability of VAEs to extract latent representations plays a key role in facilitating these processos. Here, we present several notable works in this direction.
Behavior Retrieval (Du et al. (2023) is proposed for the case where vast bask-unlabelled demonstrations are available, i.c., Dprior- Specifically, Dprior may contain suboptimal behaviors for the current task or demonstrations for a range of different but related tasks (following the same task distribution). To this end, they propose to adopt 3-VAE (i.e., (Pa, Po) in Eq. (17) to learn the latent. representations of
23

---

Published in Transactions on Machine Learning Research (08/2024)
(s,a) ~ Dprior. Then, new training data that is similar with the ones in DE can be retrieved from D Drrior the augmentation Dret- The similarity between data points can be simply measured based on their latents from the pretrained encoder Po:
f((s1, a2)) = |1211 22l12, ($1, 01) E Pprior: 02) DE, 24 ~ (-()s((),
(38)
They claim that this method would effectively filter out sub-optimal or iask-irreeevant data. As final step, a policy x[als) can be learned on Dg U Dret through BC. This representation learning protocol resembles aforementioned Scheme (2).
For robotic control tasks, state-only demonstrations from a third-person view (e.g.. videos from a demon- strator) are usually more available than the ones in the first-person view. Observations from the first- and third-person views, ie., s and s", correspond to the same true environment state Given set of demonstrations synchronized across different viewpoints [(sf, ent third-person viewpoints, a common viewpoint-agnostie representation is required to make full use of these data. To realize this, VAE-TPIL (Shang & Ryoo (202I)) proposes to learn two VAEs for the first- and third-person view data respectively, i.e. (Par, Pos) and (Par, Por), and to disentangle the latent variable into a viewpoint embedding v and a viewpoint-agnostic state embedding h. To realize this, besides usual reconstruction objective terms, auxiliary objectives are involved: (i f j: (h) Pa(s))
of { where Ti:k denotes & differ-
EET [1IPo1TT of) of [T]kk], La = E,, [I|Po(1"), - sTalll), - -F= = -E, [1|4T= - ssg(h [12a]
(39)
The intuition behind these terms are: (a) The view embeddings vf and of of states in the same view, i.c., s7 and sf. should be semantically cquivalent and so Po(hT,o3) .o)) should be ablc to recover sT. (b) The embeddings A.T and AT) corresponding to the same state from different views, i.e., T; and Tj. should be interchangeable for reconstructing 5"i since h is expected to be view-agnostic. (c) Similarly, h² and ht should be close as they are viexpoint-agnostic and correspond to the same true environment state. With the pretrained Par and Par; demonstrations across various viewpoints can be converted to viewpoint-agnostic latent representations (i.e., h). Then, method. This algorithm can be viewed as a method for the challenging setting: Learning form Observations (LfO) (introduced in Section 2.3)
pppresntattion-basee policy (aah) can be learned with any IL
GIRIL (Yu et al. (2020b)) suggests modelling the undorlying transition from (st, az) to 8t41 in DE (i.c., its inherent structure) using a CVAE, where (ot, (s)) and 8141 work as the condition e and data sample x respectively, besides imitating the policy a(a (t). Then, an intrinsic reward can be defined as Ti || s++1ll; where $1+1 ~ Po(-1st, at). (Offline) RL can be applied to the extended demonstrations [st, rr, rr. st+1) to further improve the imitator learned via BC, i..., T(aelst). Intuitively, this process encourages the agent. to explore states with high prediction errors to reduce the state uncertainty, making it possible to achieve cester-than-esee performance.
3.3.3 Managing Multi-Modal Inputs
In real-world scenarios, decision-making inputs often come from multiple modalities, including RGB images, depth images, 3D) point clouds, language instructions, and more. As a motivating example, consider the task of navigating a drone in an outdoor environment, where multiple modalities emerge due to visual information captured by varions sensors and factors like the drone's current pose and location. Coherently managing the multi-modal input essential for the real-life applications of IT.. In this context,, VAFs have proven effective for fusing or unifying inputs from multiple modalitics basod on the use of latent emboddings.
RGBD-VIB (Du et al. 2022 proposes to fuse multi-modal inputs based on their uncertainty for evaluation, ensuring that inputs with higher certainty are given more weight in decision-
To this end, VAE (z)()))) Pa. (al (2)) is trained for each modality i E [N. using the VIB framework (i.c., Eq. 366)- Note that these VAEs share the same action space but vary in their state spaces. The uncertainly measure can then be acquired based on the pretrained VAEs as u²(s) = DKL(P (Pz(P):<
Intuitively, a large uncertainty (s) indicates that the current state modality i may be out of the distribution of the training dataset. Finally, the action is a weighted sum of the output from each VAE decoder, where the weight assigned to each modality is proportional to Σ Ejn (s) n' (s), inversely related to the uncertainty.
24

---

Published in Transactions on Machine Learning Research (08/2024)
Demonstrations may come with contexts c, such the task id, language instructions, goals, etc. In this case, a conditional policy m(als,c) can be learned for specific task goals or instructions. MCIL (Lynch & Sermanet (2021) is proposed for the scenario where multiple related contexts c; E [l,...,,]] are provided for a task. For a coherent use of these contexts, MCIL learns an encoder for each con- text, iee, (Po (Po (2(o1), ....,, (zleww) to embed different contexts into a unified latent space, and a de- coder Po(a|s, 2) as the latent-conditioned policy. The decoder and encoders can be trained end-to-end by max Bes..))DD....P... .. (Hla)) [log Po =)] [8] This leads to policy that generalizes over multi-modal contexts: 2 Par (+(a), a ~ Po(-|s, 2), E [1, ................... N.. As an example, language instructions are usually costly to annotate, but other contexts like the goal images are relatively easy to obtain (from sensors). Through such a unified latent space, MCIL enables the use of (large-amount) demonstrations from easier sources aid the learning of langttoe-conddtioned policy.
Similarly with MCIL, CM-VAE-BC (Bonatti et al. 22200)) aims at learning joint low-dimensional embed- ding for multiple data modalities. It utilizes CM- VAE (Spurr et al. (2018)), ) where pair of encoder-decoder (Pa (z]s), Po. (s|2)) is trained for each modality i E [1,... .] via cross-modality reconstructions of which the objoctive is:
max PP [log Pai (s)12]] (P (2)s')) 01,033
(40)
Suppose N = 2, then (i.j) in the equation above is an enumeration over the set (1,,),, (2, Note that the equation above is variational lower bound of log Pao (s') and s' & s corresponds to the same true environment stale. This mooss-modal training regime results in a single latent space that allows embed- ding and reconstructing multiple data modalities. After the representation learning, a policy conditioned on the latent variable π (a)22) can be learned with any IL method (e.g., BC), following aforementioned Scheme (2), i.., Eq. (3
3.3.4 VAE-based Skill Acquisition and Hierarchical Imitation Learning
In the context of IL, skills are meaningful subsequences/patten within the expert trajectories, which can be potentially utilized in multiple related tasks. As an example, consider IL for cooking tasks, where the tasks can be diverse but often have overlapping patterns, such as, slicing, chopping, etc. As introduced in OPAL (see Section 3.2.4.. VAEs can be used to extract such patterns via learnt latent embeddings (i.e., using Eq. (31) which is a special case of aforementioned Scheme (3)). As in OPAL, after training, the decoder Pa(a|s, 2) can be viewed as the policy for skill = (i.e., (als, z)), and the encoder (z(z))) can be used to parse the expert data into trajectories of (s, pairs for training the high-level policy Thigh (2])) with any II. method. Then, the hierarchical policy can be executed in call-and-return manner: sampling a skill 2 ~ Thigh executing the corresponding policy a ~ (-w(-)) for h time steps, sampling the next skill, and so on. This whole process can be considered as a hicrarchical extension of BC. Beyond this straightforward use of VAEs, some works have proposed to impose additional structures or properties, such as disentanglement, the latent space for improved skill acquisition (e.g., in interpretability). As mentioned in Section 3.1 being disentangled means that each dimension/ inateniiooppatttiion parttition of the latent variable is independent and semantically meaningful. One way to realize this is to directly incorporate an auxiliary loss term to the standard VAE ELBO regularization:
max
CELBO (De) - A. (Dn), 0,0,00
9.0
(41)
where L CELBO usually takes the form of Eq.
31 ; w is the parameter of an auxiliary network; À >> 0
controls the tradeoff between the two loss terms.
SAILOR (Nasiriany et al. (2022)) defines (Dg) = BGr,T)) Ifw (H(Po(-IT)) (etpallmi..l((((Pa)pp ((Po(-12)))) o where T1 and T2 belong to the same trajectory and are scparated by t time steps, (-)(()) denotes the mean of the encoder output. Such a term can encourage the learned embedding to be predictable (For the temporal difference between skills) and consistent, which is important for downstream policy learning
8This objective ensembles the VIB framework, Le., Eq. 36). but overlooks the regulation on z. Also, training separate encoder for each context. is necessary, since dealing with different modalities requires different foundation models.
25

---

Published in Transactions on Machine Learning Research (08/2024)
with these skills. Note that the gradient from - w.,0,6 (DB) can be backpropagated to the encoder, allowing this term to shape the learned skill embeddings.
Both OPAL and SAILOR would partition trajectories into segments of length & (i.e., the assumed skill horizon) and then train a CVAE on these independent segments for skill acquisition (i.e., with Eq. (31)
CKA Pasula (2020)), on the other hand, proposes a regularization term on the entire trajectory. particular, if a trajectory T can be decomposed into m skills in a sequence, the learned skill embeddings Fimm can be regularized by mutual information term ( - the i-th segment of F. The intuition is that, as a complex behavior, T should be a combination of these skills, By maximizing the mutual information, the interpretability of the learned embeddings for the behavior t can be enhanced. The reason to assume a summation structure >;1 2; as the combination form is to enable a practical estimation of the mutual information term (see (Pasula (2020) )) Aforementioned methods either assume a fixed skill horizon of h or require that the skill segmentation of the trajectory is readily available, thus only learning skill embeddings for each segment. CompILE (Kipf ct al. (2019)) suggests method to concurrently infer both the skill boundaries and skill embeddings directly from the trajectory.
(Ei 24 r), where т; is
A major benefit of skill acquisition lies in the potential for skills to be transferred across multiple related tasks. This transfer can facilitate the learning of multi-task policy, x[a|s, 2),, where 2 and repre- sent the skill and task, respectively. In this case, (z constitutes the policy condition and disentan- glement between 2 and e are required for interpretable and controlled behavior generation. To this end, VC-VAE (Noseworthy et al. (2019)) ) is proposed for the scenario where the task variables e are provided and irrelvreelevnt skill embeddings z need to be learned. A regularization term maxş min C ( = maxş min (na - -llı is adopted, where fw is trained to predict the task from the mean of the encoder u(P)(-IT)). Intuitively, maximizing this objective over ø will encourage the learned latent space 2 to be non-informative about the task c. SKILL-IL (Bian et al. (2022)) ), on the other hand, learns z and e in the meantime as the latent variable of VAE encoded from a set of maulti-task demonstrations. To encourage the discntanglement, a Gated VAE (sce Section 3.1 is adopted, where the latent variable is partitioned inlo two subdomains for z and e respectively. Parameters related to subdomain z are updated on data comprising different skills but within the same task. Similarly, parameters related to subdomain are trained with data corresponding to the same skill but from different tasks.
3.3.5 Tackling Causal Confusion
Causal confusion/misse iitisiiinnmmsssenniffcttoonnn happens when learnt imitator policies become strongly correlated certain nuisance variables present in the task environment, rather than identifying the causal variables that expert demonstrators actually exploited in decision decision-making. Thus, more information in the environment could yield worse IL performance as there would be more nuisance factors. For instance, in a driving task, the driver must brake when encountering obstacles or pedestrians. If the demonstration images include brake light. indicators on the dashboard, imitators might incorrectly associate the braking behavior with these indicators. However, evaluation scenarios may not always feature these indicators, leading to potentially catastrophic outcomes (such as hitting a pedestrian). Therefore, to be maximally robust to distribution shifts (between the training and evaluation scenarios), a policy must rely solely on true causes of expert actions, thereby avoiding causal misidentiffcation.
As a seminal work, RCM-IL (De Haan et al. (2019) resolves causal misidentiiscation in IL by learning a disentangled representation 2 E Rd of the state s, each dimension of which represents a disentangled factor of variation, and then differentiating between nuisance and actual causal factors in z. In particular, a 33-AE is adopted to learn such a disentangled representation = following Eq. (17) (with x replaced by s). After pretraining the 3-VAE on DE, states in Dr can be replaced by their corresponding latent embeddings 2. Each of the d dimensions of a could represent either a causal nuisance factor, and it is necessary to mask out the nuisance factors. The problem is then to find the correct mask vector m E 10. With m, the policy can be defined on the masked representations as π (alm (ulm @ z), where @ denotes element-wise multiplication and m@2 keeps only actual causal factors. De Haan et al. (2019) propose methods for finding the correct m by querying experts or interacting with the environment, which does not rely on the use of VAEs and so is not detailed here. Masked (Pfrommer et al. (2023)) adopts the same protocol as RCM-IL but improves the algorithm for finding the mask vector. Specifically, it uses carefully-designedd statistical
26

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
VAE Type
VAE Usage
Evaluation Task
EIRLI
VAE
Representation Learning (2)
dm control, Procgen, MAGICAL
T-VAE
VAE
Trajectory modeling (3)
2D Navigation, 2D Circle, Minecraft
Basketball Tracking, PEMS-SF Traffic,
RC-VAE
VRNN
Trajectory forecasting (3)
Billiard Ball Trajectory
Behavior
3-VAE
Data augmentation (2)
RoboSuite, PyBullet, Real Robot
Retrieval
VAE-TPIL
VAE
Data augmentation (2)
Minecraft, PyBullet
GIRIL
CVAE
Data augmentation
OpenAI Atari, Pybullet
RGBD-VIB
VIB
Handling multi-modal input (4)
Real Robot
MCIL
VAE
Handling multi-modal imput (1)
3D Playroom
CM-VAE-BC
CM-VAE
Handling multi-modal imput (2)
AirSim, Real Robot
SAILOR
3-E
Skill acquisition (3)
Franka Kitchen, CALVIN
CKA
VAE
Skill acquisition (3)
PyBullet
ComplLE
VAE
Skill acquisition (3)
GridWorld, dm control
TC-VAE
3-VAE
Skill acquisition (3)
Synthetic Arcs, MIME Pouring
SKILL-IL
Gated VAE
Skill acquisition (2)
Craftworld, Real-world Navigation
RCM-IL
B-VAE
Addressing causal confusion (2) OpenAl Gym/MuJJCCa Atari
Addressing causal confusion (2)
Masked
3-VAE
OpenAl Gym/MuJoCo MuJoCo
OREO
VQ-VAE
Addressing causal confusion
OpenAl Atari, CARLA
Table 2: Summary of representative VAE-based IL algorithms. For each algorithm, we detail the type of VAE used, how the VAE is applicd, and the tasks on which it was evaluated. In Column 3, (1)-(4) refer to the 4 schemes of VaE-based IL introduced in Section [.3.1] As for the evaluation tasks, IL algorithms has relatively more diverse choices than offline RL algorithms (as shown in Table D By Real Robot or Real-world Navigation, we mean evaluations on real-world platforms, which vary from one study to another. We provide the references of these benchmarks here: dm cin_cottol (Tassa et al. (2018)), Procgen (Cobbe
2020)), , MAGICAL (Toyer al. (2020)). 2D Navigation/ /Circle (Lu et al. (2019)). Minecraft (Guss et al.
ct al. (019 Basketball Tracking (Felsen ct al. (2018)), PEMS-SF Traffic (Dua et al. (2019)). Billiard Ball
Trajectory (Fragkiadaki et al.] (2016)), RoboSuite (Zhu et al. (2020)). PyBullet (Coumans & Bai 20016
2021b), OpenAl (yym/Atari
AAtari/MuJoCo MuJoCo (Brockman et al. (2016)), 3D Playroom (Lynch et al. (2019)), AirSim
Shah ct al.] (2017)). Franka Kitchen [Gupta et al. (2019a)). ), CALVIN (Mees et al. (2022b)), GridWorld
Zintgraf et al. (2020)), Synthetic Ares (Noseworthy et al. (2019)). MIME Pouring (Sharma et al. (2018)
Craftworld (Devin (2024)) ), CARLA (Dosovitskiy et al. (2017)).
hypothesis testing algorithm to efficiently mask out nuisance variables among the latent dimensions produced by the 8-VAE, which does not require expensive queries to the expert or environment for intervention. Under certain assumptions, this algorithm is guaranteed not to incorrectly mask factors that causally influence the expert. Please see (PTrommer et al. 2023)) for details.
Alternatively, Park et al. 22021 present OREO to ameliorate causal misidentiffcatton in IL, of which the main idea to encourage the policy to uniformly attend to all semantic factors in the state to prevent it from strongly exploiting certain nuisance factors, In particular, a VQ-VAE is trained on states from demonstrations using Eq. (21), where x is replaced with ~ DE- Since there could be multiple (say n) semantic factors within 8, the latent representations from the VQ-VAE, i.e., *E and æD, are set to be 2D matrices in Raxd, where d is the dimension of the representation for each single factor. After pretraining, given a state 8, its multi-factor representation ED can be acquired by querying the closet code for each row of zg: (C1sk is the codcbook, Cj E Rdd, and žg E Rdd is the i-th row of 2g.)
E ~ Po(-(s), zD = (e Mngn) q(i) = arg min || - -2ll2-
(42)
jEll...
The next step is to learn a policy through BC based on the multi-factor ropresentation 2p while reducing the influence of nuisance factors. Specifically, at each state s, a set of k binary random variables M, E {0, 1). e [],... k) are iid sampled from a Bernoulli distribution, and then mask vector m corresponding to can
27

---

Published in Transactions on Machine Learning Research (08/2024)
be created as: [M Ma where ((1,, (((n) are defined as above. Finally, the IL objective is: maxx,f (a.,))-DE., m log x (a)m @ f()),, where I' is initialized with the pretrained encoder Po and updated during the policy training. Suppose the q(i)-th factor is a nuisance factor that is strongly correlated with decision-making, then it would occur frequently in the latent representations, i.c., cT in 2D. By randomly setting the corresponding mask Mg(e) to O, this nuisance factor can be effectively dropped out, thereby mitigating the issue of causal misidentification. Randomized dropping provides a simple yet effective way to regularize the uniform treatment of each generative factor, i.e, either a causal variable that explains expert actions or nuisance variable that just shows strong correlation with the demonstrations.
The representative VAE-based offline RL and IL algorithms are summarized in Table 1 and 2 respectively. We notice that most applications of VAEs in oflline policy learning are based on the learned embedding z. For example, it can be used to identify OOD actions (Section 3.2.2). transform high figh-dimensional states or continuous actions for improved learning efficiency (Soction 3.2.3), represent tasks or skills for multi-task or hicrarchical learning (Section 3.2.4), fuse input from multiple modalitics (Section 3.3.3), provide disentangled representations ions of the states for tackling causal confusion (Section 3.3.5p. and so on. On the other hand, we note that some directions of baseb offline policy learning can be further explored. For instance, introduced in Scheme (2) of Section 3.3.1 the VAE can be used to extract compact reprosentations of the states, which can then be used for learning a representation- peppeeeentation-aseed policy (a|z). However, EIRLI (Chen et al. (2021a)) suggests that reward- and value-predietion would benefit more (than policy learning) from the rep- resentation x that captures mostly coarse-grained visual differences. Thus, it's worth investigating whether VAE-based representation learning would improve offline RL/IL that requires reward- or calue-prediction.. Moreover, RCM-IT, and Masked in Section 3.3.5 assume that 3-VAEs can extract well-disentangledd represen- tations of the states the basis of mitigating causal confusion. However, as noted in Gated VAE (Section 3.1. consistent disentanglement with B-VAEs, such an unsupervised manner, has been demonstrated to be impossible. Thus, further improvements can be made on tackling causal confusion, by using more advanced variants of VAFs for learning disentangled state representations.
4 Generative Adversarial Networks in Offline Policy Learning
Similar to the previous section, we divide the content here into three parts: background on GANs (Section 1.1, the applications of GANs in IL (Section 4.2 and offline RL (Section [.3] Compared to GAN-based offline RL, GAN-based IL algorithms are relatively more. GAN-based IL primarily extends two fundamental IRL algorithms: GAIL, and AIRL, while GAN-based offline RL is mainly based on the model-based offline RL.
4.1 Background on Generative Adversarial Networks
The Generative Adversarial Network (GAN, Goodfellow et al. (0144 ) is a generative model based on the minimax optimization. It consists of a generator G and discriminator D that are trained simultaneously to compete against each other. The generator tries to capture the distribution of real data Px(x) and generate new data examples æ from initial noise discriminate generated examples from real data as accurately as possible. Formally, the generator can be modeled as a differentiable function that maps noise following certain prior distribution Pz(() to samples in the data space X: Pz(+), x G(z) Pa(.). While, the discriminator's output D(z) estimates the probability of data point x being sampled from the true data distribution the generator. The training objective of GAN is formulated as a minimax game between G and D:
(2).). The discriminator is usually a binary classifier used to
Px(+) rather than generated by
min max Be- ()) [log D(x)] + Exw [log[] DDG(=)))
(43)
G
Here, D is trained to maximize the probability of assigning correct labels to both real data and fake ones from the generator: D(x) - 1, D(((z) + 0. For a certain G, the optimal discriminator is given by: (z) = Pott) Plugging this back in Eq. (43). , the minimax problem can be converted to an optimization problem with regard to the generator G: ming JS(P(())(P)P))) where JS(-)) denotes the Sensen-Shannor divergence [Menendez et al. (1997)). Please refer to (Goodfellow al.) (2014a)) for the detailed derivation. Thus, in essence, G is trained to generate samples that match the real data distribution.
(x
28

---

Published in Transactions on Machine Learning Research (08/2024)
Next, we introduce several variants of GANs, as the necessary background for Section 4.2 and 4.3
Conditional GAN (CGAN) & InfoGAN: CGAN (Mirza & Osindero 2014 trains its generator and discriminator together with extra information y as conditions:
min max Ex (-) (log (( + (2) - DD(G(2l)))
(44)
D
Similarly, this objective can be converted to ming JSS( (S(Px( i.e, matching the condi- tional distribution of the real data. With these conditional models, CGAN has the advantage of handling not only unimodal datasets but also multimodal ones like Flickr (Plummer et al.) (015 which consists of diverse labeled image data. With the class label or text as the condition y, CGAN can be used for conditional generation. However, when the dataset contains multiple modalities but the labels y are not given, InfoGAN (Chen et al. (016)) proposes to introduce a latent code e (fol- lowing an assumed distribution Pc(-)) to the generator. In this way, the data generation process is modified into e Pc(+), Pz('), = G(2,c) ~ Pa(+lc), where is for capturing distinct modes in the dataset and targets the unstructured noise shared across the modes. The generator G is trained not only with V(D,G) G) (i.., Eq. (43) but also to maximize the mutual information between the
mode variable C and the respective samples X = G(Z, C): ming max V(D,G) G) - XT(C, Xlc)), such that C)Z,C) would not degenerate to a unimodal model. By replacing the mutual information with its practical lower bound, we have the objective for InfoGAN as below:
max max V(D,G) - G.Q D
[log Q(c[=)] + H(C)]
(45)
Here, Q(rlc) is a variational posterior neural network trained to approximate the real but intractable
posterior distribution P(c|ær, and HC) denotes the entropy of the mode variable C g
J-GAN: A large class of different divergences, including the KL, divergence and JS divergence, are the so-called f-divergences (Csiszár & Shields (2004)). Specifically, given two contimu- ous distributions P(-) and G(.), the f-divergence between them is defined as (,((()))Q())
Q(x) dr, where f: R - IR should be a convex, swver-semicontimuuss function satisfying [
P(x))
f(1) 0 T0 Given a data distribution (x(+), a corresponding generative model G can be Icarned through minimizing (Px (Px())lPa))) As proposed in f-GAN (Nowozin et al. (2016)), this can be approximately solved via minimax optimization problem:
min max Ex (gr(V(z))) + Ef G
(46)
Here, V is a differentiable function without any range constraints on the output, 9f is an output activation function specific to the f-divergence used, and f is the convex conjugate function of f (Hiriart-Urruty & Lemaréchal| (2004)). With specific choices of f, f and gy, various types of f-GAN can be achieved (see Table 6 of (Nowozin et al. (2016))). For example, the original GAN can be recovered by setting (t) - log(1 exx((t)) and (v - log(1 + exp(-t)) II In this way, f-GAN provides a generalization of multiple variants of GANs, including the original GAN, LSGAN, and WGAN.
Least Squares GAN (LSGAN): During the early training stage, the generated examples would substantially differ from the real data, but the original GAN objective provides only very small penalties for updating G, as shown in Figure 16 of (Goodfellow (2017). To overcome this vanishing gradient problem, LSGAN (Ma et al. (2017)) replaces the cross-entropy loss used in the original GAN objective with least-square losses:
min Px () [(D(x) - ]) E [((G(())) - ) min E [((G((=) - k)²] D
(47)
&
(I() = H(C) H(C] -f Po(e) log de f (c(c)) log P(c]e) dedr 2 - Po(c) log Po(e)
J (P(()) log Q(clar) dr. The same trick as in Eq. 13 is adopted here to get this incquality.
As an instance, when f(a) a log a, Dy recovers the KL divergence.
1he discriminator in the original GAN D(z) would be a function of V(x),, Le. D(x) = 1/(1 (1+++00 exx(-V(x))). V(x) e R and
thus (xx) (0, 1).
29

---

Published in Transactions on Machine Learning Research (08/2024)
Here, i.j are the labels for the generated and real examples, respectively: k is the value that G hopes for D to believe for generated examples. As shown in (Mao et al. (2017)) when j - = 2 and j - k = 1, Eq. (47 is equivalent to minimizing the Pearson x² divergence, which is a type of f-divergence, between the real and generated data distribution, i.c., Px(x) and Po(x).
Wasserstein GAN (WGAN): As mentioned carlier, for the original GAN, the optimization with respect to G is equivalent to minimizing the JS divergence JS(Px ()()) WGAN (Ar- jovsky et al. (2017) ) proposes to minimize the Wasserstein distance instead, which is defined as s Px()) [f(x)] - Pot- [f(x)]. Ilfllz & K denotes the set of K-Lipschitz functions. In this case, the objective function of WGAN as below:
min max Exx [D(x)] EawPo() [D(x)]
(48)
Here, D is used to estimate the Wasserstein distance, hence its output is not limited to [o, 1] as in the original GAN. Also, the neural network D is applied with weight clipping, ensuring that D is K-Lipschitz. Compared with the original GAN, WGAN improves the learning stability and provides meaningful learning curves for parameter fine-tuning.
Triple-GAN: To realize data classification and conditional generation in the meantime, Triple- GAN (Li et al. Here, I' and y denote the data and label, respectively. Given a dataset of (z, v), empirical joint and marginal distributions of the real data and labels can be acquired as Px, y), Px(x), and Py))). Px,y (2, y) can be approximated as cither Px (z) (cllz)) or Py Pa y), corresponding to the classification with C' and conditional generation with G, respectively. Thus, C and G can be trained to match the joint distribution of (z,y). Here is the objective:
(2017a)) proposes to introduce classifier C(y to the original GAN framework.
min max B(xx))Pxx)()) log D(x, v)] AcB ( (.wo [log(1 - D(x,,))] C,G (1 (-)a) (as [log(1 - D(z, v))] + ((a))) (- log Pc(ulx)] (49) (aag:r/ww log (y(xr)
The first three terms resemble the GAN framework, where C and G are trained to fool the dis- criminator D by generating (I, close to the real data. The last two (cross entropy) terms pro- vide extra supervision for C' using samples from the real and generated distributions.. Note that the last objective term is only utilized for training C. It shown in (Li et al.) (2017ab) that Px,y (z.y) = Pc(yla) (x(ly)))) if and only if the equilibrium among the three players (C,G, is achieved in Eq. 49
4.2 Generative Adversarial Networks in Imitation Learning
This section starts with a detailed introduction of the fundamental algorithms in this direction: GAIL and AIRL, as a tutorial, and then present extensions for each of them. Finally, we provide a spotlight on the integrated use of GANs and VAËs for IL, as examples of synthesizing the use of different generative models for offline policy learning.
4.2.1 Fundamental GAN-based Imitation Learning Algorithms: GAIL and AIRL
Nearly all works regarding applying GANs for IL can be viewed as extensions of GAIL Ho & Ermon 2016
or AIRL (Fu et al. (2017)). Both works are based on the maximum causal entropy inverse RL IRLL
framework (Ziebart et al. (210 ) of which the objective is as below:
max min H(T) + ((s.))a ()le(s. a)] B(s,))= ((ss)>>sg(((((s....)
(50)
n
Here, (s) = (a]s) / PPss s|x) is the discounted occupancy mcasure (i.c., visit frequency) of (s, a) when taking म. and (xx) =E (( log [[]ss]] denotes the policy entropy. Intuitively, this framework looks for a cost function that assigns low costs to the expert policy ae and high costs any
30

---

Published in Transactions on Machine Learning Research (08/2024)
other policy, thereby allowing the expert policy to be learned via minimizing the expected cost, ie., through the inner RL process in Eq. 50 (i.e., minz). TE is not directly accessible and usually represented as a set of expert demonstrations.
GAIL proposes that, to ensure the expressiveness of c while avoiding overfit on the set of demonstrations, e can be any real function (i.e., R5xA) but an extra regularizer dà RRSA -> IR on e should be introduced to This (-rrgularized MaxEntIRL problem is equivalent to an occupancy measure matching problem:
Eq.
50
min (T) + (px -PE), 3² (pa PRE) sup (px -Pgg) c- vi(c)
(51)
CERSXA
That is, al-regularized MaxEntIRL implicitly seeks policy, whose occupancy measure is close to the expert's, as measured by the convex conjugate of the regularizer, i.e., t²" As introduced in Section 4.1 GANs provide a scalable manner for distribution matching. GAIL proposes that, with a certain design of a (i.e., Eq. (13)
in (Ho & Ermon (0166))) )), Eq. (51) can be converted to a GAN objective:
min max H(T) (()a))p..))0o(( D(s,a)) E(s,)) O ((o a)] DE(A,1)5xA
(52)
Compared with Eq. (50 we can see that c(s, a) is defined with the discriminator as log(1 (ss, a)) the policy x works as the generator G, so D and x can be alternatively trained as in GANs to optimize Eq. 52p. Specifically, in each training episode, π is trained for several iterations to maximize the expected return Q(50: ao) = -= - (- (1
A Also,
(a)]ss)), V(so. an), with a widely-adopted RL algorithm TRPO (Schulman et al. 2015 13 and then D trained for iterations to correctly classify samples from π or TE with Eq. (52). Compared with the original MaxEntIRL, which requires solving a complete RL problem as the inner loop for each update of e, GAIL ensures both expressiveness and scalability, especially useful for tasks in high-dimenssonal state/aud state/action /action spaces. Further, GAIL provides solid theoretical results on the connection between IRL and GANs. As for the global optimality and convergence rate of GAIL, Zhang el al. (220b)) provide an analysis under the condition of using two-layer neural networks (with ReLU between the two layers) as function estimators and applying natural policy gradient for policy updates, when the underlying MDP belongs to the class of linear MDPs.
- (5,,aa)) a.)) - A log x xass [s)),
AIRL also adopts a GAN-like framework as follows:
max B(s,a)=p= () [lg((1 - D(s, a))] + ((s,a))p a looo D(s,
D
(53)
min E () D(s, a)) log (ss
However, instead of applying a sigmoid function (i.e., a)s explf(saa) ) as the activation func- exp(f(s.0))+1
tion on the last layer output (i.e., f(s,a)) of the discriminator as in GAN and GAIL, AIRL adopts a) = (xp(((.)))))(a)))) They claim that, with this design, AIRL is equivalent to MaxEntIRLL, un- (s0, du.... ,8, 81+1) follow the Boltzmann distribution (i.e., der the assumption that trajectories a:)))) However, the derivation provided in (Fu al. (017))
explf(e.a))
P(T) & po(so) D T(s411 at) exp(>"r(sx,_)) (x)p:lsss,))exp>(((((.....)
is not rigorous 14 Further, they propose to replace ass,a) with D(s,a,
where [s,,,, 6') = g(0) >h((') h(s). They state that, at optimality, g(s) and h(s) can recover the reward
exp(f(s.a.x))
and value function, respectively, if the real reward function a function of s only and the environment is
deterministic While its theoretical foundation is not robust, AIRL can be considered
GAN-based IT.
algorithm drawing inspiration from MaXEntIRL and well substantiated by empirical results.
We equivalently substitute all D(s a) in (Ho & Ermon (2016)) as 1 - D(s, a) to be in line with GAN, where the true and fake data should be labeled as and 0, respectively, by the discriminator D. a a x (ailst)) is equivalent to ming X 0 is introduced here to balance the two objective terms.
max= Q(50. a0) "0 oz
In Appendix A.1 of (Fu et al. (2017)), ), the step @ log Zo = Epg Er 20 as) is questionable. Also, in its Appendix
b0
A. they erroncously mix up the
of and jic.
in Appendix A.4 of (Fu ct al. (0177), they claim that the global minimumn of the discriminator objective is reached when Tg, which is backed up by the original GAN paper (Goodlellow et al. (2014b)). However, the objective design of AIRL is significantly distinet from the original GAN. Moreover, this claim forms the basis the proof presented in its Appendix C g(s) and Ms) recovering the reward and value functions respectively.
31

---

Published in Transactions on Machine Learning Research (08/2024)
There is an extensive amount of works developed based GAIL or AIRL. Instead of providing details each of them, we put an emphasis on the representative ones by presenting their objective designs and key novelties in tables, and briefly enumerate other works for comprehensiveness.
4.2.2 Extensions of GAIL
we provide an overview of represenlative works that extend GAIL. They either substitule the original GAN framework with more advanced GAN variants (as introduced in Section 4.1 or expand the use of GAIL to other IL settings, including the multi-agent, multi-task, hierarchical, and model-based scenarios. Next, to provide a thorough review, we briefly enumerate other related works, which are further categorized by their base algorithms.
In Table 3
Extensions of GAIL concentrate on tackling key challenges such as increasing the sample efficiency for policy learning, ensuring balanced training of the discriminator and policy, etc. (1) DGAIL (Zuo
GAIL:
et al. (2020)) and SAM (Blondé & Kalousis (2019)) replace the stochastic policy ~ m(-[s) in GATI.
with a deterministic one a = =sss) and, correspondingly, proposes a DDPG-based (Lillicrap et al. (2016) updating rule for л. As an ofl-policy algorithm, DDPG is more sample efficient than TRPO (used in GAIL).
BGAIL (Jeon et al. (2018)) improves the sample efficiency by approximating the posterior distribution of the discriminator's s paramcters in the idcal case where correct labels are assigned to both export and generated behaviors. A more accurate estimation of the cost, function can then be acquired through sampling from that distribution, consequently improving the policy training. (2) VAIL (Peng al. (2019a)) suggests limiting information flow in the discriminator of GAIL (or AIRL) using the Variational Information Bottleneck framework (Tishby & Zaslavsky (2015)), to prevent the discriminator from converging too quickly and so providing uninformative gradients to the generator. TRAIL (Zolna et al. (2020)) proposes to regularize the discriminator through an auxiliary task, with the aim to discourage the agent to exploit spurious patterns in observations which are associated with the expert label but task-irrelevant. (3) For applications demanding reliability and robustness, RS-GAAL (Lacotte et al. (2019)) presents a risk-sensitive version of GAIL by introducing a constraint to the original GAIL. objective, such that the conditional value-at-risk (Duffie & Pan 1997) of the learned policy is at least as well as that of the expert. This algorithm is supported by rigorous theoretical validation.
Multi-agent/Hieraann /Hierarchical GAIL: IGASIL (Hao et al. (009)) extends GAIL to fully-collaborative multi- agent scenarios. It adopts a similar objective design with MAGAIL's (listed in Table 3 but suggests two major improvements for learning efficiency: training the policy with DDPG for enhanced sample elficiency and adopting high-return trajectories from the policy as demonstrations for self-imitation learning. Like Option-GAIL introduced in Table OptionGAN and Directed-Info GAIL extend GAIL to the hicrar- chical learning setting, laking ad advantage of the Mixture-of-Expertss framework (Masouduia & Ebrahimpour (2014)) and directed information maximization (Masscy et al. (1990)) ), respectively. However, these two algo- rithmn are not suitable for learning from demonstrations segmented by sub-tasks. Also, Directed-Info GAIL trains TH and Ty in two separate stages, leading to suboptimality of the hicrarchical policy π = (TH, #L).
InfoG Hausman et al. (2177) and Peng et al. (202)) address the same challenge as InfoGAII. i.c., IL from multi-modal but unstructured demonstrations. Hausman et al.] (0117) achieve the same objective function as InfoGAIL but derive it through distinct perspective; Peng et al. (2022) proposes alternative manner to update Q (in InfoGAIT) through VAE framework, where Q and т serve as the encoder and decoder, respectively. AIlL (Fu et al.] (2023) extends InfoGAIL to manage demonstrations from a mixture of experts, wherein the quantity of demonstrations from each expert is imbalanced. Burn- IIL (Kuefter& & Kochenderfer (2018) improves InfoGAIL to reproduce expert behaviors over extended time horizons.
4.2.3 Extensions of AIRL
Extensions of ATRL are relatively fewer, but we follow the content structure in Section 4.2.2 providing a summarization table as Table and a brief review of other related works as below.
AIRL adopts the function design f(s, a, s') = +)))) hh(s, where g(s) is assumed to recover the reward function and h(s) can be viewed as the potential function for reward shaping (Ng et al. (1999) EAIRL
32

---

Published in Transactions on Machine Learning Research (08/2024)
Algo (GAN Type)
Objective
Key Novelty
Recovering a policy x (a)s,c) for each condition o; Demonstrations are sharcable among similar task conditions, so the data efficiency can be high: max E Bp (-le) [log(1 - D(s, alc')] discourages the mismatch between and (e,a) for consistency. Demonstrations are multi-modal but labels c are not providod, unlike CGAIL;
CGAIL (CGAN)
ming maxp - (x-((()))) + E ( Bog ((s, (p) loo(11 D(s, a|c)]]+
Zhang et al. 2019
E P (+le) Dog(1 - a(ss)
min,, Q maxp Hx) + F.p, [log(1 (s).)))]++
Lr Q) is a lower bound of the mutual information [(e; a)), by maximizing which we can disentangle trajectories corresponding to different Policies for each latent modality x(als, c) can then be recovered.
InfoGAIL (InfoGAN)
Eprg [log D(s.a) (T,
LI(T,, Q) Po(,)o() [log Q(cls, a)] H(C),
al. 2017b)
Po(() is assumed prior distribution.
min, max Eprg (T(), Epa (T()
GAIL is special case of f-GAIL, when T(x) = log D(z), f'(x)=
(*),, f is the convex conjugate of f (which defines a certain f-divergence) and satisfies: convexity and unf (f" - u} = U.
f-GAIL
log(1 c²);
(f-GAN)
Maximizing over f E J aims to find the largest f-divergence between Per and Prg: which can better guide
Zhang al. 2020a
Like and T, f is modeled as neural network but specially designed fulfill its two constraints.
wg.
The development and theoretical validation Triple-GA GAIL exactly follow Triple-GAN (see Section 4.1). C' and x are trained by matching the approximated joint occupancy measures of (8, a,ee (r(+) and pc()), with prg (-). The last two objective terms serve as extra supervision for C. Viewing e as options/ aptions/skills, C and iT then constitute hicrarchical policy.
minl, C maxp ((T) Epeg [log D(s, c)]+
ApBpa [log(1 - D(s, a, c))
Tril GAIL (Triple-GAN
Aa Ag)R [log(1
a())
Ap log C(c) a)] + Ag log C(cls, a)]: can be viewed as labels for (s,a), provided in
Fei et al.
2020
demonstrations; C(cls, a) is utilized to classify (s, a).
while =[als, is the policy for the class
The objective of D is the same as the one of GAIL. x [)) is trained by maximixing the return (sn, an; 8), where the policy gradient, i.., J8. can be appracimated recursively as: (t=
This is model-based version of GAIL. A dynamic model f learned, such that the agent can look one-step ahead to additionally adopt the gradient from J,
MGAIL (GAN)
J/ = Ramp + 7 + J [usss) J = R. + RaTs + +.Jett (fa + faTT);
ie., 73 fuuo, as part of Jr. (Typically, Já = Remo + + .) Thus, fewer expert data and policy samples would be required, compared with GAIL.
Baram et al. 2017
f the learned dynamics: s' f(s,a), R(s,aa)
is the reward defined with D, R. _ 0R and so on. Hy
mina max Epp EEM, log(1 (s))): +
A multi-agent extension of GAIL, which learns D; and ny for each agent. As agents interact with each other, D, and n, are not learned independently. The paper's 's theoretical results assume "g is known, cuabling the learning for each agent i to he treated separately. Yet, in practice, better management for the non-stationarity than MAGAIL is required.
Epng D log a)(s, ग (m TN); W (M g..... N.
MAGAIL (GAN)
All agents share the state but choose their own action a; ms(-1s).
Song et al. 2018
The other agents -i can be viewed as part of the environment for agent i. However, if m=: unknown, the training environment would be non-stationary for i, as T= are also being updated.
A hierarchical learning version of GAIL. Long-horizon tasks can be segmented to a sequence of sub-tasks, each of which can be done with an option (l.e. sub-policy). This algorithm can recover hierarchical policy, useful for long-horizon horizon tasks, from the expert data through matching the occupancy measure (o/, 0, a) between x and Tg, which follows GAIL. An EM version of Option-GAlL is proposed, in case the expert's 's options are not labeled.
min, maxp -XH(m))+ + Ep, - D(o', D a.)
Epig [log s,),), π (TH.TL)).
Option-GAIi GAIT. (GAN)
o TH (-)s a ~ ()),
The agent decides on its current option (a.k., skill) with bh based on 8 and of (i.e., the previous option), and then samples actions a with St subject to a.
Jing et al. 2021
Table 3: Summary of representative works following GAIL. In Column 1, we list the algorithms and the type of GANs they utilize. Their key objectives and novelties are listed in Column 2 and 3, respectively.
33

---

Published in Transactions on Machine Learning Research (08/2024)
Algo (GAN Type)
Objective
Key Novelty
max E Bampeg () log D(s]] B)) ()) log[1 D(s)
'This work provides generalization of AIRL by applying the the general f-divergence. The reward ra()) is trained by state marginal matching,
ming (f) (t) = (ptR (s) po(s) x
S pol5o) Il=o [(s++1 (1, az) exp( (a(+c11) (s)dr,
(s) = -,== 1(x== =);;
f-IRL (GAN)
i.c, ming Dy(PT (s))) used to estimate u, (0),(0), based on the fact:
The training alternates between maxp and ming;
Ni et al. 2020
With the learned reward function re, л is trained by RL. ming (0) is realized by gradient descents: (o oT cow p-ppt-)
(*) = (s)( (s) po(s)) So, each update of 0 requires a near- optimal D", which is time-consuming,
EE [(ux) (u( En Voro(*)))
prg((s) D(a))
po(as) I- D-a(a cov(-) denotes covariance.
This is a multi-agent version of AIRL, which trains D., n. and applies AIRL to each agent. States and joint actions are shared among agents. Theoretical results are based on Markov games and logistic stochastic best response equilibirum. However, viewing a-; as part of the environment, the problem and derivations degenerate
maxp ΣN Epr [log Di a)l Ep, [log(1 D,(s, miny Bp, Blog(1 D)( a)) log Di(s. all 1, NN: o) = exp(f(s.a)) ((.. + ()) a = (aI,.... .N); म = (-1,... ,.)) All agents share a state and have access to action decisions from the other agents, ie., a-s, during the training process.
MA-AIRL
(GAN)
Yu et al. 2019a)
the single-agent AIRL case.
This algorithm integrates multi-task learning. hierarchical learning, and ATRL. n a hierarchical policy that can be applied to multiple tasks by conditioning on corresponding c. The objective design can be viewed as AIRL on the extended state-action space (5r, As regularization, is trained to maximize the mutual /directed information [(T=; C) [[(T= Ouur|C) to build the causal relationship between and the task /option variables.
maxp Exg 21=0 log D(s,, arlc) Ex Er=
log(1 - ((t,, āz (e); (st, āt) = ((0 81), (oz, as));
ming Ex 2 log(1 D2) log Di
Ou. (C) Ta = [(s, at 1-0:T-1 A1 C) - A2 ((..
MH-AIRL
(GAN)
(L) and T(āe 5t,c) c) = TH iss 1 e.). and
Df (e))
Chen et al. (023cl)
denote the task and option variable, respectively. MH-AIRL can also be adopted, when and o
not labeled in demonstartions, via Expectation Maximization (EM) design.
Table 4: Summary of representative works following AIRL.
Qureshi et al. (019h) proposes to implement h(s) as the empowerment function (Mohamed & Rezende (2015)), maximizing which would induce an intrinsic motivation for the agent to seek the states that have the highest number of future reachable states. However, as claimed in AIRL, h(()) is designed to recover the value function at optimality, which is different from the empowerment function in definition. AIRL has been extended to various II. sctups, including model-basod, off- off-policy, multi-task, and hierarchical IL. MAIRL (Sun et al. (001 ) simply replaces the GAIL objective in MGAIL (introduced in Table 3) with the AIRL's and gets a model-based version of AIRL. Following the same intuition as DGAIL and SAM (introduced Ci-policyAARIL. suggests adopting off- off-policy RL algorithms (specially, SAC (Haarnoja 2018 ))) to train the generator æ in AIRL for improved sample efficiency. Table introduccs MH-AIRL, ct al.
in Section 4.2.2)
which exlends AIRL to support both multi-task and hierarchical learning. As related works, PEMIRL (Yu 2219bb) and SMILe /Ghasemipour et al. (2019a)) are proposed for multi-task AIRL, while o.RL
et al.
(Venuto et al. 2020)) and H-AIRL (Chen et al. (2023d)) focus on hierarchical AIRL.
4.2.4 Integrating VAES and GANs for Imitation Learning: A Spotlight
Lastly, spotlight some papers that utilize VAEs to enhance GAIL, serving as examples of integrating different generative models for advanced imitation learning. GANs are recognized for their capacity to generate sharp image samples, as opposed to the blurrier samples from VAE models. However, GANs, unlike VAEs, are susceptible to the mode collapse problem, meaning that they may only capture partial modes in a multi-moda dataset. Therefore, the abilities of GANs and VAEs are highly complementary. As mentioned in Table 3. one approach, such as CGAIL and InfoGAIL, for solving the mode collapse issuc is to train a discriminator and policy for each mode by conditioning them on the mode embedding. Such embeddings could be acquired from pretrained VAE. In particular, VAE (Pa. Po) can be trained to
34

---

Published in Transactions on Machine Learning Research (08/2024)
imitate expert trajectories (su, du. sT) ~ DE through an ELBO objective:
max DE [BowPa()rr Dog Po(Tlo)] (Ru(Po(cc))PP((O))
(54)
8,4
where Pc(c) is the predefined prior distribution, Pa(cT) and Po(Tlc) work the encoder and decoder of the VAE. log (r(e) can be decomposed as log Pa((olc) + Σ [log mo (ar|st, c) + log To 1 |st, (t, ar,e] based on the MDP model. Through this unsupervised training process, the VAE is expected to capture the multiple modes in the dataset and its encoder Po(cl7) can be employed to provide mode embeddings for the expert trajectories. Then, the mode-conditioned discriminator and policy can be trained via an objective similar with CGAIL:
min max Dg. D
log D(s a [cle]] + (lo) D(s, (H(()))))
(55)
T
The pretrained VAE policy no Talas c) can be used to initialize the GAIL policy (als,, c). Through further training with GAIL (i.e., Eq. (55)), the VAE policy can be refined by addressing the tendency to produce blurry imitations. Diverse GAIL (Wang et al. (2017) and VAE-ADAIL (Lu & Tompson (2020)) are notable examples of this paradigm. A more straightforward manner to integrate VAEs and GANs for IL is adding regularizer min, E (-) Dx (x((()) [[ (a)a((a))) to the GATI, objective, where (V)e(a))) is the VAE-based policy pretrained on DE (see Section 3.3.1). In this way, the GAIL policy ж is encouraged to be close to the potentially multi-modal VAE policy. SAIL (Liu et al. (2020)) explores in this direction. Another significant advantage of VAFs over GANs is that they can provide semantically meaningful latent embeddings for the input data. As introduced Section 3.2.3] such embeddings can be used as transformed training data to lower the difficulty of GAIL training. As an example, LAPAL (Wang et al.) (2022b)) proposes to adopt CVAE to transform high-dimensional actions into low-dimensional latent vectors and apply GAIL, on the latent space instead, in order to stabilize and accelerate the learning process.
4.3 Generative Adversarial Networks in Offline Reinforcement Learning
For model-based offline RL (i.e., Section 2.1.2). ; both a policy and a world model of the environment need to be learned from the offline dataset. The generator G in GAN framework can be trained to approximate the policy or environment model, as detailed in Section 4.3.
and 4.3.2 respectively.
4.3.1 Policy Approximation using GANs
based offline RL, avoid OOD states and actions, the learned policy To should be close to the underlying behavior policy μ4 which is typically realized through introducing a regularization term. Thus, the off- off-po learning objective in model-based offline RL (i.e., the first objective in Eq. can be improved as:
As in
max Ay Dang (-18) (Qo(s, a)] [(TO,
(56)
where λy 0 is the regularization coefficient. SGBCQ DDogg et al. 2023 proposes to directly con- strain the action distribution of ng at each state to be close to the one of by implementing f /4) as EonD, [d( ( (-|s)]lp(-|ss), where d(-) denotes statistical divergence, resembling policy constraint methods (i.e., Eq. 2 ). As introduced in Section 4.1 GANs can be used to model and minimize the JS divergence between two data distributions. Thus, SGBCQ gives out a practical framework to solve Eq.
(56 based on
CGAN as below:
max (s,) [lo D(als) + E. log(1 - D(G(((s))))
D
(57)
min AQ - P2()) G(z]s))
[log(1 (0((gg(((1GGG(s)) D(G(zls))
Compared with the CGAN objective (i.c., Eq. (44 s and a here work as the conditional information y and data point x, respectively. Intuitively, the stochastic policy a na(-Is) is implemented as generator G(z[s), 2 Pz()) and the generator is trained to maximize the expected Q-values and fool the discriminator
35

---

Published in Transactions on Machine Learning Research (08/2024)
D at the same time. Involving such GAN training process encourages the policy to generate behaviors close to the demonstrated ones in distribution.
SDM-GAN (Yang et al. (2022b)) follows similar protocol but chooses to regularize the stationary (s, a) distribution induced by Te to be close to the (s, a) distribution in D2 to avoid OOD behaviors. Specifically, they define [((To.1) in Eq. (57] as d(p] (11% (-)), where P (s, a) can be approximated as the empirical distribution of (s,a) in Das Pro (s, = limm +o0 T#I Et= P(s+ = 8, at = also po(-), at mo(-1se); 81+1 T(-sst, a:)) is the stationary (s,a) distribution of Te under the real dynamic T. Assuming the dynamic function T is well-fitted (through the process shown in Section 2.1.2 , d(p% (-)llp% (+) is upper bounded by:
(Please refer to (Yang et al. (202bb)) for the derivation and demiiiions of the function classes G1
sup (tea))p000ll(0 gr-gn
E 2 ( 0(8, ,, d')]l+ sup [B(e (l9((, a)] Eonpg ()) a)]l g-G62
(58)
The equation above is in a similar form with the Wasserstein distance (defined in Section 4.1 between samples stored in D2 and generated by no, which inspires the use of GANs for the estimation. In particular, the objective is as below:
max Bio.a) llo D(s, a)l + ((s.a) [log(1 - D(s, a))l,
D
(59)
min AQ Daug
De aw Px() (Q0(s, G(2ls)) + E Do D(s,
Here, inspired by Eq.
58 Da contains (s, (s', a') pairs collected in this way: 8 ~ Djr a =o(-)s),
((1ss, a), d' T (=(ss"),, where a (T(-ss) is implemented as a G(zls), Pz(-). The only difference between Eq. (57) and [59 is thus the way to generate samples. For Eq. (57 the (s, a) pairs are simply
generated by
D u. a Te(-|s). The same authors have proposed model-free variant of this algorithm Yang et al. (202cc) ), sharing similar motivations and designs.
AMPL Yang et al. ))22dh) adopts the same objective (i.e., Eq. 59h) for training To, but proposes that the environment models T and f can be periodically updated by collecting new data with Ty and applying supervised learning (e.g., with Eq. 7 to update T and A. Interestingly, they theoretically show that the objective for To, which combines the Q-function and distribution discrepancy, approximates an upper bound for (T,, Le, the negative expected return of the policy on the real MDP. On the other hand, DASCO (Vuong et al.] (2022)) proposes that the two terms in the objective of me may conflict with each other, since fooling the discriminator requires mimicking all in-distribution actions, which can be suboptimal, but maximizing the Q function would mean avoiding low-return behaviors. Thus, they propose to introduce auxiliary generator Gaux to generate low-return samples and modify the objective as follows:
min max ((,a))D, (log D(s, a)] + E(s,a) Gaux [log(1 D(s, DO P2()) 1oo(s, G(z[s))] (60) G,Gaux D
Here, samples in De, are generated through: 8 ~ D,,, Pz(+), (G(z|s) + Gznx(2|s)) Both G and Gaux are adopted to mimic the demonstrations, but only the primary generator G is trained to maximize the Q-values. Theoretically, they prove that the optimal solution for the primary generator G will maximize the probability mass of in-support samples that maximize the Q-function.
All aforementioned algorithms utilize very similar objective design, which can be viewed as paradigm of using GANs for policy approximations in (model-based) offline RL. GOPlan (aagg et al. (023aa) proposes an advantage-weighter CGAN objective, which is different in form from previous ones, for capturing the modal action distribution in goal-conditioned offiine planning:
min max Eg-D, [B(s.a)-Dg [wr(s, g) log D(s)agg) + B.n.D (z) (lg(( - Dss, G(=
(61)
G
Here, Dg denotes the partition of D,. that takes as the goal; w((s, g) exp(A*((e a, g)) denotes the weight function, where (s, g) is a separately trained advantage function. This mechanism encourages the policy to produce actions that closely resemble high-quality (i.c., high-advantage) actions from the offline datasot, but no theoretical guarantee is provided. Although algorithms introduced in this category are proposed for model-based offline RI., these methods can be transferred to the model-free setting seamlessly.
36

---

Published in Transactions on Machine Learning Research (08/2024)
4.3.2 World Model Approximation using GANs
Beyond policy modeling, GANs can also be used to approximate the environment model M, as illustrated in MOAN (Yang el al. (2023a)), ) TS (Hepburn & Montana (2022)), S2P (Cho et al. (2022)), which expected
to outperform traditional supervised learning methods (i.e., using Eq. 77) To be specific, MOAN proposes the following objectíve:
max D min - AN AN (- G(r, [s) + (1. (log(1 - D(s,a,
log a,, +
[log(1 - a,,, 8'))
(62)
log
This design is similar with Eq. (i.e., log s's,, that is commonly used for environment model approximation. In this CGAN frame- work, the conditional generator G(-ls,) is learned to predict 8' and r, working as the approximated reward f(s, a) and transition function ((s' s,aa). a). In particular, the generator G(-ls,a) N (mmeans,,a) std a)) is implemented
59 in form, but replaces the Q-function term with a negative log-likelihood
stochastic Gaussian model, and the standard deviation of the predictions, i.., std(ss a), can be used as uncertainty measure (i.e., a(s,a) in Section 2.1.2). MOAN proposes to further include the confidence level given by the discriminator, i.e., u(s. = std a) + /2(11 )(F(s"))G(()aa)) (s) a, f, 8)). In- tuitively, if the variance of the cstimation is high or the discriminator classifies the prediction as generated data, i.e., D + O, the uncertainty level at this point (s, a) should be high. However, this uncertainty measure design is not backed up by theoretical analysis.
TS proposes to learn environment models for stitching high-value segments from different trajectorics to form higher-quality trajectories for offline RL. To realize this, they model the distribution P(a, r, s'(ss P(s')s))Paas' s) P(rla,, s',),,,ss)) s). That is, they search for a next state s' from the neighbourhood of 8, which
can come from a different trajectory and should have a higher probability P(s"Is) & value V(s') than the original next state of s. Then, they identify the most probable intermidiate a, r. In particular, they learn P(s)(s); P(als', s), P(rla,s',s) 8) with the traditional supervised learning, CVAF, and WGAN (see Section respectivcly. The WGAN objective is: (G(2)s,a,,s) works as the approximation of P(rla,s',s))
4.1
min max Es, (D(s, r)] D...s ((a)s,,)
a, G(2)s,,,,'))
(63)
This algorithm can be viewed as model-based data augmentation method for offline RL using GANs. S2P, on the other hand, is model learning approach for vision-based offline RI. It first approximates the transition and reward model (i.e., Tss'|s, (), a(), for the underlying states through traditional supervised learning, and then learns a generator to synthesize the image T/T' that perfectly represents the corresponding state s/s'. The generator is conditioned on both the current state and previous image, i.e., I' = G(z|s, I), and trained within a WGAN framework. Here, the GAN is used for data transformation simplify the (vision-based) environment model approximation.
In Table 5 we provide a summary of GAN-based offline RL algorithms. GANs can be employed for both policy and world model approximations in model-based offline RL, primarily due to their ability to match a distribution (hidden in the offline data) by minimizing some statistical discrepancy.
5 Normalizing Flows in Offline Policy Learning
Applications of Normalizing Flows (NFs) in offline policy learning are relatively few, particularly for NF- based offline RL. Thus, in Section 5.3. we slightly broaden the scope to include NF-based online RL methods that utilize offline data.
5.1 Background on Normalizing Flows
Normalizing Flows (NFs, Kobyzev et al. (201)) allow for generating data samples following complex distri- butions and exact density estimation of given data points, which are essential in probabilistic modeling tasks. Formally, NFs transform simple, known probability distribution in the latent space (i.e., 2 ~ Pz(-)) to a complex, desired data distribution (i.e., x Px(.)) through sequence of invertible and differentiable
37

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
GAN Type
GAN Usage
Evaluation Task
SGBCQ
CGAN
Policy
D4RL (L)
SDM-GAN
GAN
Policy
DARL (L, M, A)
AMPL
GAN
Policy
DARL (I, M, A)
DASCO
GAN (auxiliary generator)
Policy
D4RL (L, M)
GOPlan
CGAN (advantage- weighted) )
Policy
MuJoCo Robotic Manipulation
MOAN
GAN
Environment Model
D4RL (L)
TS
WGAN
Environment Model
DARL (L)
S2P
WGAN
Environment Model
dm Lim_contooo
Table 5: Summary of GAN- offiine RL algorithms. Most algorithms in this category have been evaluated on D4RL (Fu et al. (2200), ) which provides multiple datasets for data-driven RL tasks, including Locomotion (L), AntMaze (M), Adroit (A), etc. MuJoCo Robotic Manipulation (Yang ct al. (2023b)) provides offline datasets for a series of goal-conditioned robotic manipulation lasks, several of which can be utilized to assess
the OOD generalization capabilities of the policy. dm control (Tassa et al. (2018)) includes 6 environments, which are typically used for evaluating vision-based RL algorithms.
transformations. The mathematical representation of the transformation from x to 2 is given by a series of bijections: F -= FN o FN .... o Fj. The data flow ( Xt, .N]] (2o = x, IN = 2) within NFs adheres to the following process: for all 1 i N, I₁ = -1) x;-1 = F, FF'((,)) (z,). The core principle behind the transformation is the change of variable formula. Specifically, the probability density functions (PDF) of x and z are related as:
pz(z)= Px det D(F
Px(x) = det det
(64)
where det (1F( denotes the determinant of the Jacobian matrix of F(z). Given the transformation
function F and the basic distribution Pz, the densily of a data sample x can be exactly acquired as Px (x) defined in Eq. 64 Thus, the ability in exact density estimation of NFs relies on the fact that F is invertible and differentiable. Converscly, the data generation can be done by first sampling latent ~ (z(+) and then apply the generator function G = F-1. As for training, it is usually through maximizing the liglliielihood of the target data distribution (Dinh et al. (2017)):
max Ex (Px log(Px(z)) = log( log(P2 F
+ log(1 det D(F(z))
(65)
For NEs, the generator G should be sulficiently expressive to accurately model the distribution of interest, while the transformation F should be invertible and the computation of the two directions mentioned above (i.c., data generation and density estimation), particularly the determinant calculation, must be efficient. Based on these requirements, various types of flows have been developed. Here, we briefly present some representative variants. As mentioned above, G is composed of a series of generator modules G, = F For each variant, we introduce its specific design of G., and we use m and n to denote the input and output of G, respectively. Note that we do not present complex mathematical details rogarding calculations or objective designs, as they are not essential for understanding the applications of NFs in oflline policy learning, which are introduced in Section 5.2 and 5.3
Coupling Flows enable highly expressive transformations via a coupling method. In particular, the input m E R is partitioned into two subspaces: Un4,m E Rd RD-dd and the generator module G₂ is defined in the format: G.(m) (H (mm;; O(m m m²) Here, H(:0)) Rd + Rd is bijection, and θ is the parameter function. И of such form is also called the coupling function. In this case, the Jacobian of G2 is simply the Jacobian of H, and for efficient computation of determinants, H is usually designed to be an element-wise bijection, i.e., H(m 0) = (H1 (muf; 0,), ... Ha(m 0a)) where each H.(:; 0,) : R + IR is a scalar bijection. The expressiveness of a coupling flow lies in the complexity of its parameter function O which, in practice, is usually modeled as noural networks. Algorithms in this category include NICE (Dinh et al. (2015)). RealNVP (Dinh et al. (2017)). Glow etc.
(Kingma & Dhariwal (2018)) ) NSF (Durkan et al. (2019)), ), Flow++ (Ho et al. (2019)
38

---

Published in Transactions on Machine Learning Research (08/2024)
Autoregressive Flows utilize autoregressive models for generation. To be specific, each entry of = -m) conditions on the previous entries of the input:
H (m;; O (m):=1)), where H (:: 0) R - IR is a scalar bijection and O is a parameter function. In this case, each n4 relies solely on mi:t: resulting in a triangular Jacobian maxtrix of G. This structure allows for efficient computation of the determinant and parallel computation of each entry in n. However, the inverse process; represented as my -n: O2 (m1:e-1)) must be performed sequentially, since the parameter for the subsoquent (n4;
step (i.e., 2.41 (miz)) uses the output of the current step, i.e., my; as input. Algorithms like LAF Kingma et al. (0016)) ) MAF (Papamakarios et al. (017 and NAF (Huang et al. (2018)) fall in
this category.
Residual Flows employ residual networks (He et al. (2016)) as the generator component, char- acterized by G((m) m + R(m). Here, R(-) is the residual block, which can be implemented as any type of neural network that is invertible. Several studies (Gomez al. (2017); Chang et al.)
(2018); Jacobsen et al. (2018)) have tried to develop invertible network architectures suitable for use residual blocks. Nonetheless, these architectures present significant challenge in efficiently calculating the Jacobian determinants. Research works such as iResNet (Behrmann et al. (019 and Residual Flow (Chen et al. (2019h) propose an invertible G; by limiting the Lipschitz constant. of the residual block to be less than 1 and compute its inverse through ixed-point iterations. However, controlling the Lipschitz constant of a neural network is also quite challenging.
ODE-based Flows aim to learn continuous dynamic system shown first-order ordinary differential equation (ODE): ze m(() = A(m((t;((t)) t [o, 1J. It's worthy noting that the residual flows mentioned above can be interpreted as a discretization of this ODE. Assuming the uniform Lipschitz continuity in m and setting the initial condition as m(0) = 2, the solution of that ODE at each time point (2) exists and is unique (Arnold (1992)). As proposed in NODE (Chen et al.) (2018b)), the map at time one (z) can serve as the generator, i.e., I (²);), which is equivalent to an "infinitely deep" neural network with stacked weights 8(t), E [0, 1]. Thus, typically, continnous ODF-type flows require fewer parameters to match the performance of discrete ones (Grathwoh] ct al. (2018)). As for the invertibility of T1, it is naturally ensured by the theorem of existence and uniqueness pertaining to the solution of the ODE. However, $- must be orientation preserving that is, its Jacobian determinant must be positive, which may constrain its representational capacity. In order give freedom for the Jacobian determinant to remain positive, the authors of ANODE Dupont et al. (019g) propose augmenting the original ODE with supplementary variables mn(t), 2i [m( m[(] f((((())) 0()), & E [0, 1], with || signifying concatenation.
d
resulting in d
There are other types of flows which are not widely used in practice and not introduced here, such Planar and Radial Flows (Rezende & Mohamed (2015); van den Berg et al. (2018) and Langevin Flows (continuous and SDE- (Welling & Tch (2011) b Chen et al. (2018a)). Coupling Flows and Autoregressive Flows facilitate straight.orwart invertible transformationn and efficient determinant computation, yet. lag in expressiveness compared to Residual Flows which are computationally costly. ODF-based Flows provide a potentially elegant and parsimonious representation with fewer parameters. However, they necessitate the resolution of ODEs during training, a process that can be computationally demanding and sensitive to the choice of numerical solver settings. Compared with VAFs and GANs, NFs allow for exact density estimation of generated sample and can avoid training issues like mode collapse, vanishing gradicnts, etc (Salimans et al. (2016)) However, its requirements for bijection functions and determinant calculations can restrict its capacity in efficiently modeling complex data distributions (Cornish et al. (2020)). )
5.2 Normalizing Flows in Imitation Learning
Na-based IL algorithms either employ NEs for exact density estimation or leverage NEs' proficiency in mod- elling complex policies to manage challenging task scenarios, which correspond to the normalizing direction (i.e., x + 2) and the generation direction (i.e., 2 + 2) of NFs respectively. Next, we present a comprehensive review of works from both categories.
39

---

Published in Transactions on Machine Learning Research (08/2024)
5.2.1 Exact Density Estimation with Normalizing Flows
As a paradigm, IL methods that adopt NFss as density estimators usually start with standard IL objectives, and optimize them as RL problems wherein the rewards necessitate estimation of specific distributions. Tn this case, NE's are introduced model those distributions and provide exact density inference for reward calculation, which greatly enhance the training stability and performance. To be specific, we present some representative works here.
As proposed in [Ghasemipour et al. (2019b)), most IL methods can be viewed as matching the agent's 's state-action distribution with the expert's, by minimizing some f-divergence Dy. As an example, CFIL (Freund et al. (2023) chooses to minimize the KL divergence between the agent 's and expert's stale-action occupancy measure (as defined in Section 4.2.1
arg max E, (1. log ()) Px (s(saa)
arg max J(I, r loog PE) Pr
arg min D2L(
(66)
where J(T,r) denotes the expected return (Sutton & Barto 2018 of the policy л under the reward function r. An MAF (Masked Autoregressive Flow, Papamakarios et al. (017)) is used to model the distributions P a) and (=(s,a) based on which the rewards can be acquired. Specifically, (s,a) can be viewed as
x in the NF framework and the densities at (s,a), i.e., PF((, a) and pr(s, a), can be estimated using Eq. (64). However, rather than optimizing these two NFs independently with corresponding MLE objectives (i.e., Eq. (5)), they couple the modelling of these two distributions based on the optimality point of the Nonsker-Varadlie form of the KL divergence (Donsker & Varadhan 1976)):
(F( a)] log (oa))oz)) [ef(ea]] DK.(0=(5..)llle((.c)) = sup (( (f(s, f:8xA-3
(67)
In the equation above, optimality occurs at (*, = log pz(e,a) agls,a) + +C where C' E R. Thus, through maximizing the right-hand side of the equation above, the log ratio used the reward function can be recovered.
Specifically, they model f as fw. log p (s, a) - log (s, a) with the two flows p and Pg. è Given this
improved estimator, the overall IL objective can be written as: (which combines Eq. 666 and (67
p (* a) (5)
-E(aa))arl) log (s. (g)
arg max min log ((s,))DD(( Ph.PE
(68)
In this case, the learning alternates between max, and min pa.n max, is realized with RL process using log (l(()))) the reward function. While, (s. a) and pf(s, a) are updated based on the expert data (i.c.,
(2(),a)
(s,a) DE) and rollout data collected by æ (i.e.,
IL- (Chang et al. (2022) focuses on Learning from Observations (LfO), where the agent only gets access to a datasct of state sequences. This work also starts with the KL divergence but processes to decouple the policy optimization from the reward learning to avoid minimax optimization as in Eq. (68) and improve the training stability. In particular, the learning objective for the policy æ is to maximize (1)))) (s)) which equals to:
E E P.
Σ (log (e(s)+1)e) - log Pa (3441111))
= E Eep.r
Σ (log PE(s(41(s)) + (((((+)))))
(69)
The equality holds when environment dynamics are deterministic and invertible. The right hand side of Eq. 699 can be optimized with RI. by setting the reward as r log Pe(st41|ss) while maximizing the entropy of the policy, i..,, H (=)sss)) As a separate stage, before the RL training, Pe(s'(s) is modeled with a conditional NF (i.e., conditioning the transformation function F on a fixed state variable 8: s' = F(z ) ~ Pz(-)) based on the demonstrations Dg. Specifically, they select NSF Durkan et al. 2019 as the density estimator.
SOIL-TDM (Boborzi al. 2021b 2022)) also focuses on LfO and starts with the same objective as 1L- However, SOIL-TDM gets rid of the requirements for deterministic and invertible dynamics by estimating
40

---

Published in Transactions on Machine Learning Research (08/2024)
Pa ($141151) as Ta (a)(s the objective for ıt is converted into:
based on the Bayes Theorem. With this new definition,
max B (0)
Σ [r(se; ae) + (a(((ss))
(70)
where r(s,, a.) = BeeinTTT(ss
[-log log Ta (8 ae) + log गं' (ag|s1+1, 8t) + log P In this case, besides the experl state transition model Pr(#1-1(81) 18t); they adopt conditional NFs to model the agent's transition dynamics (s St, ar) and the posterior distribution associated with n, i.e. x'(0/|s441)s)) st), for which they choose Real NVP (Dinh et al.] (2017)) Note that the approximation of Pe st) is trained DE, while approximations of (5 0z) and x (010s)144) are trained on rollout data collected by π during the RL process.
SLIL Zhang et al. (021bb) is based on the Likelihood--bsee Imitation Learning (LIL) framework:
max R (s [log (=(s,a)) a)] 8.1. Pa arg max (() (log P(x,a) a)]
(71)
P
Intuitively, as) represents the probability of observing an expert state-action pair (s,a) E Dg when executing the learner policy æ. Directly solving this bilevel optimization problem requires jointly learning x and Ps. which brings training instability. Thus, they propose to maximize its tight lower bound:
max (s,a)>DD [log []] + Eesp, [log PE(s)] s.t. PE arg max B.DDS [log P(s)] P
(72)
where (s) denotes the occupancy measure of s induced by the policy ग. The training process can then be divided into two stages. At stage they train an NF to model the expert state distribution PE(s) based on the demonstration set De, which is, at stage 2, adopted to optimize the policy mTaas) using Eq. (2p. Intuitively, this objective combines Behavioral Cloning (i.e., max, E(a,a)DD [log (al)() and RL with log (s(() as the reward (i.e., max, Ea-p, [log (s)))) It is worthy noting that they utilize a Continuous Normalizing Flow DCNF (Zhang et al. (2021b) ) as density estimators, due to the enhanced modeling capabilitics and fewer model restrictions that Continuous Normalizing Flows offer Grathwohl al. (2018).
GPRIL (Schroecker et al. (2019) also utilizes the LII. framework but optimizes it in different manner:
max (s,)) DE [log Pro a)) max (ea))DD [log (a(ass) + log Pro (s)] my
(73)
Here, the first term of the right-hand side is simply Behavioral Cloning. While, for the sccond term, according to (Ross & Bagnell (2010)), the gradient from it can be estimated as V Pro (s) & (""","")~,, (+1=) (Vo log To(]" Br, (-1s) models the distribution of states and actions that, following the current policy ne, will eventually reach the given state s. That is, Bag (s", aa(ss) (1) (1- 7) Ei= '' Pa (s, s" s", = === = s). In GPRIL, Br, (s", ad|s) is modeled as B (s" [s)) Bua (a"ls, s),, i.e., the product of two density functions each of which is modelled by a conditional MAF (Papamakarios et al. (2017)). To train these models, they collect training data using self-supervised roll-outs. In particular, they sample stales, actions and target-stater (i.e., "")a",s),, where the separation in time between the s" and selected randomly from a geometric distribution paramcterized by 7. In this way, the collected triplets thcoretically satisly: (s",a") ~ BRp (-1s). As a subsequent research, the work (Schroecker & Jr. (2020)) utilizes the same objective function, but introduces an alternative method for estimating Vo log (=, RL perspective. However, this algorithm is still in an early stage of development.
(s) from goal-conditioned
5.2.2 Policy Approximation using Normalizing Flows
As a potent generative model, Normalizing Flows can be directly adopted as policy networks for tackling challenging tasks. In this regard, recent works in off-policy RL have demonstrated that NFs can be used to model continuous policies (Haarnoja et al. (2018); Ward al.) (2019); Mazoure et al. (219 which can lead to faster convergence and higher returns by enhaneing exploration and supporting multi-modal action
41

---

Published in Transactions on Machine Learning Research (08/2024)
Objective Type
Evaluation
Algorithm
Flow
Flow Type
Flow Usage
Task
DE (pE(s, a), Pa (s)
CFIL
MAF
AR
KL
MuJoCo
DE (Pr(s'Is) DE (PE(''(s), (= a), (a)s',s))
IL-flOw
NSF
CP
KL
MuJoCo
SOIL-TDM
Real NVP
CP
KL
MuJoCo
DE
SLIL
DCNF
ODE
LIL
MuJoCo
(Pr(s))
DE (B, aa)ss)
Robotic
GPRIL
MAF
AR
LIL
Insertion
G (Policy Network)
High-D
Flow DAC
Real NVP
CP
AlL
MujoCo
G (Trajectory Planner)
FlowPlan
NAF
AR
KL
HES-4D
Table 6: Summary of NF-based IL algorithms. AR, CP, and ODE represent autoregressive, coupling, and ODE-based continuous flows, respectively. DE and G represent the two manners that NFs can be used, corresponding to the density estimator and generator, respectively. For DE, specifically point out the densities estimated, for which the definitions are available in Section 5.2.1 Regarding the types of IL objectives, KL, LIL, and AIL denote the KL divergence, Likelihood- based IL, and Adversarial IL, respectively. Examples for AIL inchude GAIL and AIRL, which are introduced Section 4.2.1 For the benchmarks, MuJoCo (Todorov et al. (2012) provides a series robotic locomotion tasks; Robotic Insertion (Vecerík et al. (2017) specific type robotic manipulation task built on the MuJoCo engine; High-D Krajewski al. (2018)) and HES-4D (eeyer et al. (2019h are two real-word driving dalasel.
distributions. Compared to commonly-employed diagonal Gaussian policies, where each action dimension is independent the others, those based on NFs offer greater expressiveness.
In the context of IL., Flow DAC (Boborzi et al. (2021a)) extends DAC (an off-policy IL algorithm Kostrikov et al. ) by adopting conditional version of Real NVP the policy network. Specifically, the action a at state s is generated by: 2 ~ Pz(:), G(z[s) where G denotes the NF generator and is the conditioner. The stochasticity of this policy comes from the latent distribution Pz(-), and the action output of G can be in any complex distribution. Similarly, FlowPlan (Agarwal et al. (2020)) adopts NFs as a trajectory generator/planee through sequent generating the next state embedding based on historical states, and an NAF (Huang al. (2018)) is adopted as the generative model, which is trained by minimizing the KL divergence between the expert's and generated trajectories.
In Table f provide summary of NF-based II algorithms. For each algorithm, provide key information including what type of NF and how the NF is utilized, its underlying IL framework, and evaluation tasks.
5.3 Normalizing Flows for Reinforcement Learning with Offline Data
As mentioned in the beginning of Section we broaden the scope of RL-related works to include both offline RL (Section 5 and online RL with offiine data (Section 5.3.2 We believe that the approaches developed for using oflline data in ouline RL could be effectively adapted to enhance oflline RL. Similar with NF-based IL methods, algorithms in Section and 5.3.2 can be categorized based on the usage of NFs, i.e., working as either a density estimator or a function generator.
5.3.1 Adopting Normalizing Flows Offline Reinforcement Learning
As introduced in Section 2.1.1 major challenges in offline RL include (1) extrapolation error caused by approximating the value of state-action pairs not well-covered by the training data D,, and (2) distributional
42

---

Published in Transactions on Machine Learning Research (08/2024)
shift between behavior and inference policies. The common practice to tackle these problems is to induce con- servatism, through either keeping the learned policy close to the behavior policy or constructing pessimistic value functions. However, this may lead to over-conservative and sub-optimal policies. By introducing Nor- malizing Flows, an agent can utilize offline data more effectively and circumvent out-of-distribution (OOD) actions through its generative (Akimov et al. (2022; Yang et al. (222d)) or density estimation capacity (Zhang et al. (2023)). ).
CNF AAkmmov al. 2022) and LPD (Yang et al. (0033d) ) employ similar algorithm idea using NFs
to extract latent variables underlying primitive actions in the offline dataset. Specifically, they train Normalizing Flow G' conditioned on to model the mapping from z to a, following ~ P2(.), = G(2ls), through supervised learning (i.e., Eq. (65)). After training, G can work as an action decoder to convert a latent variable 2 (following a simple base distribution) to a primitive action. In this case, only a high-level policy 2 ~ (high(11s) needs to be learned with offline RI., and the function composition G high can work as a hicrarchical policy mapping from to a. CNF and LPD are driven by different motivations. In particular, CNF pretrains Real NVP conditioned on 8 as the action decoder. To avoid OOD actions caused by the long tail effect, the latent space (i.e., the support of Pz(-)), which is also the output space of the high-level policy, is designed to be bounded, specifically an n-dim interval (-1, Through pretraining on Dp: cach latent variable in this bounded space should correspond to an in-distribution action through the decoder G. As a result, the hierarchical policy (i.e., G o Thigh) would not generate OOD actions, even without clipping the high-level policy's output (as in PLAS introduced in Section 3.2.2). which avoids possible sub- optimality. LPD, instead, focuses on improving the offiine RL performance in challenging, long-horizon tasks, by adopting NFs to model the transformation from to a fixed-length sequence of actions starting from 8, i.c., skill. Provable benefits can be gained when the learned skills are expressive enough to recover the original policy space (Yang et al.] (2023d)) ), and, as an advanced generative model, Normalizing Flows can facilitate the learning of such skills. With this pretrained transformation, the dataset can be relabeled in terms of skills as Duigh = [(56 24, Eh-1 7 sf, where h is the skill length and 2 can be aequired as (ap-hh (<bbs--lss))> = ((db)--1=))) (i.e., via the normalizing direction of the NF). Then, offline RL can be adopted to learn Thigh(=le) from Daigh- In this case, the decision horizon of the offline RL. policy is reduced by a factor of h, which can effectively mitigate cunulative distribution shifts.
APAC (Zhang et al. 2023 adopts NEs (specifically Flow-GAN (Grovrr et al. 2018)) )) as density estimators, rather than action decoders as in aforementioned algorithms, to filter out OOD actions while avoiding being over-conservative. To be specific, the density of the point (s, a) can be approximated as (a(G)))))))) where G is the NF: a = G(2[s). Points with higher density are more likely to be generated by the behavior policy and so can be viewed as "safe" in-distribution points for training use. To be specific, they modify the policy improvement step in Eq. (1 as:
T arg max Ea-D, (1) a)] s.t. Pa(G)) > c(s,a).
(74)
where c(s, a) is defined threshold. In this way, samples for updating the policy are restricted within the safe area, which could contain both observed and unobserved points in the offline dataset. Thus, the policy can potentially perform exploration out of the given dataset and avoid visiting OOD actions, simultaneously. Note that this constraint on æ is more relaxed and practical compared with the one of BEAR (introduced in
Section 2.1.1h: :: π E {T': : S x A + [0 x <a(als) = 0 whenever u(a(s) < c), to avoid being ver-conservative.
5.3.2 Adopting Normalizing Flows in Online Reinforcement Learning with Offline Data
To deepen the understanding of applying NFs to RL, we incorporate another line of works on online RL utilizing offline data. Still, these works can be categorized based on the usage of NFs: potent generators
(Mazoure et al. (2019): Yan et al. (2022); Slack et al. (2022) or density estimators (Wu et al. (021))
With the same insights as Flow DAC (introduced in Section 5.2.2). ; SAC-NF (Mazoure et al. (0199) extends SAC, an off pollco RL algorithm (Haarnoja et al.) (2018)), by adopting NFs (IAF (Kingma et al. (2016h)) as the policy network, which could polentially improve exploration and support multi-modal action distributions. On the other hand, CEIP (Yan et al. (2022)) and SAFER Sllak et al. (2022) ) leverage the offline data by first learning a transformation from latent. variables z to actions a using NFs as action
43

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
Flow
Flow Type
Flow Usage
Evaluation Task
CNF
Real NVP
CP
G (Action Decoder)
D4RL (L,M)
LPD
Real NVP
CP
G (Skill Decoder)
D4RL (M, A, K)
APAC
Flow-GAN
CP
DE (Pa(s, a))
D4RL, (T.M)
SAC-NF
IAF
AR
G (Policy Network)
MuJoCo
CEIP
Real NVP
CP
G (Action Decoder)
Kitchen, Office, FetchReach
SAFER
Real NVP
CP
G (Action Decoder) DE (Po(s,a)
Operation
NF Shaping
MAF
AR
Robotic Insertion, Pick Place
Table 7: Summary of NF-based (Offine) RL algorithms. AR and CP represent autoregressive and coupling flows, respectively. DE and G represent the two manners that NF's can be used, corresponding to the density estimator and generator, respectively. Specifically, Pa(s, a) denotes the state-action pair distribution in the offline dataset. For the benchmarks, all the evaluation tasks shown in this table are built the MuJoCo [Todorov et al. (0012) engine; D4RL (Fu et al. 2020 provides bunch of challenging (robotic) tasks specifically for offline RL, including Locomotion (L), AntMaze (M), Adroit (A), Kitchen (K); Office (Pertsch Kitchen (from D4RL), Robotic Insertion, Pick Place
et al. (2021)), FetchReach (Plappert et al. [2018)
(Vecerik et al. (2017)) ), and Operation (Slack et al. 2022 all involve the manipulation of a robot arm.
decoder and then training a high-level policy Thigh (2|s) atop the transformation function using online RL, akin to CNF. Yet, in comparison to CNF, they extend the action decoder learning from different perspectives. In CEIP, the generator conditioned on a concatenation of the current and next states, ie., u = [s, s'), instead of the state alone, which can better inform the agent of the state it should try to achieve with its current action. Moreover, they propose a manner to utilize data from various yet related tasks, which are more accessible than task-specific ones. Specifically, they first train a flow for each task i with corresponding demonstraations The generator is defined as = Gę(zlu) = exp(c(()) @ = + d,(u) where & and d, are trainable deep nets, @ refers to the Hadamard product. This structure design follows Real NVP (Dinh et al. (2017)). Subsequently, they train a combination flow on demonstrations of the target lask, with the generator defined as G(z|u) = (E" (u) ex)(c 3 2 + (Ei (u))) At this stage, they fix a and d; (i =i=10 1,..,, n) and train the weighting functions 14 and A to optimize the combination flow for the designated task via supervised learning (i.e., Eq. 65)) In this way, they efficiently utilize demonstrations from related tasks. While, SAFER emphasizes the safety aspect of the learned transformation. Specifically, the transformation should prevent selecting unsafe actions aunsafe in the environment, which are labeled in the training datasct. To achieve this, the authors propose conditioning the flow on both the state s and a safety context c. It's important to note that e is not provided directly but must be inferred from the information available in the environment. Thus, SAFER simultaneously learns (1) an inference network Po (c]A) to determine the current safety context based on a sliding window of states A and (2) a generator a = c)( (Real NVP) conditioned on s and c, with the following objective:
max B. aunenta Da, om (.lA)))) (log Po(als, c) Allg Po (s, c)] DKL((P(c(A))IP((()) 0,6
(75)
Here, co(a|s, = P2(G7 (als, c)), Pe(c) denotes the assumed prior distribution of Intuitively, the first two terms encourage safe actions while deterring unsafe ones, and, in conjunction with the final term, the variable c is compelled to encapsulate useful information regarding safety. This algorithm serves as an illustration of how NFs and VA can be integrated. Tn this setup, Pa and Pa function as the encoder and decoder of the VAE, respectively, with Po being implemented as a Normalizing Flow.
Unlike SAC-NF, CEIP, and SAFER, NF Shaping (Wu et al (2021) adopts NEs density estimators. This is method that combines (online) reinforcement learning and imitation learning by shaping the reward function (for RL) with a potential term learned from demonstrations. To be specific, they first train an NF (specifically MAF Papamakarios el al. (2017)) to estimate the density of stale-action pairs in the demonstration dataset, ie., ao(s,a) = Pz(C sa(s)). Then, they shape the reward function as F, = r(s,, Ge. St+1) + +00(0441 a:41)= 1- ((s,, where ((s, a) = fog (Pa(s, +c), f >> 0 adjusts the weight, and e is a small constant to prevent numerical issues. According to (Ng ct al. (1999 Wiewiora| (2003)) ). reward shaping can intensify the reward and greatly improve the learning efficiency. Intuitively, this potential
44

---

Published in Transactions on Machine Learning Research (08/2024)
term highlights regions where the demonstrated policy visits more frequently, signaling areas that should be explored more.
To sum up, we note that the algorithms proposed in (Akimov et al. 2022 : Yang et al. (023d Yan et al.
(2022; Slack et al. ( ) adopt similar ideas. They first learn an basbased transformation from the latent
space to the real action space based on the offline data, and then train a high-level policy over the latent variables with online or offline RL. The prelearned transformation provides a mapping from a simpler, more controllable distribution to the target distribution, which is usually complex and unknown. Also, find that Normalizing Flows are primarily used exact density estimators in IL (Section 5.2) while mainly functioning as expressive generators in (Offline) RL (Section 5.3). From Table 6 most works adopt autoregressive or coupling flows, which is probably due to their computation efficiency. As introduced in Section [.1] continuous flows can potentially be more expressive in modelling complex data distribution. Integrating these types of flows with challenging offline policy learning tasks, such as vision- based oncs, could be future direction. On the other hand, while almost all algorithms are evaluated on robotic benchmarks such as MuJoCo or D4RL, there is notable absence of comparalive analysis among these algorithms. Given the diverse perspectives from which these algorithms are developed, it is challenging to compare them at the algorithmic level, thus comprehensive and fair empirical evaluations are essential to determine their effectiveness and suitability for practical applications.
and 7 we can observe that
6 Transformers in Offline Policy Learning
Transformers can be used for offline RL (Section 6.2 or IL (Section 6.3 by modeling them as a problem of predicting the next token based on historical data. Notably, most works in this section follow similar seminal work of Inajectoy-oopimizatton--sss
algorithm design with Decision Transformer (Section 2.1.3.. Due to their architectural design, transformers are particularly well-suited and effective for modeling time-scries data such as decision trajectories, as introduced in Section 6.1
offline RL
6.1 Background on Transformers
Transformer (Vaswani et al. (2017)) is an important foundation model that has shown exceptional capability across various areas, such as natural language processing (Kalyan et al. (2021)), ) computer vision (Han et al. (2022)), time series analysis (Wen et al. (2023)), ) and so on. Recently, there has been a growing trend in developing IT. and offline RL algorithms based on transformers, with the hope to achieve sequential decision- making based on next-token predictions as in natural language procossing. As shown in Figure D the transformer follows the widely-adopted encoder-decoder structure. The encoder maps the input (II,,.. (un) to a sequence of embeddings (Z1.... 2n). and the decoder generates the output sequence autoregressively. That is, the decoder predicts one token Dm++1 at a time, based on the input embeddings (21, in) and previously-generatee outputs (V1,..., :.m)) Both the encoder and decoder are composed of a stack of L identical modules. For clarity, we present each layer in the module sequentially, following the data flow.
Positional Encoding: After the token embedding layer, which is shared by (x1,-.-, ; and (y...... Vm), cach element is converted to d-dim vector. To enable the model to make use of the order of tokens, information about the relative or absolute positions of tokens within the sequence is injected through positional encoding. For the i-th token, its positional encoding is also a d-dim vector, of which the j-th dimension is (100))77) ) if j is even and cos( (1000(-1777) otherwise. The positional encoding and token embedding are then combined through summation. Per Vaswani et al. (2017), this tion and enables the model to manage sequences longer than those experiencod in training.
positional encoding design embeds relative position informa-
Self Multi-Head Attention (MHA): This component utilizes the attention mechanism. Given s queries and key-value pairs, the attention function maps each query to a weighted-sum of the values, where the weight assigned to cach value computed as the compatibility of the query with the key paired with that value. The queries, keys, and values can be represented as Q Rax,, K RRtxd, and V E Rixd, respectively, and the outputs for all queries (i.e., O) can be computed
45

---

Published in Transactions on Machine Learning Research (08/2024)
in parallel as: (For conformability of matrix multiplication, dj dg.)
0 Autention(Q, K, V) = SoftMax (QK")
(1), vdk.
(Q Ki) ddk
V. O. = MaoiMax
V (76)
(dl
Here, SoftMax is a row operator, and the output for query i, i.e., O₁, is weighted sum of the rows of V, where the weight for row is propotional to the similarity of Qi (i.e., the i-th row of Q) and K, measured by their inner product (Q,, The dot products grow large with da. which would push the Soft function to regions with vanished gradients, so the factor 1/vde is introduced. The imput embeddings H₃ (in Figure [ can potentially be used as the matrices Q, K, and V for self attention. Howover, to enable the model to jointly attend to information from different representation subspaces, a multi-head attention mechanism is adopted:
QHA K, V) Concal Concat(head,- ,head,, WWO head; = Attention(QW KW VW.) (77)
where Q - -K=K= = H. E Baxd w.O paxd', w.K Raxd' W.Y E gaxd' WPO E gaxd, d' = dd/h, and Concat represents the concatenation operation. Wº,, W.K, .WY convert Hg to matrices in the i-th attention head, providing a distinct representation subspace. It's worthy noting that the output of MHA belongs to Raxd, that is, the same in shape as its input.
Add & Normalize: A residual connection (He et al. (0166) is employed around each attention layer, followed by a layer normalization (Ba et al. (2016)) ). As common practice, these two operations are adopted to stabilize training of (very) deep networks (by alleviating ill-posed gradients and model degeneration). Suppose the previous layer is F, which can be an MHA Feed-forward Network shown in Figure and its input is X, then the calculation of this Add & Normalize layer can be denoted as LayerNorm F(X) + X).
Point-wise Food-forward Network (FFN): FFN layers are important for transformer to achieve good performance. Dong et al. (2021) observe that simply stacking MHA modules causes a rank collapse problem (i.e., leading to token- sutt--uunffrtt)). and FFN is an important building block to mitigate this issue. Specifically, a two-layer fully-connected network with a ReLU activation func- tion in the middle is applied to each of the n token emboddings separately, leading to n new d-dim embeddings as output.
For the decoder, Ha is replaced by Hy E Rmxd i.e., outputs. For rationality, the query (of the Masked Self MHA) at each position is only allowed to attend with positions up to and including that position, as the other queries correspond to outputs yet to generate. This is realized within the Masked Self MHA by masking out corresponding compatibility values, i.e., (Qi, for the Cross MHA, its queries come from the previous decoder layer and key-value pairs are from the output of the encoder, i.c., H, E Rnxd in Figure [ In this way, every query for predicting the next token can allend over all positions in the input sequence (via the Cross MHA) and all greviously-generatee tokens (via the Masked Self MHA).
Masked Self MHA & Cross MHA:
embeddings of previously-genertte
-oo Vj > (in Eq. (6)). As
Each encoder module (listed above) outputs an x d matrix, and each decoder module outputs an m x d matrix. Here, n and m denote the counts of elements in the input and output sequences, respectively, while d represents the embedding dimension. This uniformity in matrix sizes enables the stacking of multiple encoder (or decoder) modules to create a deep model. Each encoder (or decoder) module will take the output from the previous encoder (or docoder) module. Generally, the transformer architecture can be used in three different ways: encoder-only, decoder-only, or encoder-decoder. When using only the encoder, its output serves representation of the input sequence, suitable for natural language understanding tasks such as text classification. While, if only the decoder is employed, the Cross MHA modules are removed and this architecture is suitable for sequence generation tasks like language modeling. Applications of transformers in II. or oflline RL usually adopt. this decoder-only way. The overall encoder-decoder architecture is equipped with the ability to perform both natural language understanding and generation, typically used in sequence- to-sequence modcling (c.g., neural machine translation).
46

---

Published in Transactions on Machine Learning Research (08/2024)
As a foundation model, the transformer architecture has significant advantages. Compared to fully-connected layers, the transformer is more flexible in handling variable-length inputs (i.c., with different n) and more parameter-efficient, since the amount of nelwork pa- rameters a transformer irrelevant to the sequence length n The convolutional layer uses a convo- lution kernel of size k n to connect pairs of tokens, so, to link distant token paits, stack of convolutional layers is required. However, with the attention mecha- nism, each pair of tokens can be connected in one MHA layer and the time complexity for matching their cor- responding query and key is constant (i.c., irrelevant to n). This makes the transformer excel at capluring long-range dependencies within a sequence. Recurrent neural networks produce sequence of hidden states h4, each of which is dependent on the previous hidden state h1-1 and the input at position v. This sequential nature makes parallel computation impossible. How- ever, in transformers, computations at each position are independent, allowing for parallel execution that are es- sential for processing long sequences. Last, both convo- lutional and recurrent models make structural assump- tions over the inputs, suitable for image-like and time series data, respectively. However, the transformer has few prior assumptions on the data structure and thus is a more universal model. Empirical results have shown that the transformer has superior performance on a wide-range of lasks and a larger capacity to handle huge amount of training data.
Prediction tolest Przes
Lase SenMaax
"Doodde"
Add Nurmalie
Pointorise
"Eroode""
Feed foeward Network
Nonvelize
Add Nomeelie
K2
1
Cress Malii-liud Attection
e
L
Add & Nerraiize
Add Nommaliz
Matti lied Alleatiom
Masied Allaniin
K
Ry % 2g
My 1g
Postional
Powittonal Ensading
Tokes Enbeliing
Token Endlekking
Impuc
Curpst Go "t....
Figure 1: The Transformer Architecture
As listed above, the transformer contains multiple types of modules. Numerous studies have explored modifications or replacements of these modules in the standard transformer architecture for improvements. For comprehensive review, please refer to (Lin et al. (2022c)). On the other hand, the computation and memory complexity of MHA modules are quadratic to the length of the input sequence (i.e., n), leading to inefficiency at processing extremely long sequences. Various extensions have been developed to enhance either the computational or memory efficiency. Additionally, due to the minimal structural assumptions made by the transformer about the input data, transformers are prone to overfitting when trained on small- scale datasets. Attempts like pretraining the transformer on large-scale unlabeled data (Brown al. (2020) introducing sparsity assumptions (Child et al. (2019)) such as limiting the mumber and positions of key-value pairs that each query attend to, have been made to solve these issues. Finally, extensions to adapt the transformer for particular downstream applications are possible. Studies detailed in Section 6.2 and 6.3 can
be viewed as examples of this type of extensions.
6.2 Transformers in Offline Reinforcement Learning
The content of this section is arranged as follows. First, we highlight pioncering studies on transformer- based offline RL in Section 6.2.1 along with a series of follow-up works for improvements in Section 6.2.2 and 6.2.3 Then, we explore the use of transformers in extended problem setups in Section 6.2.4 such as multi-agent and multi-task offline RL. In particular, for the multi-tast setting, we focus on developing generalist agents basod on the pretraining & fine-tuning scheme. Finally, we reflect on existing problems in this field and discuss potential future research directions in Section 6.2.5
The FEN layers applied to each position rather than the whole sequence, and the MHA layers only require updating the weights: W.P. WW wy. Wo, the sizes of which are irrelevant with n.
47

---

Published in Transactions on Machine Learning Research (08/2024)
6.2.1 Representative Works
Inspired by the great success of high-capacity sequence generation models (such as the transformer) in natural language processing (NLP), Chen et al. (2021b) propose to view offline RL as a sequence modeling problem and solve it with a Decision Transformer (DT). This algorithm exemplifies ceajectry-optimizatinn based offline RL (introduced in Section 2.1.3), we provide detailed introduction of it, along with two other seminal works, as brief tutorial on this particular branch of offline RL methods. Given offline trajectories {T = (so, Go, Ro. ... as,,ar, RT)), where R. = Ei= Ti denotes the return-to-go (RTG), the DT is trained to predict the next action based on the previous l: + transitions:
T
T , or min E, a
min E, Σ (ar ((T (T)=
Σ - log (a. 72
Rkt) k:t)
(78)
x
where Te-kit (s), Ry Ry..... St,, Rs) (j min(t k, 0)) is the input sequence; the two terms above are for tasks with continuous and discrete actions, respectively. DT adopts a GPT-like architecture (Radford et al. (2018)) ) which is a decoder-only transformer as introduced in Section 6.1 During evaluation rollouts, an initial target return Rg must. be specified, which can be the highest achievable return for the task. The inference trajectory is then generated autoregressively using the DT as follows: 50 ~ >pa(-), ag ~ T-1150, Ro), $1 ~ T[ ao), ru r(su,,a)) ... R₂ = R - Tt-1: ae ~ T(t)ee):t). .., where (po. T.r) are components of the MDP. DT greatly simplifies offline RL by eliminating the necessity to fit. value functions through dynamic programming or compute policy gradients as in classical offline RL. Using such (supervised- learning based) sequence modeling objective makes it less prone to selecting OOD actions, because multiple state and action anchors throughout the trajectory prevent the learned policy from deviating too far from the behavior policy u(als). Additionally, as a high-capacity model, the transformer has the potential to enhance both the scalability and robustness of the learned policy, similar to its applications in NLP. Per Chen et al. (021b)), DT shows competitive performance compared with prior oflline RL methods, such as Conservative Q-Learning (CQL, [umar et al. (2200), especially for tasks with sparse and delayed reward functions. However, Emmons et al. (2022) investigate what is essential for offline RL via supervised learning (RvS) through extensive experiments. Their findings indicate that a two-layer MLP model, utilizing a simpler input format (i.e., (st, R.) instead of Tz) and the same objective (i.e., Eq. (8 exhibits competitive, and in some cases superior, performance compared to DT, and DT underperforms prior offline RL algorithms on most benchmark tasks. This raises the questions whether it is necessary to adopt such a high-capacity model, i.e., the transformer, as the policy network, and under what scenarios RvS methods might outperform progaammprog based offline RL methods.
Trajectory Transformer (TT, Janner al. (2021) follows DT but utilizes more techniques from NLP, including tokenization, discretization, and bean search. In particular, they treat each dimension of the state and action as token and discretize them indepondently. Suppose the state and action have N and M dimensions respectively, the objective is min, E,, [[(T)], where L(T) is defined as follows:
M
log (s]1s,) T<)) + log {a<{ la St,
",, Te() + log m(rola, St, Tex) + log T (Rerre: an st. Tee)
(79)
This objective involves supervision on each dimension of the state/action, the reward, and the return-to-go. For the inference process, a beam search technique Freitag & Al-Onaizan 2017)) is utilized. Specifically, B most-likely samples are kept when sampling of and R,. Then, samples (a. FM Re) with the highest cumulative reward plus return-to-go (i.e, the estimated trajectory return Σ r. + R,) is selected for execution. Note that, during the inference, si:N and rt are acquired from the simulator. TT shows superior performance than DT in some benchmarks as reported in (Janner et al. (2021)), but treating each dimension separately would introduce learning and sample inefficiencies, especially for high-dimensional tasks.
Instead of modeling the policy, Q-Transformer (Chebotar el. al. (2023)) proposes to learn transformer- based Q-network. Similarly to TT, each dimension of the action is discretized and treated as separate time step. When training, they update the Q-function in a temporal difference (TD) manner, i.c., min- imizing the disagreement between the predicted Q-value Q( ((st-ka,,0 azt)) and target Q-value Q(5t- a..)).
48

---

Published in Transactions on Machine Learning Research (08/2024)
If M (i.e., the action dimension), (() rs + max Q(8- otherwise, ket max Qs at "0","""" attt). Note that, for the Bellman optimality (Puterman (014 of this per-dimension updating rule to hold, it requires the assumption that the choice of a," does not influence m Q(s a a), However, this is probably not true since dimensions of an action are likely to be correlated. Thus, this method lacks theoretical support. During evaluation, the action is also generated dimension by dimension, based on the trained Q-function.
So far, we have introduced some fundamental algorithm designs to cast offiine RL as sequence modelling problems based on transformers. Next, we present follow-up improvements of these mothods. As illustrated in (Emunons el al. (2022) ); carefully aligning the model capacity with the training data to prevent overfitting or underfitting and choosing which information to condition the policy on are critical for RvS performance. The discussions concerning the following works focus these two aspects.
6.2.2 Balancing Model Capacity with Training Data
The performance of transformer-basel offline RL relies heavily on the quantity and quality of the training data. A series of studies have been proposed focusing on augmenting the offline dataset across various task scenarios. (1) Based on TT, Bootstrapped Transformer (Wang et al. (2022a) ) proposes to generate trajectories from n,, i.e, the model being learned, and adopt them additional training data to expand the amount and coverage of the offline dataset. The self-generatdd trajectories are filtered by their log likelihood under the policy ग, which indicates the quality and reliability of the training data. (2) SS-DT (Zheng et al. (2023)) introduces a data augmentation method for semi-supervised settings, where most trajecto- ries lack action labels and are in the format (so.ro." ro.. ,8, Their approach simply involves training an inverse dynamics model, i.c., Tav((a St, St+1), on trajectorics with action labels and then applying this model to predict actions for the unlabeled trajectories. (3) Given only sub-optimal trajectories, trajectory- optimization-oenn based offline RL would fail. In this case, the agent needs to learn to stitch segments from different trajectorics for an optimal policy. Valuc-based RL does not have the same issuc as it pools infor- mation for each state across trajectories through the Bellman backup. In this case, QDT (Yamagala et al. (023 suggests learning a Q-function via CQL and replacing the RTG values (i.e., Rs in the offline dataset with the learned Q-values as augmentation for DT training.
(4) CDT (Liu et al. (2023b) ) proposes a data augmentation method for safe RL. Specifically, in a Constrained MDP (Altman| 1998)), the DT conditioned on both R(T) and C(r) = E Q (i.e., the constraint-violatin cost of the trajectory) is expected to achiove the target trajectory return while ensuring that the accumulated cost remains lower than C(r). However, during inference, unachievable (R(7), C(T)) pairs may be given as conditions, which are not represented in the offline dataset. Thus, as an augmentation, they suggest using the trajectories from the offline dataset that achieve the highest return (which is lower than the unachievable return R(T)) without violating C(r) (5) For tasks with sparse and delayed rewards, DT suffers from model degradation, since the RTG does not change within a trajectory. DTRD (Zhu et a (2023a)) proposes to learn a reward shaping function (s, a) to redistribute the delayed reward to each time step. This is achieved by solving a bi-level optimization problem as below:
as the corresponding trajectories of (R(T), C(r)) for offline training.
T
min Lvai (0) 6) s.t. R(T) Σ 1=0
ralss,at), VT, and (* arg min Creain (O, Ø))
(80)
Here, 0 is the parameter of the policy म; Leram (0, Φ) denotes the DT objective on the training dataset, which is augmented by replacing the original sparse rewards ra with ra(s;, az); (f)) Φ) the DT objective on the augmented validation dataset. Intuitively, ra is updated ensure that the policy learned from the dataset, augmented with corresponding dense rewards, is optimal. They propose a practical alternative training framework for 0 and 6, which however lacks convergence guarantee in theory.
Another line of works in this category focus on modifying the DT architect to explicitly make use of the structural patterns within the training trajectories. As mentioned in Section [..] introducing strutual assumptions/pin prriors to the input data can greatly improve the learning efficiency and mitigate the overfitting issue when learning from small-scale datasets. (1) Based on the observation that each trajectory a sequence of state-action-rewarr triplets, St.Rforrmr (Shang et al. (0222) proposes to first extract representations
49

---

Published in Transactions on Machine Learning Research (08/2024)
2e from each (r--1, 0(-; and then apply DT on (21.-.. 22) to predict az. Note that they adopt r, rather than Re, which is a counterintuitive design, as RT, decision-making typically relies on information about long- term returns.
(2) GDT (Hu et al. (2023b)) follows the same motivation as StARformer, recognizing that naively attending to all previous tokens, as in DT, can overlook the causal relationships among certain types of tokens and thus hurt the performance. In particular, they introduce relation embeddings to the compatibility calculation (i.e., Eq- (76)) for each pair of tokens (I2, With this modification, the compatibility between the query Qi and key Kj is calculated as (ei+- Kj teeeee,, where re is an embedding of the adjacency matrix that represents the causal relationships among tokens in the imput sequence. For instance, a is directly influenced by 8n and Re, so there should be edges from to and from RS to in the casual graph. In this way, they incorporate the Markovian relationship into the DT model. (3) DTd (Wang et al. (2023c) views the state, action, and RTG as distinct modalities. By analyzing attention values from a trained DT, they rank the importance of token interactions (for decision- decision-making) within or cross modalities, Specifically, 9-8, 8, a a, R R < 8 R, a R < a, where s denotes interactions (i.c., attending via the MHA) between states, and <<' denotes being less important. Consequently, they suggest a hierarchical DT structure, processing less crucial interactions before more significant ones to prevent important interactions from being distracted. The resulting structure is large in scale and outperforms DT, but its overall performance is close to traditional offline RL methods like CQL.
6.2.3 Mitigating Impacts from Environmental Stochasticity
Another category of research works that follow DT propose to change the conditional information of the DT to enhance its performance in stochastic environments. (1) In the absence of determinism, high- return offline trajectory could be the outcome of uncontrollable environmental randomness, rather than the result of the agent's actions. Thus, goals that are independent from the environmental stochasticity are the only conditions that the agent can reliably achieve. ESPER (Paster et al.) (2022)) ) and DOC (Yang et al. (2023cl) propose methods to learn such conditions automatically from the offline datasct. They share common intuition: to ensure the conditional variable to contain sufficient information on action predictions but no information regarding the environment dynamics (e.g., on predicting future rewards state transitions). Both algorithms use a contrastive learning framework, resulting in similar objectives. However, these methods require to learn at least five networks (so we choose not to show their objectives here), which lose the simplicity of DT and is even more complicated than traditional offline RL approaches. (2) SPLT (Villaflor et al. (2022)) proposes to model the policy as ((āe((s)xx: at-k, .8), 2;"), where the condition 2 is sampled from an encoder that lakes ( " St, (t) as input. Both the encoder and x have Di-like architectures and are trained jointly in CVAE framework (introduced in Section 3.1), where t works the decoder, (#t-k; Ot-k.''. , 82) and de work the condition and data point respectively. 2f is d,-dim discrete vector, each dimension of which has e categories. Extracted by the CVAE, each specific of can be viewed as a mode contained in the offline data and can be assigned to π generation condition. In the same manner, they learn environment model w(f,, S4+1, Ret1l(5t ae-k, at) 27). During inference, with (I,Ww) and by enumerating (27, 2 ddexd xdu predictions future h-length trajectories * can be obtained, and the next action is selected from the mode: arg max,7 min. R(*!) for robustness, where R(??)- hh frts + Reth. Through thorough enumeration the future predictions, the influence brought by the environmental randomness can be mitigated. (3) As in QDT (introduced in Scction 6.2.2 CGDT (Wang et al. (002d)) suggests training an additional Q-network lo guide the learning of DT. The rationale behind this method is that the Q-value, representing the expected return for each state-action pair, can help mitigate the impact of environmental stochasticity that could be wrongly captured by values.
RTG
6.2.4 Transformers in Extended Offline Reinforcement Learning Setups
DT has been extended to various offine RL setups, including model-based hierarchical, multi-agent, and multi-task offline RL. Unlike the previous three subsoctions, extensions in this part primarily expand on the problem setup rather than improving the algorithm design with respect to DT. Thus, this part is not our primary focus and we only provide taxonomy and brief introductions of the related works.
50

---

Published in Transactions on Machine Learning Research (08/2024)
Model-based Offline RL: Wang et al. (2023b) propose Environment Transformer to model the transi- tion dynamics and reward function P(s++1, r1/8t, ax) from the offline dataset, which can be viewed as simply replacing the output of DT, i.e,, at, with (st+1, rz). The learned environment model can then be utilized as a simulator for policy learning with any RL algorithm. TransDreamer (Chen et al. (2022) suggests replacing
the RNN module within Dreamer (Hafner et al. (2020)), which is a state-of-the-art (SOTA) model-based RL.
algorithm, with a transformer, for improved performance on tasks requiring modeling long-term temporal dependency. This reflects the advantage of transformers over RNNs, as mentioned in Section 6.1
Hierarchical Offline RL: HDT (Correia & Alexandre (022 employs two DTss for hierarchical decision making. The high-level DT models the distribution Thigh (Ct Ot-k: 88) for selecting subgoal, while the low-level DT captures the corresponding action distribution How((r It- Oz-k, ; (t). More- over, they propose a heuristic approach for extracting subgoals 91:T from the offline trajectories, when these subgoals are not labeled. Skill DT (Sudhakaran & Risi (2023)) adopts a similar hierarchical framework, but proposes to use a VQ-VAE (introduced in Scction 3.1 0
as the high-lovel policy to decide on the skill
choices for the low-level policy to condition on.
Tseng et al. (2022) ) is proposed to cast offline MARL as a sequence modelling problem. First, a teacher (joint) policy Ta. " lo!- R ... R!!") is learned. Its training process is the same as DT, viewing the concatenation of observations IRTGs/actions from n agents as a single joint observation (RTG/ action). Then, π is distilled to n student (individual) policies T' o' Ri ai. ,oi, R2), to enable decentralized execution. The policy distillation process is designed for π: 1i to imitate π while maintaining the structural relation among these agents as in the joint policy π, which does not rely on transformers. There are some other explorations for the multi-agent scenario: (2021 investigates combining offline pretraining using DT and online adaptation with
Multi-agent Offline RL:
MADT (Meng et al. gradient-based MARL algorithms; SCT (Li et al. (2023)) proposes approaches for handling non-cooperative MARL scenarios. However, these two works are still in early stages of development.
Multi-task Offline RL: Transformers have demonstrated exceptional generalization capabilities in NLP and CV tasks. Thus, there is a branch of research works on enhancing DT to enable policies pretrained on source tasks to be quickly adapted to target tasks through fine-tuning. The target task may be the same as the source task, differ but still fall within the same task distribution with the source task, or belong to entirely different modalities from the source task. We provide a brief review of these three categories as follows, each progressively demanding greater generalization capabilities. (1) Further adaption on the same task is required when the provided offline data is highly sub-optimal or only covers a limited part of the state space, for which ODT (Zheng et al. (2022)) introduces a technique to fine-tune the pretrained DT through further online interactions with the environment. Another possible scenario is when the high-quality and labelled offline data is scarce while there are abundant unlabeled (i.e., reward-free) trajectories. In this case, PDT (independently proposed by Cang et al. (2022) and[Xe et al. (2023) ) can be used for unsupervised pretraining the unlabeled data, followed by tun-ttunng using the limited high-quality data. (2) Adaptation to unseen tasks within the same task distribution is a primary focus in multi-task/mmct (offline) RL. To extend DT in this realm, some studies suggest conditioning the DT on task-specific information 2 ~ Pz(-) and training the DT across a range of tasks, each associated with unique embeddings Ztrain ~ (-).). Consequently, when encountering new tasks sampled from Pz(-) and conditioning on the corresponding žtest, the policy is expocted to be effective in these previously unseen tasks after few/zero-shot fine-tuning. 2 can be a task id (Lin et. al. (2022a)). ) segment of Lask trajectory (Xu el al. (202bb)), or trajectory embedding from an encoder (Lin et al. (2022b)). Alternatively, Boustati et al. [2021 suggest running the source policy in similar but counterfactual environments to gather augmented data for DT training, to enhance the robustnoss of the policy in unseen tasks. (3) So far, the transformer has been successfully adopted in NLP, CV, and RL, which makes it possible to build a transformer-based generalist agent across these three modalities. Gato (Reed et al. (2022)) shows that a single transformer with the same set of weights can handle 604 distinct tasks with varying modalities. Considering that texts, images, and decision trajectories can all be tokenized and embedded, the transformer can learn to generate all modalities within unified training framework predicting the next token based on previously generated ones. Specifically, the training objective aligns with Eq. (78,, but substitutes the training data with sequences of words for text generation and sequences of image patches for image understanding. During training, each batch of training data incorporates sequences
51

---

Published in Transactions on Machine Learning Research (08/2024)
from various modalities and each sequence is concatenated with demonstration for the same task, serving a prompt to assist the agent in differentiating between modalities. Gato utilizes a decoder-only transformer with 1.2 billion parameters and is trained on a dataset containing 1.76 trillion tokens. The empirical results show that Gato achieves performance excoeding 50% of the expert score threshold in 450 of the 604 evaluation tasks. Gato represents a significant allempt in developing generalist agents, yet its efficacy heavily depends on the quantity and quality of the training data. Especially, when learning control policies, it doesn't utilize reward signals, so, through imitation only, its performance would be capped by the provided demonstrations. Multi-Game DT (Lee et al. (0222)) adopts a similar protocol with Gato and suggests how to utilize reward signals in both the training and inference process. Another critical question regrading multi-modal learning is whether the learning in distinct modalities can enhance each other. Reid et al. (2022) empirically show that pretrained language models can be effectively adapted to oflline RL tasks with boosted performance. Following this work, Takagi (2022) provides an excellent in-depth empirical study on the effect of cross- modality pretraining for DT. They find that, after fine-tuning, parameters of the panguagg-pretrained model do not change as much as randomly-initialln
model, and these stable parameters preserves context- equivalent information such that the agent can make decisions even in the absence of contexts (i.e., without using the previous k transitions as input). Further, the way to efficiently utilize these context-equivalent information is also preserved by those stable parameters. As a result, they claim that pretrained context- like information could help the DT training by enabling the model to predict actions more accurately. Tn contrast, a transformer pretrained on images performs significantly worse than a standard DT, partly due to the significant differences in data characteristics.
6.2.5 Reflections and Future Works on Transformer-based Offline Reinforcement Learning
We notice that there is a series of works on applying transformers to online partially-observable RL (Yuan) et al. (2023a)): ). They adopt the transformer as the policy network in gradient-based or actor-critic RL algorithms, with the hope to harness its ability to process long-horizon historical information for decision making. These works focus on architectural modifications to the standard transformer, which are specifi- cally tailored for RI. For example, GTTrXL (Parisotto et al.] (2020) ) and Catformer (Davis et al. (2021)) suggest adjustments to the MHA module to facilitate a more stable RL training process; ALD (Parisotto
& Salakhutdinov (2021)) adopts model distillation to improve the computation efficiency in transformer- based distributed RL; STT (Yang et al. (2022e)) and WMG (Loynd et al. (2020)) propose adaptions for the transformer to process spatiotemporal coupling observations and factored observations, respectively, for enhanced sample efficiency. These architectural modifications, or potentially new ones, could be integrated with policy gradient offline RL algorithms or DT-kike return-conditiones supervised learning (RCSL) algo- rithms for performance improvement. Conducting an (empirical) comparison between these two categories (i.c., RCSL and policy gradient offline RL), both utilizing the same transformer architecture, would also be an int.riguing study.
Finally, we reflect on these DT--like RCSL methods. Although series of improvements have been made, there are still fundamental limitations regarding these algorithms. According to (Brandfonbrener et al. () RCSL. returns near-optimal policy under a set of assumptions that are stronger than those needed for traditional RL algorithms, and RCSL alone is unlikely to be a general solution for offline RL problems. In particular, RCSL offers guarantees only when the environment dynamics (including the state transition and reward functions) are nearly deterministic, a priori knowledge of the optimal conditioning function (i.e., the RTG values) is available, and the return distribution of the provided offline trajectories can cover the possible values of the conditioning function. These findings inspire several directions for future research. First, adapting RCSL algorithms to stochastic environments, ideally backed by theoretical optimality guarantees, is clear necessity. Second, developing effective strategies for selecting RIG values the conditions during inference, or alternatively, exploring new conditioning functions, is crucial. Last, a significant challenge lies in addressing out-of-distribution scenarios in case that the oflline dataset lacks adequate coverage of the task scenarios. Another fundamental problem regarding RCSL is whether it is necessary to learn low- return behaviors from the offline dataset and to require strong alignment between the target return (i.c., the condition of the DT) and realized return. As indicated in Eq. (78), the policy training involves imitating trajectories across range of returns. However, only high-return policy is required during inference. This
52

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
Key Novelty
Evaluation Task
DT
Solving offline RL as sequence modeling Tokenization, discretization and beam search
Atari, D4RL (L)
TT Q- Transformer
D4RL (L, M)
Transformer-based Q learning Data augmentation with sclf-generatod data Data augmentation via an inverse dynamic model Replacing RTGs with Q-values to enable stitching Data augmentation for safe RL in constrained MDP
Real Robot
Boot Transformer
D4RL (L, A)
SS-DT
D4RL (L, M2d)
QDT
D4RL (T., M2d)
CDT
Bullet-safety-gyn Atari, MiniGrid,
DTRD
Data augmentation for sparse, delayed reward setups
D4RL (L. K, M2d)
StARformer
Processing (Tr-1,0-11, as group Embedding the causal relation among 8, a, R
Atari, dm control
GDT
Atari, D4RL (L)
DTd
Processing 8, a, R as three distinct modalities
D4RL (T,, M)
ESPER
Replacing RTGs with a stochasticity-free conditioner
Stochastic Benchmark MuJoCo, D4RL (M),
DOC
Replacing RTGs with stochasticity- conditioner
FrozenLake
SPLT
Enumerating future trajs to mitigate stochasticity
CARLA
CGDT
Critic-guided DT training
D4RL (L, M)
Env Trans (MB)
Modeling the dynamic and reward functions
D4RL. (T.)
Hidden Order Discovery, dm con_cotrol, Atari
Trans Dreamer (MB)
Integrating Dreamer with a transformer architecture
HDT (HRL)
subgoal-conditionas
porision-making Skill extraction via a VQ- VAE
D4RL (L, A, K)
Skill DT (HRL)
D4RL (L, M)
A centralized training with decentralizer execution (CTDE) framework for MARL based on policy distillation
Fill-In, Equal Space,
MiDT+Dist (MA)
SMAC, Grid- World,
Highway
MADT (MA)
Applying shared DT to each agent's 's sequence A DT-based method for non-cooperative MARL
SMAC
SCT (MA)
simple-tag, simple-world
ODT (PT)
An online fine-tuning method for DT Pretraining on large-scale reward-free trajectories single transformer agent for 46 Atari games A generalist agent for 604 cross- cross- modality tasks
D4RL (L, M)
PDT (PT)
D4RL (L)
MG DT (MT)
Atari
Gato (MT)
See Reed et al. (022)
Table 8: Summary of tramsformer-based offline RL algorithms. In Column 1, we list representative (but not all) algorithms in this section, where Boot Transformer, Env Trans, MADT+Dist, and MG DT cor- Distillisillttinnn, and Multi-Game
respond to Bootstrapped Transformer, Environment Transformer, DT, respectively. These algorithms are grouped by their categorics, with abbreviations MB, HRI, MA, PT, and MT denoting model-based, hierarchical, multi-agent, pretraining based,, and multi-task offine RL, respectively. The evaluation tasks are listed in Column 3. Most works are evaluated on D4RL (Fu et al. (020%), which provides offline datasets for various tasks, including Locomotion (L), AntMaze (M), Adroit A), Kitchen (K), Maze2d (M2d). Regarding the other benchmarks, we provide their references here: Alari (Gronauer (2022)) ), Min-
Bellemare et al. (2013), Real Robot (Chebotar et al. (2023)), Bullet-safett-gyy
iGrid
Devvalier-Boisverr et al. (2018)), ) dm control (Tassa et al. (2018)), Stochastic Benchmark Paster
et al.
(2022) ), MuJoCo (Todorov et al. (2012)). FrozenLake (Foundation (2023)). CARLA (Dosovitskiy
et al.
(207)) ), Hidden Order Discovery (Chen al. 2202))) SMAC (Samvelyan et al. (2019)). Fill-In &
Equal Space & Grid-World & Highway (Meng et al. (2221) simple-tag & sirlle-world (Li et al.
2023b
53

---

Published in Transactions on Machine Learning Research (08/2024)
raises the possibility of developing a more efficient mechanism that strategically utilizes low-return samples to enhance the learning of high-return policy.
6.3 Transformers in Imitation Learning
Similarly with DT, transformer-bases IL utilizes supervised learning paradigm (see Section 6.3.1 p, where the transformer is adopted as the policy backbone (sce Section 6.3.2) We diffcrentiate works for IL and offline RL simply by if the reward or return is used as a condition of the policy. Notably, significant advantage of the transformer is its capability to process large quantities of training data across multiple modalities, which enables the development of generalist agents, as detailed in Section 6.3.3
6.3.1 A Paradigm of Transformer-based Imitation Learning
Given expert demonstrations Dg = {rg = (So. Go. sT, ar), transformer-based IL algorithms are de- signed to learn a policy æ to replicate expert behaviors. The objective function is typically formulated as follows:
min EDR [(a: - log (51, "0,==,),,,,,) or min ED log x(a St,,", ;x==))
(81)
x
decoder-only transformer (as introduced in Section 6.1 This objective is similar in form with the one of DT (i.e., Eq. (78 but replaces (st, a, Rs) with Zt- IL algorithms do not require reward signals but place an emphasis on the quality of demonstrations, which ideally should come from experts, as the learning process relies solely on imitation. Also, to make full use of demonstrations, auxiliary supervision objectives, such as prediction errors on the next state (i.e., the forward model Ioss) or the intermediate action between two consocutive states (i.c., the inverse model loss), are often employed. These objectives complement the action prediction error (i.e., Eq. (1]) to aid the policy learning process. As representative, Behavior Transformer (BeT, SShafuullhh empirically shows that a standard transformer architecture (specifically minGPT (Brown et al. (0020) can significantly outperform commonly-used policy networks, like MLP and LSTM, in learning from large-scale, human-generated demonsstrations, which typically exhibit high variance and multiple modalities. To cover the multiple modes in expert behaviors, the authors suggest dividing the action a into two components: its corresponding action center [a] and the residual action (a), such that a = [a| + (a). The set of action centers can be acquired by applying K-means clustering to the expert actions. Correspondingly, they apply policy network with two prediction heads, one for the action center and the other for the residual action. Explicitly employing clustering to identify the various modes in expert actions and utilizing dual- head predictions to reason the mode of each instance significantly aids in modelling multi-modal actions. However, this approach does not model the modaityy that may exist in the joint distribution of (s,a) a).
Here, De (st, at) st, and π is implemented
al. (022))
6.3.2 Policy Approximation using Transformers
A scries of studies (Kim et al. 2201aa) Pan et al. 2022); Kim et al. (2022) Zhu et al. (0022) Chen et al. (2023a); Kim et al. (2023); Liang et al. (2023a) have adopted transformers as the policy backbone for imitation learning in complex control tasks, such as vision- robotic manipulation and end-to-end self-driving. As introduced in Section 6.1 the transformer has various advantages over CNNs and RNNs. Firstly, the transformer is adept at processing time-series data (Kim et al. (2022)) by capturing the long- term temporal dependencies between inputs from different time steps. By using the current state as the query and historical data as keys and values, the transformer-based agent can pinpoint vital information in the history for the current decision-malle making through the attention mechanism. Secondly, the transformer can concurrently process various types of input data, such as texts and images (Kamath et al. (2023) texts and voxels (Shridhar et al. (2022)). ) point clouds (Pan et al. 22222) ) enabling it to be the backbone of an end-to-end deep learning agent. Each type of data can be embedded to vectors by (pretrained)
domain-specific neural networks. Then, instead of simply concatenating all these embeddings for subsequent processing, the transformer treats each embedding as distinct tokens. As detailed Scction 6.1] each token possesses its own query, key, and value, and can attend the other tokens for more informative aggregation. Thirdly, the transformer is efficient in processing input that contains multiple entities by treating each entity's 's
54

---

Published in Transactions on Machine Learning Research (08/2024)
representation as token and modeling their interrelations through the attention mechanism. For example, TSE (Liang et al. (2023ab) ) takes the state of the ego vehicle, the status of surrounding vehicles, and lane information as imput, explicitly reasoning the relationship between the ego vehicle and its surrounding entities for self-driving decision- makinn: Silver-Bullet-31 (Pan ct al. (2022) models the relationship of embeddings corresponding to different parts of the manipulated objects and robotic arms through the transformer for complex manipulation tasks; VIOLA (Zhu et al. (2022) extracts a scries of object-centric representations from the visual observation, introduces an extra action token (whose corresponding output is the action prediction), and applies transformer top of these tokens, so that the action token can learn to attend to and focus on task-relevant objects for improved performance in action prediction.
6.3.3 Transformer-based Generalist Imitation Learning Agents
Another rescarch focus in this field is languagge-onnditoone II, aiming at training (robot) agents to follow hu- man instructions. A language instruction, comprising a sequence of words, can be embedded into a sequence of tokens (4,... Im) through a pretrained language model or predefined embodding table. (1) A straight- forward method for language-conditioned IL, as shown in TDT (Putterman et al. (2022)), is to incorporate language tokens into the policy input, i.e., T( ... 11. mm)) Still, π is implemented as a decoder-only transformer, which, as previously mentioned, is capable of processing multiple types of input concurrently. Parcoiver-Actor (Shridhar et al. (2022)) adopts a similar design, but suggests using the Perceiver Transformer (Jaegle el al. (2022) ) as the policy backbone to manage extra long sequences of input (e.g., a sequence of umage//vxell patches from the vision input and word tokens from the language input), (2) MARVAL (Kamath ct al. (2023) uses an encoder-only transformer to process inputs comprising four modalities: instruction lexts (/1,'.. , m,), historical states and actions (Ze-k)... ,IL1),, the current state St, and the current action candidates At. As discussed in Section 6.1 without the Masked MHA component as in the transformer decoder, each input token can attend to every other token for more informative aggrega- tion. However, each forward pass predicts only a single action a,, making it less efficient in training than the decoder-only transformer. In decoder-only transformers, the output token corresponding to the state input st is used to predict a, (i=0.... 0,..., ,t). In contrast, encoder-only transformers require a special action token, i.c., CLS, as an input to attend to all other tokens and capture the fused representation of the entire sequence, and its corresponding output token is utilized to predict the current action dg. Additionally, MARVAL introduces auxiliary tasks, including predicting the masked language tokens and predicting the proportion of the trajectory that has been completed, to increase the amount of supervision for more efficient use of the demonstrattons. (3) Instead of using instruction lexts as conditions, Lang (Hejna et al. (2023) proposes lo predict corresponding instructions from the state sequence as an auxiliary task, to realize language-guider IL. Specifically, each demonstration trajectory divided into m segments, with cach segment (ST,,". ... -1)
associated with a subtask instruction 760 = (1(*) .... ; ..0)) A transformer encoder is applied to extract repre- sentations (z., .z,) from the state sequence (s1,.... ,8)). These representations are expected to encompass information essential for predicting both actions and instructions, achieved by minimizing the following equa- tion: - ETT log ग (a,l2), ... 2x) > Σ Σ) log P(1()1()) .... .15h1 21- 2,. -T.-))
This approach offers
alternative way to incorporate language instructions into action predictions and has proven effective when training with a limited amount of demonstrations.
Language-condittonel IL can be viewed as an instance of multi-task IL, where language texts serve as task contexts. Next, we introduce three approaches to multi-tase task IL, including context- condetioooo IL, pretraining & fine-tuning, and generalist agent. development. (1) Generally speaking, (h, (.,) could be substituted with any form of task context (Furuta et al. (2022) ). For example, Dasari & Gupta, (2020) suggest using demonstration videos from human operators, denoted as v, contexts for corresponding tasks. They train the policy T =(ae|st..... .... v) v) using Eq.
(81 supplemented by an inverse model auxiliary loss term. (2) Transformer Adapter (Liang et al. (2022) presents an approach for pretraining and fine-tuning. Initially, a transformer policy <(a (k at-ko ... 8t- 1, (1),, without (ask contexts, is pretrained on extensive multi-task demonstrations using Eq. (8T). It is then fine-tuned with a limited amount of spsk-specific demonstrations before application. For efficient fine-tuning, the pretrained model's parameters are kept unchanged. Instead, lightweight adapters (Houlsby et al. (2019)) are introduced between the pretrained model's layers, with only these adapters being updated using the task-specific demonstrations. The structure
55

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm
Key Novelty
Evaluation Task
CARLA, Block Push, Franka Kitchen
Dual-head predictions for the action center and
BeT
residual action, seperately.
TSE
A transformer-baseed policy for end-to-end self-driving.
SMARTS
BilverBBullet
A transformer-basee control policy for complex manipulation skills with a robotic arm.
ManiSkill
-3D
Sorting, Stacking, BUDS-Kitchen,
Adopting the transformer to identify task-relevant objects in a multi-object environment.
VIOLA
real-world tasks
TDT
Language-conditioned IL with the transformer.
Atari Fiostbite
Applying the Perceiver Transformer to manage long input sequences in Language-conddtionee IL.
RLBench,
Perceiver-Actor
real-world tasks Matterport,
Segmenting the input into modalities and applying various auxiliary lasks for more efficient. use of the demonstrations.
MARVAL
Gibson
Improving long-horizon imitation by representation
ATbyAI, Crafting,
Lang
learning based on instruction predictions.
ALFREAD
Transformer
Introducing Adapters to pretrained Transformers for lightweight task-specific fine-tuning.
World
Adapter
MctaWorld,
DualMind
A dual-phase training framework for generalist agents.
Habital
Training an interactive agent by imitation of human-human interactions and self-supervision via modality matching.
MIA
3D Playhouse
Table 9: Summary of transformer- IL algorithms. Representative (but not all) algorithms in this section with their key novelties and evaluation tasks are listed in this table. These algorithms are all grounded in the Behavioral Cloning framework, hence the algorithmic and theoretical advancements in this section are relatively modest. However, the learned transformer-basel agents are evaluated on much more diverse, realistic, and challenging tasks. First, there are some commonly-used benchmarks. CARLA (Dosovitskiy ct al. (2017)) simulated environment for sclf-driving tasks. Block Push (Florence et al. (2021)). ), Franka
Kitchen (Gupla et al.] (2019a)), RLBench (James et al. (2020)), MetaWorld (Yu et al. (2019c) provide
range of robotic manipulation tasks, featuring various types of robots and input modalities. Habitat
[Ramakrishnan et al. (2021)) provides navigation tasks for embodied Al in large scale 3D envirouments. BabyAl (errAl(CChvvliee (Chevalier-Bie (Chevalier-Boisvertt et al. (2019)), Crafting (Chen et al. (2021c)), ALFREAD (Shridhar et al. (2020))
offer instructiion-conditione tasks and datasets. Second, some al algorithms have shown superior performance on competitions. In the table, SMARTS refers to the NeurlPS 2022 Driving SMARTS Competition (Rasouli let al. (2022]). ), and ManiSkill refers to SAPIEN ManiSkill Challenge 2021 (Mu et al.) (2021)). Third, in the table, VIOLA, TDT, MARVAL, and MIA each develop their own environments or create large-scale training datasets tailored to their specific purposes. These efforts can be regarded as significant contributions alongside their algorithm designs. Readers are encouraged to read the respective papers for details.
of the adapter can be Adapter(X) = X (W where GeLU (Hendrycks & Gimpel 2023 is the activation function and WiA, are the weight matrices of the adapter, and it is inserted between the point-wise FFN layer and Add & Normalization layer of the transformer (as shown in Fig. 1 The intuition behind this paradigm is that agents can acquire a diverse set of behavioral priors through large-scale task- agnostic pretraining, which then enables them to be efficiently fine-tuned for specific tasks. (3) As introduced in Section 6.2.4 the ultimate goal of multi-task learning is a generalist agent, i.e., one model with a single set of weights that can be directly applied to a wide range of tasks. Different from Gato, which is directly trained on batches of prompl-conditioned demonstrations, DualMind (Wei et al. (2023) introduces a dual- phase method. Specifically, in Phase I, the transformer policy T(ae) St-k; Qt-k: St-1, a;11,5) is trained with sclf-supervised lcarning objectives to capture generic information of statc-action transitions, where the self-supervision involves predicting the next state, predicting the current action, and reconstructing masked actions based on the other elements. In Phase IT, a prompt p, which can be either language image tokens,
56

---

Published in Transactions on Machine Learning Research (08/2024)
is added as condition to the policy, i.c., T(aelp, St-k, 58-1, "0 To achieve this, extra Cross Attention layers are incorporated into the transformer decoder, enabling state/action tokens to attend to the prompt tokens. A small fraction of the transformer, including the Cross Attention layers, would then be trained via the conppt-conditonne II, which is similar with TDT, using expert trajectorics with associated prompls. This method emulates how humans learn to act in the world, i.e., acquiring skills in a lask-agnostic manner and then learning specific tasks based on the acquired knowledge, and achieves superior performance in a series of robotic manipulation and navigation tasks. Abramson et al. (2021) propose MIA, multi- modal interactive agent that can naturally interact and communicate with humans. The agent can be simply trained by imitation learning of human-human interactions and self-supervised learning through an auxiliary task. Specifically, a multi-modal transformer is used to extract representations from the vision and language input lokens (i.e,, sf, and is trained through the following objective:
max ED log x Taul st), ... 560 , sb)) + + x.I.D
Σ
T
[lo D(f(s) sb)) + log(1 D(F(S ttin (82)
Here, f denotes the transformer to extract reprosentations, x is the overall policy which comprises multiple components besides ʃ, is a discriminator (as in GANs) which is introduced to encourage f to generate and unmatched ones (i.e., (sK, .f)).. This modality matching auxiliary task, also used in (Mees et al. (2022a)), ) has shown to significantly improve the agent's performance.
distinct representations for matched vision and text data (i.e., (f), (.))
We summarize the representative algorithms discussed in this section in Table g highlighting their key nov- elties and evaluation tasks. As mentioned in the beginning of this section, transformer-based IL algorithms fundamentally rely on the straightforward BC framework. However, these algorithms have demonstrated promising outcomes in complex robotic manipulation tasks, both simulated and real-world. This leads to an inspiring paradigm for robust robotic learning, that is, using a potent foundation model trained by imitation learning on extensive, high-qualily demonstrations. Future research directions may include developing meth- ods to synthesize high-quality training data at lower costs or enhancing/ sehhancingrreplccnng the foundational algorithm BC with more advanced alternatives.
7 Diffusion Models in Offline Policy Learning
In this section, we present applications of Diffusion Models (DM) in IL (i.c., Section 7.2 and offline RL (i.e., Section 7.3 In particular, there an emerging body of researches on DM-based ofllin RL algorithms, while more extensions/
extensions/eexploatioon /explorations could be made for DM-based IL.
7.1 Background on Diffusion Models
Diffusion models have demonstrated superior performance across multiple domains, such as computer vi-
sion (Amit al. 2021 Baranchuk et al. (2021), natural language processing (Austin al. 2021b Hoogeboom
et al. 2021 and multi-modal learning (Avrahami et al. 2022 Ramesh et al. 2022). showcasing their impres- sive capabilities in generating detailed and diverse instances. Diffusion models contain two interconnected processes: the forward diffusion process and the backward denoising process (Yang et al. (2022a) ). Specifi- cally, the forward process is predefined and transforms the data with a certain (but unknown) distribution, i... x ~ (x),), into a (standard Gaussian) random noise 2. This process progressively corrupts the imput data by adding a varying scale of noise at cach diffusson step. Correspondingly, the reverse process uses a neural network as the denoising function gradually undo the forward transformation to reconstruct the data x from the random noise 2. There exist three main formulations of diffusion models: Score-based Gen- erative Models (SGM) (Song & Ermon 2019 2020), Denoised Diffusion Probabilistic Models (DDPM) (Sohl- DineetiiiDifffeettial--ggee Sariaal--uuaatins-bsssssse basod models (Score SDE) Song el al. 2020 2021). Next, we introduce these formulations by illustrating their
Dickstein et al. 2015 Ho et al.] 2000 Nichol & Dhariwal (021). ) and
forward/bacee backward processes and learning objectives, while discussing their connections with each other.
SGM learns (Stein) score functions (Hyäärinen (2005) ) for samples at each diffusion step, i.c., S)) & E (I.... ; T). As defined, the Stein score function of a is the gradient of its log likelihood
57

---

Published in Transactions on Machine Learning Research (08/2024)
V. log P(x), with which data samples ~ P(.) can be generated via various efficient score-based sampling schemes such as (Song & Ermon (2019) [Joicoeur-Martineal et al. (2021) )) Specifically,
the forward process of SGM starts from Zo ~ (x(+) and injects intensifying Gaussian noise to generate Ti.T (31rr is a predefined schodule of variance levels.)
F(-[zo) = (To, 8,1), 0 31 < B2 < ..... ST
(83)
The score functions for each step Sp(z,, t) are trained to approximate Vx, log P(xi), where P(x.) F(z: In particular, the training objective is as below:
min
[A(O]lE+ + V (
(84)
Here, U[] is a uniform distribution on 11, [T, A(1)'s are positive weighting functions. This objective is derived based on the definition the forward process, i.e. T2 = Zp + v VB,e, e e~ (N(0,
For detailed derivations and the definition of A(t), please refer to (Song & Ermon (019 Regarding the backward generation process, the score functions Sg(x,, t), E [T.... 1|, are soquentially used as denoising functions to generate xz-1 from dt. Finally, the data samples so Px(+) can be acquired. As mentioned, multiple score-based sampling schemes can be adopted in this process.
We take the annealed Langevin dynamics sampling scheme (Song & Ermon (2019)) as an example: (FT = 24. zo = =8)
1 2
xt = of + S( + Viie (i=0.-- ... ,N -NW-,, 1), 2.-1 = x
(85)
N and 9t are hyperparameters, denoting the number of iterations and step size for denoising x! to 2 (t it=TT + 1).
DDPM gradually adds random noise to the data over a series of time steps (zo. ,r) in the forward process, where xo = x, Ir = z. In particular, the sample al each time step is drawn from a Gaussian distribution conditioned on the sample from the previous time step: (BIr are predefined.)
F((2+-1) NV B,x.-1, -
(86)
With Eq. (86). the sample at each step can be expressed function of mn: x, Vorso / oge; where ay = o1 - 3a), e ~~ (0, (Sohl-Dickstein et al. (2015)). aT is designed to be close to 0, so that zT approximately follows (0, 1). Conversely, through the reverse denoising process, ør is converted back to To step by step: (t (T=T-4 11)
xt- Go(-lx.) = (NO((Z t). tx))))t)
(87)
where μs and Eg can be implemented as neural networks. The objective for learning this denoising function to match the joint distributions of forr in the forward and backward processes, i.e., ming DKL(F(an) ,rr))lGG((aa ".r,,,rr))))))))))) ,T)),, which equivalent to: (Sohl-Dickstcch) et al. 2015
min Ezo- Pxxxooxx()x_rr (() log P(It) Σ Go(2)-1(zz) log F(x (x.-1)
T
(88)
0
Note that P(xr) and F(z ((=))))-1) have analytical forms and this objective is an upper bound for the
negative fog-likelihood Ex [- log Po (xo)) Although this objective can be directly optimized
through Monte Carlo sampling, Ho et al. (2020) propose a reformulation it for variance reduction. Since all (forward or backward) transformations are based on Gaussian distributions, by specifying the variance schedule 31T and fixing the backward variance Eg(re, be B,I, Eq. (8 can be
converted to: (Ho et al. (2020)
min E, (1.7 [A(t) colVarso VI- vI,a
(89)
58

---

Published in Transactions on Machine Learning Research (08/2024)
Here, X(t) is positive weighting function which has closed-form but in practice, X(t) is set as 1 for all & to improve sample quality. Intuitively, the denoising function co is trained to predict the noise injected to samples at each step. Also, by setting eo (z t) (V,,S t), the objective forms (89)) can be unified. Finally, with the learned co, the sampling vI-a. (0 + vBee, e NN(,
of SGM (Eq. (84) and DDPM (Eq.
process 1 Gg(-(z)) equivalent to: ft-1 VI-3.
1 (3)
Score SDE extends the discrcto- time schemes of the previous two methods a unified continuous- time framework, building upon stochastic differential equations (SDE), In the forward process, it perturbs data to noise with diffusion process governed by the following differential equation:
dx f(x dt + g(t) dw, t EE [0,
(90)
where w is the standard Wiener process RRicciardi (9776) ). f(r,t), g(t) are the diffusion and drift functions, respectively. In particular, the forward process of DDPM and SGM can be described by the following two equations (Song et al. (2020)), respectively, with specified f(z, t) and g(t):
dr = 2 (()x dt + B(t) dw, de = ---aar /d(()) dw
(91)
dt
The reverse process be realized via solving the reverse-time SDE Anderson 1982
dr = [f(,,t) g V log (x())) dt + g(t) dw
(92)
where w denotes the standard Wiener process when time flows backwards, P((x) denotes the distri- bution of at time in the forward process. Similarly with SGM, the score function at each time Vz, log (a(a)) can be estimated with an NN-based function So( through various score matching techniques (Vincent) (2011); ; Song et al. (2019).. Further, in (Song et al. (2020)), they propose a more computationally-efitn -efffcient denosing process by removing the random noise injection g(t) dw and prove that the resulting equation is a probability flow ordinary differential equation (ODE) which shares the same marginal densities as those of the reverse-time SDE. Both equations allow sampling from the required data distribution, i.e., Px(x).
7.2 Diffusion Models in Imitation Learning
Most works regarding applying diffusson models in IL are based on the Behavioral Cloning (BC) framework [Pomerleau (1991)), which is introduced in Section 2.2.1 Diffusion models have shown superior performance in modeling complex, high-dimensional data distributions. Thus, many recent works have proposed to improve BC by implementing the policy network as a diffusion model to model the conditional distribution (ais(als) within the expert data DE- Further, as shown in Section 7.1 the objective for training DM (i.e., Eq. 88 provides an upper bound for the negative liglliielhood (i.e., the objective of BC), which naturally connects DM with BC. Next, we introduce these works in details, which are designed to address drawbacks of the original BC algorithm from multiple perspectives.
7.2.1 Policy Approximation using Diffusion Models
The first group of works (Pearce et al. (2023): : Chi et al. 2003 Reuss et al. (2023) try to improve the expressiveness of the learned policy. The BC policy Te is trained give out point estimates 17 which precludes it from capturing the multi-modality of the statc-action pairs in Dg. Moreover, To is encouraged to learn an average distribution, as the objective (i.e., Eq. (11)) is to maximize an expectation, which would result in bias towards more frequently occurring actions. Additionally, so(a))) is usually implemented as a diagonal Gaussian policy, thus the prediction of each action dimension is independent, potentially leading to uncoordinated behaviours in high- high-dimensional action spaces. On the other hand, DM has shown great potential in generating high-dimensional samples that adhere to complex, multi-modal distributions, while ensuring the precision and diversity. In this case, Diffusion BC (Pearce et al. (2023) is proposed to utilize
17 Each data point corresponds a state-action pair (s.
59

---

Published in Transactions on Machine Learning Research (08/2024)
the denoising (score) function eg in DDPM (Ho et al. (2020)) to generate actions at given states Similarly
with Eq. 899, the IL objective is as follows: (Please refer to Section 7.1 for notation definitions.)
min F... 0
[[]e ca (a = vara + v ore, 4 +)11]
(93)
Note that $ is the conditioner which not interrupted in the diffusion process. With the learned en, the estimated expert action a = a0 can be generated following the denoising process in DDPM:
B. vI a4
01-1=
VI Br (a=
(a t;s) + BBe, t= T....... ~ 1(0,
(94)
Further, Diffusion Policy (Chi ct al. (2023) is proposed to utilize DDPM for closed-loop action-sequence prediction. In particular, it models the distribution (a)(a in Dg, where and ă denote the previous T, states and future Tn actions, respectively. Ta out of the T, actions are executed without replanning. This framework allows long- horizon planning, encourages temporal consistency of the action sequence, and remains responsive to the changing environment through receding horizon planning. In order to learn the joint distribution of the T, actions conditioned on the historical states - a task that is inherently high- dimensional - Chi et al. (2023) suggest modeling it as co(a,, t which is trained and sampled with a DDPM framework as delineated in Eq. 93 and (44).
Last, Reuss et al. (2023) propose BESO for goal-condiiioned IL. Specifically, one or more future states within the same trajectory (s, a) are used the goal g. The objective is now to get goal-conditioned policy To(al (,l,),) g), which is learned as conditional score function Sa(a,, 8,g)) in a Score SDE framework (Karras let al. (2022)). ). With Sa t; s,g), the action at s targeting can be generated with a Probability Flow ODE, as introduced in Section 7.1 Goal-conditioned policies distill useful, goal-oriented behaviors, which can be integrated with a high-level planner for various downstream tasks, as in hicrarchical RL. However, incorporating the goal conditioner amplifies the multimodal nature of the demonstrations,, since the same goal might be achieved via various distinct trajectories, underscoring the necessity of employing DM. It's worthy noting that all three works suggest using transformers the denoising function (i.c., ca. Sq) for sample quality, rather than the commonly-used U-Nets (Ronneberger et al. (015 for DM.
7.2.2 Addressing Common Issues of Imitation Learning with Diffusion Models
Boyond enhancing policy expressiveness, numerous studics have explored the use of DM to address other challenges in II, including spurious correlations (Saxena et al. (2023)) and imperfect demonstrations (Wang) et al. (023e);; Yuan et al. (2023b)). To be specific, due to spurious correlations, expressive models, such as DM, may focus on distractors that are irrelevant to action prediction and thus fragile in real-world deployment, which resembles the causal misidentification issue introduced in Section 3.3.5) Thus, Saxena
et al. 20033 propose C3DM to improve the denoising process of Diffusion BC as follows:
1 3. (ar Bt VI- co (;81)) + VBe, t= TT, 11, or N(O,T) (95)
8t C(s: 0
Compared with Eq. 94 the only difference is to replace the conditioner s with 8, which is updated with the denoising process. In particular, at each demosing iteration L, the image state is zoomed into the region around the intermediate action, i.e., pos(at), to acquire more details for decision decision-makinn while ignoring distractors in other regions. Note that this work assume the access to transformation between the action and state spaces determine pos(ar). Accordingly, they modify the training process in Eq. (93) by replacing with C(s: (a) t). Further, considering that expert demonstrations in real-world scenarios are often noisy, Wang et al. (202ee) propose DP-IL to adopt DM for purifying/dennoising denoising the demonstrations. Subsequently, any IL algorithm can be applicd to these purified data. To be specific, based on the DDPM framework, they view (8, a) as the data point x (in Eq. (89)) and learn a denoising function co(x;, ()) to recover the original state-action demonstrations (i.e., zo) from samples interrupted with Gaussian noise (i.e., xx). Note that this training process is based on set of "clean" " demonstartions. Then, for imperfect data points ã, the learned DM can be used to purify them by first adding random noise to get &, (i.e., via the forward process) and then using ég in the reverse denoising process to get the purified data To- Theoretically, they prove that 5;
60

---

Published in Transactions on Machine Learning Research (08/2024)
IL
Targeted
Algorithm
DM Type
DM Usage
Evaluation Task
Framework
IL Issues
Multimodal distribution
Diffusion BC
DDPM
Policy Generator
BC
Kitchen, CSGO
Multimodal
Kitchen, Push-T,
Diffusion Policy
DDPM
Policy Generator
BC
Distribution
Block-Push,
& HD data
Robomimic
Kitchen,
Multimodal distribution
BESO
Score SDE
Policy Generator
BC
Block-Push,
CALVIN
Spurious
Autodesk,
C3DM
DDPM
Policy Generator
BC
correlation
Real Robot
Not
Noisy
DP-II
DDPM
Data Denoiser
MuJoCo
limited
data
Noisy
SMILE
DDPM
Policy Generator
BC
MuJoCo
data
Table 10: Summary of DM-based IL algorithms.
Specifically, "HD" is short for "high-dimensional" Re-
garding the evaluation tasks, Kitchen GGupta et al. Robomimic (Mandlekar et al. 2021b). CALVIN (Mees et al.
2019b). Block-Push & Push-T (Florence et al. 2022
2022b) are robotic manipulation tasks in dif-
ferent scenarios; CSGO (Pearce & Zhu 2022 is a 3D First-person Shooter video game; Autodesk (Koga
ct al. 2022) and Real Robot (Saxena et al. 2023) represent robotic manipulation tasks built on the Autodesk
Sinulatoo and real robot platforms, respectively; MuJoCo Todorov et al. (2012) provides a series of robolic locomotion tasks.
and by can be arbitrarily closed in distribution as increases, and so cø trained on 2s can be used to convert ã, into purified demonstrations as well. Different from the two-stage framework in DP-IL, SMILE (Yuan) let al. (2023b)) proposed to jointly perform the automatic filtering of noisy demonstrations and learning of the expert policy. is based on Diffusion BC (i.e., Eq. (93 and (4)), but changes the forward diffusion process of DDPM as follows:
~ (-1s), T (aals) - (anl)))))a dag, of =
32
(96)
1=1
Herc, 7 denotes the underlying policy of the provided demonstrations. Through this equation, the diffusion process of SMILE conducted on a policy level rather than on data points, and so the learned denoising function 6p can be used to compare the suboptimality among behavior policies underlying different trajec- tories (i.e., with Eq. (13) in (Yuan al. (2023b))). Trajectories generated using policies noisier than the currently learned one are then filtered out from the training dataset.
While not exclusively designed for DM-based II, the studies by Shi et al. 2023 and Sridhar et al. (0223) introduce techniques namely, AWE and MCNN that further enhance the performance of Diffusion Policy and Diffusson BC, respectively. Both techniques focus on mitigating the compounding errors IL, i.e. prediction errors compounded over the decision horizon. AWE can be applied to automatically extract waypoints within an expert trajectory. The waypoint sequences are notably shorter in horizon and can be used for waypoint-only imitation. On the other hand, MCNN does not modify the training data but enhances the policy for reduced compounding errors. During evaluation, it identifies the nearest neighbor state of the current state in the expert dataset, and then blends the corresponding expert action with the output of the learned policy network to formulate an error-constrained policy.
In Table [0 we present a summary of DM-based IL methods. This includes their base IL algorithms, the specific IL challenges they aim address, and how DM used in resolving these issues. There are several promising future research directions in this field. Firstly, while most current works rely on DDPM due to its robust performance, future studies could explore more advanced DMs, particularly those with
61

---

Published in Transactions on Machine Learning Research (08/2024)
efficient sampling schemes which are crucial for sequential decision-making -> Secondly, given that these works target various II, issues, the development of a unified (DM-based) algorithm capable of simultaneously resolving multiple issues of IL (which are well summarized in Section 2.3 is also a promising direction. Finally, integrating DM with more advanced IL frameworks, as listed in Table G could offer solutions to the fundamental issues of BC.
7.3 Diffusion Models in Offline Reinforcement Learning
Recently, there is an emerging body of advancements in applying diffusion models to offline RL (Zhu et al. (023c ). In particular, diffusion models can be adopted as the policy, planner, or data synthesizer in the context of offline RL, as introduced in Section 7.3. 7.3.3 Additionally, we present the applications of DM for extended oflline RL setups in Section 7.3.4 For each category, we introduce the seminal works in details tutorial on its paradigm, followed by a brief review of the extensions with focus on their key novelties.
7.3.1 Diffusion Models as Policies
As introduced in Section 2.1.1 in order to alleviate the value overestimation issue caused by OOD actions, four offline RL schemes developed. Among them, DM-based offline RL usually follows the policy con- straint method, that is, constraining the learned policy ny to the behavior policy p while maximizing the Q-function:
max EgwD,, [Ba-=octa Qo(s, a) - (DKx(To((ls))lltt
(97)
8
where Qo is a learned Q- Q-function for the current policy Te, and À > 0 is the Lagrangian multiplier 1D Such an optimization problem has a closed-form solution, i.e., (als) = z exp( e Qo a)), where (s) is the partition function for normalization. Suppose that x² can be represented as a parametric function, then the optimal policy can be learned through weighted regression (Wang et al. (020 ) as below:
max (s,,)),D,
log mo(als) exp( 1 Z(s)
1
A (o(s.aa) [
(98)
Therefore, the parametric policy To should be expressive enough to recover the optimal policy.
In this case, DM can be employed to model the policy лe. (1) SfBC (Chen et al. (202bb)) provides a straight.orward manner to realize this. It first. imitates the behavior policy p(aas) with a DM To (alss) (specifically, Score SDE) in an IL scheme as introduced in Section 7.2.1 Then, for any state s, N actions are sampled with (-(-ss) as candidates, and one action resampled from these candidates with exp( exp(+Q0(s Q a)) being the sampling weights. In this way, a ~ (-1s) is approximated via importance sampling. Note that (s a) can be learned with any offline RT. protocol. IDQL (Hansen-Fstruch et al. (2023) adopts the same resampling scheme but imitates u(als) with DDPM. Additionally, rather than using exp( (Q they design sampling weight based on the Implicit Q-Learning framework (Kostrikov et al.) (2022) ), which is a SOTA offline RL algorithm.
(2) Alternatively, the Q-function Qp(s. a) can be directly involved in training the DM policy Ta(als), In particular, Diffusion-QL, (Wang et al. (2023f) learns a DDPM-based policy through the following objective:
7 min (DM B(,,a)D,, [Q a)ll D [Q an)]
(99)
0
Here, (0) is defined as Eq. 93 (i.e., an IL objective) and can be replaced by the corresponding training objective if a different type of DM is employed, no is implicd by the denoising function ca, on is the action sample after being denoised for T iterations (with Eq. (94)), B(,,,) ( a)]] is incorporated for adaption to Q-functions with different scales. Intuitively, the Quuunction acts as guidance in the reverse generation process of the DM. Also, Eq. (39) mirrors Eq. (97). since both equations encourage the learned policy to maximize Q-values while being closed to the behavior policy, and it's noteworthy that DiffCPS (He et al. (2023b)) provides a theoretical connection between Eq. (99) and Fq. 977
At each time step, sample (=(s),) the entire denoising process (with iterations) shown Eq. executed, thus the sampling efficiency of DM essential. 19 As common practice, À be fine-tuned as a hyperparameterr, controlling the tradeoff between the two objective terms.
994 needs to be
62

---

Published in Transactions on Machine Learning Research (08/2024)
Following Diffusion-QL, several algorithms adopting a similar objective design with Eq. (99 have been
proposed, including SRDP (Ada et al. (2023)) ). EDP (Kang et al. (023) and Consistency-AC DDig &
Jin )2 To be specific, SRDP additionally introduces a statt-reconstruction loss term in Eq. 99 to
enable gencralization for OOD states through self-supervised learning (Liu et al. (2023a) ). EDP improves
the sampling efficiency of Diffusion-QL. by modifying the sampling process do ~ TT (T)))s)) in Eq. (97 Instead
of iterative sampling with Eq. (94), it first samples at via the forward process and then approximates Go
as Va.
-I-eL _ you co(ae, t; s), since a = Varao + vI - age and Eg is trained to estimate c. Similarly, as a
new variant of DM, Consisteney Models (Song et al. (2023)) aim to learn a consistency function fo that can map the noisy sample al any iteration 1 to its original sample in just one iteration. Consistency-AC thus replaces DDPM in Diffusion- QL with a Consistency Model to improve the sampling efficiency.
At last, there are two works: QGPO (Lu et al. (2023) ) and CPQL (Chen et al. (022f)) ) aiming to directly
learn the score function of the optimal policy: (This is derived from the definition of "², where a = 1/x.)
Va, log (arls) x Va, log + Va,
(100)
As mentioned in Section 7.1 with the score function Va, log samples ar ~ xT (-|s) can be generated via various efficient score- sampling schemes 20 Here, the score function Va, log p((els) can be learned with Score-based DM from Du. which is introduced in Section 7.1 but Va,aQ (s, az) relies on the Q- functions over intermediate samples de; which is intractable. As potential solutions, QGPO propose a contrastive learning framework to estimate aQo(s, at). Vt. While, CPQL employs Consistency Models as the policy, reframes the score function estimation in Eq. (100) into an objective similar Eq. (99), and theoretically establishes their equivalence when the weights of the two terms in Eq. (99) are properly chosen.
In Table 11 we provide a summary of these algorithms. All the algorithms in this category have been evaluated on the D4RL benchmark. Readers can refer to corresponding papers for the numeric evaluation results to compare their performance. In particular, we notice that three schemes of using diffusion models as policies for offiine RL have been developed: SfBC & IDQL, QGPO & CPQL, and the other works that follow Diffusion- DLtasiionnnL.aa Among them, QGPO & CPQL choose to directly learn the score function corresponding to the optimal policy, which is quite challenging but promising direction for future works.
7.3.2 Diffusion Models as Planners
at) needs be approximated from the dataset For model-based learning, a dynamic function To (s*ttlss (s",tt)
Dj first, and then planning over the optimal action sequence (a!.... .xhh) can be done by maximizing the return while avoiding OOD actions, of which the objective is shown as Eq. (g). Note that, for clarity, we use superscript k to indicate the time step and subscript t to denote the iteration of the diffusion process. SGP (Suh et al. (2023) proposes to implement the uncertainty mcasure ulss, a) (in Eq. (9) as the negative log-likelihood of (s.a) under a perturbed distribution, i.c., - log Pa ((s( a); Dp) where Po (2: = D... Em,eD, N(x; mi, I2)). Intuitively, this likelihood measures the distance of (s, a) to the dataset D₁, and
large distance indicates a high uncertainty of the point (s, as it may be OOD. Then, to optimize Eq. 99
for trajectory planning, the score function V. log Po Po((s, Dp) needs to be estimated. As detailed in Suh et al. (202))) ), this estimation can be obtained as a denoising function in DM (specifically, SGM).
A more widely-adopted manner for planning with DM, which is firstly proposed in Diffuser (Janner et al.] (2022)), is to fold the two processes mentioned above: transition dynamic modeling and tra- jectory optimization regarding alikk as trajectory modeling process using DM. Viewing trajectories ((=)a=)), ... (s* aa*)))))))))))))) data points (i.e,, x in Eq. (89)), DDPM is used to model the distribu- tion of 7 in p. through Eq. (89] where a denoising function for trajectory generation co(T), t) is learned. Then, the planning can be done by sampling trajectories starting from the curent state using the learned DM. The sampling process is similar with Eq. (44). However, as an RL algorithm, this process is guided by a (separately-trainee) return function of the trajectory samples, i.e., Ja(F) (Bt, Ott and Σ, are hyperpa-
Note that at represents the sample at the t-th diffusion iteration rather than the action at time step 1. 21 (+()) is trained estimate the return of the original trajectory, i.e. J(r) = Exe r(sh, at), where can be the trajectory sample at any diffusion iteration, Le., Tt-
63

---

Published in Transactions on Machine Learning Research (08/2024)
rameters and defined in Section 7.1
Br
(T. T--1 ~ Go(-ITe, 9) = NNu + Eig, Ex); Ur = VT- Bt V1-a
co((), ()) 9= -=V+Jo(f)l (101)
This process is also known as classifier-guidee sampling (CG (Dhariwal & Nichol (0211 )) which is widely adopted for conditional generations with diffusion models. Intuitively, the generation is guided by the gradient (T) along which the expected return J(7) would be maximized, aligning it with RL. In (Janer let al. (2022)), ), the authors connect this guided sampling design with the control-as-inferene
framework
of RL (Levine (2018)) and claim that trajectories from such a generation process follows the distribution P(r|O1:K- = 1); where O = 1 indicates the optimality of the time step k. During cvaluation, the lcarned DM can be applied as follows: (T is the real dynamic.)
[Tr(s") t s*, T--1 G Go(-Ir.g))l-.. at + To(a"), sh+1 ~ -T--s" 1. K
(102)
To determine a², trajectory samples at each iteration are forced to start with the current state, i.c. (T(s)) + & (t= (U=1...,,7) This follows the idea for solving inpainting problems (Sohl-Dickstein et al. (2015)), where the generation for the unobserved part is in a manner consistent with the observed constraints. Morcover, only the first action in the generated plan, i.e., To(a"), is executed without replanning, which aligus with the receding horizon control (Mayne& & Michalska (1988)) Decision Diffuser (Ajay et al. (2023)) adopts similar designs with two key modifications. First, instead of training a return function for classifie-guideed sampling, they adopt classifier-free guided sampling (CFG (Ho & Salimans 2002))), for which a conditional and
an unconditional denoising function, i.e., <o(T,, L; [T)) and co(T,, 1), are jointly trained (with Eq.
(89)
by randomly dropping out the conditioner (T). The trajectory generation process is the same as Eq. (94)
but replaces the standard denoising function (T(T..)) with t; + (1- w), 6). Increasing the value of w would decrease the diversity of samples but aligns the trajectory distribution more closely with P(T|01:x- = 1). Notably, both CG and CFG can be utilized for generating samples that satisfy specific conditions y. In the context of RL, y can be the desired return J(T), constraints to obey, goals or subgoals to achieve (for multi-task or hierarchical RL), and so on. Second, only state sequences are modelled and predicted with the DM, ie, T = (st,..., ,K) and actions are predicted with a separate inverse dynamic model, i.e., a² = (s 3k+1). This is because sequences over actions tend to be more high-frequency and less smooth, making them much harder to predict and model.
Following Diffuser and Decision Diffuser, improvements have been made in multiple aspects such the sampling process (SafeDiffuser, Discrete Diffuser), network structure (EDGI), training objective (PlanCP), and conditioners (TCD). We present brief overview of these advancements in comparison to Diffuser and Decision Diffuser as follows. Notably, only TCD follows Decision Diffuser and the others follow Diffuser. SafcDiffuscr (Xiao et al. (2023) aims to ensure the safe generation of data in the diffusion process. For each iteration 4, the diffusion dynamic u, = Ie-1-a is re-optimized to satisfy certain safety constraints and At
the resulting dynamic of used to update 7;-1 as Ti + At + u/. At is the diffusion time interval which should be small enough. They theoretically show that To generated in this manner fulfills the safety constraints with probability almost 1. Discrete Diffuser (Coleman et al. (2023)) proposes three tiffusion-guidanne sampling techniques for generation in discrete state and action spaces, where Gaussian-based DM cannot be applied. These guided sampling methods can work with discrete DM such as D3PM (Austin et al. (021a) ).
EDGI (Brehmer et al.] (2023)) proposes an equivariant network architecture for the scenario where an embodied agent operates n objects in a 3D environment. This network design takes geometric structures into account and ensures equal likelihood of a trajectory and its counterpart for which specific spatial temmpoal translations or permutations over objects are applied. Better generalization across scenarios, where such translations permutations happen, can thus be realized. PlanCP (Sun et al. 2023) ) proposes to quantify the uncertainty of DM using Conformal Prediction (Vovk et al. (2055)). In particular, . trajectories are generated from DM corresponding to M trajectorics in Dy. each pair of which has a prediction error e,. The conformity among } reflects the uncertainty of the DM. Introducing this conformity term to the DDPM training objective (i.e., Eq. (89) as an auxiliary term can potentially reduce the uncertainty of sampling with DMM Compared with Decision Diffuser, TCD (Hu et al. (2023a)) utilizes more temporal information as the conditioner of the denoising function eg to guide the sampling process. Specifically, following the insight of Decision Transformer (introduced in Section 6.2.1p. the return-to-go and reward at the current time step
64

---

Published in Transactions on Machine Learning Research (08/2024)
are involved as the prospective and immediate conditions, respectively, to reason about the sampling progress while focusing more closely on the current generation step.
7.3.3 Diffusion Models as Data Synthesizers
The performance of offline RL is limited by the provided static dataset Dj.- Once trained with Das the DM can potentially generate a variety of high-quality trajectory data, This newly generated data can then be employod to augment Dy: thereby creating a feodback loop to further improve the DM. Morcover, in an unseen but related task, the DM's inherent generality allows for generation of performef trajectories for fine-tuning the model in this novel environment. Several works have been developed in this direction following either Diffuser (Hwang et al.]
(2023); [Liang et al. (2023b)) or Decision Diffuser (He et al. (2023a ). Specifically, AdaptDiffuser (Liang et al. (2023b)) suggests a male-based method for filtering and enhancing generated trajectories T = ((st, ,a",,,)))))) aa)),... (5th aa*)) from Diffuser, based on the return values and dynamic consistency. Starting with k- = 1, a revised action at is determined using a well-trained or defined inverse dynamic model Tim
1 , (with 3² = =))). Subsequently, the next state is adjusted according to the (learued) environment dynamic function as 3h+1 T(**, a²). Trajectories with states s* significantly deviating from their counterparts & are filtered out. The remaining trajectories f are further filtered based on their returns (F) = Ek= rs (s))),tt). Finally, the chosen trajectories are of high quality and can be used for effective data augmeniation. In a more straight.orwart approach, MTDiff (He et al. (223a)) ) models trajectories = ((s", "a","",,,,,,,,, d²,r"),- .s* a ))* conditioned on expert demonstrations from the same environment, denoted as y. Then, trajectory samples from the Icarned conditional denoising function co(Te, t; y) are diroctly used for data enhancement. For unseen lasks, a small amount of deemnsstrations can be used as prompls, i.e., y, for co(T), y) to generate high-quality training data. These data can then be employed to adapt the DM policy to novel tasks.
7.3.4 Diffusion Models in Extended Offline Reinforcement Learning Setups
Diffusion models have been applied in multi-task, multi-agent, and hierarchical offline RL setups. Still, these methods are built on the foundational algorithms mentioned above, such as Diffusion-QL, Diffuser, and Decision Diffuser. MTDiff (He et al. (2023at) and MetaDiffuser (Ni et al. (2023)) aim to learn task- conditioned planners to solve a distribution of tasks, They utilize offline data categorized by task, i... U.Dp. where μi denotes the behavior policy of task i. Following Decision Diffuser, MTDiff incorporates an expert trajectory from task i, denoted as y/, as an extra condition of the planner specific to that task, represented co(T2, t; (T), y1). Through training, the agent is expected to implicitly capture the transition model and reward function stored in the prompt trajectory yi, and perform task-specific planning based on these internalized knowledge. When being evaluated in a new but related task, the denoising function conditioned on a corresponding demonstration y, can be directly used for planning. In contrast, MetaDiffuser explicitly models the reward and transition dynamics by learning an encoder, denoted as E,y to encode trajectorics into compact representations. The encoder is jointly trained with the approximate reward and dynamic models, which serve as the decoder, within a VAE framework, resembling BoReL introduced in Section 3.2.4
max E n...
af ah of [log To (1 1 a² v) + log r (1* a
.aa,v) v)]
(103)
In this way, y is trained to embed specifc reward and dynamic functions. When planning for a specific task i, the representation y = (T.); T3 ~ Dy,, can be used as an extra condition of the planner. Notably, MetaDiffuser is based on Diffuser and it adopts the learned reward and dynamic models to define the sampling guidance g in Eq. (101) to enhance the dynamics consistency of the generated trajectories while encouraging high return, detailed as Eq. (7) in (Ni et al. 2023)).
DOM2 (Li et al. (0023d) and MADiff Zhu et al. (2023b)) target at fully cooperative, multi-agent offline RL. Built upon Diffusion-QL, DOM2 learns a Q-function and DM-based policy for each agent following a fully decentralized framework. This policy is tailored to map the individual observation of each agent to its own action, which is trained with Eq. (09). Following Decision Diffuser, MADiff is developed for planning over the joint trajectories of all agents. Each agent i has a separate denoising function eğ for planning its corresponding trajectory. To encourage global information interchange for better coordination, the denoising
65

---

D4RL (L, M), Kuka Robot Published in Transactions on Machine Learning Research (08/2024)
Algorithm
DM Type
DM Usage
Evaluation Task
SIBC
Score SDE
Policy
D4RL (I., M, K)
IDQL
DDPM
Policy
D4RL (L, M)
Diffusion-QI.
DDPM
Policy
DARL (I,, M, A, K)
SRDP
DDPM
Policy
DARL (L, M)
DiffCPS
DDPM
Policy
D4RL (L, M)
EDP
DDPM
Policy
D4RL (I, M, A, K)
Cousistency-AC
Consistency Model
Policy
D4RL (L, M, A, K)
QGPO
Score-SDE
Policy
D4RL (L, M)
CPQL
Consistency Model
Policy
D4RL (L, A), dm control
DOM2 (MARL)
Score-SDE
Policy
MPE, MAMuJoCO
LDCQ (HRL)
DDPM
Policy
DARL (L, M, A, K, C)
CartPole, D4RL (L),
SGP
SGM
Planner
Pixel-Based Single Integrator, Box- Mox-Pushing
Diffuser
DDPM
Planner
D4RL (I,, K), Kuka Robot, Unitroc-go-rumee goatroo-g--runnng D4RL (L, M)
Decision Diffuser
DDPM
Planner
SafeDiffuser
DDPM
Planner (D)
Discrete Diffuser
D3PM
Planner DD)
DARL (L)
EDGI
DDPM
Planner (D)
Kuka Robot
PlanCP
DDPM
Planner (D)
D4RL (L, M)
TCD
DDPM
Planner (DD)
D4RL (L)
MetaDiffuser (MTRL)
DDPM
Planner (D)
Multi-task MuJoCo
MADiff (MARL)
DDPM
Planner (DD)
MPE, SMAC, MATP
HDMI (HRL)
DDPM
Planner (DD)
DARL (I, M), NeoRL DARL (L, M), Kuka Robot
Planner &
AifftSiffuser
DDPM
Synthesizer (D)
Planner &
MTDif (MTRL.)
DDPM
Meta-World-V2
Synthesizer (DD)
Table 11: Summary of DM-based offline RL algorithms. Some algorithms in this table can be categorized as multi-agent RL (MARL), multi-task RL (MTRL), or hierarchical RL (HRL). When DM are used as
planners, most algorithms can be viewed as extensions Diffuser (D) or Decision Diffuser (DD). Regarding
the benchmarks, nearly all algorithms in this category have been evaluated on D4RL (uu al. (2020)
which provides offline datasets for various data-driven RL tasks, including Locomotion (L), AntMaze (M),
Adroit (A), Kitchen (K); and CARLA Autonomous Driving (C). dm control Tassa al. (2018) Kuka
Robot (Janner et al. (2022)), ) Unitree-go-runinng Margolis & Agrawal (2022). CartPole (Tedrake (0023
Box Box-Puse Pushing (Manuell et al. (2020)) are scries of continuous (robotic) control tasks. Pixel-Based Single
Integrator (Chou & Tedrake (2023) requires dynamic learning and control over the pixel space. NeoRL (Qin
et al.
2022)) is an industrial benchmark on financial decision making. Lastly, Multi-task MuJoCo [Mitchel]
et al.
2021 and Meta- World-Ve V2 (Yu et al. (2019c)) are widely-used benchmarks for multi-task/ /meta RL.
MPE 22D control Lowe al. (2017)). MAMuJoCo (robotic locomotion Peng et al. (2021)), ), SMAC (video
gaming Samvelyan et (2019) and MATP (trajectory prediction Alcorn & Nguyen
2021 provide diverse
evaluation tasks for multi-agent RI.
function contains an attention layer to aggregate trajectory embeddings from other agents. Clearly, more extensions can be developed regarding DM-based MARL, such as MARL algorithms for fully competitive multi-agent tasks, and integration of DM with state-of-the-art
or mixed (partially cooperativeccompete centralized training & decentralized execution (CTDE) MARL methods.
LDCQ (Venkatraman al. (023)) and HDMI LL al. (2023c) are proposed to learn hicrarchical policy/planna from an offline dataset, which can especially beneficial for fong-horizon decision-making LDCQ learns high-level policy Thigh (2|8) for skill selection, and low-level policies Now (a)s, 2) for each skill
66

---

Published in Transactions on Machine Learning Research (08/2024)
2. In particular, they employ 3-VAE for skill discovery from the offline dataset D/1 (This objective is the same as the one of OPAL introduced in Section 3.2.4)
max R Dr Po(( 0.0
log 1s -B0=))3)B((((0) (P
(104)
Similarly with OPAL, after training, Pg used as Tow and Po is adopted to convert D₁ to set of transitions s*) for training Thigh- Thigh is modeled as DM and trained in a similar manner with SíBC (introduced in Section 7.3.1. Alternatively, for each trajectory in Du. HDMI extracts its subgoal list following a hcuristic method (Eyscnbach et al. (2019a)). Then, two DMs are adopted to model the sequence of subgoals and trajectories targeting at each subgoal, respectively. In particular, based on Decision Diffuser, HDMI learns high-level planner to generate the subgoal list T2 guided by the trajectory return, i.., Ehigh (Ta,e, t; (()),, and a low-level planner to produce the state sequence тe corresponding to each certain subgoal 2, i.c., Oow (Ta,e: t2
in the form of (s*, = ~ (-(-),) (. - n*,
It's worthy noting that fast. sampling is essential for the application of DM for offline RL or II. Besides adopting Consistency Models, for DDPM-based methods, they usually limit the number of iterations of the reverse generation process to be relatively small and adopt a carefully-designeed variance schedule, i.e., Borr: proposed in (Xiao et al. (- For SDE-based methods, they would solve probability flow ODE for the reverse gencration process (as introduced in Section 7.1 with a fast solver: the DPM-solver (Lu et al. (2022a)). All algorithıs mentioned in this section have been summarized in Table 11 and categorized based on the DM type and usage. When applying DM as planners, most algorithms are built upon DDPM and follow the design of either Diffuser (D) or Decision Diffuser (DD).
8 Discussions and Open Problems
In this section, we provide in-depth discussions on deep generative models (DGMs) in offline policy learning and our perspectives on future research directions of this area, based on the main text in Section 3 7 The discussions are centered around how DGMs have been used in offline policy learning, which provide a
comprehensive summary of this paper and insightful ideas for future works. Note that, for simplicity, we use
abbreviations of the DGMs: VAE Variational Auto-Encoder, GAN Generative Adversarial Network, NE Normalizing Flow, DM Diffusion Model.
8.1 Discussions on Deep Generative Models and Offline Policy Learning
A common usage of DGMs in offline policy learning: All DGMs can be utilized as policy functions in offline policy learning. In the context of IL, they can be the student policy Iearned by imitating the expert, while, in oflline RI,, they can either be the approximated behavior policy the RL policy. To be specific, list their mathematical forms as follows, in the order they were introduced
VAE: 2 ~ Po(1)s,, a~ 2o() GAN: 2~ ~ Pz(),, a = G(2[s);
NF: 2 ~ P2(-), == G(zls), G G1 o...o GN; Transformer: (-(Te-a)):
(105)
DM: ~ a/-1 ~ Go(-la,,s), (T=T.-.- 1), a= ag-
All these DGM-based policies show superior expressiveness compared with the one using only feed-forward neural networks, and each of them has unique aeraantgges/dssavvattteee than the others, but offers lightweight model choice and relatively stable training process; (2) As mentioned
(1) VAEs might be less expressive
in Section 4.2.4 GAN-based polices can generate sharp data samples but may suffer from the mode collapse issue, meaning that they may not be able to cover the multiple modes in the (s, a) distribution of the dataset
(=((s) and Pa(a|s, denote the prior and decoder of the VAE, respectively. Pz (z) is the assumed prior distribution of the latent variable z. G(z[s) denotes the generator in the GAN NE, and it composed of series of invertible and diflerentiable functions, i., -o in the NF. (-()))):) represents transformer-baset policy, where Ta-kit be (st-k, Ge-ks ,8) in HI. and (S k, ag-k, -.. St, Re) offline RL. Go(0t-1lac sa),)) represents the denosing process of DM, and the subscript. denotes the denoising iteration.
67

---

Published in Transactions on Machine Learning Research (08/2024)
like the other DGMs; (3) With its special architecture (e.g., the attention mechanism), the transformer can process time series or data encompassing multiple entities, as mentioned in Section 6.3.2 (4) As for NF and DM, they excel in modelling/ /eeeiirtiiggnngnggggggggggggnggg high-dimensional and complex behaviors, due to their multi- component or multi-iteration designs shown in Eq. (105). As a spotlight, we note that both VAE- and transformer-based policies have been used to make decisions on multi-modal input (e.g., information from different sensors or in different formats like images and languages); but they adopt different strategies. In particular, VA try to embed input from different modalities into a unified latent space, and then a single latent-conditioned policy (coupled with encoders for each modality) can be used to predict actions based on multiple types of input, as introduced in Section 3.3.3 Transformers, as universal foundation model, do not make assumptions on the structure of the input data, and so can be used in various task scenarios such as CV and NLP, as introduced in Section 6.1 Thus, a transformer can directly take a data sequence containing different modalities as input, treat each modality as a separate token, and integrate them via the attention mochanism, as shown in Scction 6.3.2
The unique usages of each DGM for offline policy learning: We notice that each DGM has its own unique usage for offline policy learning, (1) VAËs can learn latent representations of the data samples. These representations, which are potentially discntangled, can be used as data transformation for learning efficiency (Section 3.2.3), task or subtask representations for multi-task or hierarchical learning (Section 8.2.4. similarity measure of data samples for data augmentation (Section 3.3.2), unified representations of multi-modal input (Section 3.3.3), or disentangled representations of the states for mitigating causal confusion (Section (.3.5), etc. (2) GANs utilize an adversarial learning framework to practically realize distribution matching. Specifically, GANs can be used to minimize the discrepancy between the learned policy and expert policy, i.c., d(px(s, a)llexg (s, a)), for II, the discrepancy between the learned policy and behavior policy, i.e., d(p a))p,,,, a)), to mitigale the OOD issue for offline RL, as introduced in Section 4.2. ] and 4.3.1 respectively. (3) NFs can provide exact density estimation due to its special architecture design (i.c., invertiable and differentiable components), enabling them to be incorporated with various advanced IL frameworks, as detailed in Section 5.2.I] We highlight that whenever there is a requirement for exact density estimation, NTs can be utilized. For example, SOPT, as introduced in Section 3.2.2 proposes to approximate the behavior policy's density function, i.e., p(als), using a VAE to apply support-constraitt offline RL. However, the NF is better choice for this purpose since it can provide more accurate estimations. (4) Due to its ability for sequence modelling, transformers enable trajjetooy-gpiinizatton---- RL/IL, as introduced in Section 6.2.land [.3.1 Also, as shown in Section 6.3.3 owing to its high capacity and generalization ability, it can be used as the core component of a generalist agent to handle a range of tasks with a single set of parameters, and it allows for pretraining and fine-tuning as in CV and NLP. (5) Finally, DM can be used to purify noisy training data with its special denoising process, as illustrated in Section 7.2.2, and solve model-based offline RL as trajectory modelling, as shown in Section 7.3.2 due to its efficiency in generating nigh-dimensional data and its special conditional sampling mechanisms (i.e., classifier guided or classifier-free free guided sampling).
oflline
Integrated use of different DGMs for offline policy lcarning: The unique asages/advantages of each DGM form the basis of an integrated use of different DGMs in offline policy learning, and there are already some works exploring in this direction. Here, we list some (but not all) examples introduced in this paper. (1) In Section 4.2.4. we present a spotlight on integrating VAEs and GANs for IL to synthesize their advantages, i.e., to mitigate the mode collapse issue of GANs and the blurry sample issue of VAEs. (2) In Section 5.3 APAC adopts a Flow-GAN to model the distribution of (s,a) in Dy, which can be viewed as an integration of NF's and GANs; SAFER adopts VAE to extract the safety context from the state sequence, which is then used as a condition of an NF-based policy for safe generation.
(3) In Section 6.2.3 SPLT shows a coherent use of VA and transformers, where the VAE is used to capture the multiple modes within the data/ eattr/envvrom as latent variables, which are then used as conditioners of a Decision Transformer to mitigate the impact from environmental stochasticity. (4) MIA, as introduced in Section 6.3.3 adopts a
GAN- auxiliary loss term to improve the data usage in transformer- IL. (5) In Section 7.2.1 all
three DM-based II works implement their denoising functions as transformers, rather than comeonly-used U-nets, for sample quality and policy expressiveness.
68

---

Published in Transactions on Machine Learning Research (08/2024)
DGM
VAE
GAN
Normalizing Flow
Transformer
Diffusion Model
BC (VTB);
MaxEntIRI (AIL)
BC
IL
KI, LIL, AIL
BC
Bayesian IRL
(sell=supervision)
DP PeP/Mdel-bbrr based offline RI, Trajectory optimization
DP-based offline RL
Model-based offline RL
DP-basod offline RL
Trajectory
Offline RL
optimization
Table 12: A summary of the base offline RL/II. algorithms for DGM-based offline policy learning. Max- EntIRL (Section 4.2.1 is short for maximum causal entropy inverse RL. KL, LIL, and AlL (Section 5.2.1 denote the AL-ddvergence-basee
aakelihood-based and Adversarial IL framework, respectively. Da-based based offline RL.
offline RL (Section 2.1.1 refers to the
We provide this summary as Table 12 (1) Regarding IT,, most DGMs, including VA Transformers, and DMs, select BC as the base algorithm, of which the objective (Eq. (11)) is simply to maximize the log likelihood of expert state-action pairs. However, there are differences in the realization. Specifically, as introduced in Section [.3.1 VAE- based IL algorithms either adopt VAF ELBO, which is a lower bound of the BC objectíve, or VIB-based framework, which is close in form with the ELBO. For transformer-baset II, besides the BC term, to make full use of demonstrations auxiliary supervision objectives, such as prediction errors on the next state (i.e., the forward model loss) or the intermediate action between two consecutive states (i.e., the inverse model loss), are often employed. As for DM-basod II, the objective for training DM (i.c., Eq. (88)) provides a lower bound of the log-likelihood, which naturally connects DM with BC, as mentioned in the beginning of Section
A summary of the base /Lfffiine RL algorithms for each DGM:
7.2 On the other hand, VAFs, GANs, and NFs have been integrated with more advanced II. frameworks. As introduced in Section 4.2.1 the fundamental GAN-based IL algorithms are derived from IRX and practically implemented as Adversarial IL (AIL) frameworks; NFs, as exact density estimators, have the flexibility to be adopted in various IL frameworks such as KL, LII, and AIL, as illustrated in Section 5.2. In the representative work, AVRIL (introduced in Section 2.2.2 VAE is employed to scale Bayesian IRL, but this direction remains relatively underexplored. (2) Three categories of offline RL algorithms,, including based, basel- and
based offline RL, are introduced in Section 2.1 as necessary background. The corresponding categories for each DGM are listed in Table 12 More specifically, we note that VAE-based offline RL mainly follows policy penalty, support constraint, and pessimistic value methods, which are subcategories of dynamiic-poorrammingg based offline RL; while, DM-based offline RL mainly follows another subcategory of constraint methods.
baaaed offline RL ppolicy
Seminal works of DGM-based offline policy learning: Regarding the applications in offline policy learning, developments for different DGMs are quite unbalanced, and even for the same type of DGM, the applications in IL may be significantly more than the ones in offline RI,, or vice versa. One main factor is whether there are seminal works in that category, since many research works would follow the seminal ones for extensions. Here, we highlight the seminal works introduced in this paper: VAE offline RT. BCQ (Fujimoto et al.) (2019)), GAN IL GAIL (Ho & Ermon (2016)) and AIRL (Fu et al. (0177)) Transformer offline RL Decision Transformer (Chen et al. [2021b] and Trajectory Transformer (Jannrr et al. 2021) ), Diffusion Model offline RI- Diffuser (Janner et al. (2022) and Decision Diffuser (Ajay et al. (2023)) There are still no seminal works for NF-based offline policy learning, which explains why there are relatively fewer works in Section 5 Notably, some seminal works pioneer new paradigms for offline policy learning. For example, GAIL converts IL/IRL to distribution matching problem 23 Decision Transformer realizes offline RT. via trajectory optimization, which eliminates the necessity to fit value functions through dynamic programming or to compute policy gradients; Diffuser folds the two processes in model-based offline
23 Although GAIL- or Alsed algorithms are used for imitation learning from offline expert data, these algorithms also rely simulators in their learning process, bocause there is inner (onlinc) RL process in their algorithm designs. Similarly, most algorithms within Scction 5.2.1 require simulators, |Peg (8, a)), and RL a natural choice optimization regarding the accupancy measure GAII and AIRL), i.e. min, (s, (P,g) of the policy, ie.. (s(s.a).
they also realize imitation learning via distribution matching (like
69

---

Published in Transactions on Machine Learning Research (08/2024)
Normalizing Flow
Diffusion
DGM
VAE
GAN
Transformer
Model
High-dim/
Sequentia// multi-modal/ multi-entity input, Multi-modal actions, Insufficient demonstrations
multi-modal
Insufficient demonstrations, Compounding error, Multi-modal demonstrations
Exact density estimation, Learning from observations, Policy expressiveness
Insufficient
state-action,
demonstrations,
Spurious
IL issues
Causal con sonfusion, Multi-modal input
correlations, Noisy
demonstrations, Compounding
error
IL extensions
MT, MA, Hier, MB, Safety
MT, Generalization
Hier
N/A
Hier
OOD actions, Insufficient training data, Segment stitching, Sparse reward,
OOD actions, Low-quality training data, High-dim states, Continuous actions
OOD actions, (Vision-based) world model estimation, Segment stitching
OOD actions, Over conservatism, Insufficient training data, Sparse reward
OQD actions, Insufficient training data
Offline RI. issues
Partial
observability
MT, MA, Hier, MB,
MT, MA, Hier, MB, Safety
Offline RT. extensions
MT, Hier
Hier, MB
Hier, Safety
Safety,
Generalization
Table 13: A summary of the issues and extensions of oflline policy learning largeted by the DGMs. MT, MA, Hier, MB represents multi-task, multi-agent, hierarchical, model-based learning, respectively. Safety and gencralization refer if the learned policy can be safely applicd to risky environments be generalizable unseen environments.
RL: transition dynamic modeling and trajectory optimization, into trajectory modcling process. These paradigm shifts are closely related to corresponding DGMs and make full use of their unique advantages, which open up promising directions for offline policy learning. On the other hand, simply using DGMs as function estimators in traditional offline RL or IL methods can be less attractive, unless the application of DGMs offers superior scalability or generalization capabilitics.
A summary of the issues and extensions of offline policy learning that have been targeted by the DGMs: In Table 13 we enumerate the IL/offline RL issues that each DGM has tried to solve, well as the extended ILfoffiin RL setups that have been explored by each DGM. Specifically, there are in total six setup extensions, including multi-task (MT), multi-agent. (MA), hierarchical (Hier), model-based (MB) learning, policy safety, and policy generalization. Readers can easily find the research works targeting at specific issues/ ussues/extensions /extensions in corresponding sections. However, we note that the listed issues issues/extents exteessionss should not considered as completely resolved by the DGMs, and future works can focus on the unsolved or underexplored issues/ ssues/extensions as detailed in Section 8.2.4 Here, we provide some remarks on this table. (1) Low-quality training data is characterized by its limited coverage of the state-action space, a lack of diversity in behavior patterns, the inclusion of suboptimal/ noisy demonstrations, (2) Multi-modal demonstrations iianonsstaaiions/actionsss actions refer to demonstrations caaiinss/tttte /actions containing multiple distributional modes, while by multi-modal input, we mean input from multiple sensors or in various formats (c.g., images and texts). (3) Learning from observations refer to imitating from sequences of states only, which is notably more challenging than usual imitation learning. (4) Compared with BC, GAN-based IL can mitigate the issues of insufficient demonstrations and compounding errors. This bocause GAN-based IL methods do more than just mimicking observed behaviors they reason about the underlying reward function from demonstrations and utilizes it for further RT. training, which complements the static demonstration and enables the agent
70

---

Published in Transactions on Machine Learning Research (08/2024)
to act reasonably at unseen states. However, GAN-based IL methods rely on simulators for RL training and suffer from issues regarding sample efficiency and training stability, as introduced in Section 4.2.2 (5)
Decision Transformer has lower learning difficulty and is less susceptible to the OOD actions issue compared to traditional offiine RL algorithms, because it is based on trajectory optimization, which eliminates the need for dynamic progranminn or policy gradient. calculations, and multiple state and action anchors throughout the trajectory prevent the learned policy from deviating too far from the behavior policy. However, there are still pending issues for Decision Transformer, such as the impacts from environmental stochasticity (Section 6.2.3 and its theoretical shortcomings listed in Section 6.2.5 (6) All DGMs have been applied to mitigate the OOD actions issue of offline RL. These models (except the transformer) rely on gynamic-programing based offline RL but employ various methods to approximate the behavior policy, the support of the behavior policy, or the discrepancy between the behavior policy and the learned policy, based on their unique usages
as generative models. Readers can refer to Section 3.2.2 1.3.1 5.3.1 7.3.1 for more insights.
Comparisons on implementation practicality: Due to the use of different types of DGMs, algorithms reviewed in Section 3 could vary in terms of computation efficieney, implementation difficulty, training stability, and difficulty of hyperparameter tuning. However, there are empirical studies yet comparing these aspects across all DGMs, so the following discussions are based on their architecturalloojectiie designs and our own implementation experience, which might be subjective. (1) Computation efficiency: VAE- based algorithms (denoted as VAFs for simplicity, and similarly for other DGMs) typically have lower computational overhead due to their simpler architectures. GANs can suffer from high simple complexity, since they adopt an adversarial learning framework and actually solve a minimax optimization problem. NFs involve a series of invertible and differentiable neural network components, which can be computationally intensive. This is especially true when calculating the exact likelihoods, Jacobian matrices and their determinants. The computation cost for applying transformers is influenced by the number of modules or attention heads used in the encoder and decoder, as illustrated in Figure D and the length of the input sequence. Specifically, its time complexity is quadratic with respect to the input length. DMs require a high computational cost due to the iterative nature of their generative and diffusion processes. Notably, certain variants of NFs and DMs need to solve stochastic differential equations (SDEs) or ordinary differential equations (ODEs) during the data generation process. Generally speaking, in terms of computation efficiency, (NFs, Transformers, DMs) < GANs < VAEs, where '<' denotes 'worse than'. (2) Implementation difficulty: Both VAEs and GANs consist of only two network components and are relatively easy to implement. Although transformers have complex architectures, most deep learning frameworks, such as Tensorf low (Abadi et al. (2015)) and PyTorch (Paszke et al. (2019)), offer APIs that allow users to directly define a transformer by specifying the number of layers (a.k.a. modules) and attention heads for the encoder and decoder. On the other hand, implementing NFs and DMs can be complicated. NFs utilize invertible and differentiable neural network components, which differ from standard layers such as MLPs or RNNs. Both NFs and DMs involve a normalization direction and generation direction, and they may require the use of an ODE or SDE solver during the generation process. Thus, for implementation difficulty, (VAEs, GANs, Transformers) < (NFs, DMs). (3) Training stability: Both NFs and Transformers utilize supervised learning objectives, which results in relatively stable training processes. VAEs optimize an Evidence Lower Bound (ELBO), like Eq. (17). To avoid potential issues such as posterior collapse (Lucas) et al. 2019) ), where the latent variable z fails to capture meaningful information about data reconstruction, a careful balance must be maintained between the reconstruction accuracy and the KL divergence term.
it requires computation of
GANs suffer from low training stability because the discriminator and generator (i.e., policy) networks compete against each other, attempting to reach a Nash equilibrium. This adversarial process is inherently challenging and can lead to issucs like mode collapse (Thanh-Tung & Tran (2020g). For DMs, the training stability can vary depending on the specific variant of DM selected, the setup of the noise schedule, and the numerical solver used. Particularly, both the training and generation processes of DMs are sensitive to the configuration of the numerical solver. Thus, their training stability can be ranked as (DMs, GANs) < VAEs < (NFs, Transformers). (4) Hyperparameter tuning: The performance of VAEs relies heavily on the weight parameter B (as in Eq. 17)), which controls the importance of the regularization term. For GANs, it is crucial to balance the training and convergence rates of the discriminator and policy networks. Typically, the discriminator converges faster, so it should have larger network structure and a smaller learning rate or fewer updating epochs to maintain this balance. For Transformers, the number of attention heads and layers
71

---

Published in Transactions on Machine Learning Research (08/2024)
should be compatible with the dataset size to avoid overfitting underfitting, and the sequence length time window size must be adjusted according to the computation budget. DMs demand careful adjustments in the noise schedule, the number of diffusion steps, and the setup of numerical solvers. Similarly for NFs, if continuous flows are used, specially-designed schemes for hyperparameter tuning may be required (Vidal
et al. )-23)). Thus, regarding the difficulty of hyperparameter luning, (VAEs, Transformers) < GANs <
(NFs, DMs).
Foundation model & Algorithm & Data: Among all the applications of DGMs, transformer- based IL/offline RL have shown some of the most promising and exciting evaluation results. (1) As listed in Table g most algorithms have seen success in challenging robotic simulators, real-world tasks, or large scale competitions. (2) As shown in Section 6.3.3 the use of transformer enables the process of input encompassing extra long time sequences or multiple modalities/entitien
and (robotic) agents that are able to follow language instructions or interact with humans can be developed solely by imitation and self- supervision. (3) As introduced in Section 6.2.4. Gato (Reed et al. (2022)), ): as single transformer with the same set of weights, can handle 604 distinnt tasks with varying modalities (i.e.; images, texts, robotic control, and video gaming); further, Reid et al. (2022) Takagi (2022) explore pretraining the transformer on language tasks and finc-tuning it on RL tasks, and demonstrate promising cross-modality pretraining effects. All these advancements are based on simple supervised learning objectives and are not related to any theoretical breakthrough. However, they do rely on a high-capacity and computation-efficinn foundation model transformer, and a large amount of diverse training data from multiple modalities. This leads to an inspiring paradigm for developing generalist agents, that is, using a potent foundation model trained by (self-) supervised learning from extensive, high-quality demonstrations. It also prompts the question of whether future advancements in artificial general intelligence (AGT) should be driven by data or algorithms. Nevertheless, it is clear that the development of new foundation models or algorithms should prioritize scalability and computational efficiency, key factors in the success of the transformer.
8.2 Perspectives on Future Research Directions
Based on the discussions above and the main content in Section 3 future research directions, and we categorize the content in this section as four aspects: data, benchmarking, theories, and algorithms. We believe more open problems can be gleaned from the detailed introductions in the main content, and we have included comments on future works specific to certain DGMs in corresponding sections, such as Section 6.2.5 and the last paragraphs of Scction 3
7
we present some perspectives on
6.3
7.3.1
8.2.1 Future Works Data-centric Research
How would the offline policy learning agent cvolve with the quality and quantity of the training data? The performance of offline policy learning is closely related to the quality and quantity of the provided offline data. Therefore, one potential research direction would be investigating how the dataset's characteristics would impact the learned policy, as instructions for building datasets. (1) First, the static dataset may contain noise (i.e., disturbed or misleading information) suboptimal behaviors 24 In this case, open questions include how to measure the noise suboptimality of the provided data and how robust can the policy learning be to these data imperfections. The answers to these questions would vary with the used algorithms, DGMs, or evaluation tasks, but the imperfection measure and robustness evaluation protocol can be general and beneficial for algorithm development. These questions are important because perfect datasots are costly to build in scale and the tolerance for imperfection can greatly reduce the burden for data collection and process. (2) Second, the performance of an offline policy learning agent on evaluation tasks relies heavily on the coverage and diversity of the training data. An effective set of behaviors for policy learning should cover the possible task scenarios and provide various behavioral patterns/skin /kills for the agent to learn. How to measure the coverage and diversity of the datasct and how these properties would influence the generalization of the learned policy on evaluation tasks would be interesting questions.
T4hhi would be an issue for IL, since IL requires expert-level demonstartions. However, for offline RL, the dataset could contain suboptimal behaviors long their rewards are correctly labeled. If there are only suboptimal trajectorics in the dataset for offline RL, the learning difficulty would be high, since the agent needs to learn to stitch segments from various trajectories form an optimal strategy.
72

---

Published in Transactions on Machine Learning Research (08/2024)
A recent empirical study (Park et al. (2024) highlights that the generalization of the learned policy to out- of-distribution (OOD) states (Kirk et al. (2023)) is often the main bottleneck in offline RL. Further, they suggest that the most straightforward approach for improvement is to use a training dataset with extensive coverage, and practitioners should prioritize high coverage over high optimality when collecting training data for oflline RL. (3) One key factor contributing to the success of certain DGMs in CV and NLP their ability to leverage extensive training datasets. Notably, their training performance would continue to improve as the quantity of the training data increases. Thus, for each variant of DGM-based offline policy learning, it is essential to study their scalability, namely, how the learned agent evolves with an increasing amount of training data. Algorithms with superior scalability would be more promising in challenging, real-life applications.
How to construct a training dataset for effective offline policy learning? With efforts for the open problems mentioned above, we could establish measurements for noise, suboptimality, diversity, and coverage of the dataset and get some insights on the relationship between these mcasurements and the performance of the learned policy, which can guide the construction of training datasets for effective offline policy learning. However, there are some other problems that need to be explored. First, the original state and action spaces may be high-dimensional and contimouu,, which would require exponentially more data for policy learning. Utilizing compact representations, which can be obtained from DGMs like VAEs and encoder- only transformers, or discretization could help improve the learning efficiency, but also would introduce inaccuracy. Second, for offline RL datasets, sparse rewards can bring training difficulty but dense rewards are costly to annotate. Thus, a tradeoff needs to be achieved between them and strategically annotating part of the dataset. with well-designed rewards could be promising direction. Similarly, although there are algorithms for learning multi-task, hierarchical, safe action labels, having these labels either fully partially annotated can siguificantly aid in the policy learning process. However, given that real-life datasets can be enormous in scale, the labeling should be only for critical points or regions in the state-action space, which necessitates the development of techniques for their identification.
safe policies from data without the task, skill,
Data Augmentation & Data Synthesis. VAEs, transformers, and DMs have been used for data aug- mentation or synthesis, as shown in Section 3.2.3 6.2.2 and [7..3] respectively, of which the purpose is simply to involve more data for training. The data used for augmentation may already exist but require transformation or supplementation to be effectively utilized, or could be synthesized using DGMs. DGMs exhibit remarkable capability in synthesizing new data that shares statistical properties with the real data. Thus, this approach has the potential to bridge the gap between the demand for large datasets and the leasibility of acquiring them. Multiple research directions in this domain could be developed. First, based on the coverage or diversity indicators of the datasot, targeted data synthesis techniques to supplement the under-covered areas or avoid repeated behavioral patterns for better diversity could be developed. Second, the synthesized data usually cannot be directly employed without filtering. In this case, necessary quality measure, such as the similarity of the synthesized data with the real (optimal) data, should be established. Idcally, there should be (thcorctical) studies on the relationship between the suboptimality of the Icarned policy and the quality/quantin creation of a positive feedback loop: enhanced policies emerging from newly synthesized data, which in turn are leveraged to generate even higher quality data for further training.
of the synthesized data. All in all, it would be quite exciting to see the
8.2.2 Future Works on Benchmarking
Development of more realistic, challenging benchmarks. We notice that most DGM-based offline RL algorithms are evaluated on D4RL, as shown in the tables of Section 3 7 which is standard benchmark for offline RL. However, uidely-used benchmarks like D4RL, cannot fully show or evaluate the advantages of DGM-based offline policy learning algorithms, as we have seen agents, which are implemented as two-layer neural networks and trained with a moderate amount of data, can already achieve excellent performance on these benchmarks. To fully explore the potential and guide the development of DGM- based offline policy learning, more challenging benchmarks are required. (1) First, the benchmark datasct should be large in size to evaluate the scalability of the DGM-based algorithms. This includes evaluating the trend of performance improvement relative to the volume of training data, and determining whether DGM- algorithms can
73

---

Published in Transactions on Machine Learning Research (08/2024)
significantly outperform others when provided with extensive training data. (2) Second, the dataset should contain multiple modalities. For example, state inputs could be collected from various sources, including language instructions, visual information from different sensors, or even videos; moreover, the state-action space should feature multiple distributional modes, such as containing distinct trajectories for achieving the same goal. DGM-based algorithms are expected to effectively harness information from diverse modalities and learn policies that cover the multiple modes within the policy space. Further, benchmarks for evaluating generalist agents should include mixture of challenging tasks from different modalities, such as CV, NLP, robotic control, and strategic decision-making. (3) The task scenarios should be more realistic. A main reason that CV and NLP techniques, which are usually DGM-based, can be widely adopted in real life is that these algorithms are directly evaluated on real-life tasks masks/datasets rather than simplified simulators. Thus, realistic oflline datasel. in a large scale could greatly advance the development of offline policy learning and showcase the superiority of DGM-based algorithms. (4) The benchmark should evaluate more than just the policy return; it should also assess aspects such as the generalization, safety of the Iearned policy, and the robustness of policy learning in environments with noise or sparse rewards.
Comparisons among different DGMs. Although there are numerous offline RL or IL algorithms for cach typc of DGMs and they have shared benchmarks, comparisons among them are quite rare. It would be insightful to see fair and thorough comparisons among different DGMs with the same usage (e.g., as policy backbones) on the same set of tasks. Also, as task complexity or dataset size escalates, analyzing the trends in policy performance and computational cost across different DGMs can provide useful insights. Such comparisons provide suggestions on the choice of DGMs, conditioning on the task difficulty and dataset scale. Moreover, as mentioned in the main text of this paper, algorithms that fall in the same category, such as NF-based offline RL algorithms, are also rarely compared with cach other. Comparisons within the same category can also be valuable, they can indicate which usage of a certain DGM is more effective.
8.2.3 Future Works on Theories
Most works on applying DGMs in offline policy learning focus on algorithm designs rather than theoretical analysis, so more theoretical research can be done for this field. One promising direction is derivations of the convergence rate or performance guarantee for the seminal works in DGM-based offline policy learning, such as (Zhang et al.] (2020b)) for GAIL and (Brandfonbrener et al. (2022)) for eecurncoonditione supervised learning algorithms like Decision Transformer, based on the theoretical results from either DGMs or offline RL/IL. Moreover, we notice that there have been efforts theoretically unifying the objectíves of different DGMs for improved learning performance and deeper understanding of the relationship among DGMs, such as (Nielsen et al. (2020); Kingma & Gao (023) ). This group of works can greatly inspire development of new DGM-based offline policy learning algorithms or unified analysis of existing ones.
Next, we outline some specific theoretical problems in this field. (1) First, regarding GAN-based IL (i.e., Sec- tion 4.2), so far, no approach has been developed that is both computationally efficient and can theoretically guarantee the recovery of the expert reward function. Also, many works propose to replace the on-policy RL within GAIL/AIRL (i.e., TRPO) with off-policy RL algorithms (e.g.. DDPG) for improved sample ef- ficiency. However, this improvement. lacks theoretical backup. With off-policy RI, the policy training at each iteration is based on samples from multiple past iterations, during which the reward function keeps changing. As a result, the RL training is conducted in an unstationary MDP where typical RL algorithms would theoretically fail. (2) Second, as mentioned in Section 6.2.5 ceturn-coddtioone supervised learning (e.g., Decision Transformer) alone is unlikely to be a general solution for offline RI. problems, and it offers guarantees only when the environment dynamics are nearly deterministic. Thus, adapting these algorithms to stochastic environments, ideally backed by theoretical optimality guarantees, is clear necessity. Also, De- cision Transformer would imitate both high-retunn and low-return trajectories, and claims that the learning performance in this manner is better than imitating high-return trajectories only. A theoretical explanation on how the low-relurn behavior learning would benefit the overall performance can be insightful. (3) Third, as introduced in Section [73.2] the algorithm design of Diffuser is mainly based on intuitions from similar problems, such as the inpainting problem and recoding horizon control, but lacks thcoretical support. To be specific, in Eq. {102), to decide on the action choice a at state s, the trajectory generation is hardcoded to start with 8, and then the action å that is right after s in the generated trajectory is adopted a.
74

---

Published in Transactions on Machine Learning Research (08/2024)
The problem here is how to ensure there is causal relationship between and a, which is crucial for (RL) sequential decision-making, and whether it is necessary to generate an entire trajectory segment in case that only the first action prediction is utilized.
8.2.4 Future Works on Algorithm Designs
From the discussions in Section 8.1 and the main content in Section B
7 we can identify many potential
future directions regarding the algorithm design. Here, we list some of them as examples.
Underexplored categories in the main text. For existing categories of algorithms, some of them are still underexplored, such as GAN-based IL algorithms that do not rely on simulators (Section 4.2). NF- based II. where the NF works as the policy (Section 5.2.2), NF-basod offline RL algorithms (Section 5.3. 1
efficient algorithms on mitigating impacts from environmental stochasticity for transformer-based offline RL (Section 6.2.3.. transformer-based offline RL which integrates traditional offline RL methods with transformer architectures (Section 6.2.5), DM-based offline policy learning where DMs work data synthesizers (Section 7.3.3 and so on.
Integrated use of various DGMs for offline policy learning. As mentioned in Section 8.1 each DGM has distinct usages due to their special design and there have been a few algorithms managing to integrate two DGMs together for improved offline policy learning performance. More in-depth exploration can be done in this direction, especially for involving more DGMs in unified offline policy learning framework. For example, we observe from Table T2 that most DGM-based II. algorithms are based on BC, which is a relatively basic IL algorithu and has several fundamental issues 25 while NF's are integrated with various advanced IL frameworks due to its ability in exact density estimations. In this case, NFs can be integrated with transformer-basee DMs, where NFs extend the base IT. framework and transformer-based DMs work as a core component of a generalist agent that can deal with multi-modal input and high-dimensional generations. Further, they can be integrated with VAEs, which can be used to extract lask or subtask representations for multi-task or hierarchical learning, as introduced in Section 3.2.4 In addition, we notice that there are some rescarch working on integrating the advantages of various DGMs for computer vision tasks, such as VQ-GAN MAB=+GAN+Tranffooro (2021)) , DiffuseVAF (VAE+DM, Pandey et al. (2022)) ), DiTs (Transformer+ Peebles & Xie (2023)
[Esse et al. (2021)), TransGAN (GAN+Transforner, Jiang et al.
which can be potentially utilized in offline policy learning.
Unsolved issues and extensions of DGM- offline policy learning. Table 13 provides a summary of the issues and extensions of IT. and offline RL. that have been explored by DGM-based algorithms, which provides indications for future works. (1) By checking the outlined challenges in Section 2.3 we can pinpoint unresolved or aaddequately-adddresseee there have been GAN-based or DM-based algorithms targeting at the compounding error issue, the GAN- based ones rely on simulators and Dasbd ones (i.c., AWE and MCNN in Section 7.2.2p are still in the early stages of development. Similarly, in offine RL, reward sparsity is a significant issue, for which the explorations made by NEs (i.e., NF Shaping in Section 5.3.2) and transformers (i.e., DTRD in Section 6.2.2) have tried reward shaping and reward function learning, which are both interesting directions awaiting further development. (2) Regarding the extensions, there can be in total six (or more) sctup extensions: multi-task, multi-agent, hierarchical, model-based, safety, and generalization. Table 13 illustrates the extensions that are awaiting exploration for each DGM. The extension method can be potentially transferred among DGMs, For instance, as shown in Section 7.3.4 DM-based multi-task and hicrarchical offline RL algorithms adopt the same algorithm ideas as BOReL and OPAL (i.e., the VAE-based methods introduced in Section 3.2.4 Notably, multi-agent extensions for DGM-based offline policy learning require significant. development, and existing explorations (such as MAGAIL, MA-AIRL in Section 4.2 and DOM2, MADiff in Section 7.3. .))
issues by existing methods as future directions. For example, although
are still quite immature. In particular, developments of MARL algorithms for fully competitive or mixed
As introduced in Section 2.2.1 BC is implemented as supervised learning. However, predictions made by which is learned with BC and used for sequential decision-making, can impact future observations, thus breaching a fundamental assupttion of supervised learning (Ross & Bagnell (2010)): training inputs should be drawn from independent and identically distributed population. Consequently, errors and deviations from the demonstrated behavior tend to accumulate over time, as minor mistakes lead the agent into arcas of the obscrvation space that the expert has not ventured into, known as the compounding error. Additionally, BC learns policies merely through imitation, without engaging reasoning, which restricts the generalization capability of the learned policy.
75

---

Published in Transactions on Machine Learning Research (08/2024)
(partially cooperative/competttiiiii multi-agent scenarios based on game theory, and integrations of DGMs with state-of-the-art CTDF MARL methods are essential future directions.
Development of generalist agents. One of the most exciting future directions for DGM-based offline policy learning is developing a decision-makiin agent that can continually evolve with the amount of training data & the size of the DGM-based policy network and can handle range of tasks (including unseen ones). CV and NLP have made great progress in this direction, owing to the use of DGMs. To realize this for offiine policy learning, several things should be done. (1) First, the base IL offiine RL algorithm should be compatible with deep and broad neural networks, so that its performance can grow with the size of the foundation model. This is also the reason why mainstream transformer-based and DM-based offline policy learning methods (e.g,, Decision Transformer and Diffuser) are based on supervised learning as in CV and NLP. Future works can work on developing deep foundation models that are compatible with policy-gradient methods or dynamic programming. (2) Second, besides constructing large-scale, high-quality training datasets as mentioned in Section [.2.1] some algorithm-driven efforts can be done. For example, algorithms that are robust to suboptinal training data or capable of processing multi-modal inputs should be developed. Specifically, learning from demonstration videos (in third-person views) is quite challenging but promising research direction, as largo-scale video datasets are more accessible and vidcos usually contain more learnable information than images or vectorized data. Agents capable of learning from videos have the potential to continuously evolve by observing their surrounding environments during deployment. (3) Third, to enhance generalization, employing advanced meta-learning or multi-task learning techniques is advisable. Adopting the "pretraining + fine-tuning" approach, similar to that used in large language and vision models, is also promising, since it enables the accumulation of training efforts. Notably, the more significant the difference between the source and target tasks, the more challenging it becomes to make pretraining elfective, yet. the more data can be utilized for training the agent.
Future works in related areas. Finally, there are some related areas that remain to be explored. First, reviews on the applications of DGMs in Inverse Reinforcement Learning (IRL) or (Online) Reinforcement Learning have not yet been developed. IRL and RL are important approaches for sequential decision-making, but they both require online interactions with the environment. Second, as shown in Table [2] DGMs have been applied to dynamic- paraamim--pooraming---aaa baaed model-basod, and trajectory-optimizzaioon based offiine RL. There are some other branches in offline RL (Levine et al. (2020)) ). such as importance-samplngg bbsedd and imcertainty--stiinttioo
offline RL, which have the potential to be enhanced with DGMs. Notably, there is a subclass of dynamic-proge progrrmmiog-------oo based offline RL called one-step offline RL (Gülçehre et al. b2 ; Brandfonbrener et al.) (2021): Li et al. (2023at). ) Instead of iteratively executing policy evaluation and policy improvement as in Eq. 11, one-step offline RL starts from learning an approximation of the bohavior policy, i.c., A. from the offline datasct Dp, and its corresponding Q function Q (using temporal difference updates). Once the Q learning phase converges, policy x can be learned with certain policy improvement operators (e.g., the second line of Eq. (I) until convergence, while keeping the Q function Qi fixed. Since the Q-learning stage is only executed once and does not involve the being-learned policy n, the issues of OOD behaviors and overestimation (as mentioned in Scction 2.1.1 can be strategically avoided. Empirical results from (Brandfonbrener el al. (2021) show that one-step offline RL outperforms the mainstream iterative offline RL algorithms (introduced in Section 2.1.1]) in terms of training stability and policy efficacy, especially when the offline dataset lacks coverage of the statc-action space. However, we haven't seen works applying DGMs to one-step oflline RL yet, signaling potential areas for future research. Given the reliance of r's performance on the accurately approximated B and its corresponding Q function, DGMs could be used to model the behavior policy, leveraging their expressiveness and ability to handle high-dimensional states & multi-modal actions (as detailed in Section 8.1) DGMs can also be adopted to learn the Q function, as exemplified by Q-transformer (Chebotar et al. (2023) introduced in Section 6.2.1 Specifically, et al. (2023a) propose closed-fort policy improvement operators that can be used in one-step offline RL, based on first-order approximations of the policy objective, which offer enhanced stability over gradient descent updates. However, their theoretical analyses assume a Gaussian behavior policy. Extending these closed-form solutions to accommodate non- non-Gaussian behavior policies would enable the use of more advanced DGMs, such as diffusion models and normalizing flows, as policy approximators. Third, we notice
that new foundation models and DGMs are continually emerging, such as Mamba (Gu & Dao 2023 and
76

---

Published in Transactions on Machine Learning Research (08/2024)
Poisson Flows Xu et al. 2022c
2023
Applying these SOTA models in offiine policy learning is certainly
promising research direction.
9 Conclusion
In this paper, we show a systematic review on the applications of DGMs in offline policy learning. In Scction we provide an overview on offline policy learning methods and challenges. Then, Section B
2
7 we introduce
the applications of each DGM in both offline RL and IL, i.e. the two primary branches of offline policy learning. Each section includes both a tutorial and a survey on the respective topic. Notably, cover nearly all mainstream DGMs, including Variational Entto-Es,,,es, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models. Following these main content, we provide summary existing works and our perspectives on future research directions in Section 8 As the first review paper in this field, we wish this work to a hands-on reference for better understanding the current research progress regarding DGMs in offline policy learning and help researchers develop improved DGM-based offline RL/II IL algorithms.
Acknowledgement
We would like to thank Hanhan Zhou for participating in discussions and assisting with material collection. We also express special thanks to the editor and anonymous reviewers of TMLR for their valuable comments, which significantly improved the manuscript.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Cor- rado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Tlya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogencous systems, 2015. URL attps://wwe tensorflos. /rrg// Software available from Lensorflow.org.
Picter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning, volume 69. ACM, 2004.
Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Tim Harley, Felix Hill, Peter C. Humphreys, Alden Hung, Jessica Landon, Tim- othy P. Lillierap, Hamza Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Grog Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Creating multimodal interactive agents with imitation and superiid learning. CoRR, abs/21 /2//22222227.,.. 2021.
Suzan Ece Ada, Erhan Öztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement lcarning. CoRR, abs/2307.04720 2023.
Shubhankar Agarwal, Harshit Sikchi, Cole Gulino, and Eric Wilkinson. Imitative planning using conditional normalizing flow. CoRR, abs/2007.16162, 2020.
Amurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating ofline reinforcement learning. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.nee 2021.
Amurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In Proceedings of the 1 11th International Conference Learning Representations. OpenReview.ner 2023.
77

---

Published in Transactions on Machine Learning Research (08/2024)
Dmitry Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. In NeurIPS Offiine RL. Workshop, 2022.
Michael A. Alcorn and Anh Nguyen. ba A look-ahead multi-entity transformer for modeling coordinated agents. CoRR, abs/2104.11980, 2021.
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottle- neck. In Proceedings of the 5th International Conference on Learning Representations. OpenReview.nee net, 2017.
Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operalions research, 48(3):387 417, 1998.
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5):185-196, 1993.
Tomer Amit, Tal Shaharbany, Eliya Nachmann,, and Lior Wolf. Segdiff: Image segmentation with diffussio probabilistic models. arXiv preprint arXiv:2112.00390, 2021.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications. 12(3):313-326, 1982.
loannis Autonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas Hubert, and David Silver. Planning stochastic environments with learned model. In International Conference on Learning Representations, 2022.
Martín Arjovsky, Soumith Chintala, and Téon Bottou. Wasserstein GAN. CoRR, abs/ 11/11777100077 .07875, 2017.
Vladimir I Arnold. Ordinary differential equations. Springer Science Business Media, 1992.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusson models in discrete state-spaces. In Advonces in Neural Information Processing Systems 34, 2021a.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Newral Information Processing Systems, 34:17981-17993, 2021b.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607 Abs/1607.00654/ 2016.
Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation lcarning. In Proceedings of the 94th International Conference on Machine Learning, volume 70, pp. 390- 399, 2017.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-officient semantic segmentation with diffusion models. arXio preprint arXiv:2112. 11..0111,,, 2021.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 573-582, PMLR, 2019.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 2013.
78

---

Published in Transactions on Machine Learning Research (08/2024)
Xihan Bian, Oscar Mendez Maldonado, and Simon Hadfield. SKILL-IL: disentangling skill and knowledge in multitask imitation learning. In TEER/RSJ International Conference on Intelligent Robols and Systems, pp. 7060 7065. IEEE, 2022.
Lionel Blondé and Alexandros Kalousis. Sample-eftccint imitation learning via generative adversarial nets. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89, pp. 3138 3148, 2019.
Damian Boborzi, Christoph-Nikokar Straehle, Jens S. Buchner, and Lars Mikelsons. Learning normalizing flow policies based on highway demonstrations. In Proceedings of the 24th IEEE International Intelligent Transportation Systems Conference, pp. 22-29. IEEE, 2021a.
Damian Boborzi, Christoph-Nikoola Strachle, Jens Stefan Buchner, and Lars Mikelsons. State-only imitation learning by trajectory distribution matching. In Submission to the 10th International Conference on Learning Representations, 2021b.
Damian Boborzi, Christoph-Nikolas Straehle, Jens S. Buchner, and Lars Mikelsons. Imitation learning by state-only distribution matching. CoRR, abs/2202.04332,, 2022.
Rogerio Bonatti, Ratnesh Madaan, Vibhav Vincet, Sebastian A. Scherer, and Ashish Kapoor. Learning visuomotor policies for aerial navigation using mross-modal representations. In IEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 1637-1644 IEEE, 2020.
Ayman Boustati, Hana Chockler, and Daniel C. McNamee. Transfer learning with causal counterfactual reasoning in decision transformers. CoRR, Abs/2110.14355, 2021.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In Advances in Newral Information Processing Systems, pp. 4933 4946, 2021.
David Brandlonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-couditioned supervised learning work for offline reinforcement learning? In Advances in Neural Information Processing Systems 35, 2022.
Kianté Brantley, Wen Sun, and Mikael Henaff. Disagreement--egilliiieed imitation learning. In International Conference on Learning Representations. OpenReview.ned .nt, 2020.
Johann Brehmer, Joey Bose, Pim de Haan, and Taco Cohen. EDGI: equivariant diffusion for planning with embodied agents. CoRR, abs/2303.12410 2023.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024. URL https://openai. com/research/
video-generate sataa-ggaerattoonnoolls--------- -wwp-rooodess-------------e--eea www.wooddddmmaloo.....soooo
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tyy Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Newral Information Processing Systems 33, 2020.
Catherine Cang, Kourosh Hakhamaneshi, Ryan Rudes, Igor Mordatch, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Semi-supervised offline reinforcement learning with pre-trained decision transformers, 2022. URL https://openreevie: htt.a//wwwe.ooa...ee...ooooooota
Alex James Chan and Mihacla van der Schaar. Scalable bayesian inverse reinforcement learning. In Inter- national Conference on Learning Representations, 2021.
79

---

Published in Transactions on Machine Learning Research (08/2024)
Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible archi- tectures for arbitrarily deep residual neural networks. In Proceedings of the 32nd AAAT Conference Artificial Intelligence, pp. 2811-221211111 2818. AAAI Press, 2018.
Wei-Di Chang, Juan Camilo Gamboa Higuera, Scott Fujimoto, David Meger, and Gregory Dudek. Il-flow: Imitation learning from observation using uormalizing flows. arXiv preprinl ar arXiv:2205.09230 2022.
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autore- gressive q-functions. In Conference on Robot Learning, pp. 3909-3928. PMLR, 2023.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with trans- former world models. CoRR, abs abs/2222.09481,, 2022.
Changyou Chen, Chunyuan Li, Liquan Chen, Wenlin Wang, Yunchen Pu, and Lawrence Carin. Continuous- time flows for efficient inference and density estimation. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 823-88 8822. PMLR, 2018a.
Cynthia Chen, Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Suang-Huee Lee, Neel Alex, Steven H. Wang, Ping Luo, Stuart Russell, Pieter Abbeel, and Rohin Shah. An empirical investigation of representation learning for imitation. In Proceedings of the Newral Information Processing Systems Track on Datasets and Benchmarks 1, 2021a.
Daoming Chen, Ning Wang, Feng Chen, and Tony Pipe. Detrive: Imitation learning with transformer detection for end-to-end autonomous driving. CoRR, abs/2310.
2023a.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high- fidelity generative behavior modeling. In Proceedings of the 11th International Conference on Learning Representations. OpenRovicw.nce .nct, 2023b.
Jiayu Chen, Vaneet Aggarwal, and Tian Lan. A unified algorithm framework for unsupervised discovery of skills based on determinantal point process. In Advances in Neurol Information Processing Systems, 2023c.
Jiayu Chen, Tian Lan, and Vaneet Aggarwal. Hicrarchical adversarial inverse reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2023d.
Jiayu Chen, Dipesh Tamboli, Tian Lan, and Vaneet Aggarwal. Multi-task hierarchical adversarial inverse reinforcement learning. In Proceedings of the 40th International Conference on Machine Leurning, volume 202, pp. 4895 4920, 2023e.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097 2021b.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems 31, pp. 6572 6583, 2018b.
Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems 32, pp. 9913-9923 2019.
Valerie Chen, Abhinav Gupla, and Kenneth Marino. Ask your humans: Using human instructions to improve generalization in reinforcement learning. In Proceedings of the 9th International Conference on Learning Representations, 2021c
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Tlya Sutskever, and Picter Abboel. Infogan: In- terpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
80

---

Published in Transactions on Machine Learning Research (08/2024)
Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting contimuous control with consistency policy. CoRR, abs/2310.063433, 2023F.
Maxime Chevalier-BBisveerl, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai 2018.
gym.
Maxime
Dzmitry Bahdanau, Salem Lahlon, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with human in the loop. In International Conference on Learning Representations, volume 105, 2019.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse trans- formers. CoRR, abs/1904.10500 2019.
Daesol Cho, Dongseok Shim, and H. Jin Kim. S2P: state-conditioned image synthesis for data augmentation in offline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
Glen Chou and Russ Tedrake. Synthesizing stable reduced-order visuomotor policies for nonlincar systems via sums-of-squares oplimization. CoRR, abs/ 0tt//2304.124055 2023.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Newral Information Processing Systems, 2018.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems 28, pp. 2980-2988, 2015.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference Machine Learn- ing, volume 119, pp. 2148-2056, 2020.
Matthew Coleman, Olga Russakovsky, Christine Allen-Blanchetter and Ye Zhu. Discrete diffusion reward guidance methods for offline reinforcement learning. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023.
Robert Cornish, Anthony T.. Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity con- straints with continuously indexed normalising flows. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 2133-2143. PMLR, 2020.
André Correia and Luis A. Alexandre. Hierarchical decision transformer. CoRR, abs/2209 10449, 2022.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet. org. 2016-2021.
Imre Csiszár and Paul C. Shields. Information thoory and statistics: A tutorial. Foundations and Trends in Communicaatinn and Information Theory, 144), 2004.
Sudeep Dasari and Abhinav Gupta. Transformers for onc-shot visual imitation. In Proceedings of the 4th Conference on Robol Learning, volume 155, pp. 2071 2071-2084, 2020.
Jared Quiney Davis, Albert Gu, Krzysztof Choromanski, Tri Dao, Christopher Ré, Chelsea Finn, and Percy Liang. Catformer: Designing stable transformers via sensitivity analysis. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 2489 2499, 2021.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Newral Information Processing Systems, 32, 2019.
Coline Devin. craftingworld. http::///ghhhhh.a////////aaaaaat 2024. Accessed: 2024-02-05.
81

---

Published in Transactions on Machine Learning Research (08/2024)
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Informalion Processing Systems 34, pp. 175008794,42 2021.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. CoRR, abs92309.11 16984, 2023.
Laurent Dinh, David Krueger, and Yoshua. Bengio. NICE: non-linear independent components estimation. In 3rd International Conference on Learning Representations, Workshop Track Proceedings, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density cstimation using real NVP. Proceedings of the 5th International Conference on Learning Representations. OpenReview net, 2017.
Wenbo Dong, Shaofan Liu, and Shiliang Sun. Safe batch constrained deep reinforcement learning with generative adversarial network. Information Science, 634:259 270, 2023.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 2793 2803, 2021.
Monroe D Donsker and SR Srinivasa. Varadhan. Asymptotic evaluation of certain markov process expecta- tions for large time-iii. iii. Communicalions on pure and applied Mathematics, 29(4):389-461, 1976.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning - identifiability challenges and effective data collection strategies. In Advances in Neural Information Processing Systems 34, pp. 4607-4618, 2021.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, volume 78, pp. 1- 16, 2017.
Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retricval: Few-shot imitation learning by querying unlabeled datasets. In Robotics: Science and Systems XIX, 2023.
Yuqing Du, Daniel Ho, Alex Alemi, Eric Jang, and Mohi Khansari. Bayesian imitation learning for end- to-end mobile manipulation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.
PMLR, 2022.
Dheeru Dua, Cascy Graff, et al. Uei machine learning repository, 2017. URL http:// ics. uci. edu/ml, 7(1), 2019.
Darrell Duffic and Jun Pan. An overview of value at risk. Journal of derivatives, 4(3):7-49, 1997.
Emilion Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented noural odes. In Advances in Neural Information Processing Systems 32, pp. 3134 3144, 2019.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
Scott Emmons, Benjamin Eysenbach, Tlya Kostrikov, and Sergey Levine. Rys: What is essential for offline RI. via supervised learning? Proceedings of the 10th International Conference on Learning Representations. OpenReview.net, 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and patiern recognition, pp. 12873-12883 2021.
Ben Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advonces in Neural Information Processing Systems 32, pp. 15220- 152311
2019a.
82

---

Published in Transactions on Machine Learning Research (08/2024)
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Represenlations, OpenRe- view.net, 2019b.
Kuan Fang, Patrick Yin, Ashvin Nair, Homer Walke, Gengchen Yan, and Sergey Levine. Generalization with lossy alfordances: Leveraging broad offline data for learning visuomotor lasks. In Conference on Robol Learning, volume 205 of Proceedings of Machine Learning Research, pp. 106 117. PMLR, 2022.
Cong Fei, Bin Wang, Yuzheng Zhuang, Zongzhang Zhang, Jianye Hao, Hongbo Zhang, Xuewu Ji, and Wulong Liu. Triple-gail: A multi-modal imitation learning framework with generative adversarial nets. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, pp. 2929-2933 2020.
Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversar- ial multi-agent motion using conditional variational autoencoders. In Proceedings of the 15th European Conference on Computer Vision, volume 11215, pp. 761 776. Springer, 2018.
Pete Florcnce, Corcy Lynch, Andy Zeng, Oscar A. Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference Robol Learning, volume 164, pp. 158-168 PMLR, 2021.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Farama Foundation. Frozen lake environment. bctps://gymnasiin/ httppp///ggaaaiiii.......ooooorp waw.aaananrrgaaaarrrrroeoooooooo frozen_laken 2023.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In proceedings of the 4th International Conference on Learning Represen- lations, 2016.
Markus Freitag and Yaser Al-Onaizan, Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.00 0166 2017.
Gideon Joseph Freund, Flad Sarafian, and Sarit Kraus. A coupled flow approach to imitation learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 10357-10372. PMLR, 2023.
Huiqiao Fu, Kaiqiang Tang, Yuanyang La, Yiming Qi, Guizhou Deng, Flood Sung, and Chunlin Chen. Ess-infogail: Semi-supervised imitation learning from imbalanced demonstrations. In Advances in Neural Information Processing Systems 36, 2023.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement
learning. CoRR, abs/ abs/1170..11248, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data- driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2052 2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for oflline hindsight information matching. In Proceedings of the 10th International Conference on Learning Representations. OpenReview.net, 2022.
Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard S. Zemel. Smile: Scalable meta inverse reinforeement learning through context-conditiin policies. In Advances in Neural Information Processing Systems 32, pp. 7879-7889, 201 9a.
83

---

Published in Transactions on Machine Learning Research (08/2024)
Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Gu. A divergence minimization per- spective on imitation learning methods. In Proceedings of the 3rd Annual Conference on Robot. Learning, volume 100, pp. 1259 1277. PMLR, 2019b.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse, The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems 30, pp. 2214 2224, 2017.
lan Goodfellow. Nips 2016 tutorial: Generative adversarial networks, 2017.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014a.
Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pp. 2672-26022 26800 2014b.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenand. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 0rXiv:1810.01366 2018.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.
Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generalive models. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 3069 3076. AAAI Press, 2018.
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces. abs/2312.00752, 2023.
CoRR,
Jiayi Guan, Shangding Gu, Zhijun Li, Jing Hou, Yiqin Yang, Guang Chen, and Changjun Jiang, Uac: Offline reinforcement learning with uncertain action constraint. IEEE Transactions on Cognitive and Developmental Systems, 2023.
Çaglar Gülçehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew W. Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. CoRR, abs/2103.0952 095775, 2021.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, volume 100, pp. 1025 1037, 2019a.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement. learning. arXiv preprint 191: 11956, 2019b.
William Н. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 1442-2448 ijcai.org, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018.
84

---

Published in Transactions on Machine Learning Research (08/2024)
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In Proceedings of the 8th International Conference on Learaing Repre- sentations, 2020.
Jay Hambidge. The elements of dynamic symmetry. Courier Corporation, 1967.
Jungwoo Han and Jinwhan Kim. Selective data augmentation for improving the performance of offline reinforcement learning. In International Conference on Control, Automation and Systems (ICCAS), pp. 226. IEEE, 2022.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Ychui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE trunsactions on pallern analysis and machine intelligence, (11))87-1110, 2022.
Philippe Hansen-Fstruch, Nya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: implicit q-lcarning actor-critic method with diffusion policies. CoRR, abs/2304.10577 2023.
Xiaotian Hao, Weixun Wang, Jianye Hao, and Yaodong Yang. Independent generative adversarial self- imitation learning in cooperative multiagent systems. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1315-1323 2019.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav S. Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advonces in Newral Information Processing Systems 30, pp. 1245, 2017.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. CoRR, abs/2305.18459. 2023a.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Compuler Vision and Pallern Recognition, pp. 770- 778. IEEE Computer Society, 2016.
Longxiang He, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. CoRR, Ats/2230005333 2023b.
Joey Hejna, Pieter Abbeel, and Lerrel Pinto. Improving long-horizon imitation through instruction predic- tion. In Proceedings of the 37th AAAI Conference Artificial Intelligence, pp. 7857-7865, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error lincar units (gelus), 2023.
Charles A. Hepburn and Giovanni Montana. Model-based trajectory stitching for improved offline reinforce- ment learning. CoRR, abs/2211.11000 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference Learning Representations. OpenReview.net, 2017.
Jean-Baptiste Hiriart- Urruty and Claude Lemaréchal. Fundementals of conver analysis. Springer Science Business Media, 2004.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advonces in Neural Infor- mation Processing Systems 29, pp. 4565-4573, 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.12595 2022.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abboel. Flow++: Improving flow-basod generative models with variational dequantization and architecture design. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp.
PMLR, 2019.
85

---

Published in Transactions on Machine Learning Research (08/2024)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
KS Holkar and Laxman Waghmare. An overview of model predictive control. International Journal of control and automation, (34):47-63, 2010.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454 12465, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges-
mundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiiet transfer learning for NLP. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 2790 2799, 2019.
Jifeng Hu, Yanchao Sun, Sili Huang, Siyuan Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuusr with temporal condition guidance for offline reinforcement learning. CoRR, abs/2306. Abs/2306.04875, 04875, 2023a.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. CoRR, abs/ 230/2333.33777 2023b.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron C. Courville. Neural autoregressive flows. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2083 2092. PMLR, 2018.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglon, Mohammadamin Barckatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. Tn Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 4476 4486, 2021.
Dongyoon Hwang, Minho Park, and Jiyoung Lec. Sample generations for reinforcement learning via diffusion models. OpenReview.ned. 2023.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6:695-709, 2005.
Jacobsen, Arnold W. M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In Proceedings of the 6th International Conference on Learning Representations, 2018.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In Proceedings of the 10th International Conference on Learning Representations. OpenReview.net, 2022.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robolics Automation Letters, 5(2):3019 3026, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning one big sequence modeling problem. In Advances in Neural Information Processing Syslems 34, pp. 1273 1286, 2021.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 9902 9915. PMLR, 2022.
Wonscok Jeon, Scokin Sco, and Keo-Eung Kim. A bayesian approach to generative adversarial imitation learning. In Advances in Neural Information Processing Systems 31, pp. 7440 7450, 2018.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that scale up. Advances in Neural Information Processing Systems, 34:14745-14747 2021.
86

---

Published in Transactions on Machine Learning Research (08/2024)
Mingxuan Jing, Wenbing Huang, Fuchum Sun, Xiaojian Ma, Tao Kong, Chuang Gan, and Lei Li. Adversar- ial option-aware hierarchical imitation learning. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 5097 5106. PMLR, 2021.
Alexia Jolicoeur-Martinens Rémi Piché- Taillefrr, Toannis Mitliagkas, and Remi Tachet des Combes. Adver- sarial score matching and improved sampling for image generation. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.nes net. 2021.
Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. AMMUS : A survey of transformer- based pretrained models in natural language processing. CoRR, abs/2108.05544 05882, 2021.
Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parckh. A new path: Scaling vision-and-languagee navigation with syn- thetic instructions and imitation learning. In IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 10813 10823, 2023.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for oflline reinforcement learning. CoRR, 00ss/2305.20083 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diflusion-based generative models. In Advances in neural information processing systems 35, 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based
offine reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Transformer- based deep imitation learning for dual-arm robot manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8965-8972, 2021a.
Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Memory- based gaze prediction in deep imitation learning for robot manipulation. In International Conference on Robotics and Automation, pp. 2427-2433.
IEBE, 2022.
Heccheol Kim, Yoshiyuki Ohmura, Akihiko Nagakubo, and Yasuo Kuniyoshi. Training robots without robots: Deep imitation learning for master-to-robot policy transfer. IEEE Robolics Aulomation Letters, 8 (5):2906-291)) 2023.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforeement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 5496-5505. PMIR, 2021b.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmen- tation. Advances in Neural Information Processing Systems 36, 36, 2023.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems 31, pp. (0236-10245) 2018
Diederik P. Kingma and Max Welling. Aulo-encoding variational bayes In Proceedings of the 2nd Interna- tional Conference on Learning Representations, 2014.
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 3443300773 2019.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinícius Flores Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmoct Kohli, and Peter W. Battaglia. Compile: Compositional imitation learning and exccution. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3418 3428. PMLR, 2019.
87

---

Published in Transactions on Machine Learning Research (08/2024)
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Arlificial Intelligence Research, 76:201-201 2240, 2023.
Tvan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review current methods. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLI- GENCE, 43(1))33964 3979, 2021.
Yotto Koga, Heather Kerrick, and Sachin Chitta. On CAD informed adaptive robotie assembly. In IEEE/RS) International Conference on Intelligent Robots and Systems, pp. 10207-10214 IEEE, 2022.
Nya Kostrikov, Kumar Krishna. Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminnato--acoo-crrit Addressing sample inefficiency and reward bias in adversarial imitation learn- ing. In Proceedings of the 7th International Conference on Learning Representations. OpenReview. net, 2019.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, In Proceedings of the 10th International Conference Learning Representations. OpenReview.ned .nt,, 2022.
Robert Krajewski, Julian Bock, Laurent Klocker, and Lutz Eckstein. The highd dataset: A dronc datasct of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In Proceedings of the 21st International Conference on Intelligent Transportation Systems, pp. 2125..... IEEE, 2018
Alex Kuefler and Mykel J. Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp. 1071-1078, 2018
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- q-learning via bootstrapping error roduction. In Advances in Neural Information Processing Systems 32, pp. 11761- 11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Tn Aduances in Neural Information Processing Systems 39, 2020.
Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In Proceedings of the 22nd International Conference on Artificial Intelli- gence and Statistics, volume 89, pp. 2154 21621113, 2019.
Kuang-Hun Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In Advances in Newral Information Processing Systems 35, 2022.
Su-Jin Lee, Tae Yoon Chun, Hyoung Woo Lim, and Sang- Ho Lee. Path tracking control using imitation learning with variational auto-encoder. In Proceedings of the 19th International Conference on Control, Automation and Systems, pp. 501-505. IEFE, 2019.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint Xrxiv:180..0 00909, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. CoRR,, abs/2005.01643. 2020.
Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in Neural Information Processing Systems 30, pp. 1088-4098, 2017a.
Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. Offline re- inforcement learning with closed-form form policy improvement operators. In International Conference on Machine Learning, volume 202, 2023a.
88

---

Published in Transactions on Machine Learning Research (08/2024)
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hicrarchical planning through goal-conditioned offline reinforcement learning. IEEE Robolics Aulomation Letters, 7(4):10216-1022) 102233, 2022.
Tao Li, Juan Guevara, Xinghong Xie, and Quanyan Zhu. Self-confirming transformer for locally consistent online adaptation in multi-agent reinforcement learning. CoRR, abs/2310.04579, 2023b.
Wenhao Li, Xiangleng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 20035-20064, 2023c.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demon-
strations. In Advances in Neural Information Processing Systems 30, pp. 3812 3822, 2017b.
Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies in offiine multi-agent
reinforcement learning. CoRR, abs/ 0b8/2307.00172, 2023d.
Anthony Liang, Ishika Singh, Karl Pertsch, and Jesse Thomason. Transformer adapters for robot learning, In CoRL. 2022 Workshop on Pre-training Robol Leurning, 2022. URL https:://peerrevie http:/ppprnriiiieeeeeeeeeeeeeeee www. forum?
id= -WRRYBFF
Hebin Liang, Zibin Dong, Yi Ma, Xiaotian Hao, Yan Zheng, and Jianye Hao. A hierarchical imitation
learning-baser decision framework for autonomous driving. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 4695-4701. ACM, 2023a.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In Proceedings of the 40th International Conference Machine Learning, volume 202, pp. 07725-00755 2023b.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Tn Proceedings of the 4th International Conference Learning Representations, 2016.
Qinjie Lin, Han Liu, and Biswa Sengupta. Switch trajectory transformer with distributional value approxi- mation for multi-task reinforcement learning. CoRR, Abs/2203.074133, 2022a.
Runji Lin, Ye Li, Xidong Feng, Zhaowei Zhang, Xian Hong Wu Fung, Haifeng Zhang, Jun Wang, Yali Du, and Yaodong Yang. Contextual transformer for offline meta reinforcement learning. CoRR, abs/2211.086 12.0006,, 2022b.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al Open, 2022c.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. In Pro- ceedings of the 8th International Conference on Learning Representations. OpenReview.net, 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. TEEE Transactions on Knowledge and Data Engineering, 35(1): 857 876, 2023a.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In Proceedings of the 40th International Con- ference Machine Learning, volume 202, pp. 21611-21638 2023b.
Francesco
Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 4114-4124, 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbecl, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative competitive
coppratiiv-compcitive environments. In Advances in Neural Information Processing Systems 30, pp. 6379 6390, 2017.
89

---

Published in Transactions on Machine Learning Research (08/2024)
Ricky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan, and Matthew J. Hausknecht. Work- ing memory graphs. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 6404 6414, 2020.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems 35, 2022a.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.
PMLR, 2023.
Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen J. Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2022b.
Xiaoyu Lu, Jan Stuchmer, and Katja Hofmann. Trajectory VAE for multi-modal imitation, 2019. URL http::///ppenerreeeeeeeee///otts
Yiren Lu and Jonathan Tompson. ADAIL: adaplive adversarial imitation learning. CoRR, abs/2008. 2000. 2020.
James Lucas, George Tucker, Roger B. Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. In Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop. OpenReview.net, 2019.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action-quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp. PMLR, 2023.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Roboties: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Proceedings of the 3rd Annual Conference on Robot Learning, volume 100, pp. 1113-1132, 2019.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative qlearning for offline reinforcement learning. In Advances in Neurul Information Processing Systems 95, 2022.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Leurning, volume 164, pp. 1678 1690, 2021a.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint orXiv:2108.0s: 03298, 2021b.
Lucas Manuelli Yunzhu Li, Peter R. Florence, and Russ Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Proceedings of the 4th Conference on Robot Learning, volume 155, pp. 693-710, 2020.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In IEEE International Conference on Computer Vision, 2017.
Gabriel Margolis and Pulkit Agrawal. Walk these ways: Coid-conddt policies yield diversified quadrupedal agility. In Conference on Robot Learning, volume 1, pp. 2, 2022.
90

---

Published in Transactions on Machine Learning Research (08/2024)
Saced Masoudnia and Reza. Ebrahimpour. Mixture of experts: literature survey. Artificial Intelligence Review, 42:275-222 223, 2014.
James Massey et al. Causality, feedback and directed information. In The International Symposium on Information Theory and Its Applications, pp. 303-305, 1990.
David Q Mayne and Hannah Michalska. Receding horizon control of nonlinear systems. Proceedings of the 27th IEEE Conference on Decision and Control, pp. 464-465. IEEE, 1988.
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R. Devon Hjelm. Leveraging exploration in off-policy algorithms via normalizing flows. In Proceedings of the 3rd Annual Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 430-444. PMLR, 2019.
Oier Mees, Lukás Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics Automation Letters, 7(4):11205-113) 112122 2022a.
Oier Mees, Lukas Hermann, Frick Rosete-Beatt Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for lorg-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327 7334, 2022b.
ML Menéndez, JA Pardo, L Pardo, and MC Pardo. The jensen-shannon divergence. Journal of the Franklin Institule, 334 1997.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pro-trained multi-agent decision transformer: One big sequence model lackles all SMAC lasks. CoRR, hts/2112.0028/// 2021.
Gregory P Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzaler and Carl K Wellington. Lasernet: An efficient probabilistic 3d object detector for autonomous driving. In Proceedings of the IEFE/CVP conference computer vision and pattern recognition, pp. 12677 12686, 2019.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1xXiv:1411.170 2014.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline reiforeeino learning with advantage weighting. Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 7780-7791, 2021.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically moti- vated reinforcement learning. In Advances in Newral Information Processing Systems 28, pp. 1125-2133, 2015.
Tongzhou MU, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasots. CoRR, abs/20008 /2006.09359, 2020.
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Proceedings of the 6th Annual Conference on Robot Learning, 2022.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the 17th. International Conference on Machine Learning, pp. 663-670. Morgan Kaufmann, 2000.
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In Proceedings of the 16th International Conference on Machine Learning, pp. 278 287, 1999.
91

---

Published in Transactions on Machine Learning Research (08/2024)
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In Proceedings of the 40th. International Conference on Machine Learning, volume 202, pp. 26087 26105, 2023.
Tianwei Ni, Harshit S. Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse rein- forcement learning via state marginal matching. In Proceedings of the 4th Conference on Robol Learning, volume 155, pp. 529 551, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 8162 8171. PMLR, 2021.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections bridge the gap between vaes and flows. In Advances in Neural Information Processing Systems 33, 2020.
Michael D. Noseworthy, Rohan Paul, Subhro Roy, Dachyung Park, and Nicholas Roy. Task-conditioneed variational autoencoders for learning movement primitives. Tn Proceedings of the 3rd Annual Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 933 944. PMLR, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Newral Informalion Processing Systems 29, pp. 271- 279, 2016.
OpenAl. Gpt-4 technical report, 2023. URL
pdf
Yingwei Pan, Ychao Li, Yiheng Zhang, Qi Cai, Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Silver- bullet-3d at maniskill 2021: Learning-frone frerringgroo--deennssraaiiion and heuristic rule-based methods for object manipulation. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
Kushagra Pandey, Avidcop Mukherjoe, Piyush Rai, and Abhishek Kumar. DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Leurning Research, 2022. ISSN 2835-8856. URL http:///ppaaaoeeeeeeeeeeeerrrrrs
George Papamakarios, Tain Murray, and Theo Pavlakou. Maskod autoregressive flow for density estimation. In Advances in Neural Information Processing Systems 30, pp. 2338 2347, 2017.
Emilio Parisotto and Ruslan Salakhutdinov. Efficient transformers in reinforcement learning using actor- learner distillation In Proceedings of the 9th International Conference on Learning Representations. Open- Review.net, 2021.
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçchhre, Siddhant M. Jayaku- mar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 7487-7498. PMLR, 2020.
Jongjin Park, Younggyo Seo, Chang Liu, Li Zhao, Tao Qin, Jinwoo Shin, and Tie Van Liu. Object-aware regularization for addressing causal confusion in imitation learning. Advances in Neural Information Processing Systems, $4:3029-3042, 2021.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rI? arXiv preprint arXiv:2406.0983 09329, 2024.
Keiran Paster, Sheila A. McIlraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. In Advances in Neural Information Processing Systems 35, 2022.
Pranay Pasula. Complex skill acquisition through simple skill adversarial imitation learning. CoRR, 2000/2777.1088,, 2020.
92

---

Published in Transactions on Machine Learning Research (08/2024)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01170s 2019.
Tim Pearce and Jun Zhu. Counter-strike deathmatch with large-scale behavioural cloning. In 2022 IEEE Conference on Games (CoG), pp- 104-111. TEEE, 2022.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Minglei Sun, Raluca Georgescu, Sergio Val- carcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffissio models. In Proceedings of the 1 11th International Conference on Learning Repre- sentations, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Bei Peng, Tabish Rashid, Christian Schröder de Witt, Pierre-Alexandre Kamicnny, Philip H. S. Torr, Wen- delin Boehmer, and Shimon Whiteson. FACMAC: factored multi-agent centralised policy gradients. In Advances in Newral Information Processing Systems 34, pp. 12208 12208-12221, 2021.
Jian-Wei Peng, Min-Chun Hu, and Wei-Ta Chu. An imitation learning framework for generating multi-modal trajectories from unstructured demonstrations. Neurocomputing, 500:712-723 2022.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discrimina- tor bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. In Proceedings of the 7th International Conference on Learning Representations, 2019a.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Maraatage-weighteer regression: Simple and scalable off- oof-policy reinforcement learning. CoRR, abs/1910. htss/1900.00177, 2019b.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstrationn-guide reinforcement learning with learned skills. In Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 729-777777 -399. PMLR, 2021.
Samuel Pfrommer, Yatong Bai, Hyunin Lee, and Somayeh Sojoudi. Initial state interventions for decon- founded imitation learning. In Proceedings of the 62nd IEEE Conference on Decision and Control, pp. 2319-2319. IEEE, 2023.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi- goal reinforcement lcarning: Challenging robotics environments and request for research. CoRR, 0bss/180.00940 2018.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. CoRR, abs/ 1505.04870, 2015.
Dean Pomerleau. ALVINN: autonomous land vehicle in neural network. In Advances in Neural Infor- mation Processing Systems, pp. 305-313. Morgan Kaufmann, 1988.
Dean Pomerlean. Efficient training of artificial neural networks for antonomous navigation. Newral Compu- tation, 3(1):88 97, 1991.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline rein- forcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Martin L Puterman. Markov decision processes: discrete stochastic dynomic programming. John Wiley & Sons, 2014.
93

---

Published in Transactions on Machine Learning Research (08/2024)
Aaron L Putterman, Kevin Lu, Igor Mordatch, and Pieter Abbeel. Pretraining for language conditioned imitation with transformers, 2022. URL httpps://openreviev httppp//wpppaarrreeeeeeee/aaaaaa wet//faoruumniieccceee2aaaaa
Mengshi Qi, Jie Qin, Yu Wu, and Yi Yang. Imitative non-autoregreesive modeling for trajectory lorecasting and imputation. In LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12733 12742. Computer Vision Foundation / IEEE, 2020.
Rongjun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Huu Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offiine reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
Ahmed Hussain Qureshi, Byron Boots, and Michael C. Yip. Adversarial imitation via variational inverse reinforcement learning, In Proceedings of the 7th International Conference on Learning Representations. OpenReview.nel, 2019.
Lawrence Rabiner and Biinghwang Juang. An introduction to hidden markov models. IEEE ASSP Magazine, 3(1):4-16, 1986.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sutskever, et al. Improving language understanding by generative pre-training. mikecaptain.com com, 2018.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Annual Conference on Learning for Dynamics and Control, volume 144, pp. 1154 1168, 2021.
Rouhollah Rahmatizadel, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In IEEE International Conference on Robotics and Aulomation, pp.
IEFF, 2018.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 2596-2591, 2007.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M. Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matten matttrrrttttrrrrtttt 3d dataset (HM3D): 1000 large-scale 3d environments for embodied AI. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Cascy Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 2204:2204 2022.
Amir Rasouli, Randy Goebel, Matthew E. Taylor, Iuliia Kotseruba, Soheil Alizadeh, Tianpei Yang, Mont- gomery Alban, Florian Shkurti, Yuzheng Zhuang, Adam Seibior, Kasra Rezace, Animesh Garg, David Meger, Jun Luo, Liam Paull, Weinan Zhang, Xinyu Wang. and Xi Chen. Neurips 2022 competition: Driving smarts, 2022.
Scott E. Reod, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenbery, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Fdwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions Machine Learning Research, 2022.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? CoRR, abss2201.11211 2022.
Allen Z. Ren, Sushant Veer, and Anirudha Majumdar. Generalization guarantees for imitation lcarning. In Proceedings of the 4th Conference on Robot Learning, volume 155, pp. 1426-14644444 1442. PMLR, 2020.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. In Roboties: Science and Systems, 2023.
94

---

Published in Transactions on Machine Learning Research (08/2024)
Shidch Rezacifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning anti-exploration. In Thirby-Sizth AAAT Conference Artificial Intelligence, pp. 8106 8114. AAAI Press, 2022.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 1530-1538, 2015.
Luigi M Ricciardi. On the transformation of diffusion processes into the wiener process. Journal of Mothe- matical Analysis and Applications, 51(1)118,,199 1976.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resohution image synthesis with latent diffusion models. In IEEE/CVF Conference Computer Vision and Pollern Recognition, pp. 19674-10680 LEEE, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Compuler-Assisteee Intervention MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241 Springer, 2015.
Frick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task-agnostic offline reinforcement learning. In Conference Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 1838 1849. PMLR, 2022.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 661-668. JMLR. Workshop and Con- ference Proceedings, 2010.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Slalistics, volume 15, pp. 627 635, 2011.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the 11th Avneua Conference on Computational Learning Theory, pp. 101-103. ACM, 1998.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29, pp. 2226 2234, 2016.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR, abs/ 0bs/1902.040433 2019.
Vaibhav Saxena, Yotto Koga, and Danfei Xu. Constrainedd-ooneett conditional diffusion models for imitation learning. CoRR, abs/2311. /2311.01199, 2023.
Julian Schrittwieser, Ioannis Antonoglon, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604 609, 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglon, and David Silver. Online and offline reinforcement learning by planning with a learned model. In Advances in Neural Information Processing Systems, pp. 27580 27591, 2021.
Yannick Schroecker and Charles L. Isbell Jr. Universal value density estimation for imitation learning and goal-conditioned reinforcement learning. CoRR, abs/2000.066473, 2020.
Yannick Schroccker, Mel Vocerík, and Jonathan Scholz. Generative predecessor models for sample-efficient imitation learning. In Proceedings of the 7th International Conference on Learning Representations. Open- Review.net, 2019.
95

---

Published in Transactions on Machine Learning Research (08/2024)
John Schulman, Sergey Levine, Picter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Leurning, volume 37, pp. 1889 1897, 2015.
Nur Muhammad Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning $k$ modes with one stonc. In Advances in Neural Information Processing Systems 35, 2022.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelityy visual and physical simulation for autonomous vehicles. In Field and Service Robotics, Results of the 11th International Conference, volume of Springer Proceedings in Advonced Robolics, pp. 621-635. Springer, 2017.
Ohad Shamir. Are resnets provably better than linear predictors? In Advances in Neural Information Processing Systems 31, pp. 505 514, 2018.
Jinghuan Shang and Michael S. Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 214 221.
IEEE, 2021.
Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael S. Ryoo. Starformer: Transformer with state-action-reward representations for visual reinforcement learning. In Proceedings of the 1 17th European Conference on Computer Vision, volume 13699, pp. 462 479, 2022.
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (MIM)): large scale demonstrations data for imitation. In Proceedings of the 2nd Annual Conference on Robot Learning, volume 87, pp. 906-915, 2018.
Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint- based imitation learning for robotic manipulation. CoRR, abs/ abs/2330..1333 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dicter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 10737 10746, 2020.
Mohit Shridhar, Lucas Mamuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 785-799. PMLR, 2022.
Dylan Z Slack, Yinlam Chow, Bo Dai, and Nevan Wichers. Safer: Data-efficient and safe reinforcement learning via skill acquisition. In ICML Decision Awareness in Reinforcement Learning Workshop, 2022.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9767 9779. PMLR, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jascha Sohl-Dickstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium lhermodynamics.. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2256 2265, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep con- ditional generative models. In Advances in Neural Information Processing Systems 28, pp. 3483-3491, 2015.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Advances in Neural Information Processing Systems 91, pp. 1472-7448) 2018.
96

---

Published in Transactions on Machine Learning Research (08/2024)
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in neural information processing systems, 32, 2019.
Yang Song and Stelano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems 33, 2020.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, volume 115, pp. 574 584, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXio preprinl arXiv:2011.13456, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. In Advances in Neural Information Processing Systems 34, pp. 1415-1428, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Πlya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 32211 32252. PMLR, 2023.
Adrian Spurr, Jie Song, Seonwook Park, and Otanrr Hilliges. Cross-modal deep variational hand pose estimation. In IEEE Conference on Compuler Vision and Pallern Recognition, pp. 99-98. Computer Vision Foundation / IEEE Computer Society, 2018.
Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, James Weimer, and Insup Lee. Memory-consistent neural networks for imitation learning. CoRR, abs/2330.06617,, 2023.
Shyam Sudhakaran and Sebastian Risi. Skill decision transformer. CoRR, abs/ Jbs/2301.13573, 2023.
H.J. Terry Suh, Glen Chou, Hongkai Dai, Lujie Yang, Abhishek Gupta, and Russ Tedrake. Fighting certainty with gradients: Offline reinforcement learning via diffusion score matching. In 7th Annual Conference Robol Learning, 2023.
Jiankai Sun, Lantao Yu, Pinqian Dong, Bo Lu, and Bolei Zhou. Adversarial inverse reinforcement learning with self-attention dynamics model. IEEE Robotics and Automation Letters, 6(2):1880 1886-1886611111 2021.
Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Talpur Nobel, Mykel Kochenderfer, and Mac Schwager. Con- formal prediction for uncertainty-oware planning with diffusion dynamics model. Tn Advances in Newral Information Processing Systems 36, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shiro Takagi. On the effect of pre-training for transformer in different modality on offline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1801 100690, 2018.
Russ Tedrake. Underactuated Robotics. Course Notes for MIT 6.832, 2023. URL https://underacuatte.. csail eit
Hoang Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans. In 2020 Interna- tional Joint Conference on Neural Networks, pp. 1-10. IEEE, 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop, pp. 1-5. IEEE, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-base control. In IEFE/RSJ International Conference Intelligent Robots and Systems, 2012.
97

---

Published in Transactions on Machine Learning Research (08/2024)
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. CoRR, abs/1807.0m hts//180.00158, 2018.
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The MAGICAL benchmark for robust imitation. In Advances in Neural Information Processing Systems 39, 2020.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent rein- forcement learning with knowledge distillation. In Advances in Neural Information Processing Systems 35, 2022.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offiine reinforcement learning. In Proceedings of the 9th, International Conference on Learning Representations. OpenReview.nef, 2021.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. In Proceedings of the 34th Conference on Uncertainty in Artificial, pp. 393 402, 2018.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Newral Information Processing Systems 30, pp. 630666366000 6315, 2017.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.0377 03748, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Aduances in neural information processing systems, 30, 2017.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demonstrations for deep rein- forcement learning on robotics problems with sparse rewards. CoRR, abs/ htss/177..08811 2017.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John M. Dolan, Jeff G. Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. CoRR, abs/2330.065500 2023.
David Venuto, Jhelum Chakravorty, Léonard Boussioux, Junhao Wang, Gavin McCracken, and Doina Pre- cup. oirl: Robust adversarial inverse reinforcement learning with temporally extended actions. CoRR, abs/2002.090433, 2020.
Alexander Vidal, Samy Wu Fung, Luis Tenorio, Stanley Osher, and Levon Nurbekyan. Taming hyper- parameter tuning continuous normalizing flows using the jko scheme. Scientific Reports, 13(1):5001, 2023.
Adam R. Villaflor, Zhe Huang, Swapnil Pande, John M. Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 22270 22283, 2022.
Pascal Vincent. A connection between score matching and denoising autocncoders. Neural Computation, 23 (7):1661 1674, 2011.
Vladimir Vork, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, vol- une 29. Springer, 2005.
Matthew J. Vowels, Necati Cihan Camgöz, and Richard Bowden. Gated variational autoencoders: Incor- porating weak supervision to encourage disentanglement. In Proceedings of the 15th IEFF International Conference on Automatic Face and Gesture Recognition, pp. 125-132. IEEE, 2020.
Quan Vuong, Aviral Kumar, Sorgey Levine, and Yevgen Chebotar. DASCO: dual-generator adversarial support constrained offline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
98

---

Published in Transactions on Machine Learning Research (08/2024)
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offine rein- forcement learning with reverse model-based imagination. In Advances in Newral Information Processing Systems 34, pp. 29420 29432, 2021a.
Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped transformer for oflline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022a.
Mianchu Wang, Rui Yang, Xi Chen, and Meng Fang. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. CoRR, abs/ 0bs/2330.20025, 2023a.
Pengqin Wang, Meixin Zhu, and Shaojie Shen. Environment transformer and policy optimization for model- based offlin reinforcement learning, 2023b.
Tianyu Wang, Nikhil Karnwal, and Nikolay Atanasov. Latent policies for adversarial imitation learning. CoRR, abs/2206.1129h) 2022b.
Yiqi Wang, Mengdi Xu, Laixi Shi, and Yuejie Chi. A trajectory is worth three sentences: multimodal transformer for offline reinforcement lcarning. In Uncertainty in Artificial Intelligence, volume 216, pp. 2226 2236, 2023c.
Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. arxiv preprint or Xiv:22312.139 13716, 2023d.
Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. Tn Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 10961-10970. PMLR, 2021b.
Yunke Wang, Minjing Dong, Bo Du, and Chang Xu. Imitation learning from purified demonstration. arXiv preprint arXtw:2310.0713 07,444,,, 2023e.
Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. In Proceedings of the 1 11th International Conference on Learning Represen- tations. OpenReview.net, 2023f.
Ziyu Wang, Josh Merel, Scott E. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems 30, Long Beach, CA, USA, pp. 5320-5329, 2017.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems 33, 2020.
Patrick Nadeem Ward, Ariella Smofsky, and Avishck Jocy Bose. Improving exploration in soft-actor-critic with normalizing flows policies. CoRR, abs/ his//9966.0271) 2019.
Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang. and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pp. 3574 3580. ijeai.org, 2021.
Yao Wei, Yanchao Sun, Ruijie Zheng, Sai Vemprala, Rogerio Bonatti, Shuhang Chen, Ratnesh Madaan, Zhongjie Ba, Ashish Kapoor, and Shuang Ma. Is imitation all you need? generalized decision-making with dual-phase training. In JEEE/CWE International Conference on Computer Vision, pp. 11221-16231. 2023.
Matthias Weissenbacher, Samarth Sinha, Animesh Garg, and Yoshinobu Kawahara. Koopman q-laarning Offline reinforcement learning via symmetries of dynamics. In Proceedings of the 39th International Con- ference on Machine Learning, volume 162, pp. 23645-23000 2022.
99

---

Published in Transactions on Machine Learning Research (08/2024)
Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, pp. 681-688, 2011.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Trans- formers in time series: A survey. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, pp. 6778 6786, 2023.
Eric Wiewiora. Potential-based shaping and q-value initialization are equivalent. Journal of Artificial Intel- ligence Research, 19:205 208, 2003.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Bchavior regularized offline reinforcement learning. CoRR, abs/1911.11361 2019a.
Yuchen Wu, Melissa Mozifian, and Florian Shkurti. Shaping rewards for reinforcement learning with imper- fect demonstrations using generalive models. In IEEE International Conference Robolics and Auloma- tion, pp. 6628 6634. IEEE, 2021.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Tmitation learning from imperfect demonstration. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 6818-6827. PMLR, 2019b.
Wei Xiao, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. CoRR, abs/2306.00144 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemmaa with denoising diffusion gans. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsupervised pretraining for decision transformer. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 38187-38203, 2023.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Thitt AAAI Conference on Artificial Intelligence, pp. 8753-8760. AAAI Press, 2022a.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 14631-24641) 2022b.
Yilun Xu, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. Poisson flow generative models. In Advances in Neural Information Processing Systems 35, 2022c.
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi S. Jaakkola. PFGM++ unlocking the potential of physics-inspired generative models. Tn proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38566 38591. PMLR, 2023.
Taku Yamagata, Ahmed Khalil, and Raúl Santos-Rodriguez. Q-learning decision transformer: Leverag- ing dynamic programming for conditional sequence modelling in offline RL. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 38989 39007, 2023.
Kai Yan, Alexander G. Schwing, and Yu-Xiong Wang. CEIP: combining explicit and implicit priors for reinforcement learning with demonstrations. In Advonces in Neural Information Processing Systems 35, 2022.
Junming Yang, Xingguo Chen, Shengyuan Wang, and Bolci Zhang. Model-based offline policy optimization with adversarial network. In Proceedings of the 26th European Conference on Artificial Intelligence, volume 372, pp. 2850 2857, 2023a.
100

---

Published in Transactions on Machine Learning Research (08/2024)
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuun Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. CoRR, abs/2209.00796 2022a.
Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned rI? In Proceedings of the 40th International Conference Machine Learning, volume 202, pp. 39543-39571, 2023b.
Shentao Yang, Yihao Feng, Shujian Zhang, and Mingyuan Zhou. Regularizing a model- policy sta- tionary distribution to stabilize offine reinforcement learning. In Proceedings of the 39th International 2022b.
Conference on Machine Learning, volume 162, pp.
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202 00002222200073,, 2022c.
Shentao Yang, Shujian Zhang, Yihao Feng, and Mingyuan Zhou, A unified framework for alternating offline model training and policy learning. In Advances in Neural Information Processing Systems 35, 2022d.
Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. In Proceedings of the 11th International Conference on Learning Representations. OpenReview.nue net, 2023c.
Yiming Yang, Dengpeng Xing, and Bo Xu. Efficient spatiotemporal transformer for robotic reinforcement learning. IEEF Robolics Aulomation Letters, ((3):7982-7983) 2022e.
Yiqin Yang, Hao Hu, Wenzhe Li, Siyuan Li, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Flow to control: Offline reinforcement learning with lossless primitive discovery. In Proceedings of the 37th AAAT Conference on Artificial Intelligence, pp. 10843 10851. AAAI Press, 2023d.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. Tn Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 7194 7201, 2019a.
Lantao Yu, Tianhe Yu, Chelsca Finn, and Stefano Ermon. Meta-inverse reinforcement learning with prob- abilistic context variables. In Advances in Neural Information Processing Systems 32, pp. 11749 11760, 2019b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the 3rd Annual Conference on Robot Learning, volume 100, pp. 1094 1100, 2019c.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Advances in Neural Information Processing Systems 33, 2020a.
Xingrui Yu, Yuemiing Lyu, and Ivor W. Tsang. Intrinsic reward driven imitation learning via generative model. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 10925-10935, PMLR, 2020b.
William Yuan, Jiaxing Chen, Shaofei Chen, Lina Lu, Zhenzhen Hu, Peng Li, Dawei Feng, Furong Liu, and Jing Chen. Transformer in reinforcement lcarning for docision-making: A survey. TechRariv 2023a.
Ye Yuan, Xin Li, Yong Heng, Leiji Zhang, and Mingzhong Wang. Good better best: Self-motivated imitation learning for noisy demonstrations. CoRR, 2bs1/22311188 2023b.
Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. CoRR,, abs/2309.02472 2023.
Chi Zhang, Sanmukh R. Kuppannagari, and Viktor K. Prasanna. BRAC+: improved behavior regularizod actor critic for offline reinforcement learning. In Asian Conference on Machine Learning, volume 157 of Proceedings of Machine Learning Research, pp. 219. PMLR, 2021a.
101

---

Published in Transactions on Machine Learning Research (08/2024)
Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement. learning. In Thirty-Sinth AAAT Conference on Artificial Intelligence, pp. 9022 9030. AAAI Press, 2022.
Jing Zhang, Chi Zhang, Wenjia Wang, and Bing-Yi Yi Jing. APAC: authorized probability-controlled actor- critic for offline reinforcement learning, CoRR, abs/2301.12130, 2023.
Xin Zhang, Yanhua Li, Xun Zhou, and Jun Luo. Unveiling taxi drivers' strategies via cgail: Conditional generative adversarial imitation learning. In IEEE International Conference on Data Mining, pp. 1480 1485. IEEE, 2019.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adver- sarial imitation learning. In Advances in Neural Information Processing Systems 33, 2020a.
Xin Zhang, Yanhua Li, Ziming Zhang, Christopher Brinton, Zhemming Liu, Zhi-Li Zhang, Hui Lu, and Zhihong Tian. Stabilized likelihood- based imitation learning via denoising continuous normalizing flow. In Submission the 10th International Conference on Learning Representations, 2021b.
Yufeng Zhang, Qi Cai, Zhuoran Yang, and 'Zhaoran Wang. Generative adversarial imitation learning with ncural network paramcterization: Global optimality and convergence rate. In Proceedings of the 40th International Conference on Machine Learning, pp. 11044 11054. PMLR, 2020b.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 77042-27052 2022.
Qinqing Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover. Semi-supervised offiine reinforcement learning with action-free trajectories. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 42339 42362, 2023.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS:: latent action space for offline reinforcement learning. In Proceedings of the 4th Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 1719 1735. PMLR, 2020.
Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Towards long-delayed sparsity: Learning a better transformer through reward redistribution. In Proceedings of the 32nd International Joint Conference Artificial Intelligence, pp. 4693 4701, 2023a.
Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu, VIOLA: imitation learning for vision-based ma- nipulation with object proposal priors. CoRR, abs/2210.11339, 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martin. robosuite: A modular simulation framework and benchmark for robot learning. CoRR, abs/2009.12293 2020.
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffiusion models. CoRR, abs/ abs/2305.17330, 2023b.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang, Diffusion models for reinforcement learning: A survey. CoRR, abs/ 2os//231101123,, 2023c.
Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference Machine Learning, pp. 1255-1280 2010.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-learning, In Proceedings of the 8th International Conference on Learning Representations. OpenReview.net, 2020.
Konrad Zolna, Scott E. Roed, Alexander Novikov, Sergio Gómez Colmenarejo, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learning. In Proceed- ings of the 4th Conference on Robot Learning, volume 155, pp. 247-263. PMIRR 2020.
102

---

Published in Transactions on Machine Learning Research (08/2024)
Guoyu Zuo, Kexin Chen, Jiahao Lu, and Xiangsheng Huang. Deterministic generative adversarial imitation learning. Neurocompuling, 388:60- 69, 2020.
103


Reviews:
[{"review_id": "zHn2iKE0LP", "reviewer": "Reviewer_HMCN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The discussion is thorough, and the covered topics are wide, the discussion for future works is also comprehensive. \n\nWeakness: I believe the authors missed one particular important class of algorithms in offline RL, one-step algorithms. For instance, in [1] offline can be formulated using one-step framework with appropriate approximation. What is the potential of one-step algorithm in with deep generative models as approximator? I think the current paper should at least explain it (and other related papers) in the future work section. \n\n\n[1] Offline Reinforcement Learning with Closed-Form Policy Improvement Operators, ICML23.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a thorough review for and topic of generative models + OPL, covers the topic such as variational auto-encoders, generative adversarial networks, normalizing flows, transformers and diffusion models. Concretely, this paper is the first to review Deep Generative Models (DGMs) for Offline Policy Learning, covering a wide range of topics, including five major DGMs and their applications in Offline Reinforcement Learning and Imitation Learning. It distills key algorithmic schemes and paradigms, highlighting seminal research works as tutorials. Additionally, it showcases the evolution of DGM-based offline policy learning in parallel with advancements in generative models and provides valuable insights and future directions in the concluding summary.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't have other questions, just answer the question I presented above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eLLffAfczs", "reviewer": "Reviewer_x4Zp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is written and organized clearly and is easy to follow\n- The summary tables present a clear overview of different algorithms, the generative model variants that they used, and the evaluation tasks, which is quite useful. \n- Discussion section and table 13 provides insight on the main issues of each family of models and their extensions, which is interesting. \n\nWeaknesses:\n\n- As a survey paper, it provides limited technical novelty or results, the main contribution is in providing a tutorial and guidance for researchers interested in the related topics. \n- The survey itself does not have major weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of different deep generative models used for offline policy learning (offline RL and imitation learning). 5 types of models are discussed: VAE, GAN, Normalizing Flows, Transformers and Diffusion models. The paper first gives a background discussion for each of these model families, then for each of them a section is dedicated to discuss the relevant works in offline RL and imitation learning settings. The paper is the first review/survey paper on using deep generative models for offline policy learning, and serves as a tutorial for researchers interested in this topic.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would be nice to also have in the discussion section to compare these families of models in aspects such as general computation efficiency, implementation difficulty, training stability, difficult or easy to tune hyperparameters, etc.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VCABgCpNwL", "reviewer": "Reviewer_hunT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* This paper is exceptionally comprehensive and current. There is truly impressive coverage of an extremely diverse field which has also seen enormous change in the past 3 years. Given the recent explosion of research papers in this area, review papers like this are critical for distilling and synthesizing insights from the literature. It will be a very helpful reference for new researchers entering this field.\n* The authors have given careful thought as to how to present results both comprehensively but also succinctly. The authors have done a nice job of using a few key papers to highlight a broad approach, and then presenting other works as deltas on these few key works. Similarly, for every section, there is a table provided that helps classify and cluster research works together. This helps a reader who may not have time to read the entire paper to pinpoint key works to look for within the section. I particularly appreciated the emphasis on which environments different algorithms were tested on within the tables.\n* Several sections are well written, with key insights being clearly elucidated in the text.\n* Caveats are clearly outlined for where there is conflicting evidence about the value of a particular approach.\n* Concluding sections detailing similarities and differences between generative model classes are well written, and the authors illuminate areas of the field that are under-explored (determined as a result of the literature review).\n\nWeaknesses\n* Some of the paper's organization is confusing. In particular, the introduction of the generative modelling methods *before* their respective sections seems not ideal, especially because their notation was not shared. It would be easier for the reader if these techniques were introduced in the sections that they belong to. Instead, the offline policy learning methods could be outlined where the generative models are currently outlined since they are used across many sections. This alternative organization of the paper would make navigating it easier for readers.\n  * For example, in the section on policy constraint methods in VAEs, the paper notes \"Offline RL algorithms based on diffusion models typically adhere to policy constraint methods. Therefore, we will provide further details on policy constraint methods in Section 7.\" This is a good example of how a restructuring to put the offline policy method details *first* would help with motivating and signposting how generative models can be used, irrespective of their model class.\n  * This restructuring would also help clarify the similarities and differences of how different generative models can be used in similar offline policy learning algorithms. For example, as the paper notes in the concluding sections, all DGMs can be used to represent the policy function. Motivating this as part of the background (by introducing offline policy learning algorithms first) would be helpful, as it provides a framework for the reader to consider the different opportunities for DGMs to play a role.\n  * This restructuring would also provide an opportunity to enumerate what the challenges for offline policy learning are which can be addressed by DGMs (actually very nicely summarized in Table 13). This is introduced in individual sections but ends up being very repetitive as the same challenges come up across many sections. Introducing these challenges before going into the specific DGMs would help motivate the rest of the review article.\n  * As another example, in \"7.2.1 Diffusion Models as Policies\", Section 3.1.1 is mentioned (policy constraint methods). The paper states \"DM can be employed to model the policy pi_theta\". If the policy constraint method was mentioned at the beginning, this would be a more natural comparison, and the paper could include an explanation for why DMs would be preferable to VAEs for modeling pi_theta.\n\n* Generative models are never defined. Four out of 5 models are fundamentally based on sampling from a probabilistic distribution (VAEs, GANs, NFs, DMs) while the fifth, the transformer, is not. As a result, the section on transformers seemed somewhat out of place with respect to the other works, since it is generative in a fundamentally different way (it predicts next tokens based on previous tokens, rather than learning a distribution such that sampling from it gives data points that resemble true data points from training).\n* Some sections are not about offline policy learning. Section 6.1.5 in particular focuses on online, partially observable RL. This is not within scope for the paper, and should therefore be removed. Likely the online RL with offline data section of NFs should also be removed, but this is more tangentially related so there could be an argument for keeping it.\n* Some terms are not defined. For example \"return-to-go\" and \"MAF\" in 5.1.1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This review paper is truly a tour de force. It covers applications of 5 different kinds of deep generative models to offline policy learning (including offline reinforcement learning and imitation learning). The paper is a mix of tutorials, reference guides, and systematic paper review. For the generative models, it covers Variational Autoencoders (VAEs) and their variants, Generative Adversarial Networks (GANs), Normalizing Flows (NFs), Transformers, and Diffusion Models (DMs). The paper is structured such that the technical background for all generative models is presented first, and then each paper section is focused on one of these generative models, and its application to offline policy learning. Every section includes a summary table highlighting various aspects of the cited papers within the section. The paper concludes by drawing similarities and differences across different generative model classes in how they are used within offline policy learning, and highlighting areas for future work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for securing my recommendation\n* As noted in the weaknesses section, the paper would be much clearer with a restructuring of how offline policy learning is introduced relative to deep generative models. Concretely, offline policy learning would be introduced as the \"Background\" section, with the details of each DGM introduced in their respective sections instead. \n* In Section 8.1, equation 102 is very difficult to read. It would be helpful to split it into multiple equations for each of VAEs, GANs, etc. listed separately.\n* \"Regarding IL, most DGMs, including VAEs, GANs, and DMs, select BC as the base algorithm...\"\n  * This doesn’t match the table – GANs are using MaxEntIRL. Transformers are using BC. Please fix this typo.\n* Please make the tables consistent with whether they use the rows or columns to define different algorithms. Currently it is a mix between the two and this made interpreting the tables more complicated than it needed to be.\n* Please provide a definition of generative models and give some motivation for why transformers (which are a fundamentally different kind of \"generative\") are discussed in the context of the review.\n\nStrengthening the work\n* In section 8.1, there are two very nice tables (Table 12 and 13) which provide comparisons between DGMs for different offline policy learning setups. Moving these to the background section with offline policy learning would be more useful than having them at the end, as they could signpost for the reader which sections to look at if they want to solve a particular problem in offline policy learning.", "broader_impact_concerns": "No concerns about broader impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have consensus that this is a well written overview/survey of the topic. The authors engaged during discussion, provided clarifications and substantially revised the draft in response to feedback.
Decision: Accept (poster)