Title: Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Offline Reinforcement Learning

Model-Based Reinforcement Learning

Latent Action

Constrained Policy

Generative Model

Applied Reinforcement Learning
Abstract: In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.

Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning

Abstract

In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-ofthe-art methods, especially outperforming on datasets with visual observations. 1

1 Introduction

Deep-learning methods are widely used in applications around computer vision and natural language processing, related to the fact that datasets are abundant. But when used for control of physical systems, in particular with reinforcement learning, obtaining data involves interaction with an environment. Learning through trial-and-error and extensive exploration of an environment can be done in simulation, but hard to achieve in real world scenarios [1, 2, 3]. Offline reinforcement learning tries to solve this by using pre-collected datasets eliminating costly and unsafe training in real-world environments [4, 5, 6].

Using online reinforcement learning methods in an offline setting often fails. A key issue is the distributional shift : the state-action distribution of the offline dataset, driven by the behavior policy, differs from the distribution generated by a learned policy. This leads to actions being inferred for states outside the training distribution. Therefore, value-based methods are prone to overestimating values due to evaluating policies on out-of-distribution states. This leads to poor performance and unstable training because of bootstrapping [6, 7, 8]. Offline reinforcement learning methods address

(b) Policy training

(c) Environment interaction

1

↑2

个3

42

r1

V2

r2

办业

V3

r3

u2

A

中业

中业

42

a1

a1

7D

a2

01

01

02

02

A

03

03

01

1

01

1

02

1

03

this issue with different approaches and can be categorized into model-free and model-based methods, similar to online reinforcement learning.

Modelfree offline reinforcement learning usually follows one of the following paradigms: constrain the learned policy to the behavior policy [9, 10, 11, 7]; or introduce some kind of conservatism to the Bellman update [12, 13, 14, 8]. Modelbased reinforcement learning methods transform the offline to an online learning setting: They approximate the system dynamics and try to resolve the evaluation of out-of-distribution states by using the generalization capabilities of the model and generating additional samples. But as the training distribution is fixed, the estimation capabilities of the model are limited. Therefore, these model-based methods also rely on a conservative modification to the Bellman update as a measure to counteract value overestimation which is mostly achieved through uncertainty penalties [15, 16, 17, 18, 19, 20, 21]. Apart from the typical approach of using an auto-regressive model to estimate the dynamics, other model-based methods treat the objective as trajectory modeling [22, 23, 24]. These methods aim to combine decision making and dynamics modeling into one objective. Instead of learning a policy, they sample from the learned trajectory model for planning. We will refer to the first kind of methods, which learn a dynamics model to train a policy, as model-based reinforcement learning.

We aim to solve the problem of value overestimation in model-based reinforcement learning by jointly modeling action and state distributions, without the need for uncertainty penalties or changes to the Bellman update. Instead of learning a conditional dynamics model p ( s | a ) , we estimate the joint state-action distribution p ( s, a ) . This is similar to methods that frame offline reinforcement learning as trajectory modeling, but we use an auto-regressive model and still learn a policy. By formulating the objective as a generative model of the joint distribution of states and actions, we create an implicit constraint on the generated actions, similar to [10, 25]. The goal of this approach is to address the shift in the entire distribution, rather than looking at out-of-distribution actions and states separately. We achieve this using a recurrent state-space model with a latent action space, which we call the recurrent latent action state-space model. Using a latent action space allows us to learn a policy that uses the latent action prior as an inductive bias. This approach keeps the policy close to the original data while allowing it to change when needed, which makes learning the policy much faster. To achieve this, we treat policy optimization as a constrained optimization problem, similar to enforcing a support constraint [9]. We provide a high level overview of our method in Figure 1.

Overall, we summarize our contribution as follows:

Weintroduce latent action state-space models for model-based offline reinforcement learning, treating it as auto-regressive generative modeling of the joint distribution of states and actions.

We formulate policy optimization as a constrained optimization problem, using the latent action space to generate actions within the support of the dataset's action distribution and jump-start policy learning by using the generative action decoder.

We evaluate our approach on one benchmark with image observations (V-D4RL [20]) and on another one with low-dimensional feature observations (D4RL [26]).

We evaluate the effect of our approach on value overestimation.

2 Preliminaries

We consider a partial observable Markov decision process (POMDP) defined by M = ( S , A , O , T, R, Ω , γ ) with S as state space, A as action space, O as observation space, s ∈ S as state, a ∈ A as action, o ∈ O as observation, T : S × A → S as transition function, R : S → R as reward function, Ω : S → O as emission function and γ ∈ (0 , 1] as discount factor. The goal is to find a policy π : O → A that maximizes the expected discounted sum of rewards E [ ∑ T t =1 γ t r t ] [27].

In online reinforcement learning, an agent iteratively interacts with the environment M and optimizes its policy π . In offline reinforcement learning, however, the agent cannot interact with the environment and must refine the policy using a fixed dataset D = { ( o 1: T , a 1: T , r 1: T ) N n =1 } . Therefore, the agent must understand the environment using limited data to ensure the policy maximizes the expected discounted sum of rewards when deployed [6]. Auto-regressive model-based offline reinforcement learning tries to learn a parametric function to estimate the transition dynamics T . The transition dynamics model is then used to generate additional trajectories which can be used to train a policy. The majority of these approaches learn a dynamics model directly in observation space T θ ( o t | o t -1 , a t -1 ) [16, 17, 18, 19, 28], while others use a latent dynamics model T θ ( s t | s t -1 , a t -1 ) [20, 21].

3 Constrained Latent Action Policies

A main issue in offline reinforcement learning is value overestimation, which we address by ensuring the actions generated by the policy stay within the dataset's action distribution. Unlike previous modelbased methods, we formulate the learning objective as a generative model of the joint distribution of states and actions. We do this by combining a latent action space with a latent dynamics model. Next, we use the generative properties of the action space to constrain the policy to the dataset's action distribution. A general outline of our method, Constrained Latent Action Policies (C-LAP ), is shown in Appendix B. It starts with learning a generative model, followed by actor-critic agent training on imagined trajectories, similar to the methods in [29, 30, 20, 21].

o

1

o

2

o

3

s

1

s

2

s

3

h

1

h

2

h

3

a

1

a

2

a

3

u

1

u

2

u

3

Generative model Model-based offline reinforcement learning requires learning a model that is accurate in areas with low data coverage but also generalizes well. Therefore, it's crucial to balance

staying within the dataset's distribution and generalizing to unseen states. We propose a generative model

that jointly models the observation and action distribution of a static dataset D = { ( o 1: T , a 1: T , r 1: T ) N n =1 } by using latent states s t along with latent actions u t . Unlike other modelbased offline reinforcement learning methods that learn a conditional model p ( o 1: T | a 1: T ) and rely on ensemble based uncertainty penalties on the Bellman update to generate trajectories within the data distribution [21, 18, 16, 17], our approach uses a latent action space to impose an additional implicit constraint. By implementing a policy in the latent action space, generated actions will stay within the dataset's action distribution, thus enabling generalization within the limits of the learned model [10]. We empirically validate this claim in Appendix F. To obtain a state space model with Markovian assumptions on the latent states s t we impose the following structure:

We implement the probabilistic model modifying the design of a recurrent state-space model [31]. Thus, the latent dynamics model p ( s t | s t -1 , u t -1 ) is based on the deterministic transition f ( h t -1 , s t -1 , a t -1 ) using the latent action decoder p θ ( a t -1 | s t -1 , u t -1 ) to generate actions. In the following, we mostly omit deterministic states h t for notational brevity. The resulting recurrent latent action state-space model is shown in Figure 2 and consists of the following components, specifically

The latent state prior predicts the next latent state s t given the previous latent state s t -1 and action u t -1 using the deterministic transition and the action decoder. The latent action prior predicts latent actions u t given latent state s t . Latent states as well and as latent actions are decoded using their respective decoder. Similar to [25, 10] actions are reconstructed given latent state and latent action.

Directly maximizing the marginal likelihood is intractable, hence we maximize the evidence lower bound (ELBO) on the log-likelihood log p ( o 1: T , a 1: T ) instead. To approximate the true posterior, we introduce

as inference models. The latent state posterior encodes observations o t to latent states s t by using the deterministic transition. The latent action posterior encodes actions a t to latent actions u t conditioned on latent states s t . All parameters of the generative model are indicated by θ and parameters of the inference model by ϕ .

We derive the ELBO

which can be organized into individual terms for reconstruction and consistency of actions and observations. The derivation can be found in Appendix A.

Maximizing the objective enables us to learn a model which can generate trajectories close to the data distribution D by sampling from both priors. As we want to use the model to learn a policy via latent imagination, we add a reward p θ ( r t | s t ) and termination p θ ( t t | s t ) model. Hence, the complete model training objective is

Constrained latent action policy We use the sequence model to generate imagined trajectories and use an actor-critic approach to train the policy. To predict state values we learn a value model v ξ ( s t ) alongside the policy. Therefore we use the n-step return of a state

as regression target for v ξ ( s t ) [27, 29]. Polices trained on trajectories generated by a model are prone to end up with degrading performance if the model only has access to a limited data distribution, as in the case of offline reinforcement learning. Compounding modeling errors and value overestimation of edge-of-reach states [19] are reasons for the decline. Since we train a generative action model, generated actions are implicitly constrained to the datasets action distribution by sampling from the action decoder a t ∼ p θ ( a t | s t , u t ) . Hence, states outside the datasets observation distribution are hard to reach and our approach is resilient to value overestimation of edge-of-reach states. Compounding modeling errors are still a source of diminishing performance, but can be counteracted by increasing the representation power of the model or generating only short trajectories. To leverage the generative action model, we learn a policy π ψ ( u t | s t ) in the latent action space similar to [10, 25]. But, as both, the latent action prior p θ ( u t | s t ) and the policy π ψ ( u t | s t ) are flexible, it is not ensured that they share the same support. Thus, we formulate policy optimization as a constrained optimization problem

similar to a support constraint [9]. We implement the constraint explicitly through parametrization to stay within the support, but do not impose any restrictions inside the supported limits (Figure 3). This is different to using a divergence measure which on one hand does not strictly ensure support limits and on the other hand is more restrictive as it also imposes a constraint on the shape of a distribution. Here and in the following ˆ u t stands for a latent action sampled from the policy π ψ ( u t | s t ) .

The policy is trained to maximize the n-step return V k N ( s τ ) while staying in support of the latent action prior. Since the latent action prior p θ ( u t | s t ) is normally distributed as N ( µ θ ( s t ) , σ θ ( s t )) , we can express the constraint as

with ˜ ϵ as a parameter setting support as multiples of σ θ centered around µ θ . From the properties of a normal distributed variable follows that

We implement the constraint explicitly by parameterizing the policy as a linear function g dependent on π ψ ( u t | s t ) and p θ ( u t | s t ) :

The support of the policy distribution is chosen to be bounded

and g ( π ψ ( u t | s t ) , p θ ( u t | s t )) as a linear combination of the latent action predicted by the policy ˆ u t and the distribution parameters σ θ and µ θ of the latent action prior:

4 Experiments

In the next section, we evaluate the effectiveness of our approach. It is divided into three parts: first, we assess the performance using standard benchmarks; then, we study how different design choices affect value overestimation; lastly, we analyze the influence of the support constraint parameter. We additionally provide the final performances in two tables in Appendix E.

We limit our benchmark evaluation to the most relevant state-of-the-art offline reinforcement learning methods to answer the following questions: 1) How do latent action state-space models compare to state-space models? 2) How comparable are model-free methods focusing on latent action spaces to latent action state-space models? 3) Does C-LAP suffer from value overestimation? 4) How does the support constraint affect the performance? 5) How does the performance differ between visual observations and observations with low-dimensional features? To focus on the latter, we separately evaluate the performance on low-dimensional feature observations using the D4RL benchmark [26], and on image observations using the V-D4RL benchmark [20].

4.1 Benchmark results

D4RL Since most offline model-based reinforcement learning methods are designed for observations with low-dimensional feature observations, there exist many options for comparison. We make a selection to include the most relevant methods focusing on latent actions and auto-regressive model-based reinforcement learning. Therefore, we include the following methods: PLAS, which is a model-free method using a latent action space [10]. MOPO, a probabilistic ensemble-based offline model-based reinforcement learning method using a modification to the Bellman update to penalize high variance in next state predictions [16]. And MOBILE, which is similar to MOPO, but penalizes high variance in value estimates instead [18]. We compare the algorithms on three different locomotion environments, namely halfcheetah, walker2d and hopper, with four datasets (medium-replay, medium, medium-expert, expert) each and the antmaze navigation environment with four datasets (umaze, umaze-diverse, medium-play, medium-diverse). The results, shown in Figure 4, display the mean and standard deviation of normalized returns over four seeds during the phase of policy training, with steps denoting gradient steps. The dashed lines indicate the asymptotic performance for MOPO and MOBILE. A detailed summary of all implementation details is provided in the Appendix D.

When comparing C-LAP to PLAS, we find that learning a joint generative model of actions and observations outperforms a generative model of only actions when used with actor-critic reinforcement learning. Both methods can use the generative nature of the model to speed up policy learning, which becomes especially clear in the results on all locomotion expert and medium-expert datasets. Compared to MOPO and MOBILE, C-LAP shows a superior or comparable performance on all datasets except halfcheetah-medium-replay-v2, halfcheetah-medium-v2 and hopper-medium-v2. Especially outperforming on the antmaze environment, where MOPO and MOBILE fail to solve the task for any of the considered datasets. The asymptotic performance of MOBILE on locomotion environments sometimes exceeds the results of C-LAP, but needs three times as many gradient steps.

Overall the results indicate that latent action state-space models with constrained latent action polices not only match the state-of-the-art on observations with low-dimensional features as observations, but also jump-start policy learning by using the action decoder to sample actions that lead to high rewards already after the first gradient steps: If the dataset is narrow, generated actions when sampling from the latent action prior will fall into the same narrow distribution. For instance, in an expert dataset, sampled actions will also be expert-level actions. During policy training, instead of sampling from this prior, we restrict the support of the policy dependent on the latent action prior. Thus, sampled latent actions from the policy will always be decoded to fall into the dataset's action distribution. So even a randomly initialized policy in the beginning of the training can generate a high reward by using the latent action decoder. This effect is especially prominent in narrow datasets such as expert datasets.

V-D4RL There are currently few auto-regressive model-based reinforcement learning methods that specifically target visual observations, with none emphasizing latent actions. In our evaluation, we include LOMPO [21] and Offline DV2 [20]. Both methods use a latent state space model and an uncertainty penalized reward. However the specifics of the penalty calculations are different: while LOMPO uses standard deviation of log probabilities as penalty, Offline DV2 uses mean disagreement. Additionaly, LOMPO trains an agent on a mix of real and imagined trajectories with an off-policy actor-critic approach, whereas Offline DV2 exclusively trains on imagined trajectories and backpropagates gradients through the dynamics model. Further implementation details are included in Appendix D.

C-LAP demonstrates superior performance across all datasets, especially significant on cheetahrun-medium_expert, walker-walk-medium_expert and walker-walk_expert. Datasets with a large

diversity of actions, such as medium-replay datasets, exhibit a weaker inductive bias for a generative action model. Hence, they require more additional policy steps, as can be seen for both the D4RL and V-D4RL benchmarks.

4.2 Value overestimation

Limiting value overestimation plays a central role in offline reinforcement learning. To evaluate the effectiveness of C-LAP, we report value estimates alongside normalized returns on all walker2d datasets in Figure 6. A similar analysis for all considered baselines is provided in Appendix G. To further analyze the influence of different action space design choices, we include the following ablations: a variant no constraint , which does not formulate policy optimization as constrained objective, but uses a Gaussian policy distribution to potentially cover the whole Gaussian latent action space; and a variant no latent action , which does not emphasize latent actions, but uses a regular state-space model as in Dreamer [29]. Besides that, we added dashed lines to indicate the dataset's average return and average maximum value estimate. The no latent action variant fails to learn

an effective policy: normalized returns are almost zero and the dataset's reference returns remain unattained; value estimates are significantly exceeding the dataset's reference values, indicating value overestimation. The no constraint variant can use the generative action decoder to limit generated actions to the dataset's action distribution, but the Gaussian policy is free to move to regions which are unlikely under the action prior. Thus, nullifying the implicit constraint imposed by the action decoder, resulting in collapsing returns and value overestimation. Only C-LAP achieves a high return and generates value estimates which are close to the dataset's reference. The value estimates on walker2d-medium-replay-v2 are higher than the dataset's reference, as the agent's performance is also exceeding the reference performance. The results confirm the importance of limiting value overestimation in offline reinforcement learning, and demonstrate that constraining latent action policies can be an effective measure for achieving this.

4.3 Support constraint parameter

To evaluate the influence of the support constraint parameter ˜ ϵ on the performance of C-LAP, we perform a sensitivity analysis across all walker2d datasets (Figure 7). Except for the more diverse medium-replay-v2 dataset, adjusting ˜ ϵ from 0 . 5 to 3 . 0 only has a minor impact on the achieved return. However, when choosing an unreasonable large value such as ˜ ϵ = 10 . 0 or removing the constraint altogether (Figure 6), we observe a collapse during training. This highlights a key insight: constraining the policy to the support of the latent action prior is essential. And in many cases, using a smaller support region closer to the mean (small ˜ ϵ ) proves sufficient.

5 Related Work

Offline reinforcement learning methods fall into two groups: model-free and model-based. Both types aim to tackle problems like distribution shift and value overestimation. This happens because the methods use a fixed dataset rather than learning by interacting with the environment.

Model-free Current model-free methods typically work by limiting the learned policy or by regularizing value estimates. TD3+BC [11] adds a behavior cloning regularization term to the policy update objective to enforce actions generated by the policy to be close to the dataset's action distribution. Similarly, SPOT [9] includes a regularization term in the policy update, derived from a support constraint perspective. It also uses a conditional variational auto-encoder (CV AE) to estimate the behavior distribution. Following a comparable intention, BEAR [8] constraints the policy to the support of the behavior policy via maximum mean discrepancy. BCQ [7] and PLAS [10] use a CVAE similarly but don't use a regularization term. Instead, they constrain the policy implicitly by making the generative model part of the policy. Beside these methods, many other approaches exist, with CQL [12] and IQL [14] being some of the most well-known. CQL uses a conservative policy update by setting a lower bound on value estimates to prevent overestimation, while IQL avoids out-of-distribution values by using a modified SARSA-like objective in combination with expectile regression to only use state-action tuples contained in the dataset.

Model-based Model-based offline reinforcement learning methods learn a dynamics model to generate samples for policy training. This basically converts offline learning to an online learning problem. Model-based methods mainly address model errors and value overestimation by using

a probabilistic ensemble and adding an uncertainty penalty in the Bellman update. MOPO [16] uses a probabilistic ensemble as in [32] and adds the maximum standard deviation of all ensemble predictions as uncertainty penalty. Similar to that, MOReL [17] adheres to the same methodology, but uses pairwise maximum difference of the ensemble predictions as penalty instead. Analogously, MOBILE [18] estimates the values for all by the ensemble predicted states and uses the standard deviation of value estimates as penalty. Edge of reach [19] comes to the conclusion that value estimation on edge of reach states are the overarching issue compared to model errors. In the end, they come up with a comparable solution to MOBILE, but use an ensemble of value networks alongside the ensemble of dynamic models. COMBO [28] pursues a different approach, as they integrate the conservatism of CQL into value function updates, removing the need for uncertainty penalties. Besides that, some methods use a different class of models: instead of learning a predictive model in observation space, they use latent state-space models to make predictions on latent states. Among these methods is LOMPO [21], which builds up on Dreamer [29], but integrates an ensemble to predict stochastic states and use the standard deviation of the log probability of the ensemble predictions as an uncertainty penalty similar to previous methods. The policy is trained on a mix of imagined and real world samples, hence they use an off-policy actor-critic style approach for policy learning. Offline DV2 [20] uses a similar model, but is based on a different penalty. Namely, they use the difference between the individual ensemble mean predictions and mean over all ensembles as uncertainty penalty. Furthermore, the policy is trained only on imagined trajectories with gradients calculated by back-propagating through the dynamics model. Overall, Offline DV2 is the method most comparable to our approach, but still different in many ways as we propose a latent action state-space model compared to a usual state-space model, and frame policy learning as constrained optimization. So far all discussed models operate in an auto-regressive fashion, but another class of methods exists, which casts offline model-based reinforcement learning as trajectory modeling. Instead of learning a policy, these kind of approaches integrate decision making and modeling of the underlying dynamics into a single objective and use the model for planning. Among them are Diffuser [24], which employs guided diffusion for planning; TT [22], which builds on advances in transformers; and TAP [23], which uses a VQ-VAE with a transformer-based architecture to create a discrete latent action space for planning.

6 Conclusion

We present C-LAP, a model-based offline reinforcement learning method. To tackle the issue of value overestimation, we first propose an auto-regressive latent-action state space model to learn a generative model of the joint distribution of observations and actions. Second, we propose a method for policy training to stay within the dataset's action distribution. We explicitly parameterize the policy depending on the latent action prior and formulate policy learning as constrained objective similar to a support constraint. We find that C-LAP significantly speeds-up policy learning, is competitive on the D4RL benchmark and especially outperforms on the V-D4RL benchmark, raising the best average score across all dataset's from previously 31 . 5 to 58 . 8 .

Limitations Depending on the dataset and environment the effectiveness of C-LAP differs: Datasets which only contain random actions are challenging for learning a generative action model, thus we do not include them in our evaluation. The effect of jump-starting policy learning with the latent action decoder to already achieve high rewards in the beginning of policy training is prominent in narrow datasets, but less effective for diverse datasets. While training the model of C-LAP does not require additional gradient steps, it still takes more time compared to LOMPO [21] and Offline DV2 [20] as the latent action state-space model is more complex than a usual latent state-space model.

References

Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. CoRR , abs/1904.12901, 2019.

Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32:1238 - 1274, 2013.

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. 2020 IEEE Symposium Series on Computational Intelligence (SSCI) , pages 737-744, 2020.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6:503-556, 04 2005.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement Learning: State of the Art , 01 2012.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR , abs/2005.01643, 2020.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR , abs/1812.02900, 2018.

Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. CoRR , abs/1906.00949, 2019.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning, 2022.

Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: latent action space for offline reinforcement learning. CoRR , abs/2011.07213, 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. CoRR , abs/2106.06860, 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. CoRR , abs/2006.04779, 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. CoRR , abs/2110.01548, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR , abs/2110.06169, 2021.

Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen J. Roberts. Revisiting design choices in model-based offline reinforcement learning. CoRR , abs/2110.04135, 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. CoRR , abs/2005.13239, 2020.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel : Model-based offline reinforcement learning. CoRR , abs/2005.05951, 2020.

Yihao Sun, Jiajin Zhang, Chengxing Jia, Hao-Chu Lin, Junyin Ye, and Yangze Yu. Modelbellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning , 2023.

Anya Sims, Cong Lu, and Yee Whye Teh. The edge-of-reach problem in offline model-based reinforcement learning, 2024.

Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations, 2023.

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. CoRR , abs/2012.11547, 2020.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems , 2021.

Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, and Yuandong Tian. Efficient planning in a compact latent action space, 2023.

Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis, 2022.

Arthur Allshire, Roberto Martín-Martín, Charles Lin, Shawn Manuel, Silvio Savarese, and Animesh Garg. LASER: learning a latent action space for efficient reinforcement learning. CoRR , abs/2103.15793, 2021.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR , abs/2004.07219, 2020.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. CoRR , abs/2102.08363, 2021.

Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. CoRR , abs/1912.01603, 2019.

Philip Becker-Ehmck, Maximilian Karl, Jan Peters, and Patrick van der Smagt. Learning to fly via deep model-based reinforcement learning. CoRR , abs/2003.08876, 2020.

Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. CoRR , abs/1811.04551, 2018.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR , abs/1805.12114, 2018.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021 , 2021.

A ELBO derivation

We start with Equation (1) to jointly model the dataset's observation and action distribution

and the defintions in Equation (3), namely:

We introduce the inference model

and by replacing p ( o 1: T , a 1: T | s 1: T , u 1: T ) and p ( s 1: T , u 1: T ) in Equation (13) we get:

We include the inference model and resolve using Jensen's Inequality:

B Algorithm

We provide the general algorithm of C-LAP.

Algorithm 1: C-LAP

C Computational Resources

We use a different hardware setup for experiments with visual observations and experiments with low-dimensional features as observations. We run all experiments on a shared local cluster. C-LAP experiments with visual observations take around 10 hours on a RTX8000 GPU and experiments with low-dimension feature observations around 11 hours on a A100 GPU. We aim to execute most of our code on GPU and parallelize our implementation. Environment evaluations represent a bottleneck as they require execution on CPU. Overall, it takes around 70 combined GPU days to reproduce the benchmark results of all methods. This does not include the compute required for the evaluation of preliminary implementations or hyper-parameter settings.

D Implementation Details

We implement all methods in JAX [33] using Equinox [34]. We provide the hyper-parameters of CLAP in Table 2 and the constraint values used for the D4RL benchmark in Table 3 and for the V-D4RL benchmark in Table 4. In general we consider constraint values in the range ˜ ϵ ∈ { 0 . 5 , 1 . 0 , 2 . 0 , 3 . 0 } .

We implement MOPO and MOBILE following [18] and use the respective hyper-parameters provided in the publication. For MOPO, we select the max-aleatoric version. As no hyper-parameters are provided for the expert datasets of the D4RL benchmark we sweep through the range specified in Table 1 and use the one selected in the table. For the antmaze environment we additionally evaluate

the discount factors of { 0 . 99 , 0 . 995 , 0 . 997 } to compensate for the sparse reward, but do not find hyper-parameters to solve the task.

We implement PLAS following [10] and use the default hyper-parameters for the expert datasets.

We implement LOMPO and Offline DV2 using the continuous state implementation from Dreamer [29], which is different from the original implementation using discrete states. This change keeps the model closer to C-LAP, which is also using continuous states. For LOMPO and Offline DV2, we take hyper-parameters from [20].

As all our implementations might differ from the original one, we include the original scores from the paper alongside our results in Table 5 and Table 6.

MOPO

MOBILE

Hyper-parameter range

penalty coefficient

∈ {

0

.

5

,

2

.

5

,

5

.

0

,

10

.

0

}

rollout steps:

∈ {

1

,

5

}

dataset ratio:

∈ {

0

.

05

,

0

.

5

,

0

.

8

}

penalty coefficient

∈ {

1

.

5

,

2

.

5

,

3

.

5

}

rollout steps:

∈ {

1

,

5

}

dataset ratio:

∈ {

0

.

05

,

0

.

5

,

0

.

8

}

halfcheetah-expert-v2

penalty coefficient:

1

.

0

rollout steps:

5

dataset ratio:

0

.

5

penalty coefficient:

2

.

5

rollout steps:

5

dataset ratio:

0

.

5

walker2d-expert-v2

penalty coefficient:

10

.

0

rollout steps:

1

dataset ratio:

0

.

8

penalty coefficient:

2

.

5

rollout steps:

1

dataset ratio:

0

.

5

hopper-expert-v2

penalty coefficient:

5

.

0

rollout steps:

5

dataset ratio:

0

.

05

penalty coefficient:

3

.

5

rollout steps:

5

dataset ratio:

0

.

5

0 . 998 (antmaze-umaze-diverse, antmaze-medium-play)

E D4RL and V-D4RL benchmark results

We provide the benchmark results in the usual offline reinforcement learning table format. As our implementations might differ from the original one, we include the original scores from the paper alongside our results in tables. We also include the average over all datasets except the expert datasets to make it comparable to the average scores provided in the respective publication.

F Action distribution

To evaluate the claim that latent actions generated by the latent action prior are close to the dataset's action distribution we use the following approach: We randomly select one trajectory from the hopper-expert-v2 dataset and employ k-nearest neighbors to identify the 20 nearest observations and their corresponding actions within the whole dataset for each step. We then sample from the action prior and decoder to generate actions based on the nearest observations at each step. Thereafter, we split the selected trajectory into sections from leaving the ground to touching the ground and fit normal distributions to the aggregated dataset actions and reconstructed prior actions. Thus, we end up with an approximation of the dataset's action distribution and an approximation of the action distribution generated by the prior for each step in the trajectory. Figure 8 shows the aggregated distributions (from leaving the ground to touching the ground) for one exemplary trajectory of the hopper-expert-v2 dataset.

G Value overestimation for the considered baselines

We further report value estimates alongside normalized returns on all walker2d datasets for the considered baselines (Figure 9). As they estimate Q-values, we calculate the corresponding value estimates by averaging over 10 actions sampled from their respective policies. MOPO and MOBILE have a low value estimate, which can be attributed to the incorporated uncertainty penalties. PLAS's value estimates are only stable for walker2d-expert-v2 datasets, but collapse for the other considered datasets. Overall, it seems that value overestimation is not the cause for degrading performance for these methods.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that C-LAP is competitive to state-of-the-art methods and especially outperforming on datasets with visual observations. These claims are based on evaluations on the D4RL and V-D4RL benchmark and the comparison to other relevant methods. We claim that we aim to solve value overestimation with our approach, which we empirically show on the walker2d datasets.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include the limitations in a separate paragraph in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We outline the derivation of the ELBO in the main section (Sec. 3) and provide a detailed derivation in the appendix A. We explain the derivation of the constrained latent action policy step-by-step in the main section (Sec. 3).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include the explanations of your approach in the main section (Sec. 3). We report the used hyper-parameters and an outline of the algorithm in the appendix (Appendix B, D). We provide the code. We use publicly available benchmarks.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code and include configuration files to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, we shortly explain the experimental setting in the experiments section (Sec. 4). We provide a general algorithm, a section on implementation details and the hyper-parameters in the appendix (Appendix A, B, D). Furthermore, we share the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the number of used seeds and report mean and standard deviation in our evaluations in the experiments section (Sec. 4).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We explain the required computational resources appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We give access to the complete method details and thoroughly share details of the experiments.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We proposed method falls in the category of foundational research in the field of offline reinforcement learning and the method is not tied to an application. Of course, applications exist where reinforcement learning might have a negative societal impact, but our method only has a limited influence on that.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no high risk of misuse of our method.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers which published the benchmarks which we use. We cite the authors of JAX and Equinox, as our implementation is based on their framework. We give credit to Twemoji in figure 1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide code and include configuration files to reproduce experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes C-LAP, a model-based offline RL method that mitigates the distributional shift problem without using uncertainty penalties or modifying the Bellman update. It learns a joint distribution of latent states and latent actions and constrains the latent action policy to the dataset distribution by using a linear transformation. Experiments on D4RL datasets and V-D4RL datasets report performance comparable to other strong offline RL baselines.\n\nSTRENGTHS:\n1. This paper addresses the issue of distributional shift in offline RL by incorporating latent actions to enhance policy learning. The empirical results demonstrate that the proposed method outperforms other offline RL baselines by requiring fewer gradient steps to learn a policy.\n2. The idea of combining a latent action prior and a bounded policy to obtain the constrained policy is novel and intuitive.\n\nWEAKNESSES:\n1. Figure 1 requires a clearer presentation of the model diagram. It would be better to simultaneously show the prior and posterior in Figure 1(a), and the latent action prior $p_\\theta(u_t|s_t)$ and the policy $\\pi_\\psi(u_t|s_t)$ in Figure 1(b) to improve understanding. During the policy training phase, are the imagined trajectories rolled out from the latent state prior $p_\\theta(s_t|s_{t−1}, u_{t−1})$ or the latent state posterior $q_\\phi(s_t|s_{t−1}, a_{t−1}, o_t)$? If it is the latent state prior, why is the output of the action decoder used? If it is the latent state posterior, $o_t$ is unavailable.\n2. Could the authors provide some distribution results to support the claim that the latent actions generated by the action prior are close to the dataset's action distribution? Does the predictive power of using latent actions degrade in performance compared to models using real actions? Some quantitative or qualitative results can be provided.\n3. It would be better to highlight the differences between the proposed method and Dreamer in terms of model designs and algorithms.\n4. The experiments mainly focus on Mujoco tasks. How does the proposed method perform on navigation tasks such as AntMaze?\n5. Why do some experiments in Figure 4, 5 show high returns at the beginning of policy learning (zero gradient step)?\n6. In table 5 and table 6, the performance of C-LAP is inferior to some baseline models in medium or medium-relay datasets. This results seem to indicate that the effectiveness of the proposed method depends on the quality of the offline dataset.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThis paper discusses the limitations, but that is not enough.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper approaches the problem of model-based policy learning from static datasets. It firstly identifies that this offline MBRL setup inherits two major concerns from its components: the problem of value overestimation from out-of-distribution actions (common in offline RL) and the bias originated by learning policies from model predictions. In order to address these issues, it proposes C-LAP, a MBRL method that besides learning a latent state space model - as common in the literature - also learns a latent action space from the static dataset. This generative model of states AND actions is further leveraged by the policy optimization objective to constrain the policy actions to share support with the latent action prior, in order to prevent out-of-distribution samples. The paper provides experiments in two offline RL benchmarks, with feature and pixel-based observations, and showing good improvements in the pixel-based observation setting.\n\nSTRENGTHS:\n- The paper addresses the open problem of offline model-based reinforcement learning, which is very relevant to the RL community nowadays. The paper is well-written, very clear and organized: the methodology is carefully and formally described; the baselines are clearly and concisely described (and justified).\n\n- Despite the incremental nature of the paper (while mixing up different building blocks from previous literature), the work is really clever on identifying that a latent modeling of actions would implicitly constrain the action space to the data distribution, which would naturally provide a regularization effect on learning policies with static datasets. Therefore, the general methodology is very well motivated and applied to the problem.\n\n- The asymptotic results in the V-D4RL are really strong and the ablations in Section 4.2 presents very clearly the effect of the proposed method in the problem of value overestimation. These results give solid evidence that the method is indeed addressing the previously raised problems.\n\nWEAKNESSES:\n- The claim in L200 regarding a “significant speed up in policy learning” is questionable, since comparing the number of gradient steps to achieve a certain performance is not fair. Some methods can leverage less computation in the updates than others. Also, one can carefully tune the hyperparameters (such as the learning rate, batch size) to minimize the number of gradient steps required, at the cost of more computation. Therefore, showing the gradient steps in the x-axis is not ideal and does not seem to provide useful information. It would be interesting to bring the computational cost for claiming speed up. Optionally, the work could remove such a claim.\n\n- There is an unclear trend in the results of the proposed method: in many cases the starting policy already performs close to the best, while other methods start from performance 0. For instance, refer to the expert cases in D4RL. Why does this happen? \n    - The most interesting case is the walker 2d-expert-v2 in Figure 6, where one ablation starts optimal and then goes down. It would be interesting to justify or at least hypothesize what is happening.\n\n- The bound of Eq. 11 is not justified in the paper and looks arbitrary. From my understanding no prior is placed in the policy so this looks like an assumption. It would be interesting to elaborate better about this.\n\n- The last concern is perhaps the most general and crucial to be cleared out during rebuttal. It is clear that the method worked much better for visual domains than feature domains (the results in the D4RL dataset do not show any asymptotic improvement – it is actually worse than MOBILE as per Table 5). In both cases, the action space is the same. It is also low-dimensional. This raises the question: is the model really implementing latent variables for actions or is it actually implementing a hierarchical latent model for the state space? The work explicitly defines “u_t” as latent actions, but it could also be interpreted as a higher-level latent state variable. Given that the gradients from the action decoder backpropagates through both s_t and u_t, I couldn’t find an explanation on why this is not a valid explanation of the method, and this would better justify the discrepancy in the results.\n\n**Minor Concerns/Further Suggestions**\n\n- It would be great to show the value overestimation plots (Figure 6) for the considered baselines, in order to compare the difference among them. It is unclear if they perform worse than the proposed method because of value overestimation or other reason.\n\n- It would also be nice to show a sensitivity analysis of the policy constraint hyperparameter epsilon across a few environments, to help understanding how the proposed method behaves and how much it is environment specific.\n\n- In L105, the paper contrasts the proposed C-LAP to previous works stating that they “rely on uncertainty estimates to generate trajectories within the data distribution”, but I would consider rephrasing that since C-LAP can also be understood as leveraging the uncertainty of the latent action prior to generate trajectories. Perhaps the distinction is that previous methods explicitly use uncertainty estimates, while C-LAP relies on a more implicit constraint.\n\n**Typo:**\n\n- A missing “)” in Eq. 12.\n\nQUESTIONS:\nPlease refer to the weaknesses section. Each Major Concern contains questions to be addressed during the rebuttal.\n\nLIMITATIONS:\n- The main limitation in my perspective is that it is not clear why the method presents much stronger results in the visual benchmark (which is supposed to be harder) than in the feature-based benchmark. This opens questions about whether the introduced latent modeling is indeed doing what is hypothesized in the paper.\n\n- There are some limitations in how the results are presented and discussed in the work (please refer to first two bullet points in the Weaknesses section).\n\n\n**Summary of the Review**:\n\nOverall, the paper is well-written, clearly describes the proposed methodology and offers strong results in the visual benchmarks. The work is based upon a very interesting insight on what latent variables could potentially offer to restrict out-of-distribution actions in offline RL settings. The paper does present some concerns, most related to better explaining the presented results and adjusting claims accordingly. There are also other minor concerns/further suggestions that do not prevent acceptance but would also enrich the paper (which I would appreciate with a higher score). That being said, I believe the paper is already above the acceptance threshold and I am willing to increase my score if my concerns (mostly questions) are properly discussed in the rebuttal phase.\n\n\n\n\n\n\n\n# Post-Rebuttal\n\nThank you, authors, for properly addressing my questions and concerns during the rebuttal. After rebuttal, I am increasing my score.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduce C-LAP (Constrained Latent Action Policies), a novel approach to model-based offline reinforcement learning in POMDPs. Notably, C-LAP does not employ explicit reward penalty terms regarding the action space.\n\nThe paper proposes a methodology for learning latent variables for both states and actions based on the state-space models (SSMs). It then frames policy optimization as a constrained optimization problem, aiming to maximize N-step returns while ensuring that latent actions remain within the support of the latent action prior.\n\nThis approach allows the policy to maximize returns predicted by the model without incorporating reward penalty terms, such as uncertainty penalization. Empirically, C-LAP demonstrates superior performance compared to baseline methods that incorporate uncertainty penalties in their reward structure.\n\nSTRENGTHS:\nThe manuscript presents a novel and effective model-based method for offline reinforcement learning in POMDPs. Prior approaches in the domain of model-based offline RL often incorporate penalty terms to mitigate OOD actions and prevent overestimation of the value function. This penalization, however, can restrict the maximization of value functions related to possibly promising actions.\n\nIn this work, the authors extend a standard state-space model (SSM), exemplified by model-based approaches such as Dreamer, to include latent action variables. By constraining the latent action space to adhere to an action prior, the generation of OOD actions is effectively precluded. \nThis modification is both simple and effective, addressing a critical challenge in the field.\n\nThe effectiveness of the proposed method is validated across both MDP and POMDP settings using standard benchmarks such as D4RL and V-D4RL. Moreover, the authors provide a thorough ablation study to identify the critical components contributing to performance improvements. As described in Figure 6, omitting the constraint on the latent action prior is the most crucial to performance, underscoring its significance as a key contribution of this research.\n\nWEAKNESSES:\nDespite the novelty of this work, the motivation for adopting latent action spaces is insufficiently articulated. \nThe paper does not clearly address why constraining with respect to the latent action space is advantageous over the actual action space. \nSpecifically, it remains unclear whether C-LAP offers any benefits compared to SSMs that include a support constraint term directly on the actual action space, as expressed by the constraint  $\\mathbb{E}_{s \\sim p, a\\sim \\mu(s)}[\\pi (a|s) ] \\ge \\epsilon$\n(where $\\mu$ denotes the behavior policy) for an instance.\nI believe that a more thorough explanation for the motivation could significantly strengthen the contributions and arguments of this work.\n\n(Minor comment) In Eq. (7), a constraint term would be a typo. It would be more accurate to revise this as $\\mathbb{E}_{p,\\pi}[p(h_t|s_t)]\\ge \\epsilon$, given that $h_t$ is sampled from $\\pi(s_t)$.\n\nQUESTIONS:\nQ1. The manuscript suggests potential applicability of the C-LAP framework beyond the traditional models explored. Have the authors considered leveraging foundation models, such as transformers or diffusion models, within the latent space formulation of C-LAP? If so, could you elaborate on how C-LAP might integrate with these advanced model architectures?\n\nQ2. The proposed C-LAP method appears to be promising for environments characterized by high-dimensional action spaces, including those analogous to language action spaces. Could the authors provide insights into the adaptability of C-LAP to such high-dimensional scenarios? What modifications, if any, would be necessary to optimize its performance in these contexts?\n\nLIMITATIONS:\nThe authors included their limitation in Section 4.2. They address societal impacts of their work in the checklist #10. Broader Impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}