Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning

Authors: marvin  alles
philip  becker-ehmck
patrick  van der smagt
maximilian  karl
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:13:07

Abstract: In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.

Main paper: 
 introduction: 
 Deep-learning methods are widely used in applications around computer vision and natural language processing, related to the fact that datasets are abundant. But when used for control of physical systems, in particular with reinforcement learning, obtaining data involves interaction with an environment. Learning through trial-and-error and extensive exploration of an environment can be done in simulation, but hard to achieve in real world scenarios [1, 2, 3] . Offline reinforcement learning tries to solve this by using pre-collected datasets eliminating costly and unsafe training in real-world environments [4, 5, 6] .Using online reinforcement learning methods in an offline setting often fails. A key issue is the distributional shift: the state-action distribution of the offline dataset, driven by the behavior policy, differs from the distribution generated by a learned policy. This leads to actions being inferred for states outside the training distribution. Therefore, value-based methods are prone to overestimating values due to evaluating policies on out-of-distribution states. This leads to poor performance and unstable training because of bootstrapping [6, 7, 8] . Offline reinforcement learning methods address this issue with different approaches and can be categorized into model-free and model-based methods, similar to online reinforcement learning. The policy is used in the real world, again using the generative action decoder. 1Model-free offline reinforcement learning usually follows one of the following paradigms: constrain the learned policy to the behavior policy [9, 10, 11, 7] ; or introduce some kind of conservatism to the Bellman update [12, 13, 14, 8] . Model-based reinforcement learning methods transform the offline to an online learning setting: They approximate the system dynamics and try to resolve the evaluation of out-of-distribution states by using the generalization capabilities of the model and generating additional samples. But as the training distribution is fixed, the estimation capabilities of the model are limited. Therefore, these model-based methods also rely on a conservative modification to the Bellman update as a measure to counteract value overestimation which is mostly achieved through uncertainty penalties [15, 16, 17, 18, 19, 20, 21] . Apart from the typical approach of using an auto-regressive model to estimate the dynamics, other model-based methods treat the objective as trajectory modeling [22, 23, 24] . These methods aim to combine decision making and dynamics modeling into one objective. Instead of learning a policy, they sample from the learned trajectory model for planning. We will refer to the first kind of methods, which learn a dynamics model to train a policy, as model-based reinforcement learning.We aim to solve the problem of value overestimation in model-based reinforcement learning by jointly modeling action and state distributions, without the need for uncertainty penalties or changes to the Bellman update. Instead of learning a conditional dynamics model p(s | a), we estimate the joint state-action distribution p(s, a). This is similar to methods that frame offline reinforcement learning as trajectory modeling, but we use an auto-regressive model and still learn a policy. By formulating the objective as a generative model of the joint distribution of states and actions, we create an implicit constraint on the generated actions, similar to [10, 25] . The goal of this approach is to address the shift in the entire distribution, rather than looking at out-of-distribution actions and states separately. We achieve this using a recurrent state-space model with a latent action space, which we call the recurrent latent action state-space model. Using a latent action space allows us to learn a policy that uses the latent action prior as an inductive bias. This approach keeps the policy close to the original data while allowing it to change when needed, which makes learning the policy much faster. To achieve this, we treat policy optimization as a constrained optimization problem, similar to enforcing a support constraint [9] . We provide a high level overview of our method in Figure 1 .Overall, we summarize our contribution as follows:• We introduce latent action state-space models for model-based offline reinforcement learning, treating it as auto-regressive generative modeling of the joint distribution of states and actions.• We formulate policy optimization as a constrained optimization problem, using the latent action space to generate actions within the support of the dataset's action distribution and jump-start policy learning by using the generative action decoder.• We evaluate our approach on one benchmark with image observations (V-D4RL [20] ) and on another one with low-dimensional feature observations (D4RL [26] ).• We evaluate the effect of our approach on value overestimation. 
preliminaries: 
 We consider a partial observable Markov decision process (POMDP) defined by M = (S, A, O, T, R, Ω, γ) with S as state space, A as action space, O as observation space, s ∈ S as state, a ∈ A as action, o ∈ O as observation, T : S × A → S as transition function, R : S → R as reward function, Ω : S → O as emission function and γ ∈ (0, 1] as discount factor. The goal is to find a policy π : O → A that maximizes the expected discounted sum of rewards E[T t=1 γ t r t ] [27]. In online reinforcement learning, an agent iteratively interacts with the environment M and optimizes its policy π. In offline reinforcement learning, however, the agent cannot interact with the environment and must refine the policy using a fixed dataset D = {(o 1:T , a 1:T , r 1:T ) N n=1 }. Therefore, the agent must understand the environment using limited data to ensure the policy maximizes the expected discounted sum of rewards when deployed [6] . Auto-regressive model-based offline reinforcement learning tries to learn a parametric function to estimate the transition dynamics T . The transition dynamics model is then used to generate additional trajectories which can be used to train a policy. The majority of these approaches learn a dynamics model directly in observation space T θ (o t | o t-1 , a t-1 ) [16, 17, 18, 19, 28] , while others use a latent dynamics model T θ (s t | s t-1 , a t-1 ) [20, 21] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We include the limitations in a separate paragraph in the conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
constrained latent action policies: 
 A main issue in offline reinforcement learning is value overestimation, which we address by ensuring the actions generated by the policy stay within the dataset's action distribution. Unlike previous modelbased methods, we formulate the learning objective as a generative model of the joint distribution of states and actions. We do this by combining a latent action space with a latent dynamics model. Next, we use the generative properties of the action space to constrain the policy to the dataset's action distribution. A general outline of our method, Constrained Latent Action Policies (C-LAP ), is shown in Appendix B. It starts with learning a generative model, followed by actor-critic agent training on imagined trajectories, similar to the methods in [29, 30, 20, 21] . o 1 o 2 o 3 s 1 s 2 s 3 h 1 h 2 h 3 a 1 a 2 a 3 u 1 u 2 u 3that jointly models the observation and action distribution of a static dataset D = {(o 1:T , a 1:T , r 1:T ) N n=1 } by using latent states s t along with latent actions u t . Unlike other modelbased offline reinforcement learning methods that learn a conditional model p(o 1:T | a 1:T ) and rely on ensemble based uncertainty penalties on the Bellman update to generate trajectories within the data distribution [21, 18, 16, 17] , our approach uses a latent action space to impose an additional implicit constraint. By implementing a policy in the latent action space, generated actions will stay within the dataset's action distribution, thus enabling generalization within the limits of the learned model [10] . We empirically validate this claim in Appendix F. To obtain a state space model with Markovian assumptions on the latent states s t we impose the following structure:EQUATIONEQUATIONWe implement the probabilistic model modifying the design of a recurrent state-space model [31] . The latent state prior predicts the next latent state s t given the previous latent state s t-1 and action u t-1 using the deterministic transition and the action decoder. The latent action prior predicts latent actions u t given latent state s t . Latent states as well and as latent actions are decoded using their respective decoder. Similar to [25, 10] actions are reconstructed given latent state and latent action.Directly maximizing the marginal likelihood is intractable, hence we maximize the evidence lower bound (ELBO) on the log-likelihood log p(o 1:T , a EQUATIONwhich can be organized into individual terms for reconstruction and consistency of actions and observations. The derivation can be found in Appendix A. We use the sequence model to generate imagined trajectories and use an actor-critic approach to train the policy. To predict state values we learn a value model v ξ (s t ) alongside the policy. Therefore we use the n-step return of a stateEQUATIONas regression target for v ξ (s t ) [27, 29] . Polices trained on trajectories generated by a model are prone to end up with degrading performance if the model only has access to a limited data distribution, as in the case of offline reinforcement learning. Compounding modeling errors and value overestimation of edge-of-reach states [19] are reasons for the decline. Since we train a generative action model, generated actions are implicitly constrained to the datasets action distribution by sampling from the action decoder a t ∼ p θ (a t | s t , u t ). Hence, states outside the datasets observation distribution are hard to reach and our approach is resilient to value overestimation of edge-of-reach states.Compounding modeling errors are still a source of diminishing performance, but can be counteracted by increasing the representation power of the model or generating only short trajectories. To leverage the generative action model, we learn a policy π ψ (u t | s t ) in the latent action space similar to [10, 25] . But, as both, the latent action prior p θ (u t | s t ) and the policy π ψ (u t | s t ) are flexible, it is not ensured that they share the same support. Thus, we formulate policy optimization as a constrained optimization problemEQUATIONsimilar to a support constraint [9] . We implement the constraint explicitly through parametrization to stay within the support, but do not impose any restrictions inside the supported limits (Figure 3 ). This is different to using a divergence measure which on one hand does not strictly ensure support limits and on the other hand is more restrictive as it also imposes a constraint on the shape of a distribution.Here and in the following ût stands for a latent action sampled from the policy π ψ (u t | s t ).The policy is trained to maximize the n-step return V k N (s τ ) while staying in support of the latent action prior. Since the latent action prior p θ (u t | s t ) is normally distributed as N (µ θ (s t ), σ θ (s t )), we can express the constraint asEQUATIONwith ε as a parameter setting support as multiples of σ θ centered around µ θ . From the properties of a normal distributed variable follows thatEQUATIONWe implement the constraint explicitly by parameterizing the policy as a linear function g dependent on π ψ (u t | s t ) and p θ (u t | s t ):EQUATIONThe support of the policy distribution is chosen to be boundedEQUATIONand g(π ψ (u t | s t ), p θ (u t | s t )) as a linear combination of the latent action predicted by the policy ût and the distribution parameters σ θ and µ θ of the latent action prior:EQUATION)4 ExperimentsIn the next section, we evaluate the effectiveness of our approach. It is divided into three parts: first, we assess the performance using standard benchmarks; then, we study how different design choices affect value overestimation; lastly, we analyze the influence of the support constraint parameter. We additionally provide the final performances in two tables in Appendix E.We limit our benchmark evaluation to the most relevant state-of-the-art offline reinforcement learning methods to answer the following questions: 1) How do latent action state-space models compare to state-space models? 2) How comparable are model-free methods focusing on latent action spaces to latent action state-space models? 3) Does C-LAP suffer from value overestimation? 4) How does the support constraint affect the performance? 5) How does the performance differ between visual observations and observations with low-dimensional features? To focus on the latter, we separately evaluate the performance on low-dimensional feature observations using the D4RL benchmark [26] , and on image observations using the V-D4RL benchmark [20] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We outline the derivation of the ELBO in the main section (Sec. 3) and provide a detailed derivation in the appendix A. We explain the derivation of the constrained latent action policy step-by-step in the main section (Sec. 3).Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
benchmark results: 
 D4RL Since most offline model-based reinforcement learning methods are designed for observations with low-dimensional feature observations, there exist many options for comparison. We make a selection to include the most relevant methods focusing on latent actions and auto-regressive model-based reinforcement learning. Therefore, we include the following methods: PLAS, which is a model-free method using a latent action space [10] . MOPO, a probabilistic ensemble-based offline model-based reinforcement learning method using a modification to the Bellman update to penalize high variance in next state predictions [16] . And MOBILE, which is similar to MOPO, but penalizes high variance in value estimates instead [18] . We compare the algorithms on three different locomotion environments, namely halfcheetah, walker2d and hopper, with four datasets (medium-replay, medium, medium-expert, expert) each and the antmaze navigation environment with four datasets (umaze, umaze-diverse, medium-play, medium-diverse). The results, shown in Figure 4 , display the mean and standard deviation of normalized returns over four seeds during the phase of policy training, with steps denoting gradient steps. The dashed lines indicate the asymptotic performance for MOPO and MOBILE. A detailed summary of all implementation details is provided in the Appendix D.When comparing C-LAP to PLAS, we find that learning a joint generative model of actions and observations outperforms a generative model of only actions when used with actor-critic reinforcement learning. Both methods can use the generative nature of the model to speed up policy learning, which becomes especially clear in the results on all locomotion expert and medium-expert datasets.Compared to MOPO and MOBILE, C-LAP shows a superior or comparable performance on all datasets except halfcheetah-medium-replay-v2, halfcheetah-medium-v2 and hopper-medium-v2. Especially outperforming on the antmaze environment, where MOPO and MOBILE fail to solve the task for any of the considered datasets. The asymptotic performance of MOBILE on locomotion environments sometimes exceeds the results of C-LAP, but needs three times as many gradient steps.Overall the results indicate that latent action state-space models with constrained latent action polices not only match the state-of-the-art on observations with low-dimensional features as observations, but also jump-start policy learning by using the action decoder to sample actions that lead to high rewards already after the first gradient steps: If the dataset is narrow, generated actions when sampling from the latent action prior will fall into the same narrow distribution. For instance, in an expert dataset, Figure 4 : Evaluation on low-dimensional feature observations using D4RL benchmark datasets. We plot mean and standard deviation of normalized returns over 4 seeds.sampled actions will also be expert-level actions. During policy training, instead of sampling from this prior, we restrict the support of the policy dependent on the latent action prior. Thus, sampled latent actions from the policy will always be decoded to fall into the dataset's action distribution. So even a randomly initialized policy in the beginning of the training can generate a high reward by using the latent action decoder. This effect is especially prominent in narrow datasets such as expert datasets.V-D4RL There are currently few auto-regressive model-based reinforcement learning methods that specifically target visual observations, with none emphasizing latent actions. In our evaluation, we include LOMPO [21] and Offline DV2 [20] . Both methods use a latent state space model and an uncertainty penalized reward. However the specifics of the penalty calculations are different: while LOMPO uses standard deviation of log probabilities as penalty, Offline DV2 uses mean disagreement. Additionaly, LOMPO trains an agent on a mix of real and imagined trajectories with an off-policy actor-critic approach, whereas Offline DV2 exclusively trains on imagined trajectories and backpropagates gradients through the dynamics model. Further implementation details are included in Appendix D.C-LAP demonstrates superior performance across all datasets, especially significant on cheetahrun-medium_expert, walker-walk-medium_expert and walker-walk_expert. Datasets with a large diversity of actions, such as medium-replay datasets, exhibit a weaker inductive bias for a generative action model. Hence, they require more additional policy steps, as can be seen for both the D4RL and V-D4RL benchmarks.Figure 5 : Evaluation on visual observations using V-D4RL benchmark datasets. We plot mean and standard deviation of normalized returns over 4 seeds.Limiting value overestimation plays a central role in offline reinforcement learning. To evaluate the effectiveness of C-LAP, we report value estimates alongside normalized returns on all walker2d datasets in Figure 6 . A similar analysis for all considered baselines is provided in Appendix G.To further analyze the influence of different action space design choices, we include the following ablations: a variant no constraint, which does not formulate policy optimization as constrained objective, but uses a Gaussian policy distribution to potentially cover the whole Gaussian latent action space; and a variant no latent action, which does not emphasize latent actions, but uses a regular state-space model as in Dreamer [29] . Besides that, we added dashed lines to indicate the dataset's average return and average maximum value estimate. The no latent action variant fails to learn Figure 6 : Ablation study, comparing C-LAP to the following variants: no constraint, C-LAP without enforcing the policy constraint dependent on the action prior; no latent action, C-LAP without a latent action space similar to Dreamer [29] . We plot mean and standard deviation of normalized returns and value estimates over 3 seeds. Moreover we add the dataset's average return and average maximum value estimate indicated by dashed lines.an effective policy: normalized returns are almost zero and the dataset's reference returns remain unattained; value estimates are significantly exceeding the dataset's reference values, indicating value overestimation. The no constraint variant can use the generative action decoder to limit generated actions to the dataset's action distribution, but the Gaussian policy is free to move to regions which are unlikely under the action prior. Thus, nullifying the implicit constraint imposed by the action decoder, resulting in collapsing returns and value overestimation. Only C-LAP achieves a high return and generates value estimates which are close to the dataset's reference. The value estimates on walker2d-medium-replay-v2 are higher than the dataset's reference, as the agent's performance is also exceeding the reference performance. The results confirm the importance of limiting value overestimation in offline reinforcement learning, and demonstrate that constraining latent action policies can be an effective measure for achieving this.To evaluate the influence of the support constraint parameter ε on the performance of C-LAP, we perform a sensitivity analysis across all walker2d datasets (Figure 7 ). Except for the more diverse medium-replay-v2 dataset, adjusting ε from 0.5 to 3.0 only has a minor impact on the achieved return. However, when choosing an unreasonable large value such as ε = 10.0 or removing the constraint altogether (Figure 6 ), we observe a collapse during training. This highlights a key insight: constraining the policy to the support of the latent action prior is essential. And in many cases, using a smaller support region closer to the mean (small ε) proves sufficient.Figure 7 : Sensitivity analysis of the support constraint parameter ε for the considered D4RL walker2d datasets. We plot mean and standard deviation of normalized returns over 4 seeds.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We include the explanations of your approach in the main section (Sec. 3). We report the used hyper-parameters and an outline of the algorithm in the appendix (Appendix B, D). We provide the code. We use publicly available benchmarks. 
related work: 
 Offline reinforcement learning methods fall into two groups: model-free and model-based. Both types aim to tackle problems like distribution shift and value overestimation. This happens because the methods use a fixed dataset rather than learning by interacting with the environment.Model-free Current model-free methods typically work by limiting the learned policy or by regularizing value estimates. TD3+BC [11] adds a behavior cloning regularization term to the policy update objective to enforce actions generated by the policy to be close to the dataset's action distribution. Similarly, SPOT [9] includes a regularization term in the policy update, derived from a support constraint perspective. It also uses a conditional variational auto-encoder (CVAE) to estimate the behavior distribution. Following a comparable intention, BEAR [8] constraints the policy to the support of the behavior policy via maximum mean discrepancy. BCQ [7] and PLAS [10] use a CVAE similarly but don't use a regularization term. Instead, they constrain the policy implicitly by making the generative model part of the policy. Beside these methods, many other approaches exist, with CQL [12] and IQL [14] being some of the most well-known. CQL uses a conservative policy update by setting a lower bound on value estimates to prevent overestimation, while IQL avoids out-of-distribution values by using a modified SARSA-like objective in combination with expectile regression to only use state-action tuples contained in the dataset.Model-based Model-based offline reinforcement learning methods learn a dynamics model to generate samples for policy training. This basically converts offline learning to an online learning problem. Model-based methods mainly address model errors and value overestimation by using a probabilistic ensemble and adding an uncertainty penalty in the Bellman update. MOPO [16] uses a probabilistic ensemble as in [32] and adds the maximum standard deviation of all ensemble predictions as uncertainty penalty. Similar to that, MOReL [17] adheres to the same methodology, but uses pairwise maximum difference of the ensemble predictions as penalty instead. Analogously, MOBILE [18] estimates the values for all by the ensemble predicted states and uses the standard deviation of value estimates as penalty. Edge of reach [19] comes to the conclusion that value estimation on edge of reach states are the overarching issue compared to model errors. In the end, they come up with a comparable solution to MOBILE, but use an ensemble of value networks alongside the ensemble of dynamic models. COMBO [28] pursues a different approach, as they integrate the conservatism of CQL into value function updates, removing the need for uncertainty penalties. Besides that, some methods use a different class of models: instead of learning a predictive model in observation space, they use latent state-space models to make predictions on latent states. Among these methods is LOMPO [21] , which builds up on Dreamer [29] , but integrates an ensemble to predict stochastic states and use the standard deviation of the log probability of the ensemble predictions as an uncertainty penalty similar to previous methods. The policy is trained on a mix of imagined and real world samples, hence they use an off-policy actor-critic style approach for policy learning. Offline DV2 [20] uses a similar model, but is based on a different penalty. Namely, they use the difference between the individual ensemble mean predictions and mean over all ensembles as uncertainty penalty. Furthermore, the policy is trained only on imagined trajectories with gradients calculated by back-propagating through the dynamics model. Overall, Offline DV2 is the method most comparable to our approach, but still different in many ways as we propose a latent action state-space model compared to a usual state-space model, and frame policy learning as constrained optimization. So far all discussed models operate in an auto-regressive fashion, but another class of methods exists, which casts offline model-based reinforcement learning as trajectory modeling.Instead of learning a policy, these kind of approaches integrate decision making and modeling of the underlying dynamics into a single objective and use the model for planning. Among them are Diffuser [24] , which employs guided diffusion for planning; TT [22] , which builds on advances in transformers; and TAP [23] , which uses a VQ-VAE with a transformer-based architecture to create a discrete latent action space for planning. 
conclusion: 
 We present C-LAP, a model-based offline reinforcement learning method. To tackle the issue of value overestimation, we first propose an auto-regressive latent-action state space model to learn a generative model of the joint distribution of observations and actions. Second, we propose a method for policy training to stay within the dataset's action distribution. We explicitly parameterize the policy depending on the latent action prior and formulate policy learning as constrained objective similar to a support constraint. We find that C-LAP significantly speeds-up policy learning, is competitive on the D4RL benchmark and especially outperforms on the V-D4RL benchmark, raising the best average score across all dataset's from previously 31.5 to 58.8.Limitations Depending on the dataset and environment the effectiveness of C-LAP differs: Datasets which only contain random actions are challenging for learning a generative action model, thus we do not include them in our evaluation. The effect of jump-starting policy learning with the latent action decoder to already achieve high rewards in the beginning of policy training is prominent in narrow datasets, but less effective for diverse datasets. While training the model of C-LAP does not require additional gradient steps, it still takes more time compared to LOMPO [21] and Offline DV2 [20] as the latent action state-space model is more complex than a usual latent state-space model. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We proposed method falls in the category of foundational research in the field of offline reinforcement learning and the method is not tied to an application. Of course, applications exist where reinforcement learning might have a negative societal impact, but our method only has a limited influence on that.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: We start with Equation (1) to jointly model the dataset's observation and action distribution log p(o 1:T , a 1:T ) = log p(o 1:T , a 1:T | s 1:T , u 1:T )p(s 1:T , u 1:T ) ds du,and the defintions in Equation ( 3), namely:EQUATIONp(s 1:T , uEQUATIONWe introduce the inference modelEQUATIONand by replacing p(o 1:T , a 1:T | s 1:T , u 1:T ) and p(s 1:T , u 1:T ) in Equation ( 13) we get:EQUATIONWe include the inference model and resolve using Jensen's Inequality:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONWe provide the general algorithm of C-LAP. a t ∼ p(a t | s t , g(π ψ (u t | s t ), p θ (u t | s t )) Predict rewards r τ ∼ p θ (r τ | s τ ), termination t τ ∼ p θ (t τ | s τ ) and values v ξ ∼ v ψ (u τ | s τ ) Compute value estimates V k N (s τ ) via Equation (6) Update policy parameters ψ ← ψ + α ψ ∇ ψ t+H τ =t V k N (s τ ) Update critic parameters ξ ← ξ + α ξ ∇ θ t+H τ =t 1 2 ∥v ξ (s τ ) -V k N (s τ )∥ end for end forWe use a different hardware setup for experiments with visual observations and experiments with low-dimensional features as observations. We run all experiments on a shared local cluster. C-LAP experiments with visual observations take around 10 hours on a RTX8000 GPU and experiments with low-dimension feature observations around 11 hours on a A100 GPU. We aim to execute most of our code on GPU and parallelize our implementation. Environment evaluations represent a bottleneck as they require execution on CPU. Overall, it takes around 70 combined GPU days to reproduce the benchmark results of all methods. This does not include the compute required for the evaluation of preliminary implementations or hyper-parameter settings.We implement all methods in JAX [33] using Equinox [34] . We provide the hyper-parameters of C-LAP in Table 2 and the constraint values used for the D4RL benchmark in Table 3 and for the V-D4RL benchmark in Table 4 . In general we consider constraint values in the range ε ∈ {0.5, 1.0, 2.0, 3.0}.We implement MOPO and MOBILE following [18] and use the respective hyper-parameters provided in the publication. For MOPO, we select the max-aleatoric version. As no hyper-parameters are provided for the expert datasets of the D4RL benchmark we sweep through the range specified in Table 1 and use the one selected in the table. For the antmaze environment we additionally evaluate Table 1 : MOPO and MOBILE hyper-parameters for the expert datasets in the D4RL benchmark. We provide the benchmark results in the usual offline reinforcement learning table format. As our implementations might differ from the original one, we include the original scores from the paper alongside our results in tables. We also include the average over all datasets except the expert datasets to make it comparable to the average scores provided in the respective publication.To evaluate the claim that latent actions generated by the latent action prior are close to the dataset's action distribution we use the following approach: We randomly select one trajectory from the hopper-expert-v2 dataset and employ k-nearest neighbors to identify the 20 nearest observations and their corresponding actions within the whole dataset for each step. We then sample from the action prior and decoder to generate actions based on the nearest observations at each step. Thereafter, we split the selected trajectory into sections from leaving the ground to touching the ground and Table 6 : Results on the V-D4RL benchmark. Showing normalized returns and standard deviations at the end of policy training.fit normal distributions to the aggregated dataset actions and reconstructed prior actions. Thus, we end up with an approximation of the dataset's action distribution and an approximation of the action distribution generated by the prior for each step in the trajectory. Figure 8 shows the aggregated distributions (from leaving the ground to touching the ground) for one exemplary trajectory of the hopper-expert-v2 dataset.We further report value estimates alongside normalized returns on all walker2d datasets for the considered baselines (Figure 9 ). As they estimate Q-values, we calculate the corresponding value estimates by averaging over 10 actions sampled from their respective policies. MOPO and MOBILE have a low value estimate, which can be attributed to the incorporated uncertainty penalties. PLAS's value estimates are only stable for walker2d-expert-v2 datasets, but collapse for the other considered datasets. Overall, it seems that value overestimation is not the cause for degrading performance for these methods. Justification: We claim that C-LAP is competitive to state-of-the-art methods and especially outperforming on datasets with visual observations. These claims are based on evaluations on the D4RL and V-D4RL benchmark and the comparison to other relevant methods. We claim that we aim to solve value overestimation with our approach, which we empirically show on the walker2d datasets.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: We give access to the complete method details and thoroughly share details of the experiments.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no high risk of misuse of our method.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers which published the benchmarks which we use. We cite the authors of JAX and Equinox, as our implementation is based on their framework. We give credit to Twemoji in figure 1 .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Emoji graphics used in Figure1are licensed under CC-BY 4.0 by Twemoji. 


Decision: This paper introduces Constrained Latent Action Policies (C-LAP), an approach for model-based offline RL designed to handle value overestimation. C-LAP employs a latent action state-space model to generate actions that are constrained within the support of a learned action distribution, mitigating the risk of generating out-of-distribution actions. By formulating the policy learning as a constrained optimization problem, C-LAP eliminates the need for additional uncertainty penalties, thereby speeding up the learning. The approach is tested on benchmarks with both low-dimensional and visual observations, demonstrating competitive performance, particularly in visual tasks. In summary, C-LAP demonstrates a robust solution to the challenges of offline reinforcement learning by maintaining close distances to the original data distribution while enabling efficient policy training.

Reviews: review: paper_summary: paper_summary: The work introduce C-LAP (Constrained Latent Action Policies), a novel approach to model-based offline reinforcement learning in POMDPs. Notably, C-LAP does not employ explicit reward penalty terms regarding the action space.

The paper proposes a methodology for learning latent variables for both states and actions based on the state-space models (SSMs). It then frames policy optimization as a constrained optimization problem, aiming to maximize N-step returns while ensuring that latent actions remain within the support of the latent action prior.

This approach allows the policy to maximize returns predicted by the model without incorporating reward penalty terms, such as uncertainty penalization. Empirically, C-LAP demonstrates superior performance compared to baseline methods that incorporate uncertainty penalties in their reward structure.
strength_weakness: strength_weakness: Despite the novelty of this work, the motivation for adopting latent action spaces is insufficiently articulated. 
The paper does not clearly address why constraining with respect to the latent action space is advantageous over the actual action space. 
Specifically, it remains unclear whether C-LAP offers any benefits compared to SSMs that include a support constraint term directly on the actual action space, as expressed by the constraint  $\mathbb{E}_{s \sim p, a\sim \mu(s)}[\pi (a|s) ] \ge \epsilon$
(where $\mu$ denotes the behavior policy) for an instance.
I believe that a more thorough explanation for the motivation could significantly strengthen the contributions and arguments of this work.

(Minor comment) In Eq. (7), a constraint term would be a typo. It would be more accurate to revise this as $\mathbb{E}_{p,\pi}[p(h_t|s_t)]\ge \epsilon$, given that $h_t$ is sampled from $\pi(s_t)$.
questions: questions: Q1. The manuscript suggests potential applicability of the C-LAP framework beyond the traditional models explored. Have the authors considered leveraging foundation models, such as transformers or diffusion models, within the latent space formulation of C-LAP? If so, could you elaborate on how C-LAP might integrate with these advanced model architectures?

Q2. The proposed C-LAP method appears to be promising for environments characterized by high-dimensional action spaces, including those analogous to language action spaces. Could the authors provide insights into the adaptability of C-LAP to such high-dimensional scenarios? What modifications, if any, would be necessary to optimize its performance in these contexts?
limitations: limitations: The authors included their limitation in Section 4.2. They address societal impacts of their work in the checklist #10. Broader Impacts.
paper_summary: paper_summary: The paper approaches the problem of model-based policy learning from static datasets. It firstly identifies that this offline MBRL setup inherits two major concerns from its components: the problem of value overestimation from out-of-distribution actions (common in offline RL) and the bias originated by learning policies from model predictions. In order to address these issues, it proposes C-LAP, a MBRL method that besides learning a latent state space model - as common in the literature - also learns a latent action space from the static dataset. This generative model of states AND actions is further leveraged by the policy optimization objective to constrain the policy actions to share support with the latent action prior, in order to prevent out-of-distribution samples. The paper provides experiments in two offline RL benchmarks, with feature and pixel-based observations, and showing good improvements in the pixel-based observation setting.
strength_weakness: strength_weakness: - The claim in L200 regarding a “significant speed up in policy learning” is questionable, since comparing the number of gradient steps to achieve a certain performance is not fair. Some methods can leverage less computation in the updates than others. Also, one can carefully tune the hyperparameters (such as the learning rate, batch size) to minimize the number of gradient steps required, at the cost of more computation. Therefore, showing the gradient steps in the x-axis is not ideal and does not seem to provide useful information. It would be interesting to bring the computational cost for claiming speed up. Optionally, the work could remove such a claim.

- There is an unclear trend in the results of the proposed method: in many cases the starting policy already performs close to the best, while other methods start from performance 0. For instance, refer to the expert cases in D4RL. Why does this happen? 
    - The most interesting case is the walker 2d-expert-v2 in Figure 6, where one ablation starts optimal and then goes down. It would be interesting to justify or at least hypothesize what is happening.

- The bound of Eq. 11 is not justified in the paper and looks arbitrary. From my understanding no prior is placed in the policy so this looks like an assumption. It would be interesting to elaborate better about this.

- The last concern is perhaps the most general and crucial to be cleared out during rebuttal. It is clear that the method worked much better for visual domains than feature domains (the results in the D4RL dataset do not show any asymptotic improvement – it is actually worse than MOBILE as per Table 5). In both cases, the action space is the same. It is also low-dimensional. This raises the question: is the model really implementing latent variables for actions or is it actually implementing a hierarchical latent model for the state space? The work explicitly defines “u_t” as latent actions, but it could also be interpreted as a higher-level latent state variable. Given that the gradients from the action decoder backpropagates through both s_t and u_t, I couldn’t find an explanation on why this is not a valid explanation of the method, and this would better justify the discrepancy in the results.

**Minor Concerns/Further Suggestions**

- It would be great to show the value overestimation plots (Figure 6) for the considered baselines, in order to compare the difference among them. It is unclear if they perform worse than the proposed method because of value overestimation or other reason.

- It would also be nice to show a sensitivity analysis of the policy constraint hyperparameter epsilon across a few environments, to help understanding how the proposed method behaves and how much it is environment specific.

- In L105, the paper contrasts the proposed C-LAP to previous works stating that they “rely on uncertainty estimates to generate trajectories within the data distribution”, but I would consider rephrasing that since C-LAP can also be understood as leveraging the uncertainty of the latent action prior to generate trajectories. Perhaps the distinction is that previous methods explicitly use uncertainty estimates, while C-LAP relies on a more implicit constraint.

**Typo:**

- A missing “)” in Eq. 12.
questions: questions: Please refer to the weaknesses section. Each Major Concern contains questions to be addressed during the rebuttal.
limitations: limitations: - The main limitation in my perspective is that it is not clear why the method presents much stronger results in the visual benchmark (which is supposed to be harder) than in the feature-based benchmark. This opens questions about whether the introduced latent modeling is indeed doing what is hypothesized in the paper.

- There are some limitations in how the results are presented and discussed in the work (please refer to first two bullet points in the Weaknesses section).


**Summary of the Review**:

Overall, the paper is well-written, clearly describes the proposed methodology and offers strong results in the visual benchmarks. The work is based upon a very interesting insight on what latent variables could potentially offer to restrict out-of-distribution actions in offline RL settings. The paper does present some concerns, most related to better explaining the presented results and adjusting claims accordingly. There are also other minor concerns/further suggestions that do not prevent acceptance but would also enrich the paper (which I would appreciate with a higher score). That being said, I believe the paper is already above the acceptance threshold and I am willing to increase my score if my concerns (mostly questions) are properly discussed in the rebuttal phase.







# Post-Rebuttal

Thank you, authors, for properly addressing my questions and concerns during the rebuttal. After rebuttal, I am increasing my score.
paper_summary: paper_summary: This paper proposes C-LAP, a model-based offline RL method that mitigates the distributional shift problem without using uncertainty penalties or modifying the Bellman update. It learns a joint distribution of latent states and latent actions and constrains the latent action policy to the dataset distribution by using a linear transformation. Experiments on D4RL datasets and V-D4RL datasets report performance comparable to other strong offline RL baselines.
strength_weakness: strength_weakness: 1. Figure 1 requires a clearer presentation of the model diagram. It would be better to simultaneously show the prior and posterior in Figure 1(a), and the latent action prior $p_\theta(u_t|s_t)$ and the policy $\pi_\psi(u_t|s_t)$ in Figure 1(b) to improve understanding. During the policy training phase, are the imagined trajectories rolled out from the latent state prior $p_\theta(s_t|s_{t−1}, u_{t−1})$ or the latent state posterior $q_\phi(s_t|s_{t−1}, a_{t−1}, o_t)$? If it is the latent state prior, why is the output of the action decoder used? If it is the latent state posterior, $o_t$ is unavailable.
2. Could the authors provide some distribution results to support the claim that the latent actions generated by the action prior are close to the dataset's action distribution? Does the predictive power of using latent actions degrade in performance compared to models using real actions? Some quantitative or qualitative results can be provided.
3. It would be better to highlight the differences between the proposed method and Dreamer in terms of model designs and algorithms.
4. The experiments mainly focus on Mujoco tasks. How does the proposed method perform on navigation tasks such as AntMaze?
5. Why do some experiments in Figure 4, 5 show high returns at the beginning of policy learning (zero gradient step)?
6. In table 5 and table 6, the performance of C-LAP is inferior to some baseline models in medium or medium-relay datasets. This results seem to indicate that the effectiveness of the proposed method depends on the quality of the offline dataset.
questions: questions: Please see the weaknesses section.
limitations: limitations: This paper discusses the limitations, but that is not enough.

Comments: title: Please respond to author response if you haven't
comment: Dear authors,

Thank you for the detailed explanations and additional empirical evidence. My concerns were addressed, and I am increasing my score accordingly.
Thank you to the authors for their response. Your reply adequately addressed my concerns, and  I will maintain my rating.
Dear Reviewers,

Before the author-reviewer discussion period ends, please make sure that you have read the author responses, acknowledge that your review reflects them, and engage with the authors if necessary.

Thank you!
Thank you for your detailed responses to my concerns. I will maintain my rating and encourage the authors to incorporate the discussed details on the motivations into the revision.