Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

Authors: chen-hao  chao
chien  feng
wei-fang  sun
cheng-kuang  lee
simon  see
chun-yi  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 11:02:07

Abstract: Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.

Main paper: 
 introduction: 
 Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] has emerged as a prominent method for modeling stochastic policies. Different from standard RL, MaxEnt RL integrates the entropy of policies as rewards, which leads to a balanced exploration-exploitation trade-off during training. This approach has demonstrated improved robustness both theoretically and empirically [17] [18] [19] . Building on this foundation, many studies leveraging MaxEnt RL have shown superior performance on continuous-control benchmark environments [8, 9] and real-world applications [20] [21] [22] .An active research domain in MaxEnt RL concentrates on the learning of the soft Q-function [8] [9] [10] [11] [12] [13] [14] [15] . These methods follow the paradigm introduced in soft Q-learning (SQL) [8] . They parameterize the soft Q-function as an energy-based model [23] and optimize it based on the soft Bellman error [8] calculated from rewards and the soft value function. However, this approach presents two challenges. First, sampling from an energy-based model requires a costly Monte Carlo Markov Chain (MCMC) [24, 25] or variational inference [26] process, which can result in inefficient interactions with environments. Second, the calculation of the soft value function can involve computationally infeasible integration, which requires an effective approximation method. To tackle these issues, various methods [8] [9] [10] [11] [12] [13] [14] [15] were proposed, all grounded in a common design philosophy. To address the first challenge, these methods suggest operating on an actor-critic framework and optimizing it through alternating steps of policy evaluation and policy improvement. For the second challenge, they resort to Monte Carlo methods to approximate the soft value function using sets of random samples. Although these two issues can be circumvented, these methods still have their drawbacks. The actor-critic design introduces an additional optimization process for training the actor, which may lead to optimization errors in practice [27] . Moreover, the results of Monte Carlo approximation may be susceptible to estimation errors and variances when there are an insufficient number of samples [28] [29] [30] .Instead of using energy-based models to represent MaxEnt RL frameworks, this paper investigates an alternative method employing normalizing flows (i.e., flow-based models), which offer solutions to the aforementioned challenges. Our framework is inspired by the recently introduced Energy-Based Normalizing Flows (EBFlow) [31] . This design facilitates the derivation of an energy function from a flow-based model while supporting efficient sampling, which enables a unified representation of both the soft Q-function and its corresponding action sampling process. This feature allows the integration of the policy evaluation and policy improvement steps into a single objective training process. In addition, the probability density functions (pdf) of flow-based models can be calculated efficiently without approximation. This characteristic permits the derivation of an exact representation for the soft value function. Our experimental results demonstrate that the proposed framework exhibits superior performance on the commonly adopted MuJoCo benchmark [32, 33] . Furthermore, the evaluation results on the Omniverse Isaac Gym environments [34] indicate that our framework excels in performing challenging robotic tasks that simulate real-world scenarios. 
background and related works: 
 In this section, we walk through the background material and the related works. We introduce the objective of MaxEnt RL in Section 2.1, describe existing actor-critic frameworks and soft value estimation methods in Section 2.2, and elaborate on the formulation of Energy-Based Normalizing Flow (EBFlow) in Section 2.3.In this paper, we consider a Markov Decision Process (MDP) defined as a tuple (S, A, p T , R, γ, p 0 ), where S is a continuous state space, A is a continuous action space, p T : S × S × A → R ≥0 is the pdf of a next state s t+1 given a current state s t and a current action a t at timestep t, R : S × A → R is the reward function, 0 < γ < 1 is the discount factor, and p 0 is the pdf of the initial state s 0 . We adopt r t to denote R(s t , a t ), and use ρ π (s t , a t ) to represent the state-action marginals of the trajectory distribution induced by a policy π(a t |s t ) [8] .Standard RL defines the objective as π * = argmax π t E (st,at)∼ρπ [r t ] and has at least one deterministic optimal policy [35, 36] . In contrast, MaxEnt RL [4] augments the standard RL objective with the entropy of a policy at each visited state s t . The objective of MaxEnt RL is written as follows:EQUATIONwhere H(π( • |s t )) ≜ E a∼π(•|st) [-log π(a|s t )] and α ∈ R >0 is a temperature parameter for determining the relative importance of the entropy term against the reward. An extension of Eq. ( 1) defined with γ is discussed in [8] . To obtain π * MaxEnt described in Eq. ( 1), the authors in [8] proposed to minimize the soft Bellman error for all states and actions. The solution can be expressed using the optimal soft Q-function Q * soft : S × A → R and soft value function V * soft : S → R as follows:EQUATIONEQUATIONIn practice, a policy can be modeled as π θ (at |s t ) = exp( 1 α (Q θ (s t , a t ) -V θ (s t ))with parameter θ, where the soft Q-function and the soft value function are expressed as Q θ (s t , a t ) and V θ (s t ) =The optimization processes defined by the objective functions L(θ) and L(ϕ) in Eqs. ( 4) and ( 5) are known as the policy evaluation steps and policy improvement steps [9] , respectively. Through alternating updates according to ∇ θ L(θ) and ∇ ϕ L(ϕ), the critic learns directly from the reward signals to estimate the soft Q-function, while the actor learns to draw samples based on the distribution defined by the critic.Although the introduction of the actor enhances sampling efficiency, calculating the soft value function in Eq. (3) still requires Monte Carlo approximations for the computationally infeasible integration operation. Prior soft value estimation methods can be categorized into two groups: soft value estimation in Soft Q-Learning (SQL) and that in Soft Actor-Critic (SAC), with the former yielding a larger estimate than the latter, derived from Jensen's inequality (i.e., Proposition A.1 in Appendix A.1). These two soft value estimation methods are discussed in the following paragraphs.Soft Value Estimation in SQL. Soft Q-Learning [8] leverages importance sampling to convert the integration in Eq. (3) into an expectation, which can be estimated using a set of independent and identically distributed (i.i.d.) samples. To ensure the estimation variance is small, the authors in [8] proposed to utilize samples drawn from π ϕ . Let {a (i) } M i=1 be a set of M samples drawn from π ϕ . The soft value function is approximated based on the following formula:EQUATIONEq. ( 6) has the least variance when π ϕ (• |s t ) ∝ exp(Q θ (s t , •)/α) [29] . In addition, as M → ∞, the law of large numbers ensures that this estimation converge to V θ (s t ) [41] .Soft Value Estimation in SAC. Soft Actor-Critic [9] and its variants [10, 11, 13, 14, 12] reformulated the soft value function V θ (s t ) = α log exp (Q θ (s t , a)/α) da as its equivalent formE a∼π θ [Q θ (s t , a) -α log π θ (a|s t )] based on the relationship that π θ (a|s t ) = exp( 1 α (Q θ (s t , a)) - V θ (s t )). By assuming that the policy improvement loss L(ϕ) is small (i.e., π θ ≈ π ϕ ), the soft value function V θ can be estimated as follows:EQUATIONAn inherent drawback of the estimation in Eq. ( 7) is its reliance on the assumption π ϕ ≈ π θ . In addition, the second approximation involves Monte Carlo estimation with M samples {a (i) } M i=1 , where the computational cost increases with the number of samples M . (9) In EBFlow, the energy functionE θ (x) is defined as -log(p z (g θ (x)) i∈Sn | det(J g i θ (x i-1 ))|) and the normalizing constant Z θ = exp(-E θ (x))dx = i∈S l | det J g i θ | -1 is independent of x.The input-independence of Z θ holds since g i θ is either a first-degree or zero-degree polynomial for any i ∈ S l , and thus its Jacobian is a constant to x i-1 . This technique was originally proposed to reduce the training cost of maximum likelihood estimation for normalizing flows. However, we discovered that EBFlow is ideal for MaxEnt RL. Its unique capability to represent a parametric energy function with an associated sampler g -1 θ , and to calculate a normalizing constant Z θ without integration are able to address the challenges mentioned in Section 2.2. We discuss our insights in the next section.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of this work are discussed in the main manuscript. The discussion covers the assumptions made in this paper and the computational efficiency of the proposed framework. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
methodology: 
 In this section, we introduce our proposed MaxEnt RL framework modeled using EBFlow. In Section 3.1, we describe the formulation and discuss its training and inference processes. In Section 3.2, we present two techniques for improving the training of our framework. Ultimately, in Section 3.3, we offer an algorithm summary.We propose a new framework for modeling MaxEnt RL using EBFlow, which we call MEow. This framework possesses several unique features. First, as EBFlow enables simultaneous modeling of an unnormalized density and its sampler, MEow can unify the actor and the critic previously separated in MaxEnt RL frameworks. This feature facilitates the integration of policy improexvement steps with policy evaluation steps, and results in a single objective training process. Second, the normalizing constant of EBFlow is expressed in closed form, which enables the calculation of the soft value function without resorting to the approximation methods mentioned in Eqs. ( 6) and (7) . Third, given that normalizing flow is a universal approximator for probability density functions, our policy's expressiveness is not constrained, and can model multi-modal action distributions.In MEow, the policy is described as a state-conditioned EBFlow, with its pdf presented as follows:EQUATIONwhere the soft Q-function and the soft value function are selected as follows:EQUATIONSuch a selection satisfies V θ (s t ) = α log exp(Q θ (s t , a)/α)da based on the property of EBFlow.In addition, both Q θ and V θ have a common output codomain R, which enables them to learn to output arbitrary real values. These properties are validated in Proposition 3.1, with the proof provided in Appendix A.2. The training and inference processes of MEow are summarized as follows. Proposition 3.1. Eq. ( 11) satisfies the following statements: (1) Given that the Jacobian ofg θ is non-singular, V θ (s t ) ∈ R and Q θ (s t , a t ) ∈ R, ∀a t ∈ A, ∀s t ∈ S. (2) V θ (s t ) = α log exp (Q θ (s t , a)/α) da.Training. With Q θ and V θ defined in Eq. ( 11), the loss L(θ) in Eq. ( 4) can be calculated without using Monte Carlo approximation of the soft value function target. Compared to the previous MaxEnt RL frameworks that rely on Monte Carlo estimation (i.e., Eqs. ( 6) and ( 7)), our framework offers the advantage of avoiding the errors induced by the approximation. In addition, MEow employs a unified policy rather than two separate roles (i.e., the actor and the critic), which eliminates the need for minimizing an additional policy improvement loss L(ϕ) to bridge the gap between π θ and π ϕ . This simplifies the training process of MaxEnt RL, and obviates the requirement of balancing between the two optimization loops.Inference. The sampling process of π θ can be efficiently performed by deriving the inverse of g θ , as supported by several normalizing flow architectures [43] [44] [45] [46] [47] [48] . In addition, unlike previous actor-critic frameworks susceptible to discrepancies between π θ and π ϕ , the distribution established via g -1 θ (z|s t ), where z ∼ p z , is consistently aligned with the pdf defined by Q θ . As a result, the actions taken by MEow can precisely reflect the learned soft Q-function.In this subsection, we introduce a number of training and inference techniques aimed at improving MEow while preserving its key features discussed in the previous subsection. For clarity, we refer to the MEow framework introduced in the last section as 'MEow (Vanilla)'. This experiment adopt the affine coupling layers [47] as the nonlinear transformations.Learnable Reward Shifting (LRS). Reward shifting [61] [62] [63] [64] [65] is a technique for shaping the reward function. This technique enhances the learning process by incorporating a shifting term in the reward function, which leads to a shifted optimal soft Q-function in MaxEnt RL. Inspired by this, this work proposes modeling a reward shifting function b θ : S → R with a neural network to enable the automatic learning of a reward shifting term. For notational simplicity, the parameters are denoted using θ, and the details of the architecture are presented in Appendix A.5.1. The soft Q-function Q b θ augmented by b θ is defined as follows:EQUATION)The introduction ofQ b θ results in a correspond- ing shifted soft value function V b θ (s t ) ≜ α log exp(Q b θ (s t , a)/α)da = V θ (s t ) + b θ (s t ) (i.e., Proposition A.3 in Appendix A.2), which can be calculated without Monte Carlo estimation.Moreover, with the incorporation of b θ , the policy π θ remains invariant since exp(1 α (Q b θ (s t , a t ) -V b θ (s t ))) = exp( 1 α ((Q θ (s t , a t ) + b θ (s t )) -(V θ (s t ) + b θ (s t )))) = exp( 1 α (Q θ (s t , a t ) -V θ (s t ))), which allows the use of g -1 θ for efficiently sampling actions. As evidenced in Fig. 1 , this method effectively addresses the issues of the significant growth and decay of Jacobian determinants of g θ (discussed in Appendix A.3). In Section 4.4, we further demonstrate that the performance of MEow can be significantly improved through this technique.Shifting-Based Clipped Double Q-Learning (SCDQ). As observed in [66] , the overestimation of value functions often occurs in training. To address this issue, the authors in [66] propose clipped double Q-learning, which employs two separate Q-functions and uses the one with the smaller output to estimate the value function during training. This technique is also used in MaxEnt RL frameworks [9] [10] [11] [12] [13] . Inspired by this and our proposed learnable reward shifting, we further propose a shifting-based method that adopts two learnable reward shifting functions, b θ can be obtained using Eq. ( 12), while the soft value function V clip θ is written as the following formula:EQUATIONθ (s t ) . (13) This design also prevents the production of two policies in MEow, as having two policies can complicate the inference procedure. In our ablation analysis presented in Section 4.4, we demonstrate that this technique can effectively improve the training process of MEow.Deterministic Policy for Inference. As observed in [8] , deterministic actors typically performed better as compared to its stochastic variant during the inference time. Such a problem can be formalized as finding an action a that maximizes Q(s t , a) for a given s t . Since A is a continuous space, finding such a value would require extensive calculation. In the MEow framework, this value can be derived by making assumptions on the model architecture construction. Our key observation is that if the Jacobian determinants of the non-linearities (i.e., g i θ ∈ S n ) are constants with respect to its inputs, and that argmax z p z (z) can be directly obtained, then the action a that maximizes Q(s t , a) can be efficiently derived according to the following proposition.Proposition 3.2. Given that | det(J g i θ (a i-1 |s t ))| is a constant with respect to a i-1 , then g -1 θ (argmax z p z (z)|s t ) = argmax a Q θ (s t , a).The proof is provided in Appendix A.2. It is important to note that g i θ can still be a non-linear transformation, given that | det(J g i θ (a i-1 |s t ))| is a constant. To construct such a model, a Gaussian prior with the additive coupling transformations [46] can be used as non-linearities. Under such a design, an action can be derived by calculating g -1 θ (µ|s t ), where µ represents the mean of theInput: Learnable parameters θ and shadow parameters θ ′ . Target smoothing factor τ . Learning rate β.EQUATIONθ (•), and b ▷ Extend the Replay Buffer.3: at = g -1 θ (z|st), z ∼ pz(•). 4:st+1 ∼ pT (•|st, at).D ← D ∪ {(st, at, rt, st+1)}.▷ Update Policy.(st, at, rt, st+1) ∼ D. 8:Q θ (st, at) = α log(pz (g θ (at|st)) i∈Sn | det(J g i θ (a i-1 t |st))|). ▷ Eq. (11) 9: V θ ′ (st+1) = -α log i∈S l | det(J g i θ ′ (st+1))|.▷ Eq. ( 11)10:EQUATIONθ (st) and Q(2)EQUATIONθ (st). ▷ Eq. ( 12)11:EQUATIONθ ′ (st+1) . ▷ Eq. ( 13)12:L(θ) = 1 2 (Q (1) θ (st, at) -(rt + γV clip θ ′ (st+1))) 2 + 1 2 (Q (2) θ (st, at) -(rt + γV clip θ ′ (st+1))) 2 . ▷ Eq. (4) 13: θ ← θ + β∇ θ L(θ). 14: θ ′ ← (1 -τ )θ ′ + τ θ.Gaussian distribution. We elaborate on our model architecture design in Appendix A.5.1, and provide a performance comparison between MEow evaluated using a stochastic policy (i.e., a t ∼ π θ (• |s t )) and a deterministic policy (i.e., a t = argmax a Q θ (s t , a)) in Section 4.4.We summarize the training process of MEow in Algorithm 1. The algorithm integrates the policy evaluation steps with the policy improvement steps, resulting in a single loss training process. This design differs from previous actor-critic frameworks, which typically perform two consecutive updates in each training step. In Algorithm 1, the learning rate is denoted as β. A set of shadow parameters θ ′ is maintained for calculating the delayed target values [67] , and is updated according to the Polyak averaging [68] of θ, i.e., θ ′ ← (1 -τ )θ ′ + τ θ, where τ is the target smoothing factor. 
experiments: 
 In the following sections, we first present an intuitive example of MEow trained in a two-dimensional multi-goal environment [8] in Section 4.1. We then compare MEow's performance against several continuous-action RL baselines in five MuJoCo environments [32, 33] in Section 4.2. Next, in Section 4.3, we evaluate MEow's performance on a number of Omniverse Isaac Gym environments [34] simulated based on real-world robotic application scenarios. Lastly, in Section 4.4, we provide an ablation analysis to inspect the effectiveness of each proposed technique. Among all experiments, we maintain the same model architecture, while adjusting inputs and outputs according to the state space and action space for each environment. We construct g θ using the additive coupling layers [46] with element-wise linear transformations, utilize a unit Gaussian as p z , and model the learnable adaptive reward shifting functions b θ as multi-layer perceptrons (MLPs). For detailed descriptions of the experimental setups, please refer to Appendix A.5.In this subsection, we present an example of MEow trained in a two-dimensional multi-goal environment [8] . The environment involves four goals, indicated by the red dots in Fig. 2 (a). The reward function is defined by the negative Euclidean distance from each state to the nearest goal, and the corresponding reward landscape is depicted using contours in Fig. 2 (a). The gradient map in Fig. 2 (a) represents the soft value function predicted by our model. The blue lines extending from the center represent the trajectories produced using our policy. As illustrated in Fig. 2 (a), our model's soft value function predicts higher values around the goals, suggesting successful learning of the goal positions through rewards. In addition, the trajectories demonstrate our agent's correct transitions towards the goals, which validates the effectiveness of our learned policy.To illustrate the potential impact of approximation errors that might emerge when employing previous soft value estimation methods, we compare three calculation methods for the soft value function: (I) Our approach (i.e., Eq. ( 11)): V θ (s t ), (II) SQL-like (i.e., Eq. ( 6)):α log( 1 M M i=1 exp(Q θ (st,a (i) )/α) π ϕ (a (i) |st)),and (III) SAC-like (i.e., Eq. ( 7)):1 M M i=1 (Q θ (s t , a (i) ) -α log π ϕ (a (i) |s t )), where {a (i) } M i=1 is sampled from π ϕ . The approximation errors of the soft value functions at the initial state are calculated using the Euclidean distances between (I) and (II), and between (I) and (III), for various values of M . As depicted in Fig. 2 (b), the blue line and the orange line decreases slowly with respect to M . These results suggest that Monte Carlo estimation converges slowly, making approximation methods such as Eqs. ( 6) and ( 7) challenging to achieve accurate predictions.In this experiment, we compare MEow with several commonly-used continuous control algorithms on five MuJoCo environments [32] from Gymnasium [33] . The baseline algorithms include SQL [8] , SAC [9] , deep deterministic policy gradient (DDPG) [69] , twin delayed deep deterministic policy gradient (TD3) [66] , and proximal policy optimization (PPO) [70] . The results for SAC, DDPG, TD3, and PPO were reproduced using Stable Baseline 3 (SB3) [71] , utilizing SB3's refined hyperparameters. The results for SQL were reproduced using our own implementation, as SB3 does not support SQL and the official code is not reproducible. Our implementation adheres to SQL's original paper. Each method is trained independently under five different random seeds, and the evaluation curves for each environment are presented in the form of the means and the corresponding confidence intervals.As depicted in Fig. 3 , MEow performs comparably to SAC and outperforms the other baseline algorithms in most of the environments. Furthermore, in environments with larger action and state dimensionalities, such as 'Ant-v4' and 'Humanoid-v4', MEow offers performance improvements over SAC and exhibits fewer spikes in the evaluation curves. These results suggest that MEow is capable of performing high-dimensional tasks with stability. To further investigate the performance difference between MEow and SAC, we provide a thorough comparison between MEow, SAC [9] , Flow-SAC [10, 11] , and their variants in Appendix A.4.2. The results indicate that the training process involving policy evaluation and policy improvement steps may be inferior to our proposed training process with a single objective. In the next subsection, we provide a performance examination using the simulation environments from the Omniverse Isaac Gym [34] . In this subsection, we examine the performance of MEow on a variety of robotic tasks simulated by Omniverse Isaac Gym [34] , a GPU-based physics simulation platform. In addition to 'Ant' and 'Humanoid', we employ four additional tasks: 'Ingenuity', 'ANYmal', 'AllegroHand', and 'Franka-Cabinet'. All of them are designed based on real-world robotic application scenarios. 'Ingenuity' and 'ANYmal' are locomotion environments inspired by NASA's Ingenuity helicopter and ANYbotics' industrial maintenance robots, respectively. On the other hand, 'AllegroHand' and 'FrankaCabinet' focus on executing specialized manipulative tasks with robotic hands and arms, respectively. A demonstration of these tasks is illustrated in Fig. 5 .In this experimental comparison, we adopt SAC as a baseline due to its excellent performance in the MuJoCo environments. The evaluation results are presented in Fig. 4 . The results demonstrate that MEow exhibits superior performance on 'Ant (Isaac)' and 'Humanoid (Isaac)'. In addition, MEow consistently outperforms SAC across the four robotic environments (i.e., 'Ingenuity', 'ANYmal', 'AllegroHand', and 'FrankaCabinet'), indicating that our algorithm possesses the ability to perform challenging robotic tasks simulated based on real-world application scenarios.In this subsection, we provide an ablation analysis to examine the effectiveness of each technique introduced in Section 3.2.Training Techniques. Fig. 6 compares the performance of three variants of MEow: 'MEow (Vanilla)', 'MEow (+LRS)', and 'MEow (+LRS & SCDQ)', across five MuJoCo environments. The results show that 'MEow (Vanilla)' consistently underperforms, with its total returns demonstrating negligible or no growth throughout the training period. In contrast, the variants incorporating translation functions demonstrate significant performance enhancements. This observation highlights the importance of including b θ in the model design. In addition, the comparison between 'MEow (+LRS)' and 'MEow (+LRS & SCDQ)' suggests that our reformulated approach to clipped double Q-learning [66] improves the final performance by a noticeable margin.Inference Technique. Fig. 7 compares the performance of two variants of MEow: 'MEow (Stochastic)' and 'MEow (Deterministic)'. The former samples action based on a t ∼ π θ (• |s t ) while the latter derive action according to a t = argmax a Q θ (s t , a) = g -1 θ (µ|s t ). As shown in the figure, MEow with a deterministic policy outperforms its stochastic variant, suggesting that a deterministic policy may be more effective for MEow's inference.  
conclusion: 
 In this paper, we introduce MEow, a unified MaxEnt RL framework that facilitates exact soft value calculations without the need for Monte Carlo estimation. We demonstrate that MEow can be optimized using a single objective function, which streamlines the training process. To further enhance MEow's performance, we incorporate two techniques, learnable reward shifting and shiftingbased clipped double Q-learning, into the design. We examine the effectiveness of MEow via experiments conducted in five MoJoCo environments and six robotic tasks simulated by Omniverse Isaac Gym. The results validate the superior performance of MEow compared to existing approaches.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code and installation instructions are available in an anonymous repository, with the link provided in Appendix A.5. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental configurations, detailed hyperparameter setups, and hardware requirements for the experiments presented in this paper are elaborated in Appendix A.5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The evaluation curves in Figs. 3, 4, 6, 7, A1, and A2) present the mean performance, with the shaded areas indicating the 95% confidence intervals. Each of them is derived from five independent runs with different seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no risk for misuse (e.g., pretrained language models, image generators, or scraped datasets). Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators of assets are properly credited through citations, and the licence is included in the asset (see Appendix A.5). Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The code, installation instructions, and running commands are summarized in an anonymous repository, with the link provided in Appendix A.5. The environments are publicly available, and the experiments are performed on them with the default setup.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As discussed in Section 3.2, deterministic policies typically offer better performance compared to their stochastic counterparts. Although our implementation of MEow supports deterministic inference, this capability is based on the assumptions that the Jacobian determinants of the non-linear transformations are constants with respect to their inputs, and that argmax z p z (z) can be efficiently derived. These assumptions may not hold for certain types of flow-based models. Therefore, exploring effective architectural choices for MEow represents a promising direction for further investigation.On the other hand, the training speed of MEow is around 2.3× slower than that of SAC, even though updates according to L(ϕ) are bypassed in MEow. According to our experimental observations, the computational bottleneck of MEow may lie in the inference speed of the flow-based model during interactions with environments. While this speed is significantly faster than many iterative methods, such as MCMC or variational inference, it is still slower compared to the inference speed of Gaussian models. As a result, enhancing the inference speed of flow-based models represents a potential avenue for further improving the training efficiency of MEow.Finally, our hyperparameter sensitivity analysis, as presented in A.4.5, indicates that our current approach requires different values of τ to achieve optimal performance. Since hyperparameter tuning often demands significant computational resources, establishing a more generalized parameter setting or developing an automatic tuning mechanism for τ presents an important direction for future exploration.(2) V θ (s t ) = α log exp (Q θ (s t , a)/α) da.1 = π(a|s t )da = p z (g θ (a|s t )) i∈Sn det J g i θ (a i-1 |s t ) i∈S l det(J g i θ (s t )) da. ⇔ i∈S l det(J g i θ (s t )) -1 = p z (g θ (a|s t )) i∈Sn det J g i θ (a i-1 |s t ) da. ⇔ log i∈S l det(J g i θ (s t )) -1 = α log p z (g θ (a|s t )) i∈Sn det J g i θ (a i-1 |s t ) da. ⇔ -α log i∈S l det(J g i θ (s t )) = α log p z (g θ (a|s t )) i∈Sn det J g i θ (a i-1 |s t ) da. ⇔ V θ (s t ) = α log exp (Q θ (s t , a)/α) da.In Section 3.2, we demonstrate that argmax a Q θ (s t , a) can be efficiently obtained through g -1 θ (argmax z p z (z)|s t ). To provide theoretical support for this result, we include a proof for Proposition 3.2.Proposition 3.2 Given that | det(J g i θ (a i-1 |s t ))| is a constant with respect to a i-1 , then g -1 θ (argmax z p z (z)|s t ) = argmax a Q θ (s t , a). Proof. Let c ≜ i∈Sn | det(J g i θ (a i-1 |s t ))|. argmax a Q θ (s t , a) = argmax a α log p z (g θ (a|s t )) i∈Sn det J g i θ (a i-1 |s t ) = argmax a α log (p z (g θ (a|s t )) c) (i) = argmax a p z (g θ (a|s t )) c = argmax a p z (g θ (a|s t )) (ii) = argmax a p z (z) (iii) = g -1 θ (argmax z p z (z)|s t ),where (i) is because logarithm is strictly increasing, (ii) is due to z = g θ (a|s t ), and (iii) is due to a = g -1 θ (z|s t ).In Section 3.2, we incorporate the learnable reward shifting function b θ in Q θ and V θ . This incorporation results in redefined soft Q-function Q b θ and soft value functionV b θ . In Proposition A.3, we verify that V b θ (s t ) = α log exp(Q b θ (s t , a)/α)da = V θ (s t ) + b θ (s t ). Proposition A.3. Given that Q and V satisfy V (s t ) ≜ α log exp (Q(s t , a)/α) da. The augmented functions, Q b (s t , a t ) = Q(s t , a t ) + b(s t ) and V b (s t ) ≜ α log exp(Q b (s t , a)/α)da, where b(s t ) is the the reward shifting function, satisfy V b (s t ) = V (s t ) + b(s t ). Proof. V b (s t ) = α log exp Q b (s t , a)/α da. = α log exp ((Q(s t , a) + b(s t )) /α) da = α log exp(b(s t )/α) exp (Q(s t , a)/α) da = α log exp (Q(s t , a)/α) da + α log (exp(b(s t )/α)) = α log exp (Q(s t , a)/α) da + b(s t ) = V (s t ) + b(s t ).In this section, we provide the motivation for employing the learnable reward shifting function described in Section 3.2. We show that while Q θ and V θ defined in Eq. ( 11) have the theoretical capability to learn arbitrary real values (i.e., Proposition 3.1), they may experience numerical instability in practice. This instability arises due to the exponential growth of i∈Sn | det(J g i θ (a i-1 t |s t ))| and the exponential decay of i∈S l | det(J g i θ (s t ))|. We first examine the relationship between V θ (s t ) and i∈S l | det(J g i θ (s t ))| according to the following equations:V θ (s t ) = -log i∈S l det J g i θ (s t ) ⇔ exp(-V θ (s t )) = i∈S l det J g i θ (s t ) .The above equation suggests that the value of i∈S l | det(J g i θ (s t ))| decreases exponentially with respect to V θ (s t ), which may lead to numerical instability during training. On the other hand, the relationship between Q θ (s t , a t ) and i∈Sn | det(J g i θ (a i-1 t |s t ))| can be expressed according to the following equations:Q θ (s t , a t ) = log p z (g θ (a t |s t )) + log i∈Sn det J g i θ (a i-1 t |s t ) . ⇔ Q θ (s t , a t ) -log p z (g θ (a t |s t )) = log i∈Sn det J g i θ (a i-1 t |s t ) . ⇔ exp (Q θ (s t , a t ) -log p z (g θ (a t |s t ))) = i∈Sn det J g i θ (a i-1 t |s t ) .The equations indicate that i∈Sn | det(J g i θ (a i-1 t |s t ))| increases exponentially with respect to Q θ (s t , a t ) -log p z (g θ (a t |s t )). Therefore, increasing Q θ may also lead to an exponential growth ofi∈Sn | det(J g i θ (a i-1 t |s t ))|.LRS makes our model less susceptible to numerical calculation errors since the learnable reward shifting function b θ , unlike Q θ and V θ , is not represented in logarithmic scale. Consider a case where FP32 precision is in use, 'MEow (Vanilla)' could fail to learn a target V θ * (s t ) > 38, ∀s t , since i∈S l | det(J g i θ (s t ))| = exp(-V θ * (s t )) < 2 -126 cannot be represented using FP32 precision. Therefore, without shifting the reward function, the loss sometimes becomes undefined values, and can lead to ineffective training (e.g., the green lines in Fig. 6 ). The reward shifting term can be designed as a (state-conditioned) function or a (non-state-conditioned) value. It can also be learnable or non-learnable. All of these designs (i.e., b θ (s t ), b(s t ), b θ , and b) can be directly applied to MEow since none of them influences the action distribution. Based on our preliminary experiments, we identified that a learnable state-conditioned reward shifting delivers the best performance. soft value calculation during training, the first and second frameworks adopt the value estimation method in SAC (i.e., Eq. ( 7)). For the third and the fourth frameworks, their training adopts the exact value calculation (i.e., Eq. ( 11)), which is the same as MEow. The results are presented in Fig. A2 .As depicted in Fig. A2 , MEow exhibits superior performance and stability compared to other actor-critic frameworks in the 'Hopper-v4', 'Ant-v4', and 'Walker2d-v4' environments, and shows comparable performance with ECGA in most environments. In addition, the results that compare the frameworks with flow-based models as actors (i.e., ECFA and FCFA) to those with Gaussians as actors (i.e., ECGA and FCGA) suggest that Gaussians are more effective for modeling actors. This finding is similar to that in [10] . On the other hand, the comparisons between ECGA and FCGA, and between FCGA and FCFA, do not show a clear trend. These findings suggest that both flow-based and energy-based models can be suitable for modeling the soft Q-function. Furthermore, the comparison between FCFA and MEow reveals that the training process involving alternating policy evaluation and improvement steps may be inferior to our proposed training process with a single objective. In this section, we use a one-dimensional example to demonstrate that flow-based models are capable of learning multi-modal action distributions. We employ a state-conditioned neural spline flow (NSF) [50] as the model, and train it in a single-step environment with one-dimensional state and action spaces. Fig. A3 (a) illustrates the reward landscape with the state and action denoted on the xaxis and y-axis, respectively. Fig. A3 (b ) illustrates the probability density function (pdf) predicted by the model. The result demonstrates the capability of flow-based models to effectively learn multi-modal distributions.In this section, we examine the performance of SAC with the proposed LRS technique. Since the original implementation of SAC involves the clipped double Q-Learning technique, SAC with LRS is equivalent to SAC with the shifting-based clipped double Q-Learning (SCDQ) technique discussed in Section 3.2 of the main manuscript. The performance of 'SAC' and 'SAC (+LRS)' is presented in Fig. A4 . The results indicate that applying LRS does not improve SAC's performance. Therefore, for a fair evaluation, the original implementation of SAC is adopted in the comparison in Section 4 of the main manuscript.In this section, we provide a performance comparison of SAC and MEow trained with different target smoothing parameter values (i.e., τ = 0.005, 0.003, 0.0005, and 0.0001). The results shown in ......... Software and Hardware Setups. For the experiments performed on Omniverse Isaac Gym, the implementation is built on SKRL [79] due to its compatibility with Omniverse Issac Gym [34] . The computation was carried out on NVIDIA L40 GPUs equipped with 48GB of memory. The training takes approximately 22 hours per 1 million training steps, with each GPU capable of executing three training sessions simultaneously. For 'Ant', 'Humanoid', 'Ingenuity', and 'ANYmal', each training step consists of 128 parallelizable interactions with the For 'AllegroHand' and 'FrankaCabinet', each training step consists of 512 parallelizable interactions with the environments.Hyperparameter Setups. The shared and the environment-specific hyperparameters of MEow are summarized in Tables A1 and A3 , respectively. Those of SAC are summarized in Tables A2 and A4 , respectively. Both SAC and MEow were tuned using the same search space for τ and α to ensure a fair comparison. Specifically, a grid search was conducted with τ values ranging from 0.1 to 0.00025 and α values from 0.8 to 0.0005 for both algorithms. The setups with the highest average return were selected for each environment.This work represents a new research direction for MaxEnt RL. It discusses a unified method that can be trained using a single objective function and can avoid Monte Carlo estimation in the calculation of the soft value function, which addresses two issues in the existing MaxEnt RL methods. From a practical perspective, our experiments demonstrate that MEow can achieve superior performance compared to widely adopted representative baselines. In addition, the experimental results conducted in the Omniverse Isaac environments show that our framework can perform robotic tasks simulated based on real-world application scenarios. These results indicate the potential for deploying MEow in real robotic tasks. Given MEow's potential to be extended to perform challenging tasks, it is unlikely to have negative impacts on society.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims in the abstract and Section 1 accurately reflect the contributions of this paper. The theoretical results are discussed in Section 3. The experiments in Section 4 provide empirical justification for these claims. Finally, Section 5 summarizes both the theoretical and empirical contributions of this work. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper proposes a new MaxEnt Reinforcement Learning (RL) framework using Energy-Based Normalizing Flows (EBFlow). Traditional MaxEnt RL methods typically use actor-critic frameworks and alternate between policy evaluation and policy improvement steps. The proposed framework integrates these steps into a single objective, eliminating the need for Monte Carlo approximation while enabling efficient modeling of multi-modal action distributions. Experiments on the MuJoCo benchmark and Omniverse Isaac Gym environments show superior performance compared to existing MaxEnt RL methods.

**Strengths** 
- The idea of adopting EBFlow for MaxEnt RL is novel.
- The presentation is clear and well-organized, and the method is described in detail.
- The experimental results are convincing.

**Weaknesses**
- The calculation of the determinants is more time-consuming than existing methods like SAC.

Reviews: review: paper_summary: paper_summary: The paper presents a new method for MaxEnt RL based on the recently proposed energy-based normalizing flows (EBFlow). The adoption of EBFlow allows us to overcome two major issues with training MaxEnt RL algorithms: (I) sampling from an energy-based model (of the policy) and (ii) approximating the soft value function (with MC methods).
strength_weakness: strength_weakness: * **New and old challenges**: while the approach resolves some issues with the current MaxEnt RL methods, it still suffers from typical deep RL issues (e.g. overestimation, for which the authors proposed SCDQ) and it introduces additional challenges (e.g. exploding Jacobian determinants, for which the authors introduce LRS). Normalizing flows also introduce additional constraints on the network's architecture, a problem which I believe is strongly mitigated by the introduced LRS (?)
* **Required clarification on LRS**: see Questions
questions: questions: * Typo at line 149, "Jocobian"
* I am not sure if the role of the Learnable Reward Shifting (LRS) is clear to me. The authors say that they were inspired by the reward-shifting literature. However, I don't see the LRS term working the same way as classical reward shaping, as a state-dependent non-linear function ($b_\theta$), without any particular constraints (I don't see any stated) wouldn't preserve the properties of the original functions. I think the role of the learnable reward shifting is to have some form of "baseline" in the Q and V functions that simplifies the estimation process (since it's using a more flexible learning architecture), and so it avoids the explosion of the Jacobian. Could the authors provide clarification on this?
* I would be interested in seeing a comparison between the learned reward shifting term and the value function (V) in a plot (it would suffice to see this in the Appendix)
limitations: limitations: Limitations are addressed in Appendix. I think the authors should mention this in the main text.
paper_summary: paper_summary: The paper introduces a new MaxEnt RL framework called Meow based on Energy-based Normalizing Flow, which integrates the policy evaluation steps and the policy improvement steps and results in a single objective training process. Besides Meow enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation and supports the modeling of multi-modal action distributions.
strength_weakness: strength_weakness: 1.	The calculation of the derterminant is usually time-consuming. So the reviewer wonders how the training time of Meow is compared with that of the classical off-policy RL methods like SAC.
2.	In the experiments, Meow is just compared with several classical RL methods for continuous control and lacks the comparison with advanced RL methods [1, 2] with other generative models like diffusion model and consistency model, which also supports the modeling of multi-modal action distribution.

[1] Yang L, Huang Z, Lei F, et al. Policy representation via diffusion probability model for reinforcement learning[J]. arXiv preprint arXiv:2305.13122, 2023.

[2] Yue Y, Kang B, Ma X, et al. Boosting offline reinforcement learning via data rebalancing[J]. arXiv preprint arXiv:2210.09241, 2022.
questions: questions: 1.	How is the training time of Meow compared with other methods?
2.	How is Meow compared with other advanced RL methods with the utilization of generative-model-based policy?
3.	Could the authors explain why Meow can learn a good reward shifting rather than a trivial shifting, like for any $s_t$, $b(s_t)=0$?
limitations: limitations: The authors have clearly presented the limitations in the paper.
paper_summary: paper_summary: The paper presents a new framework for Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) using Energy-Based Normalizing Flows (EBFlow). Traditional MaxEnt RL methods, particularly for continuous action spaces, typically utilize actor-critic frameworks and alternate between policy evaluation and policy improvement steps. The proposed EBFlow framework integrates these steps into a single objective, eliminating the need for Monte Carlo approximation while enabling efficient modeling of multi-modal action distributions. The framework was experimentally validated on the MuJoCo benchmark and high-dimensional robotic tasks in Omniverse Isaac Gym, demonstrating superior performance compared to existing baseline methods.
strength_weakness: strength_weakness: - There are some unreasonable aspects in the organization of the content. For example, Section 3.2 appears somewhat abrupt; it is suggested to move the content of A.4 into the main text. The discussion about inference in Section A.3 is also very important and should be included in the main text.
- The experimental section does not sufficiently highlight the advantages of the MEow method. The experiments in Section 4.1 on toy examples are not enough. Can comparisons be made from the perspectives of inference time, training time, representation capability, etc.?
- While the integration of policy evaluation and improvement steps simplifies the training process conceptually, the implementation details of the EBFlow framework may introduce additional complexity that needs to be managed.
questions: questions: - GFlowNet points out that the MaxEnt RL paradigm has deficiencies in terms of diversity. Specifically, MaxEnt RL tends to achieve target states that are "further" from the initial state. Does MEow also inherit this defect of the MaxEnt RL paradigm?
- Energy-based normalizing flow is a generative model, and its original loss is to maximize log-likelihood. In Algorithm 1, this loss does not seem to be present. In my view, MEow uses the network structure of energy-based normalizing flow, but the loss still follows the MaxEnt RL paradigm. In this case, could the title of this paper be potentially misleading?
- Figure A5 shows that the MEow network uses a hypernetwork module. Why was this module introduced? Is this a common architecture for energy-based normalizing flow?
- What are the learning dynamics observed during the training process? Are there any notable differences in the convergence behavior compared to traditional actor-critic methods?
- How does the proposed framework scale with increasing complexity of the environment or the state-action space? Are there any specific limitations or challenges observed during experimentation?
- How robust is the EBFlow framework to different hyperparameters?
limitations: limitations: The author has already discussed the limitations ofthe method and its potential impacts in the paper, and provided possible solutions.
paper_summary: paper_summary: This paper proposes to use the energy-based normalizing flow (EBFlow) to represent the policy and value functions in maximum entropy (MaxEnt) RL. Specifically, the paper builds the connection between EBFlow and MaxEnt RL by linking the conditional unnormalized density that depends on the input to the action value function Q and the constant that is independent of the input to the value function V. As a consequence, the Boltzmann policy defined by Q and V can be expressed exactly using a mapping in EBFlow. By updating the current Q estimate to approximate the action value of the Boltzmann policy, the algorithm would spontaneously update the Boltzmann policy. Hypothetically, the algorithm should converge to the optimal policy under entropy regularization. Using an existing implementation of EBFlow in the relevant literature, the paper proposed MEow based on this formulation. Experiments on four MuJoCo environments and six Issac Gym environments demonstrate the learning performance of MEow.
strength_weakness: strength_weakness: 1. There is a potential concern about the empirical investigation: The proposed method is tuned over the target smoothing parameter, while the baselines are not tuned. Why is the target smoothing parameter tuned per environment? What would happen if the proposed method also used the same, fixed smoothing parameter?
2. This is a minor point. The organization of the paper can be improved. Specifically, the background section is too long, which takes up the space for more interesting discussions presented in the appendix. If space allows, the main text should include the network architecture in Figure A5, as it may be unfamiliar to a lot of readers. In addition, the limitation, especially the computational aspect of the proposed approach, should be discussed in the main paper.

Other suggestions that do not affect the evaluation:
* It would be great if there is at least some qualitative comparison between the proposed method and baselines using MaxEnt RL in the multi-goal environment.
* Typos between Lines 254 and 276: The reference to Figure 1 should be to Figure 2.
questions: questions: Some questions that may affect the evaluation:
1. Are deterministic policies used for other algorithms in the evaluation in Figures 3 and 4?
limitations: limitations: The limitations of the work are discussed in the appendix, which should be moved to the main text. Another unmentioned potential limitation is the sensitivity to the target network smoothing parameter.
paper_summary: paper_summary: This paper proposes a new framework for Maximum Entropy Reinforcement Learning using Energy-Based Normalizing Flows (EBFlow). The authors argue that this framework offers three main advantages: Firstly, the soft value function can be obtained without estimation. Second, the framework combines policy evaluation and policy improvement in one iteration. Finally, it does not use Monte Carlo methods and interacts with the environment effectively. To achieve this goal, Learnable Reward Shifting (LRS) technique which involves reward shifting for the value function using a reward shifting network and Shifting-Based Clipped Double Q-Learning (SCDQ) technique that employs two reward shifting networks for the value function and utilizes the minimum value function similar to double Q-learning have been proposed.
strength_weakness: strength_weakness: 1. Despite the fact that the proposed method is the first to apply EBFlow to reinforcement learning (RL), the novelty of the algorithm may appear weak, as there are several existing RL methods based on flow-based models. Besides applying EBFlow to the RL structure, were there are any additional considerations or advantages that were taken into account to enhance its performance or applicability in the context of RL? If such points exist, highlighting them would help to emphasize the contributions of the authors' proposed technique.

2. Introducing new frameworks generally necessitates providing sufficient mathematical proof to validate their efficacy, even for basic techniques or simple conjectures. It would be beneficial if the paper included mathematical proofs demonstrating that the proposed framework converges to the optimal policy. For example, the techniques presented in the paper, namely Learnable Reward Shifting (LRS) and Shifting-Based Clipped Double Q-Learning (SCDQ), have a significant impact on performance. However, the paper does not provide adequate mathematical justification for these techniques. Including such proofs would greatly enhance the confidence in the proposed method.

3. It would be beneficial for the paper to compare its methods with more recent state-of-the-art algorithms. The comparison with SAC, published in 2018, feels a bit outdated. Given that reinforcement learning is a rapidly evolving field, including comparisons with more recent methods would provide a clearer understanding of where this framework stands in the current landscape. Such comparisons would greatly enhance the paper by placing the proposed method in the context of the latest advancements.
questions: questions: 1. Please answer the questions in the weakness section

2. According to the appendix, the proposed method is said to be 2.3 times slower than SAC. Given this difference in speed, a time-wise comparison of the algorithm's performance would be very helpful to understand its efficiency relative to SAC. 

3. In Figure 3, SAC and MEow show similar performance in the Mujoco domain, but there is a noticeable difference in performance in the Isaac Gym domain in Figure 4. Could you provide a further analysis on the reasons for the occurrence of this difference? It would provide valuable insight into the strengths and limitations of the proposed method.

Minor Issues\
Typo in line 149: "Jocobian" should be "Jacobian."\
Misreference in section 4.1: It should refer to Fig. 2, not Fig. 1.
limitations: limitations: They have handled it well.

Comments: title: Follow-up questions on hyperparameter tuning
comment: Thank you for the details. I encourage the author to present the full details about the Isaac Gym experiments, including SAC's environment-specific hyperparameters and the sensitivity analysis from the grid search. If these results are going to be included in the camera-ready version if accepted, I am willing to further raise my rating.
We sincerely appreciate the reviewer’s response and valuable feedback. Thank you again for your thoughtful review.
We sincerely appreciate the reviewer’s response and valuable feedback. In the Isaac Gym experiments, both SAC and MEow were tuned using the same search space for $\tau$ and $\alpha$ to ensure a fair comparison. Specifically, a grid search was conducted with $\tau$ values ranging from 0.1 to 0.00025 and $\alpha$ values from 0.8 to 0.0005 for both algorithms. The setups with the highest average return were selected for each environment. We agree with the reviewer that the differences in the hyperparameters present an interesting direction for further investigation. Thank you again for your thoughtful review.
We sincerely appreciate the reviewer’s response and valuable feedback. Thank you again for your thoughtful review.
We sincerely appreciate the valuable feedback and recognition of our work's contribution and novelty. Thank you once again for your thorough review.
We sincerely appreciate the reviewer’s response and valuable feedback. We would be glad to incorporate the additional explanation of LRS into the manuscript in the final revision. Thank you again for your thoughtful review.
Many thanks to the authors for thoroughly supplementing the key experiments and discussions. I have raised the score to 7.
I am satisfied with the authors' response, which clarifies my main concerns with the work.

I think that a more appropriate description of how LRS is useful should be included in the paper, to improve clarity about the authors' contribution. A description similar to the one provided here would suffice.

The additional results presented also seem to reflect my original understanding and the authors' response about how LRS works and its useful in their method.

I increased my score to 7 and will recommend acceptance of the work.
Thank you for providing the additional experiments and for addressing the questions with such a detailed rebuttal. Your response, along with the referenced paper, has more clarified the aspects I inquired about. I appreciate your efforts in improving the clarity of the work, and I will be increase my score accordingly.
Thank you for the rebuttal and additional experiment results on SAC’s sensitivity on the target smoothing factor, $\tau$. I think the difference in the sensitivity to $\tau$ of SAC and MEow is very interesting and worth further investigation. In addition, I wonder what the hyperparameter tuning process is in the Issac gym environments. Are both SAC and MEow tuned (for both $\tau$ and $\alpha$)? If yes, how?
Thanks for your responses and the efforts on the additional experiments. However, as shown in Table 2 of "Boosting offline reinforcement learning via data rebalancing", the experiments in online RL setting are also considered. In that case, the reviewer thinks this paper cannot be viewed as separate research work from yours. However, the contribution and novelty of this paper should be acknowledged. Hence, I am sticking to my score.