Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
Authors: Chen-Hao Chao
Chien Feng
Wei-Fang Sun
Cheng-Kuang Lee
Simon See
Chun-Yi Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Maximum Entropy Reinforcement Learning
Normalizing Flows
Energy-Based Models
Energy-Based Normalizing Flow
Primary area: reinforcement_learning

Abstract: Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.

Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
Chen-Hao Chao" 112
Chien Feng"!
Wei-Fang Sun?
Cheng-Kuang Lee²
Simon See
Chun-Yi Lee
Elsa Lab, National Tsing Hua University, Hsinchu City, Taiwan NVIDIA AI Technology Center, NVIDIA Corporation, Santa Clara, CA, USA
Abstract
Existing Maximum-Entropyy (MaxEnt) Reinforcement Learning (RL. methods for continuous action spaces are typically formulated based on actor-critic frame- works and optimized through alternating steps policy evaluation and policy improvement. In the policy evaluation steps, the critic updated to capture the soft Q-function. the policy improvement steps, the actor is adjusted in accor- dance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Bart -Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in single objective training process. Our method enables the calcula- tion of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to aidely-adopted representative baselines.
Introduction
Maximum- Entuuop (MaxEnt) Reinforcement Learning (RL) [H17 has emerged as prominent method for modeling stochastic policies. Different from standard RL, MaxEnt RL integrates the en- tropy of policies rewards, which leads to balanced exploration-exploitatoo trade-off during train- ing. This approach has demonstrated improved robustness both theoretically and empirically [17]19] Building on this foundation, many studies leveraging MaxEnt RL have shown superior performance on continuous-contró) benchmark environments g and real-world applications 22--22).
An active research domain in MaxEnt RL concentrates the learning of the soft Q-function 11 These methods follow the paradigm introduced in soft Q-learning (SQL) [8]. They parameterize the soft Q-function
the soft Bellman error [8] calculated from rewards and the soft value function. However, this approach presents two challenges. First, sampling from energy-based model requires costly Monte Carlo Markov Chain (MCMC) [24] 25 or variational inference (26) process, which can result in inefficient interactions with environments. Second, the calculation of the soft value function involve computationally
energy-based model [23 and optimize based
"qqual contribution.
Coorresponding author. Email: cyleeêcs nthu. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

infeasible integration, which requires an effective approximation method. To tackle these issues, various methods [8|15] were proposed, all grounded in a common design philosophy. To address the first challenge, these methods suggest operating on an actor-critic framework and optimizing it through alternating steps of policy evaluation and policy improvement. For the second challenge, they resort to Monte Carlo methods to approximate the soft value function using sets of random samples. Although these two issues can be circumvented, these methods still have their drawbacks. The actor-critic design introduces an additional optimization process for training the actor, which may lead to optimization errors in practice [27]. Moreover, the results of Monte Carlo approximation may be susceptible to estimation errors and variances when there an insufficient number of samples [28 30).
Instead of using energy- based models to represent MaxEnt RL frameworks, this paper investigates an alternative method employing normalizing flows (i.e., flow-based models), which offer solutions to the aforementioned challenges. Our framework is inspired by the recently introduced Energy-Basel Normalizing Flows (EBFlow) 31. This design facilitates the derivation of an energy function from a flow-based model while supporting efficient sampling, which enables unified representation of both the soft Q-function and its corresponding action sampling process. This feature allows the integration of the policy evaluation and policy improvement steps into single objective training process. In addition, the probability density functions (pdf) of flow-based models can be calculated efficiently without approximation. This characteristic permits the derivation of exact representation for the soft value function. Our experimental results demonstrate that the proposed framework exhibits superior performance the commonly adopted MuJoCo benchmark [32 B3]. Furthermore, the evaluation results on the Omniverse Isaac Gym environments (34) indicate that our framework excels in performing challenging robotic tasks that simulate real-world scenarios.
2 Background and Related Works
In this section, we walk through the background material and the related works. We introduce the objective of MaxEnt RL in Section 2.1] describe existing actor-critic frameworks and soft value estimation methods in Section 2.2 and claborate on the formulation of Energy- -Based Normalizing
Flow (EBFlow) in Section
2.3
2.1 Maximum Entropy Reinforcement Learning
In this paper, we consider a Markov Decision Process (MDP) defined as a tuple (S, A. Pr, R, R,7, Po)). where S is a continuous state space, AA is a continuous action space, Pr S x S x A + R>0 is the pdf of next state St-1 given a current state s; and a current action a, at timestep t, R is the reward function, 0 < 7 < 1 is the discount factor, and is the pdf of the initial state sg. We adopt rt to denote R(s,, a and use P- (s; a,) to represent the state-action marginals of the trajectory distribution induced by a policy я (a [s:) [8]
A R
Standard RL defines the objective as n" argmax, E, Eig [rr] and has at least one deter- ministic optimal policy [35] 36). In contrast, MaxEnt RL (4) augments the standard RL objective with the entropy of policy at each visited state s.. The objective of MaxEnt RL is written follows:
MaxEnt argmax
(ss,)))) [r. + aH(T( (s.))
where H(TT( (ss))) 2 Ea-z(()))l- log π (a]s:)) and a E Is is a temperature parameter for deter- mining the relative importance of the entropy term against the reward. An extension of Eq. ( defined with y is discussed in [8]. To obtain Max described in Eq. (T) the authors in [ proposed to minimize the soft Bellman error for all states and actions. The solution can be expressed using the
optimal soft Q-function Qioh : SS x A + R and soft value function V : - IR as follows:
Maxton (ar(st) exp (2 ( (St, (s)) Veore (())((s)))
where
(2)
Qiait (sc, au) = r T [Vinne (s1+1),, Vinre(s)) = a log / exp Qion (s a) da. (3)
(Qu at) - (sx))) with parameter In practice, a policy can be modeled as no (a (s:) = (Q
8, where the soft Q-function and the soft value function are expressed Qo ar) and Vo(ss)

---

log I exp (2 Qo(ss, a1)) da,, respectively. Given an experience reply buffer D that stores transition tuples (s,, ag; Tt, 8t+1), the training objective of Qa (which can then he used to derive Vo and my can be written the following equation according the soft Bellman errors:
[()) (8, at (..D))))
(Qo(s., a.) (rr + (oo(s
(4)
Nonetheless, directly using the objective in Eq. (4) presents challenges for two reasons. First, drawing samples from energy-based based model (i.e., To To (s))s)) cxp(Qo aa)/a)) )) requires a costly MCMC or variational inference process [26 37], which makes the interaction with the environment inefficient. Second, the calculation of the soft value function involves integration, which require stochastic approximation methods [28-30] to accomplish. To address these issues, the previous MaxEnt RL methods [+ adopted actor-critic frameworks and introduced number of techniques estimate the soft value function. These methods are discussed in the next subsection.
2.2 Actor-Critic Frameworks and Soft Value Estimation in MaxEnt RL
Previous MaxEnt RL methods 8 [5 employed actor-critic frameworks, in which the critic aims to capture the soft Q-function, while the actor learns to sample actions based on this soft Q-function. Available choices for modeling the actor include Gaussian models [9]. Gaussian mixture mod- els 38), variational autoencoders (VAE) [T]] 13 39,, normalizing flows [TO] [. and amortized SVGD (A-SVGD) 8 40,, all of which support efficient sampling. The separation of the actor and the critic prevents the need for costly MCMC processes during sampling. However, this design induces additional training steps aimed to minimize the discrepancy between them. Let (a:(()) x exp( exp(+QQess, (se, at)) and (a (s)) denote the pdfs defined through the critic and the actor, respectively. The objective of this additional training process is formulated according the reverse KL divergence [TI.[To( (To [sr)] between T and πT, and is typically reduced as follows [
(())= = Eg,-D [-Bar>x [Qass a.) log T (at(sa))]]]
(5)
The optimization processes defined by the objective functions L()) and L(ő) in Eqs. (4) and ( known the policy evaluation steps and policy improvement steps (9], respectively. Through alternating updates according to (eC(O) and VoL(o), the critic learns directly from the reward signals to estimate the soft Q-function, while the actor learns to draw samples based on the distribution defined by the critic.
Although the introduction of the actor enhances sampling efficiency, calculating the soft value function in Eq. (3) still requires Monte Carlo approximations for the computationally infeasible integration operation. Prior soft value estimation methods be categorized into two groups: soft value estimation in Soft Q-Learning (SQL) and that in Soft Actor-Critic (SAC), with the former yielding larger estimate than the latter, derived from Jensen's inequality (i.e., Proposition A. in Appendix A. These two soft value estimation methods are discussed in the following paragraphs.
Soft Value Estimation in SQL. Soft Q-Learning g leverages importance sampling convert the integration in Eq. (3) into an expectation, which can be estimated using a set of independent and identically distributed (i.i.d) samples. To ensure the estimation variance is small, the authors in [ proposed to utilize samples drawn from Tg. Let {
Ti=1 be set of M samples drawn from Ts- The soft value function is approximated based on the following formula:
(s()s) a log f exp (Qu(se, a)/a) da aloo / TT(aa(s))
exp (Qo(se, a)/a) da (a(a)s:)
exp (Qo(s,,a)/a) (a(s))
M
exp (Qo a (1))aa)
(6)
= aa log Earns
so a
M
(a (s{)
Eq. (6) has the least variance when T (s)) x exp(Qa(s: //a) law of large numbers ensures that this estimation converge to (s,), 41
29]. In addition, as M + oo, the
Soft Value Estimation in SAC. Soft Actor-Critic 9 and its variants IC || 13 12 re-
formulated the soft value function Va(sz) = log f exp (Qu(s,, a)/a) da as its equivalent form Ea~x, (Q, a) log ma(ss)) based the relationship that ne (als:) exp( a))
3

---

Vo(s:)). By assuming that the policy improvement loss L()) is small (i.e., To = To), the soft value
function V can be estimated as follows:
(S+) Ea-ws (Qu(s a) a log To (a(ss))
1 M (Qo(s,,a!") - a log TH (a())())))
M Σ
(7)
E Ba QQ((s a) a m
An inherent drawback of the estimation in Eq. is its reliance on the assumption To ~ no. In addition, the second approximation involves Monte Carlo estimation with M samples { where the computational cost increases with the number of samples M.
2.3 Chergy-Based Normalizing Flows
Normalizing flows (i.e., flow-based models) are universal representations for pdf [42]. Given input data E P latent variable z E IR with prior pdf Px. and an invertible function go 96 .... ...o ....o..
modeled as a neural network with T. layers, where s TR - RD. Vi E (1....... L.). According to the change of variable theorem and the distributive property of the determinant operation, parameterized pdf Pa can be described as follows:
po(x)= = Pz (go (x)) II det (J&z (x(x)))))) i=1
(8)
where x² A is the input, x' = =g6= ... go (x) is the output of the i-th layer. and Jgj (x (x &
T (x represents the Jacobian of the i-th layer of go with respect to x 1, To draw samples from pg. one can first sample z from Pz and then derive 3. 9 (z). To facilitate efficient computation of the pdf and the inverse of ga. one can adopt existing architectural designs 43 for ge. Popular examples involve autoregressive layers 43H45) and coupling layers 446.48), which utilizes specially designed architectures to speed up the calculation.
Energy-Bin Based Normalizing Flows (EBFlow) B1 recently introduced to reinterpret flow-based models as energy based models. In contrast to traditional normalizing flow research [46] 47 [50] that focuses on the use of effective non-linearities, EBFlow emphasizes the use of both línear and non-linear transformations in the invertible transformation go- Such a concept was inspired
fully-connected as well as energy-based training techniques (58-60). Let S {i g is linear) and S, {i g. is non-linear) represent the sets of indices of the linear and non-line linelinear transformations in ge, respectively. As shown in [31], the Jacobian determinant product Eq. (8) can be decomposed according S,, and S,. This decomposition allows flow-based model to be reinterpreted as an energy- -baed model, as illustrated in the following equation:
by the development of normalizing flows with convolution layers 48
51
54
layers (53] linear independent component analysis (ICA) models 57
58
po (x) = Pz (x)(x))
II det (J (x" (x-
II det JJ
exp (-Eo ZØ
iES,
SES;
Unnormalized Density Const.
Unnormalized Density
Const.
In EBFlow, the energy function (x is defined as - log(pz (xx(x)) ILESs, det(J ()l) and the normalizing constant Zo = exp(-Ee(xx)
Il,es, |et J - is independent of x. The input-independence of Ze holds since d is either first-degree or zero-degree polynomial for any d E S), and thus its Jacobian a constant to - This technique was originally proposed to reduce the training cost of maximum likelihood estimation for normalizing flows. However, we discovered that EBFlow is ideal for MaxEnt RL. Its unique capability to represent a parametric energy function with an associated sampler 9, and to calculate a normalizing constant Zp without integration are able to address the challenges mentioned in Section 2.2 We discuss our insights in the next section.
3 Methodology
In this section, we introduce our proposed MaxEnt RI. framework modeled using EBFlow. In Sec- tion 3.1 we describe the formulation and discuss its training and inference processes. In Scction 3.2

---

we present two techniques for improving the training of our framework. Ultimately, in Section 3.3 we offer an algorithm summary.
3.1 MaxEnt RL via EBFlow
We propose a new framework for modeling MaxEnt RL using EBFlow, which we call MEow. This framework possesses several unique features. First, EBFlow enables simultaneous modeling of an unnormalized density and its sampler, MEow can unify the actor and the critic previously separated in MaxEnt RL frameworks. This feature facilitates the integration of policy improexvement steps with policy evaluation steps, and results in a single objective training process. Second, the normalizing constant of EBFlow is expressed in closed form, which enables the calculation of the soft value function without resorting to the approximation methods mentioned in Eqs. (6) and (7). Third, given that normalizing flow is a universal approximator for probability density functions, our policy's expressiveness is not constrained, and can model multi-modal action distributions.
In MEow, the policy described as a state-conditioned EBFlow, with its pdf presented follows:
To Ta(ar((:) Ps (ga (a: (s:)) II det (J (a) (a.
II iES)
det(J (s:))
S
Unnormalized Density
Norm. Const.
(10)
exp - (o(ss, an)
exp -- Vo(s))
Unnormalized Density
Norm. Const.
where the soft Q-function and the soft value function are selected follows:
Qo(s;, ar) A a aloog Pa (a (t)) II i.S.
det (Joz (aj (a)- Vo(s:) A -a log II
det o6 (so)
KES.
(11)
Such selection satisfies Vo(s,) = a log ʃ exp(Qo(s:: a)/a))da based on the property of EBFlow. In addition, both Qo and Vo have common output codomain R. which enables them to learn to output arbitrary real values. These properties are validated in Proposition 3.1] with the proof provided in Appendix A.2 The training and inference processes of MEow are summarized as follows.
Proposition 3.1. Eq. (TTy satisfies the following statements: (1) Given that the Jacobian of go is non-singular, Vo (S))
P and Qo(si, ar) R. Va, Vs, S. (2) Vo(st)
log exp (Q; a)/a) da.
Training. With Qe and Vo defined in Eq. 11 the loss L(0) in Eq. (4) can be calculated without using Monte Carlo approximation of the soft value function target. Compared to the previous MaxEnt RL frameworks that rely on Monte Carlo estimation (i.e., Eqs. (6) and (7). our framework offers the advantage of avoiding the errors induced by the approximation. policy rather than two separate roles (i.c., the actor and the critic), which eliminates the need for minimizing an additional policy improvement loss L(ø) to bridge the gap between and T This simplifies the training process MaxEnt RL, and obviates the requirement of balancing between the two optimization loops.
addition, MEow employs a unified
Inference. The sampling process of тe can be efficiently performed by deriving the inverse of ge. as supported by several normalizing flow architectures 443. In addition, unlike previous actor-critic frameworks susceptible discrepancies between Te and Rg. the distribution established via 9p (z s,), where Pa. is consistently aligned with the pdf defined by Qo. As result, the actions taken by MEow can precisely reflect the learned soft Q-function.
3.2 Techniques for Improving the Training and Inference Processes of MEow
In this subsection, we introduce number of training and inference techniques aimed at improving MEow while preserving its key features discussed in the previous subsection. For clarity, we refer to the MBow framework introduced in the last section 'MEow (Vanilla)'.
5

---

Learnable Reward Shifting (LRS). Reward shift-
(8)
Juochian determinant product of the non linear trandoemations
(b)
Jucohian determinnnt product of the linrar trnnffemmtionns
ing [5T is technique for shaping the reward func-
tion. This technique enhances the learning process
50613
Venta
1.00
by incorporating a shifting term in the reward func- tion, which leads to a shifted optimal soft Q-function in MaxEnt RL. Inspired by this, this work proposes modeling a reward shifting function by S - I with a neural network to enable the automatic learning of reward shifting term. For notational simplicity, the parameters are denoted using 8, and the details of the architecture are presented in Appendix(A.5.1 The soft Q-function Q3 augmented by be is defined as follows:
4.013
10k-1
3.e13
1.0e2
2.0e33
1.0c3
Lie13
1.0c4
0.0
0.2
Stope(1a5)
0.6
0.6 0.8
1.0
0.0
0.2
24
(1c))
0.5
1.
Figure 1: The Jacobian determinant products for (a) the non-linear and (b) the linear trans- formations, evaluated during training in the Hopper-v4 environment. Subfigure (b) is pre- sented log scale for better visualization. This experiment adopt the affine coupling layers [47 as the nonlinear transformations.
@h ax) (s a+) + (s.).
(12)
The introduction of Q% results in a correspond- ing shifted soft value function (5)(s) A
= (o(ss) + (s:) (i.e., Proposition A.3 in Appendix A.2). which can be calculated without Monte Carlo estimation. Moreover, with the incor- poration of bø. the policy Th remains invariant since
a log
a.) - (s)))) =
( ax) + hg(s,) - (Ve(s,) + (8{))))) = ( a,) vo(s))), which allows the use of 90 for efficiently sampling actions. As evidenced in Fig. this method effectively addresses the issues of the significant growth and decay of Jacobian determinants of go (discussed in Appendix (..)). In Section 4.4 we further demonstrate that the performance of MEow can be significantly improved through this technique.
Shifting-Based Clipped Double Q-Learning (SCDQ). As observed in [66]. the overestimation of value functions often occurs in training. To address this issue, the authors in (66] propose clipped double Q-learning, which employs two separate Q-functions and uses the one with the smaller output to estimate the value function during training. This technique is also used in MaxEnt RL frameworks [913). Inspired by this and our proposed learnable reward shifting, we further propose a shifting- based method that adopts two learnable reward shifting functions, ((1) and ( without duplicating the soft O-function Qo and soft value function V,, defined by go. The soft Q-functions
Q and !² with corresponding learnable reward shifting functions ((()) and 61²) can be obtained using Eq. 122 while the soft value function Vellip is written as the following formula:
(s:), velli (s.) min (Vo(s.) + ( Vo(s.)
(s:))) Vo(ss)+ (6) (s.), 68² (13)
This design also prevents the production of two policies in MEow, as having two policies can complicate the inference procedure. In our ablation analysis presented in Section 4.4 we demonstrate that this technique can effectively improve the training process of MEow.
Deterministic Policy for Inference. As observed in [8) deterministic actors typically performed better as compared to its stochastic variant during the inference time. Such problem can be formalized as finding an action a that maximizes Q(s,, a) for a given s, Since A is continuous space, finding such value would require extensive calculation. In the MEow framework, this value the model architecture construction. Our key observation is
can be derived by making assumptions that if the Jacobian determinants of the non- non-lincarities (i.c., Se E S,) are constants with respect to its inputs, and that argmax, (z(z) can be directly obtained, then the action that maximizes ((s,, a) can be efficiently derived according to the following proposition.
Proposition 3.2. Given that (Jt 9b (a*=)|s:)] is a constant with respect to a then 9o = (argmaxx Pa (2))s)) argmax, Qu(S,, a).
The proof is provided in Appendix A.2 It is important note that g can still be non-linear transformation, given that det(J oú (a {t)] is a constant. To construct such model, a Gaussian prior with the additive coupling transformations [46] can be used non-linearities. Under such design, an action can be derived by calculating 9ø (H](s)), where μ represents the mean of the

---

Algorithm Pseudo Code of the Training Process of MEoww
Input: Learnable parameters 0 and shadow parameters 0' Target smoothing factor Learning rate
Neural networks go(- [)), ()),), and (.)) (-). Temperature parameter Discount factor
1: for each training step do
2: Extend the Replay Buffer.
3: a. 90 (z)ss), (x(-).
4: S:41 ~ pr (-|s:; ac).
D DU [(ss, Tr,
6: Update Policy.
7:
(1,41) ~ D.
Qo(s;, ar) = a log (go (ar Iees det([J (a,
(a;-111) (s:)))
Eq.
(8((841) log Ih,s det(J of (8t-1))l.
Eq.
( ar) Qo(s., a.) b, (s:) Q
0 (s., a+) = Qo(s: as) (s.).
Eq.
11:
(8.41) = (sr(1)))) + min (0 (s441) (s.41)
Eq. 13
12:
C(0) (s)
(st, (r+ + ( (Q) (st, (st, (r+ + vo
(st+1) (sc+1)))
())) Eq.
04
13:
0 e- 0 + (V)(()).
14: 8' + (1 r) +0..
15: end for
Gaussian distribution. We elaborate on our model architecture design Appendix|A.5.1 A.5.1| and provide performance comparison between MEow evaluated using stochastic policy (i.e., a, ne( (s:)) and a deterministic policy (i.e., a = argmax, Qe(s,, a)) in Section 4.4
3.3 Algorithm Summary
We summarize the training process of MEow in Algorithm [ The algorithm integrates the policy evaluation steps with the policy improvement steps, resulting single loss training process. This design differs from previous actor-critic frameworks, which typically perform two consecutive updates each training step. In Algorithml [] the learning rate denoted as B. set of shadow parameters #' is maintained for calculating the delayed target values [67], and is updated according to the Polyak averaging (68 of 8, i.e., 0' (1 T)0' + 70, where is the target smoothing factor.
Experiments
In the following sections, we first present intuitive example of MEσw trained in two-dimensional multi-goal environment ( in Section 4. We then compare MEow's performance against several continuous-action RL baselines in five MuJoCo environments 32 33 in Section 4.2 Next, in Sec- tion 4.3 we evaluate MEow's 's performance on number of Omniverse Isaac Gym environments (34) simulated based on real-world robotic application scenarios. Lastly, in Section 4.4 we provide an ablation analysis to inspect the effectiveness of each proposed technique. Among all experiments, we maintain the same model architecture, while adjusting inputs and outputs according to the state space and action space for each environment. We construct using the additive coupling layers (46) with element-wise linear transformations, utilize a unit Gaussian as Pz. and model the learnable adaptive reward shifting functions by as multi-layer perceptrons (MLPs). For detailed descriptions of the experimental setups, please refer to Appendix A.3
4.1 Evaluation on Multi-Goal Environment
In this subsection, we present an example of MEow trained in a two-dimensional multi-goal en- vironment 88) The environment involves four goals, indicated by the red dots in Fig. (a). The reward function is defined by the negative Euclidean distance from each state to the nearest goal, and the corresponding reward landscape is depicted using contours in Fig. 2 (a). The gradient map in Fig. 2 (a) represents the soft value function predicted by our model. The blue lines extending from the center represent the trajectories produced using our policy.

---

MEow
SAC
SQL
PPO
TD3
DDPG
Hopper-e4
helttChettah.v4
Walker2d-vd
Ant-vá
Humanoid-va
15:00
6306
7500
200
30.00
5000
3530
L500
4500
500
5000
3305
1000
400
240
2004
120
10:0
280
1000
-100
23
20
1.0
Steps (55)
Staps (1o6)
Steps (1e6)
Steps (106)
Sieps (1o6)
Figure 3: The results in terms of total returns versus the number of training steps evaluated on five MuJoCo environments. Each curve represents the mean performance, with shaded areas indicating the 95% confidence intervals, derived from five independent runs with different seeds.
As illustrated in Fig. 2 (a), our model's soft value function predicts higher values around
(0)
(b)
++-XI CIT
the goals, suggesting successful learning of the goal positions through rewards. In addi- tion, the trajectories demonstrate our agent's s correct transitions towards the goals, which val- idates the effectiveness of our learned policy. To illustrate the potential impact of approx- imation errors that might emerge when em- ploying previous soft value estimation meth- ods, we compare three calculation methods for the soft value function: (I) Our approach (i.e., Eq. (1b)::
40-
Estionation Ervor
21,655
60
20.700
40
3.0
69
2000
4,000
6,000
5,000
1890
5.0
6.0
4.0 3.0
Figure (a) The soft value function and the trajec- tories generated using our method on the multi- environment. (b) The estimation error evaluated at the initial state under different choices of M.
(s:), (II) SQL-like (i.e.,
EM1 exp(Qu(s,. Ma)
Eq. (6)): a' log( J7
log (a sampled from Tg. The approximation errors of the soft value functions at the initial state are calcu- lated using the Euclidean distances between (I) and (II), and between (I) and (III), for various values of M. As depicted in Fig. 2 (b), the blue line and the orange line decreases slowly with respect to M. These results suggest that Monte Carlo estimation converges slowly, making approximation methods such as Eqs. (6) and challenging to achieve accurate predictions.
and (III) SAC-like (i.e., Eq. D MV M EE (Qo(s,, a(!)
[s))),
where (a()) J==1 is
4.2 Performance Comparison on the MuJoCo Environments
In this experiment, we compare MEow with several commonly-used continuous control algorithms on five MuJoCo environments (32 from Gymnasium [33] The baseline algorithms include SQL (8).
SAC [9. deep deterministic policy gradient (DDPG) [69 twin delayed deep deterministic policy gradient (TD3) [66], and proximal policy optimization (PPO)
The results for SAC, DDPG, TD3, and PPO were reproduced using Stable Baseline 3 (SB3) 71 utilizing SB3's s refined hyperparameters. The results for SQL were reproduced using our own implementation, as SB3 does not support SQL and the official code is not reproducible. Our implementation adheres to SQL's original paper. Each method is trained independently under five different random seeds, and the evaluation curves for each environment are presented in the form of the means and the corresponding confidence intervals.
As depicted in Fig. 3 MEow performs comparably to SAC and outperforms the other baseline algorithms in most of the environments. Furthermore, in environments with larger action and state dimensionalities. such as 'Ant-v4' and 'Humanoid-ves -v"", MEow offers performance improvements over SAC and exhibits fewer spikes in the evaluation curves. These results suggest that MEow is capable of performing
sigh-iimnniioall tasks with stability. To further investigate the performance difference between MEow and SAC, we provide a thorough comparison between MEow, SAC (9). Flow-SAC TO III and their variants Appendix A.442 The results indicate that the training process involving policy evaluation and policy improvement steps may be inferior to our proposed training process with a single objective. the next subsection, we provide performance examination using the simulation environments from the Omniverse Isaac Gym (34)
8

---

MEow
SAC
Ani (Issse)
Humanoid (Isanc)
Ingenuity
(Isanc)
Humanoid (Isaac)
Ingenuity
4000
Return
3000
1000
1000
1200
3.0
(1c5)
755
18.0
50
100
1.35
state dim. 660 action dim.. $
static dim. 10N action dim. 21
state dim. action dim.
(N-128)
(28)
(N--128)
ANYymal
750
AllegroHand
FrunkaCabinet
ANYmal
AllegroHand
FrankaCabinet
390
Y00
Retum
150
1000
Steps (1e5)
Steps (1e5)
55 100
Steps (le5)
stati dim. action dim. 12
stafic dim. 72
state dim. action dim.
(M--113)
action dim. 16
Figure 4: comparison on six Isaac Gym environ- ments. Each curve represents the mean performance of five runs, with shaded areas indicating the 95% con- fidence intervals. 'Steps' in the x-axis represents the number of training steps, each of which consists N parallelizable interactions with the environments.
Figure 5: A demonstration of the six Isaac Gym environments introduced in Section 4.3 The dimensionalities of the state and action for each environment are denoted below each subfigure.
4.3 Performance Comparison on the Omniverse Issac Gym Environments
In this subsection, we examine the performance of MEow on a variety of robotic tasks simulated by Omniverse Isaac Gym (34. a GPU-based physics simulation platform. In addition to 'Ant' and 'Humanoid', we employ four additional tasks: 'Ingenuity', 'ANYmal', 'AllegroHand' and 'Franka- Cabinet'. All of them are designed based on real-world robotic application scenarios. 'Ingenuity' and 'ANYmal' are locomotion environments inspired by NASA's Ingenuity helicopter and ANYbotics industrial maintenance robots, respectively. On the other hand, 'AllegroHand" and "FrankaCabinet" focus on executing specialized manipulative tasks with robotic hands and arms, respectively. A demonstration of these tasks is illustrated in Fig. 5
In this experimental comparison, we adopt SAC as a baseline due to its excellent performance in the MuJoCo environments. The evaluation results are presented in Fig. 4 The results demonstrate that MEow exhibits superior performance on 'Ant (Isaac)' and 'Humanoid (Isaac)'. In addition, MEow consistently outperforms SAC across the four robotic environments (i.e., 'Ingenuity'. 'ANYmal', 'AllegroHand' and "FrnkaCabinet"" ), indicating that our algorithm possesses the ability to perform challenging robotic tasks simulated based real-world application scenarios.
4.4 Ablation Analysis
In this subsection, we provide an ablation analysis to examine the effectiveness of each technique
introduced in Section 3.2
Training Techniques. Fig. @oompares the performance three variants MEow: 'MEow (Vanilla)', 'MEow (+LRS)', and 'MEow (+LRS & SCDQ)', across five MuJoCo environments. The results show that 'MEow (Vanilla)' consistently underperforms, with its total returns demonstrating negligible or no growth throughout the training period. In contrast, the variants incorporating translation functions demonstrate significant performance enhancements. This observation highlights the importance of including by in the model design. In addition, the comparison between 'MEow (+LRS)' and 'MEow (+LRS & SCDQ)' suggests that our reformulated approach to clipped double Q-learning 66 improves the final performance by noticeable margin.
Inference Technique. Fig. compares the performance of two variants of MEow: 'MEow (Stochas- tic)' and 'MEow (Deterministic)" The former samples action based a, ~ TT(- while the latter derive action according to a, argmax, Qu(st, a) 9p - (uls.). As shown in the figure, MFoww with deterministic policy outperforms its stochastic variant, suggesting that a deterministic policy may be more effective for MEow's inference.

---

MEow (+LRS SCDQ)
MEow (+LRS)
(Vanilla)
Hopper-v4
HallChhetta--44
Walker2d-u4
Ant-vé
Humamoid-o4
30.01
4500
3000
100
1500
Stups: (1o6)
Stups (1v6)
(1o6)
Steps Stups(lo6) (1o6)
Stups (166)
Figure 6: The performance comparison of MEow's 's variants (i.c., 'MEow (Vanilla)', 'MEow (+LRS)', and 'MEow (+LRS & SCDQ)") five MuJoCo environments, Each curve represents the mean performance of five runs, with shaded areas indicating the 95% confidence intervals.
MPow (Deterinnistic)
MFow (Stachastie)
Hoppos-y4
HallCheazah-vd
Walker24-44
Ant-v4
Humanoid-v4
1250
1.50
-090
100:
180
300
-500
150
3000
100
500
100
1500
10
00
1.3 20
40
6.2
26 30
Stops (1e6)
Steps (1e6)
Steps (1e6)
Steps (1e6)
Steps (Le6)
Figure 7: Performance comparison between MEow with a deterministic policy and MEow with stochastic policy on five MuJoCo environments. Each curve represents the mean performance of five runs, with shaded areas indicating the 95% confidence intervals.
5 Conclusion
In this paper, we introduce MEow, a unified MaxEnt RL framework that facilitates exact soft value calculations without the need for Monte Carlo estimation. We demonstrate that MEow can be optimized using a single objective function, which streamlines the training process. To further enhance MEow's performance, based clipped double Q-learning, into the design. We examine the effectiveness of MEow via experiments conducted in five MoJoCo environments and six robotic tasks simulated by Omniverse Isaac Gym. The results validate the superior performance of MEow compared to existing approaches.
incorporate two techniques, learnable reward shifting and shifting-
Limitations and Discussions
As discussed Section 3.2 deterministic policies typically offer better performance compared to their stochastic counterparts. Although our implementation of MEow supports deterministic inference, this capability based on the assumptions that the Jacobian determinants the non-linear transformations constants with respect to their inputs, and that argmax, Pz(z)) (z) be efficiently derived. These assumptions may not hold for certain types of flow-based models. Therefore, exploring effective architectural choices for MEow represents promising direction for further investigation.
On the other hand, the training speed MEow is around 2.3x slower than that of SAC, even though updates according to L(Ø) are bypassed in MFow. According to our experimental observations, the computational bottleneck of MEow may lie in the inference speed of the flow-based model during interactions with environments. While this speed is significantly faster than many iterative methods, such as MCMC or variational inference, it still slower compared to the inference speed of Gaussian models. As a result, enhancing the inference speed of flow-based models represents potential avenue for further improving the training efficiency of MEow.
Finally, our hyperparameter sensitivity analysis, presented in A.4.5 indicates that our current approach requires different values of T to achieve optimal performance. Since hyperparameter tuning often demands significant computational resources, establishing more generalized parameter setting or developing an automatic tuning mechanism for presents an important direction for future exploration.
10

---

Acknowledgement
The authors gratefully acknowledge the support from the National Science and Technology Council (NSTC) in Taiwan under grant numbers MOST L11-2223-E-002-00 10----0-030--11 NSTC 113-2221-E-002 212-MY3, and NSTC 113-2640-E-002--00 The authors would also like to express their appreciation for the computational resources from NVIDIA Corporation and NVIDIA AI Technology Center (NVAITC) used in this work. Furthermore, the authors extend their gratitude to the National Center for High-Performance Computing (NCHC) for providing the necessary computational and storage resources.
References
[1] HJ Kappen. Path Integrals and Symmetry Breaking for Optimal Control Theory. Journal of Statistical Mechanics: Theory and Experiment, 2005.
[2] Brian Ziebart, Andrew Maas, J. Bagnell, and Anind Dey. Maximum Entropy Inverse Rein- forcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2008.
[3] Mare Toussaint. Robot Trajectory Optimization using Approximate Inference. In Proceedings of the International Conference on Machine Learning (ICML), 2009.
[4] Brian D. Ziebart. Modeling Purposefuul Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, USA, 2010.
[5] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. In Proceedings of the International Joint Conference Artificial Intelligence (IJCAI), 2012.
[6] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the Noise in Reinforcement Learning via Soft Updates. In Proceedings of the Conference on Uncertainty Artificial Intelligence (UAI), 2016.
[7] Brendan O' O'Donoghue, Rémi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. PGQ: Com- bining Policy Gradient and Q-learning. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[8] Tuomas Haarnoja, Haoran Tang, Picter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Ban Based Policies. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
[9] Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
[10] Tuomas Haarnoja, Kristian Hartikainen, P. Abbeel, and Sergey Levine. Latent Space Policies for Hierarchical Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). 2018.
[11] Bogdan Mazoure, Thang Doan, Audrey Durand, R Devon Hjelm, and Joelle Pineau. Leveraging Exploration in Off- offfppoicy Algorithms via Normalizing Flows. In Proceedings of the Conference on Robot Learning (CoRL), 2019.
[12] Patrick Nadeem Ward, Ariella Smofsky, and A. Bose. Improving Exploration in Soft-Actor- Critic with Normalizing Flows Policies. 2019.
[13] Dinghuai Zhang, Aaron Courville, Yoshua Bengio, Qinqing Zheng, Amy Zhang, and Ricky Q Chen. Latent State Marginalization as a Low-cost Approach to Improving Exploration. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[14] Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang. Bo An, Haipeng Chen, and Sanjay Chawla. S2AC: Energy-Basse Reinforcement Learning with Stein Soft Actor Critie. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
11

---

[15] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2021.
[16] Wenjie Shi, Shiji Song, and Cheng Wu. Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning. In Proceedings the International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[17] Benjamin Eysenbach and Sergey Levine. Maximum Entropy RL Provably Solves Some Robust RL Problems. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[18] Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a Posteriori Policy Optimisation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[19] Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning.
ArXiv, abs/1902.00017, 2019.
[20] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft Actor-Critic Algo- rithms and Applications. ArXiv, abs/1812.05905, 2018.
[21] Kwan-Woo Woo Park, MyeongSeop Kim, Jung-Su Su Kim, and Jac-Han Park. Path Planning for Multi-Arm Manipulators Using Soft Actor-Critic Algorithm with Position Prediction of Moving Obstacles via LSTM. Applied Sciences, 2022.
[22] Junior Costa de Jesus, Victor Augusto Kich, Alisson Henrique Kolling, Ricardo Bedin Grando, Marco Antonio de Souza Leite Cuadros, and Daniel Fernando Tello Gamarra. Soft Actor-Critic for Navigation of Mobile Robots. Journal of Intelligent and Robotic Systems, 2021.
[23] Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, and Fu Jie Huang. A Tutorial Energy-Based Learning. 2006.
[24] Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 1996.
[25] Gareth O. Roberts and Jeffrey Rosenthal. Optimal scaling of discrete approximations to Langevin diffusions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1998.
[26] Qiang Liu and Dilin Wang. Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm. In Proceedings of the International Conference Neural Information Processing Systems (NeurIPS), 2016.
[27] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. 2015.
[28] Malvin H. Kalos and Paula A. Whitlock. Monte Carlo methods. Vol. 1: basics. Wiley- Interscience, 1986. ISBN 0471898392.
[29] Surya T. Tokdar and Robert E. Kass. Importance Sampling: A Review. Wiley Interdisciplinary Reviews: Computational Statistics, 2, 2010.
[30] Michael B. Giles. Multilevel Monte Carlo Methods. Acta Numerica, 24:259 328, 2013.
[31] Chen-Hao Chao, Wei-Fang Sun, Yen-Chang Hsu, Zsolt Kira, and Chun Yi Lee. Training Energy- Based Normalizing Flow with Mcore-Matching Objectives. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2023.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of the International Conference on Intelligent Robots and Systems (IROS), 2012.
12

---

[33] Mark Towers, Jordan K. Terry. Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, 2023. HtRL.https:////
[34] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, N. Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. Proceedings of the International Conference Neural Information Processing Systems (NeurlPS) Dataset and Benchmark Track, 2021.
[35] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Progrumming. John Wiley & Sons, Inc., 1994. ISBN 0471619779.
[36] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, 2018. ISBN 0262039249.
[37] Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynam- ics. In Proceedings of the International Conference on Machine Learning (ICML), 2011.
[38] Iman Nematollahi, Erick Rosete-Beas, Adrian Roefer, Tim Welschehold, Abhinav Valada, and Wolfram Burgard. Robot Skill Adaptation via Soft Actor-Critic -ritic Gaussian Mixture Models. Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2022.
[39] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. 2013.
[40] Dilin Wang and Qiang Liu. Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning. Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[41] Michel Dekking. A Modern Introduction Probability and Statistics. 2007.
[42] George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research (JMLR), 2019.
[43] Mathieu Germain, Karol Gregor, lain Murray, and H. Larochelle. MADE: Masked Autoencoder for Distribution Estimation. 2015.
[44] Diederik P. Kingma, Tim Salimans, and Max Welling. Improved Variational Inference with In- verse Autoregressive Flow. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2016.
[45] George Papamakarios, Iain Murray, and Theo Pavlakou. Masked Autoregressive Flow for Density Estimation. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2017.
[46] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. Workshop at the International Conference on Learning Representations (ICLR), 2015.
[47] Laurent Dinh, Jascha Narain Sohl-Dickstein, and Samy Bengio. Density Estimation using Real NVP. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[48] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convo- lutions. Proceedings of the International Conference on Neural Information Processing Systems (NeurlPS), 2018
[49] Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus H. Gross, and Jan Novák. Neural Importance Sampling. ACM Transactions on Graphics (TOG), 2018
[50] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural Spline Flows. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2019.
13

---

[51] Emiel Hoogeboom, Rianne van den Berg, and Max Welling, Emerging Convolutions for Generative Normalizing Flows. Proceedings of the International Conference on Machine Learning (ICML), 2019.
[52] Xuezhe Ma and Eduard H. Hovy. MaCow: Masked Convolutional Generative Flow. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2019.
[53] You Lu and Bert Huang. Woodbury Transformations for Deep Generative Flows. Proceedings of the International Conference on Neural Information Processing Systems (NewrlPS), 2020.
[54] Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, and Stefano Ermon. ButterfyyFlow Building Invertible Layers with Butterfly Matrices. Proceedings of the International Conference Machine Learning (ICML), 2022.
[55] L. Gresele, G. Fissore, A. Javaloy, Schölkopf, and A. Hyvärinen. Relative Gradient Optimiza- tion of the Jacobian Term in Unsupervised Deep Learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
[56] T. Anderson Keller, Jorn W. T. Peters, Priyank Jaini, Emiel Hoogeboom, Patrick Forr'e, and Max Welling, Self Normalizing Flows. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[57] A. Hyvärinen and E. Oja. Independent Component Analysis: Algorithms and Applications. Neural Networks: the Official Journal of the International Neural Network Society, 13 4-5: 411-30 11-30, 2000.
[58] A. Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research (JMLR), 2005.
[59] Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey E. Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research (JMLR), 4: 1235-1260, 2003.
[60] Will Grathwohl, Kuan-Chieh Jackson Wang, Jörn-Henrik Jacobsen, David Kristjanson Du- venaud, and Richard S. Zemel. Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling. Proceedings of the International Conference on Ma- chine Learning, 2020. URtptttps: :/api. eeemanticsche car org/
[61] Jette Randløv and Preben Alstrøm. Learning to Drive a Bicycle Using Reinforcement Learning and Shaping. In Proceedings of the International Conference on Machine Learning (ICML), 1998.
[62] A. Ng. Daishi Harada, and Stuart Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the International Conference Machine Learning (ICML), 1999.
[63] Adam Laud. Theory and Application of Reward Shaping in Reinforcement Learning. 2004.
[64] Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma. Jian Guo, and Bolei Zhou. Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping. In Proceedings of the
International Conference on Neural Information Processing Systems (NeurIPS), 2022.
[65] Zhe Zhang and Xiaoyang Tan. Adaptive reward shifting based behavior proximity for offline reinforcement learning. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2023.
[66] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
[67] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Ku- maran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control through Deep Reinforcement Learning, Nature, 518:529-523 5333, 2015.
14

---

[68] Xiang Li, Wenhao Yang, Jiadong Liang, Zhihua Zhang, and Michael I. Jordan. A Statisti- cal Analysis of -urpektt Averaged Q-Learning. In Proceedings of the International Conference Artificial Intelligence and Statistics (AISTAT), 2021.
[69] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Proceedings of the International Conference Learning Representations (ICLR), 2016.
[70] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal Policy Optimization Algorithms. ArXiv,
2017.
[71] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable- Balele-Bssin:::: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research (JMLR), 22(268):11 2021.
[72] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. 2019.
[73] Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for Activation Functions. 17X15:1710.05941, 2017.
[74] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. abs/
Layer Normalization. ArXiv,
2016.
[75] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. ArXiv, abs/1207.0580. 2012.
[76] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine Learning Research (JMLR), 23 1274):1-18, 2022.
[77] Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernández-Lobato. normflows: A PyTorch Package for Nor- malizing Flows. Journal of Open Source Software, 8(86):5361, 2023.
[78] Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[79] Antonio Serrano-Muñoz. Dimitrios Chrysostomou, Simon Bøgh, and Nestor Arana- Arexolaleiba. skrl: Modular and Flexible Library for Reinforcement Learning. Journal of Machine Learning Research (JMLR), 94 2023.
15

---

A Appendix
In this Appendix, we begin with discussion of the soft value estimation methods used in SQL and SAC in Section A.1 We then derive number of theoretical properties of MEow in Section A.2 Next, we discuss the issue of numerical instability Section A.3] Then, we present additional experimental results in Section A.4 and summarize the experimental setups in Section A.5 Finally, we elaborate on the potential impacts of this work in Section A.6
A.1 The Soft Value Estimation Methods in SAC and SQL
In this section, we elaborate on the soft value estimation methods mentioned in Section 2.2 We first show that (s,) approximated using Eq. (6) is greater than that approximated using Eq. 7 for any given state 8, in Proposition A. and Remark A.2 Then, we discuss their practical implementation.
Proposition A.I. For any St E E and a E Rso, the following inequality holds:
exp (Qa(s;, a)/a)
a' log R Raw
2 Bawny (8o a) log (a)ss)))
(A1)
Ta(a)):)
Proof.
exp (Qo(st,a)/0) (a(a)s))
(6)
2 aBanzo log (exp (xxp(Q((s),a)/0) a)/aa (a(a))))
log Eawns
alEa~T [Qa(s a)/a a log T (a/s:)]
EawTs [Qo(s;, a) log (a|s:))
where (1) is due to Jensen's 's inequality.
Remark A.2. The inequality in Eq. Al
preserves after applying the Monte Carlo estimation. Namely,
a:log 1
exp (Qo(s,, faa)
1 M
Σ
(Qo(s), a log (a si)).
(A2)
M
(a(1 st)
where (a()) MM Ji=1 represents set of samples drawn from Tg.
Unlike the estimation in Eq. the estimation in Eq. 6 is guaranteed to converge to (s())) as M + co. However, empirically, the estimation method in Eq. (7) is preferred and widely used the contemporary MaxEnt framework. One potential reason could be the required number of samples needed for effective approximation. According to [8) M = 32 is an effective choice for Eq. (6). whereas M works well for Eq. (7). adopted by many previous works. 12 14).
A.2 Theoretical Properties of MEow
In this section, examine number of key properties of the MEow framework. We begin by presenting a proposition to verify MEow's capability in modeling the soft Q-function and the soft value function. Then, present proposition to derive deterministic policy in MEow. Finally, offer a discussion of the impact of incorporating learnable reward shifting functions.
Proposition 3.1 Eq. 17 satisfies the following stutements: (1) Given that the Jacobian of go is non-singular, (S() R and Qo(si, ar) R Va, E Vs. E S. (2) Volst)
log eex (s; a)/a) da.
Proof. (1) Given that the Jacobian of ga is non-singular, Vo(s:) E R and Qa(s;, ar) E R, Va, e A, Vs, S.
the Jacobian of go is non-singular, then both Ihes, |ett[J 9i (s:)) E Ryo and lless, det oj (s:)) Ezo. This suggests that a log Iliss, det(J (sr)) R and log Pa (a (s)) l.ss [et((J (s:)) e R. As result, Vo(s;) ER and Qo a.) R.
16

---

(2) Vo(s)) log exp (Qo(s), a)/s a)/a) da.
= / r(als))da= = / (go(a)))))
II dot (J (a
II iESt
dct ob (sx)) da.
iES,
II VES
det(J ob (8.1)) = / Pa (go(a))e))
II iES..
det (Joz (a (aa))) da.
+
+ a log
II JEE
det(J u3 (Bii)
= a log / Ps (go(a)))))
II det (Jos (at-1(s))) da
kES.
a log II
det(J o5 (sx)) = a logg / Pz (go(aals))
II
det (J (a (a' (11) da
iES;
iES,
+ Vo(ss) a log / exp (Qo(s,, a)/a) da.
In Section
3.2 we demonstrate that argmax, (s(s;, a) can be efficiently obtained through So (argmax, Pa (z)]. To provide theoretical support for this result, we include proof for Propo- sition 3.2
Proposition 3.2 Given that dett(J 3/ (s))l a constant with respect to a a'-1, then Oo (argmax, Pa (2)]s:) argmax, (s a).
Proof. Let c 2 Iliss, ddet(J (s:))
argmax (s(s a log Pa (go(a))))) iES.
II
det (Jog (a--)(s))
argmax a log (Px (go (alss)))
argmax Pa (al())
argmax Pa (go (a(s:)
(86)
ar argmax Ps (z)
(ais)
(argmax Pz (z) (s;),
where (i) is because logarithm is strictly increasing, (ii) is due galss)) and (fii) is due 90 (z]s:)-
In Section 3.2 we incorporate the learnable reward shifting function by in Qo and Vg. This incorpora- tion results redefined soft Q-function Q and soft value function Vo. In Proposition A.3. we verify that V/ (s:) a log exp(Q"(ss, a)/o Vo(s:) + bo(s.).
Proposition A.3. Given that Q and V satisfy (s::) A a log exp (Q(st, a)/a) da. The augmented functions, (s(, a.) Q(s,, a.) + b(se) and (>(ss) a log exp(Q" a)/a)da where b(s)) is the the reward shifting function, satisfy (s,) (s.)) b(s)).
17

---

Proof.
(s;) a log
exp a))/a) da.
(fexp((Q( a) b(s:)) /a) da
a log
exp(b(s:)/o) /
=a log
exp (Q(s,, a) da exp a))a) da + a log (exp(b(s:)/a))
=oloo
= =a log
(Q a)/aa da ((s.)
Vsss) b(s)).
A.3 The Issue of Numerical Instability
In this section, we provide the motivation for employing the learnalee reward shifting function described in Section 3.2 We show that while Qo and Vo defined in Eq. (TT) have the theoretical capa- bility to learn arbitrary real values (i.e., Proposition...1 3.1). they may experience numerical instability in practice. This instability arises due to the exponential growth Ihes.. det sj "r)=)sss)] and the exponential decay of T,Es det(( oi (s:))]- We first examine the relationship between (s.) and Iiees det(J 9b (sx)) according to the following equations:
(s.)) log II det (Joz (8t) KES,
exp( vo((-V(s:))
II det (J) SES:
The above equation suggests that the value of Hies, |ett(J (8:)) decreases exponentially with respect to (st), which may lead to numerical instability during training. On the other hand, the relationship between (s as) and Hies., det(J og (af [ss))] can be expressed according to the following equations:
Qo(se, a.) log (go (ac (st)) + log II det (J (ai 1(s:)
KES.
Qø(se, a.) log (a)(s))):) log I det Jug (af ((se))
ieS
+ exp (Qa(s), a.) - log (gn(ar(s:))) = II det (J (atfaa11s)) (ss)
LES.
The equations indicate that Higs, dJt gj (ai [st))] (s))) increases exponentially with respect to Qs a.) - log Pv. (so(aa/s)) (s.)). Therefore, increasing Qo may also lead to an exponential growth of likes, det([ (|)))
[st))]-
LRS makes our model less susceptible to numerical calculation errors since the learnable reward shifting function be. unlike Qo and Vg. is not represented logarithmic scale. Consider a case where FP32 precision is in use, 'MEow (Vanilla)' could fail to learn target Vg- (st) ) > 38, Vs,, since Ihes. det(J o3 (s:)) = exp(-V,- (s:)) 2- -126 cannot be represented using FP32 precision. Therefore, without shifting the reward function, the loss sometimes becomes undefined values, and can lead to ineffective training (e.g.. the green lines in Fig. 6). The reward shifting term can be designed as (state-conditioned) function or (non-state-condiiionee)) value. It can also learnable or non-learnable. All of these designs (i.e., (s.)). b(st). bo. and b) can be directly applied to MEow since none of them influences the action distribution. Based on our preliminary experiments, we identified that learnable state-conditioned reward shifting delivers the best performance.
18

---

MBow (Additiv)
MEow (Affine)
Hopper-e4
helttChettab-v4
Walker2d-vd
Ant-vá
Humanoid-va
17500
6000
1903
4000
430
600
3000
2000
2003
2000
500
190
1000
1800
1000
015
Staps (le6)
60
Steps (1e6)
13
Steps (1o6)
20
Steps((oo5) (1o5)
20
Stops (1o6)
20
2.
Figure Al: Performance comparison between MEow with additive coupling transformations in go and MEow with affine coupling transformations in ge on five MuJoCo environments. Each curve represents the mean performance, with shaded areas indicating the 95% confidence intervals, derived from five independent runs with different seeds.
MEow
Enengy Critic Garessian Actor
Energy Chitic- Fllw
Flow CriteeGGussian
Flow Criti-Flow
Hopper-x4
HalfChoetab-va
Walker2d-od
(000)
7.00
Rotern
6691
-100
3090
1100
2100
1000
4399
003
Steps (1c6)
Steps (166)
Steps (1e6)
Steps (106)
Steps (lef)
Figure A2: Performance comparison between 'MEow', 'Energy Critic+Gaussian Actor' (ECGA), Energy Critic+Flow Actor' (ECFA), 'Flow Critic+Gaussian Actor' (FCGA), and 'Flow Critic+Flow Actor' (FCFA) five MuloCo environments. Each curve represents the mean performance, with shaded areas indicating the 95% confidence intervals, derived from five independent runs with different seeds.
A.4 Supplementary Experiments
In this section, we provide additional experimental results. In Section A.4.1
we offer a comparison
between MEow with Zo modeled using additive coupling layers and that using affine coupling layers. In Section A...2 we compare the performance of MEow with four distinct types of actor-critio frameworks formulated based on prior works 1111. Section A.4.3] we provide an example illustrating the ability of flow-based models to represent multi-modal distributions as policies. In
Section A.4.4 we present performance comparison between SAC and variant with LRS. Finally,
in Section A.4.5 we provide sensitivity examination for the target smoothing parameter.
A.4.1 Comparison of Additive and Affine Transformations
In this section, evaluate the performance of MEow with two commonly-adopier non-linear transformations, additive 46) and affine [47] coupling layers, for constructing ga- The results are presented in Fig. AT The results show that MEow with additive coupling layers achieves better performance than that with affine coupling layers. Based on this observation, we adopt additive coupling layers for constructing go throughout the experiments in Section of the main manuscript.
A.4.2 Influences of Parameterization in MaxEnt RL Actor-Critic Frameworks
In this section, we compare the performance of MEow against four different actor-critic frameworks formulated based on prior works 19111. The first framework is the same as SAC D. with the critic modeled as an energy- energy-based model and the actor as a Gaussian. The second framework follows the approaches of [I]. where the critic is also an energy- based model, but the actor is flow-based model. The third and fourth frameworks both utilize a flow-based model for the critic, with the actor modeled denoted as: 'Energy Crasic+Gaussian Actor' (ECGA), 'Energy Critie+Flow Actor' (ECFA), 'Flow Critic+Gaussian Actor' (FCGA), and FFow Criw Actor' (FCFA), respectively. Regarding the
Gaussian and a flow-based model, respectively. These frameworks are
19

---

SAC (+LRS)
SAC
Hopper-v4
helttChettab-v4
Walker2d-vd
Ant-vá
Humanoid-va
1005
16.00
6000
140
$500
-000
600
3000
100
2004
250
3000
40
1600
50
Staps (le6)
174
07
Steps (1e6)
000
Steps (1o6)
Steps
103
Stops (1o6)
2.0
3.0
Figure A4: Performance comparison between 'SAC' and 'SAC (+LRS)'. Each curve represents the mean performance, with shaded areas indicating the 95% confidence intervals, derived from five independent runs with different seeds.
soft value calculation during training, the first and second frameworks adopt the value estimation method in SAC (i.e., Eq. (7). For the third and the fourth frameworks, their training adopts the exact value calculation (i.e., Eq. Ti). which the same as MEow. The results are presented Fig. A2
As depicted in Fig. A2 MEow exhibits superior performance and stability compared to other actor-critic frameworks in the 'Hopper-v4', 'Ant-v4', and "Walker2d-v4" environments, and shows comparable performance with ECGA in most environments. In addition, the results that compare the frameworks with flow-based models as actors (i.c., ECFA and FCFA) to those with Gaussians as actors (i.e., ECGA and FCGA) suggest that Gaussians more effective for modeling actors. This finding is similar to that in [T]. On the other hand, the comparisons between ECGA and FCGA, and between FCGA and FCFA, do not show clear trend. These findings suggest that both flwwbased and energy-based models can be suitable for modeling the soft Q-function. Furthermore, the comparison between FCFA and MEow reveals that the training process involving alternating policy evaluation and improvement steps may be inferior to our proposed training process with a single objective.
A.4.3 Modeling Multi-Modal Distributions using Flow-based Models
In this section, we use a one-dimensional ex- ample demonstrate that flow-based models are capable of learning multi-modal action distributions. We employ a condecoondi neural spline flow (NSF) [50] as the model, and train it in single-step environment with one-dimensional state and action spaces. Fig.
(a)
(b)
Action
A3 (a) illustrates the reward landscape with the state and action denoted on the x- axis and y-axis, respectively. Fig. A3 (b) lustrates the probability density function (pdf) predicted by the model. The result demon-
State
State
Figure A3: (a) The reward landscape of the one-step environment described Section A.4.3 (b) The conditional pdf prediction using an NSF model.
strates the capability of flow-based models to effectively learn multi-moda. distributions.
A.4.4 Applying Learnable Reward Shifting to SAC
In this section, examine the performance of SAC with the proposed LRS technique. Since the original implementation of SAC involves the clipped double Q-Learning technique, SAC with LRS is equivalent to SAC with the shifting-basee clipped double Q-Learning (SCDQ) technique discussed in Section 3.2 of the main manuscript. The performance of 'SAC' and 'SAC (+LRS)' is presented in Fig. A4 The results indicate that applying LRS does not improve SAC's performance. Therefore, for a fair evaluation, the original implementation of SAC is adopted in the comparison in Section 4 of the main manuscript.
A.4.5 Sensitivity Examination for the Target Smoothing Parameter
In this section, we provide a performance comparison of SAC and MEow trained with different target smoothing parameter values (i.c., 0.005, 0.003, 0.0005, and 0.0001). The results shown in
20

---

0.005
0.003
0,0005
0.0001
Hoppes-s4
HallCTioctah-od
Walker-y4
Ant-v4
Husianod-vvl
2090
7030
11800
MEow
350
2000
150
2000
10300
1.X
25
25
30
50 Stops (1oo5)
Steps (1e6
Steps (1)6
Steps (1ed)
Steps (1
SAC
200C
1300
2000
10:0
1900
Stepss (le6)
Stags (1ef)
Steps (1e5)
Stepp (1e6)
Steps (1e5)
Figure A5: A performance comparison between MEow and SAC under different trained on five MuloCo environments. Each curve represents the mean performance, with shaded areas indicating the 95% confidence intervals, derived from five independent runs with different seeds.
Fig. A5 indicate that SAC performs the best when 0.005, while MEow requires different values to achieve good performance across different tasks. Although both algorithms exhibit significant performance variations with different 7 values, SAC demonstrates more consistent trend in terms of the total returns among the tested values of T.
A.5 Experimental Setups
In this section, elaborate the experimental configurations and provide the detailed hy- perparameter setups for the experiments presented Section 4 of the main manuscript. The code is implemented using PyTorch [72) and is available in the following repository: hhtpps //github. Serm/ChienFeenn hong-hubbmeoss
A.5.1 Model Architecture
Among all experiments presented in Section of the main manuscript, maintain the same model architecture, while adjusting inputs and outputs according to the state space and action space for each environment. An illustration of this architecture is presented in Fig. A6 The model architecture comprises three main components: (I) normalizing flow, (II) hypernetwork, and (III) reward shifting function. For the first component, the transformation go includes four additive coupling layers [46] followed by an element-wii wise linear layer. The prior distribution Pz is modeled as unit Gaussian. For the second component, the hypernetwork involves two types of multi-layer perceptrons (MLPs), labeled as (a) and (b) in Fig. A6 which produce weights for the non-linear and linear transformations, respectively. Both MLPs employ swish activation functions 773 and have hidden layer size of 64. The MLPs labeled as (a) incorporate layer normalization 77 and a dropout layer 75 with
dropout rate of 0.1. For the third component, the reward shifting functions (i.e., 16") and b(²2) implemented using MLPs with swish activation and a hidden layer size of 256. The parameters used in these components function L() defined in Eq. (4), with the soft Q-function and the soft value function replaced by Qj and V&. respectively. Please note that, for the sake of notational simplicity and conciseness, the parameters of each network are all represented using instead of distinct symbols (e.g., O Ban-b). fun-a), and (un)))))
collectively referred to 0, and are optimized using the same objective
A.5.2 Experiments on the Multi-Goal Environment
The experiments in Section 4.1 are performed on two-dimensional multi-goal environment [8] This environment consists of four goals positioned at [0, 5), (0, -5), [5. 0). and [-5, 0). denoted as gı, g2. g3. and g1. respectively. The reward is the sum of two components, r (st) and r2(a,), which are
21

---

(III
216
Reward Shifting Functions
(a ax)
296
(sx))
bi
Qa ar)
Hypernetwork
Drognu
Diegouu
Norm
Nom
Normalizing Flow
AUARY
MARKNN
(b)
Q0
Liwar
Linexr
Prior
Figure A6: The architecture adopted in MEow. This architecture consists of three primary components: (I) normalizing flow, (II) hypernetwork, and (III) reward shifting function. The hypernetwork includes two distinct types of networks, labeled (a) and (b), which are responsible for generating weights for the non-linear and linear transformations within the normalizing flow, respectively. Layer normalization is denoted 'L. Norm' in (a).
Table Al: Shared hyperparameters of MEow.
Table A2: Shared hyperparameters of SAC.
Parameter
Value
Parameter
Value
optimizer
Adam
optimizer
Adam 78
learning rate (8)
0.001
learning rate (0)
0.0003
gradient clip value
30
gradient clip value
discount (7)
0.99
discount (?)
0.99
buffer size
10
buffer size
10
formulated follow:
(st) max ||s gill and r2(ar) -30 ||ael|
(A3)
According to Eq. A3 ri (St) encourages policies to reach states near the goals. On the other hand, ra:): encourages policies to produce actions with small magnitudes.
In this experiment, we adopt temperature parameter = 2.5, target smoothing factor r = 0.0005, learning rate 8 0.001, discount factor y 0.9, and a total of 4, 000 training steps. The computation was carried out on NVIDIA TITAN V GPUs equipped with 12GB of memory. The training takes approximately four minutes.
A.5.3 Experiments on the MuJoCo Environments
Software and Hardware Setups. For the experiments on the MuJoCo environments, our imple- mentation is built on CleanRL (76), with the normalizing flow component adapted from [77]. The computation was carried out on NVIDIA V100 GPUs equipped with 16GB memory. The training takes approximately 13 hours per 1 million steps, with each GPU capable of executing four training sessions simultaneously.
Hyperparameter Setups. The shared and the environment-specife hyperparameters of MEow are summarized Tables [A and A3 respectively. The hyperparamers for the baseline methods are directly borrowed from Stable Baseline 3 (SB3) [71].
22

---

Table A3: A list of environment-speciffc hyperparameters used in MEow.
Enviromment
Target Smoothing Parameter (r) Temperature Parameter (a)
Hopper-v4
0.005
0.25
HalfChectah-v4
0.003
0.25
MuJoCo
Walker2d-v4
0.005
0.1
Ant-v4
0.0001
0.05
Humanoid-v4
0.0005
0.125
Ant
0.0005
0.075
Humanoid
0.00025
0.25
Ingenuity
0.0025
0.025
Omniverse Isaac Gym
ANYmal
0.025
0.00075
AllegroHand
0.001
0.1
FrankaCabinet
0.075
0.1
Table A4: A list of environment-sspeciiee hyperparameters used in SAC.
Environment
Target Smoothing Parameter (7)
Temperature Parameter (a)
Ant Humanoid
0.0025
0.4
0.0025
0.025
Ingenuity
0.0025
0.1
Omniverse Isaac Gym
ANYmal
0.0025
AllegroHand
0.0025
0.1
FrankaCabinet
0.025
0.1
A.5.4 Experiments on the Omniverse Isaac Gym Environments
Software and Hardware Setups. For the experiments performed on Omniverse Isaac Gym, the implementation built on SKRL 779 due to its compatibility with Omniverse Issac Gym 34 The computation was carried out NVIDIA L40 GPUs equipped with 48GB of memory. The training takes approximately 22 hours per million training steps, with each GPU capable of executing three training sessions simultaneously. For 'Ant', 'Humanoid', 'Ingenuity', and 'ANY mal', each training step consists of 128 parallelizable interactions with the environments. For 'AllegroHand" and 'FrankaCabinet,, each training step consists of 512 parallclizable interactions with the environments.
Hyperparameter Setups. The shared and the environment-specife hyperparameters of MEow summarized in Tables Al and A3, respectively. Those of SAC are summarized in Tables A2 and A4 respectively. Both SAC and MEow were tuned using the same search space for and fair comparison. Specifically, a grid search was conducted with 7 values ranging from 0.1 to 0.00025 and a values from 0.8 0.0005 for both algorithms. The setups with the highest average return were selected for each environment.
ensure
A.6 Broader Impacts
This work represents new research direction for MaxEnt RL. It discusses a unified method that can be trained using single objective function and can avoid Monte Carlo estimation in the calculation of the soft value function, which addresses two issues in the existing MaxEnt RL methods. From practical perspective, our experiments demonstrate that MEow can achieve superior performance compared to widely adopted representative baselines. In addition, the experimental results conducted in the Omniverse Isaac environments show that our framework can perform robotic tasks simulated based on real-world application scenarios. These results indicate the potential for deploying MEow in real robotic tasks. Given MEow's 's potential to be extended to perform challenging tasks, it is unlikely to have negative impacts on society.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and Section accurately reflect the contribu- tions of this paper. The theoretical results are discussed in Section The experiments in Section 4 provide empirical justification for these claims. Finally, Section S summarizes both the theoretical and empirical contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations this work are discussed in the main manuseript. The discussion covers the assumptions made this paper and the computational efficiency of the proposed framework.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions and the proofs of our theoretical results are presented in detail in Appendices A.3
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully disclose all the information needed to reproduce the experimen- tal results. The experimental configurations, detailed hyperparameter setups. and hardware requirements for the experiments presented in this paper are elaborated in Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
25

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and installation instructions are available in an anonymous repository, with the link provided in Appendix A.5
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
Sttpsi//uuidess//deeuueee wwt.ssubbiiiiiooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/puble wwtpccpwwwpcpcudeeseespppppppppe htt.guidddde//eecmmmmmcceae sondesuumisssioPoll11e
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental configurations, detailed hyperparameter setups, and hardware requirements for the experiments presented in this paper elaborated in Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
A1 and A2 present the mean performance, with the shaded areas indicating the 95% confidence intervals. Each of them is derived from five independent runs with different seeds.
Justification: The evaluation curves in Figs. 4
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The hardware requirements (e.g., the computational hardware configurations
and the execution time) for each experiment are elaborated in Appendix A.5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All authors have reviewed the NeurIPS Code of Ethics and confirmed that the research conducted in this paper complies with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses its potential impacts in Appendix A.6
27

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no risk for misuse (e.g., pretrained language models, image generators, or scraped datasets).
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators of assets are properly credited through citations, and the licence is included in the asset (see Appendix A.5)
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperssit paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code, installation instructions, and running commands are summarized in an anonymous repository, with the link provided Appendix [A.5 The environments are publicly available, and the experiments performed on them with the default setup.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "WUKOul1XQV", "reviewer": "Reviewer_he3t", "summary": "The paper presents a new method for MaxEnt RL based on the recently proposed energy-based normalizing flows (EBFlow). The adoption of EBFlow allows us to overcome two major issues with training MaxEnt RL algorithms: (I) sampling from an energy-based model (of the policy) and (ii) approximating the soft value function (with MC methods).", "questions": "* Typo at line 149, \"Jocobian\"\n* I am not sure if the role of the Learnable Reward Shifting (LRS) is clear to me. The authors say that they were inspired by the reward-shifting literature. However, I don't see the LRS term working the same way as classical reward shaping, as a state-dependent non-linear function ($b_\\theta$), without any particular constraints (I don't see any stated) wouldn't preserve the properties of the original functions. I think the role of the learnable reward shifting is to have some form of \"baseline\" in the Q and V functions that simplifies the estimation process (since it's using a more flexible learning architecture), and so it avoids the explosion of the Jacobian. Could the authors provide clarification on this?\n* I would be interested in seeing a comparison between the learned reward shifting term and the value function (V) in a plot (it would suffice to see this in the Appendix)", "limitations": "Limitations are addressed in Appendix. I think the authors should mention this in the main text.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* **Contribution**: I think the contribution of this work is significant and novel. The issues discussed with MaxEnt RL are important. The idea proposed is novel, it naturally emerges from the EBFlow framework and it's seamlessly applied to the MaxEnt RL domain. The approach solves the problems mentioned and it demonstrates superior performance compared to other standard approaches.\n* **Presentation**: the presentation is clear and well-organised, and the method is described in detail.", "weaknesses": "* **New and old challenges**: while the approach resolves some issues with the current MaxEnt RL methods, it still suffers from typical deep RL issues (e.g. overestimation, for which the authors proposed SCDQ) and it introduces additional challenges (e.g. exploding Jacobian determinants, for which the authors introduce LRS). Normalizing flows also introduce additional constraints on the network's architecture, a problem which I believe is strongly mitigated by the introduced LRS (?)\n* **Required clarification on LRS**: see Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Questions**\n\n**Q1.** Typo at line 149, \"Jocobian\".\n\n**Response:** Thank you for pointing this out. We will correct the typo in the final revision.\n\n---\n\n**Q2.** I am not sure if the role of the Learnable Reward Shifting (LRS) is clear to me. The authors say that they were inspired by the reward-shifting literature. However, I don't see the LRS term working the same way as classical reward shaping, as a state-dependent non-linear function ($b_\\theta$), without any particular constraints (I don't see any stated) wouldn't preserve the properties of the original functions. I think the role of the learnable reward shifting is to have some form of \"baseline\" in the Q and V functions that simplifies the estimation process (since it's using a more flexible learning architecture), and so it avoids the explosion of the Jacobian. Could the authors provide clarification on this? \n\n**Response:** Based on our formulation, the original $Q_\\theta$ and $V_\\theta$ functions are the residuals between their respective targets and $b_\\theta$. We agree with the reviewer that this property of $b_\\theta$ helps $Q_\\theta$ and $V_\\theta$ in predicting their targets during the optimization process.\n\nAnother important aspect of LRS is that it reduces the magnitude of the soft value function $V_\\theta$, which makes our model less susceptible to numerical calculation errors. An analysis of this issue is provided in Section A4. This problem arises due to limited numerical precision. Consider a case where FP32 precision is in use, `MEow (Vanilla)’ could fail to learn a target $V_\\{\\theta^{\\*}\\} (s_t)>38, \\forall s_t$, since $\\exp(-V_\\{\\theta^{\\*}\\}(s_t))=\\Pi_\\{i \\in S_l\\} \\| \\det (J_\\{g_\\{\\theta^{\\*}\\}^i\\}(s_t)) |<2^{-126}$ cannot be represented using FP32 precision. Therefore, without shifting the reward function, the loss sometimes becomes undefined values, and can lead to ineffective training (e.g., the green lines in Fig. 6). The reward shifting term can be designed as a (state-conditioned) function or a (non-state-conditioned) value. It can also be learnable or non-learnable. All of these designs (i.e.,  $b_\\{\\theta\\} (s_t)$,  $b\\(s_t\\)$,  $b_\\{\\theta\\}$,  and $b$) can be directly applied to MEow since none of them influences the action distribution. Based on our preliminary experiments, we identified that a learnable state-conditioned reward shifting worked the best and proposed this technique.\n\n---\n\n**Q3.** I would be interested in seeing a comparison between the learned reward shifting term and the value function (V) in a plot.\n\n**Response:** We appreciate the reviewer's suggestion. The plot is offered in Fig. 4 of the attached PDF file in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjOPpjc4jT", "reviewer": "Reviewer_HRPY", "summary": "The paper introduces a new MaxEnt RL framework called Meow based on Energy-based Normalizing Flow, which integrates the policy evaluation steps and the policy improvement steps and results in a single objective training process. Besides Meow enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation and supports the modeling of multi-modal action distributions.", "questions": "1.\tHow is the training time of Meow compared with other methods?\n2.\tHow is Meow compared with other advanced RL methods with the utilization of generative-model-based policy?\n3.\tCould the authors explain why Meow can learn a good reward shifting rather than a trivial shifting, like for any $s_t$, $b(s_t)=0$?", "limitations": "The authors have clearly presented the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed method is innovative and interesting, which utilizes the special property of Energy-based Normalizing Flow, and thus enables the consistency between the actor and critic and a single objective training process.\n2.\tThe paper is overall well-written. The background and related works are introduced appropriately.\n3.\tThe experimental results are comprehensive, involving the mujoco and Issac gym benchmarks.", "weaknesses": "1.\tThe calculation of the derterminant is usually time-consuming. So the reviewer wonders how the training time of Meow is compared with that of the classical off-policy RL methods like SAC.\n2.\tIn the experiments, Meow is just compared with several classical RL methods for continuous control and lacks the comparison with advanced RL methods [1, 2] with other generative models like diffusion model and consistency model, which also supports the modeling of multi-modal action distribution.\n\n[1] Yang L, Huang Z, Lei F, et al. Policy representation via diffusion probability model for reinforcement learning[J]. arXiv preprint arXiv:2305.13122, 2023.\n\n[2] Yue Y, Kang B, Ma X, et al. Boosting offline reinforcement learning via data rebalancing[J]. arXiv preprint arXiv:2210.09241, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Comments**\n\n**C1.** **(1)** The calculation of the determinant is usually time-consuming. **(2)** So the reviewer wonders how the training time of MEow is compared with that of the classical off-policy RL methods like SAC. \n\n**Response:** \n**(1)** Several existing normalizing flow architectures (e.g., [1] used in this work) support efficient calculation of Jacobian determinants, as discussed in Section 2.2 (Lines 138-140). In general, the Jacobian determinant of these architectures can be calculated with time complexity $O(D^2L)$, where $D$ represents the dimension of actions and $L$ is the number of layers in $g_\\theta$ (defined in Line 132). This complexity is the same as forward passing a normalizing flow model.\n\n**(2)** A discussion regarding the training time of MEow and SAC is offered in Section A.7. Although the training time of our method is longer than that of SAC, SAC can not model multi-modal action distribution. For a fair comparison, we compared the training time of SAC-Flow (i.e., [2]) and MEow. We found that the training time of SAC-Flow is 1.13x slower than MEow due to the additional policy improvement updates (i.e., Eq. (5)).\n\n[1] Dinh et al. NICE: Non-linear Independent Components Estimation, ICLR Workshop 2015.\\\n[2] Haarnoja et al. Latent Space Policies for Hierarchical Reinforcement Learning, ICML 2018.\n\n---\n\n**C2.** In the experiments, MEow is just compared with several classical RL methods for continuous control and lacks the comparison with advanced RL methods [1, 2] with other generative models like diffusion model and consistency model, which also supports the modeling of multi-modal action distribution. \n\n**Response:** We appreciate the reviewer’s suggestion and have included a performance comparison between MEow and DIPO [1] in Table 1 in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). The results show that MEow outperforms DIPO, even when DIPO uses 100 sequential forward passes during action sampling.\n\nTo the best of our knowledge, consistency models have not been applied to online RL setups. In addition, combining consistency models with online MaxEnt RL may be challenging due to the intractability of the entropy calculation.\n\nOn the other hand, reference [2] addresses offline RL tasks, which differ from the online RL setting discussed in this paper. The primary distinction is that in offline RL, agents cannot interact with the environment during training. Therefore, reference [2] represents a separate research direction and is distinct from the main focus of this paper.\n\nFinally, we would like to highlight that several online MaxEnt RL methods developed for modeling multi-modal policies are discussed in Section 2.2 and compared in Section 4.2. We use a number of representative MaxEnt RL methods, such as SQL, as our baselines in Fig. 3. We also compare our method with more advanced variants like SAC-Flow (i.e., [3]), denoted as ECFA in Fig. A3. Furthermore, Table 1 in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp) demonstrates that our method outperforms the latest MaxEnt online RL framework (i.e., S2AC [4]), where S2AC also supports modeling multi-modal action distributions.\n\n[1] Yang et al. Policy representation via diffusion probability model for reinforcement learning, 2023. \\\n[2] Yue et al. Boosting offline reinforcement learning via data rebalancing, 2022.\\\n[3] Haarnoja et al. Latent Space Policies for Hierarchical Reinforcement Learning, ICML 2018.\\\n[4] Messaoud et al. S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor-Critic, ICLR 2024.\n\n---\n\n**Questions**\n\n**Q1.** How is the training time of MEow compared with other methods?\n\n**Response:** The training time comparison is discussed in the response of C1.\n\n---\n\n**Q2.** How is MEow compared with other advanced RL methods with the utilization of generative-model-based policy? \n\n**Response:** Please refer to the responses to C2 and Table 1 in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). \n\n---\n\n**Q3.** Could the authors explain why MEow can learn a good reward shifting rather than a trivial shifting, like for any $s_t$, $b(s_t)=0$?\n\n**Response:** We appreciate the reviewer's question regarding MEow's ability to learn a good reward shifting rather than a trivial one. The inclusion of a learnable reward shifting term serves a crucial purpose: it prevents numerical errors in the Jacobian determinant calculations, as discussed in Section 3.2 and analyzed in Appendix A4. The efficacy of this approach is demonstrated in Fig. 1 of the main manuscript, where we observe the successful resolution of these numerical issues. Furthermore, Fig. 6 in the manuscript illustrates that this method contributes to improved overall performance. These results collectively indicate that MEow learns a meaningful and beneficial reward shifting, rather than a trivial one. \n\nIn addition, according to our observation, $b_\\theta$ does not learn to be a constant. Its value changes according to different visited states. To verify this, we sample a trajectory of states and plot the corresponding values of $b_\\theta$ for the Hopper-v4 environment during the inference phase in Fig. 5 of the attached PDF file in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vU87a3sY9Z", "reviewer": "Reviewer_UfJ6", "summary": "The paper presents a new framework for Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) using Energy-Based Normalizing Flows (EBFlow). Traditional MaxEnt RL methods, particularly for continuous action spaces, typically utilize actor-critic frameworks and alternate between policy evaluation and policy improvement steps. The proposed EBFlow framework integrates these steps into a single objective, eliminating the need for Monte Carlo approximation while enabling efficient modeling of multi-modal action distributions. The framework was experimentally validated on the MuJoCo benchmark and high-dimensional robotic tasks in Omniverse Isaac Gym, demonstrating superior performance compared to existing baseline methods.", "questions": "- GFlowNet points out that the MaxEnt RL paradigm has deficiencies in terms of diversity. Specifically, MaxEnt RL tends to achieve target states that are \"further\" from the initial state. Does MEow also inherit this defect of the MaxEnt RL paradigm?\n- Energy-based normalizing flow is a generative model, and its original loss is to maximize log-likelihood. In Algorithm 1, this loss does not seem to be present. In my view, MEow uses the network structure of energy-based normalizing flow, but the loss still follows the MaxEnt RL paradigm. In this case, could the title of this paper be potentially misleading?\n- Figure A5 shows that the MEow network uses a hypernetwork module. Why was this module introduced? Is this a common architecture for energy-based normalizing flow?\n- What are the learning dynamics observed during the training process? Are there any notable differences in the convergence behavior compared to traditional actor-critic methods?\n- How does the proposed framework scale with increasing complexity of the environment or the state-action space? Are there any specific limitations or challenges observed during experimentation?\n- How robust is the EBFlow framework to different hyperparameters?", "limitations": "The author has already discussed the limitations ofthe method and its potential impacts in the paper, and provided possible solutions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- By combining policy evaluation and policy improvement into a single training process, the proposed method simplifies the optimization process and avoids potential optimization errors inherent in alternating updates.\n- The use of normalizing flows facilitates efficient sampling and exact calculation of probability density functions, which addresses the inefficiencies associated with Monte Carlo methods in traditional MaxEnt RL approaches.\n- Experimental results indicate that the EBFlow framework outperforms widely-adopted baselines on both the MuJoCo benchmark and complex robotic tasks, showcasing its effectiveness and robustness.", "weaknesses": "- There are some unreasonable aspects in the organization of the content. For example, Section 3.2 appears somewhat abrupt; it is suggested to move the content of A.4 into the main text. The discussion about inference in Section A.3 is also very important and should be included in the main text.\n- The experimental section does not sufficiently highlight the advantages of the MEow method. The experiments in Section 4.1 on toy examples are not enough. Can comparisons be made from the perspectives of inference time, training time, representation capability, etc.?\n- While the integration of policy evaluation and improvement steps simplifies the training process conceptually, the implementation details of the EBFlow framework may introduce additional complexity that needs to be managed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Comments**\n\n**C1.** There are some unreasonable aspects in the organization of the content. For example, Section 3.2 appears somewhat abrupt; it is suggested to move the content of A.4 into the main text. The discussion about inference in Section A.3 is also very important and should be included in the main text. \n\n**Response:** We appreciate the reviewer's suggestion and would be glad to incorporate the contents of Sections A.3 and A.4 into the main manuscript to improve the paper's quality. We will implement this arrangement in the final revision.\n\n---\n\n**C2.** The experimental section does not sufficiently highlight the advantages of the MEow method. The experiments in Section 4.1 on toy examples are not enough. Can comparisons be made from the perspectives of inference time, training time, representation capability, etc.?\n\n**Response:** We appreciate the reviewer’s suggestion and have expanded the experiments in Section 4.1 to include a comparison of inference and training times in the Humanoid environment (MuJoCo benchmark), as shown in the following table. The results demonstrate that performing Monte Carlo estimation and MCMC sampling at each step imposes a computational burden during both the training and inference phases.\n\n||Training time (sec.)|Inference time (sec.)| \n|-|-|-|\n|MEow|0.022 $\\pm$ 0.000|0.004 $\\pm$ 0.000|\n|MC-based  ($M=10$)|0.110 $\\pm$ 0.000|0.040 $\\pm$ 0.004|\n|MC-based  ($M=100$)|1.064 $\\pm$ 0.001|0.388 $\\pm$ 0.004|\n|MC-based  ($M=200$)|3.181 $\\pm$ 0.012|0.760 $\\pm$ 0.004|\n\n**Table.** Training and inference time comparison. $M$ denotes the number of samples used for soft value estimation and the number of steps in the MCMC sampling process. The results are evaluated on NVIDIA V100 GPUs.\n\nRegarding the representation capability of our method, we have provided an analysis in Section A.5.4. The results indicate that our method possesses the ability to model multi-modal action distributions.\n\n---\n\n**C3.** While the integration of policy evaluation and improvement steps simplifies the training process conceptually, the implementation details of the EBFlow framework may introduce additional complexity that needs to be managed.\n\n**Response:** The implementation of EBFlow can be simplified by integrating existing normalizing flow model libraries (e.g., [1,2]). Since log determinant calculation is already supported by many normalizing flow packages, the only required modification is the addition of an if-else condition during the forward pass to separate the energy function and the normalizing constant. We have made our code implementation available in an anonymous repository, with the corresponding link provided in Lines 699-700 of Section A.6.\n\n[1] Stimper et al. normflows: A PyTorch Package for Normalizing Flows, Journal of Open Source Software 2023.\\\n[2] Durkan et al. nflows: normalizing flows in PyTorch. 2020.\n\n---\n\n**Questions**\n\n**Q1.** GFlowNet points out that the MaxEnt RL paradigm has deficiencies in terms of diversity. Specifically, MaxEnt RL tends to achieve target states that are \"further\" from the initial state. Does MEow also inherit this defect of the MaxEnt RL paradigm? \n\n**Response:** We would like to highlight that the primary focus of this paper is on investigating the parameterization and its implications on MaxEnt RL. Comparing different RL frameworks (e.g., GFlowNet versus MaxEnt RL) falls outside the scope of this work. We acknowledge that further investigation into the topic of diversity within this context is an interesting direction for future research, and welcome any additional references the reviewer may provide to enrich this potential line of inquiry. \n\n---\n\n**Q2.** Energy-based normalizing flow is a generative model, and its original loss is to maximize log-likelihood. ... but the loss still follows the MaxEnt RL paradigm. In this case, could the title of this paper be potentially misleading? \n\n**Response:** We believe that our paper title accurately reflects the key concept of the proposed methodology. As discussed in EBFlow’s original paper [1], a normalizing flow can be represented as an energy-based model and optimized using energy-based training objectives. Specifically, the authors in [1] employ score matching (which minimizes Fisher divergence) as an alternative to maximum likelihood training (which minimizes KL divergence) to optimize EBFlow. MEow can be viewed as an extension of EBFlow to the MaxEnt RL domain, where a policy is also represented as an energy-based model [2]. By synthesizing these concepts, we parameterize the policy using an EBFlow and demonstrate that the policy can be optimized by minimizing the soft Bellman errors.\n\n[1] Chao et al. Training Energy-Based Normalizing Flow with Score-Matching Objectives. NeurIPS 2023.\\\n[2] Haarnoja et al. Reinforcement Learning with Deep Energy-Based Policies. ICML 2017.\n\n---\n\n**Q3.** Figure A5 shows that the MEow network uses a hypernetwork module. Why was this module introduced? Is this a common architecture for energy-based normalizing flow?\n\n**Response:** Hypernetworks are commonly employed in conditional normalizing flow models for modeling the weights in the transformations (see [1] for more details). In our approach, the policy is modeled as a state-conditioned normalizing flow, with hypernetworks utilized to encode state information. As for implementation, we adopt the existing implementation of conditional normalizing flow [2], which incorporates a hypernetwork architecture.\n\n[1] Kobyzev et al. Normalizing Flows: An Introduction and Review of Current Methods. TPAMI 2019.\\\n[2] Stimper et al. normflows: A PyTorch Package for Normalizing Flows, Journal of Open Source Software 2023.\n\n---\n\nThe responses to **Q4**-**Q6** are provided in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cva110P0nD", "reviewer": "Reviewer_PzAH", "summary": "This paper proposes to use the energy-based normalizing flow (EBFlow) to represent the policy and value functions in maximum entropy (MaxEnt) RL. Specifically, the paper builds the connection between EBFlow and MaxEnt RL by linking the conditional unnormalized density that depends on the input to the action value function Q and the constant that is independent of the input to the value function V. As a consequence, the Boltzmann policy defined by Q and V can be expressed exactly using a mapping in EBFlow. By updating the current Q estimate to approximate the action value of the Boltzmann policy, the algorithm would spontaneously update the Boltzmann policy. Hypothetically, the algorithm should converge to the optimal policy under entropy regularization. Using an existing implementation of EBFlow in the relevant literature, the paper proposed MEow based on this formulation. Experiments on four MuJoCo environments and six Issac Gym environments demonstrate the learning performance of MEow.", "questions": "Some questions that may affect the evaluation:\n1. Are deterministic policies used for other algorithms in the evaluation in Figures 3 and 4?", "limitations": "The limitations of the work are discussed in the appendix, which should be moved to the main text. Another unmentioned potential limitation is the sensitivity to the target network smoothing parameter.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strengths of this paper are as follows:\n1. The connection between EBFlow and MaxEnt RL revealed in this paper is very interesting and, to my knowledge, novel. Given that MaxEnt is a popular approach in the RL literature, the issue addressed in this paper, learning the action value function and the Boltzmann policy, as well as the proposed approach, will be interesting to a broad set of audiences in the RL community.\n2. The paper includes abundant comparative and ablation experiments. Notably, the ablation study covers the effect of the two components in the practical implementation, the difference between the deterministic and stochastic evaluation, etc.\n3. The paper is well-written and mostly clear.", "weaknesses": "1. There is a potential concern about the empirical investigation: The proposed method is tuned over the target smoothing parameter, while the baselines are not tuned. Why is the target smoothing parameter tuned per environment? What would happen if the proposed method also used the same, fixed smoothing parameter?\n2. This is a minor point. The organization of the paper can be improved. Specifically, the background section is too long, which takes up the space for more interesting discussions presented in the appendix. If space allows, the main text should include the network architecture in Figure A5, as it may be unfamiliar to a lot of readers. In addition, the limitation, especially the computational aspect of the proposed approach, should be discussed in the main paper.\n\nOther suggestions that do not affect the evaluation:\n* It would be great if there is at least some qualitative comparison between the proposed method and baselines using MaxEnt RL in the multi-goal environment.\n* Typos between Lines 254 and 276: The reference to Figure 1 should be to Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Comments**\n\n**C1.** **(1)** There is a potential concern about the empirical investigation: The proposed method is tuned over the target smoothing parameter, while the baselines are not tuned. **(2)** Why is the target smoothing parameter tuned per environment? What would happen if the proposed method also used the same, fixed smoothing parameter? \n\n**Response:** \n**(1)** Our baselines use the refined hyperparameters of Stable Baseline 3 (SB3) as mentioned in Line 285. Further modifications to the hyperparameters may not result in performance improvement. For a fair evaluation, we provide the results of SAC (the best-performing baseline) tuned with different target smoothing factors used in MEow (i.e., $\\tau=0.005$, $0.003$, $0.0005$, and $0.0001$) in Fig. 1 of the attached PDF file in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). The results show that SB3’s original hyperparameter ($\\tau=0.005$) is the most effective.\n\n**(2)** We appreciate the reviewer's question regarding the target smoothing parameter $\\tau$. According to Fig. 1 in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp), our current approach requires different values of $\\tau$ to achieve good performance. However, we acknowledge the importance of determining a more generalized parameter setting. As part of our future work, we plan to expand our experimental framework to explore a broader range of $\\tau$ values to identify a generally applicable $\\tau$ across diverse environments. \n\n---\n\n**C2.** This is a minor point. The organization of the paper can be improved. Specifically, the background section is too long, which takes up the space for more interesting discussions presented in the appendix. If space allows, the main text should include the network architecture in Figure A5, as it may be unfamiliar to a lot of readers. In addition, the limitation, especially the computational aspect of the proposed approach, should be discussed in the main paper. \n\n**Response:** We appreciate the reviewer's suggestion and concur that a more concise Section 2, coupled with the inclusion of discussions on MEow's model architecture and limitations in the main manuscript, will enhance the paper's quality. These modifications will be incorporated in the final revision.\n\n---\n\n**C3.** Other suggestions.\n**(1)** It would be great if there is at least some qualitative comparison between the proposed method and baselines using MaxEnt RL in the multi-goal environment. \n**(2)** Typos between Lines 254 and 276: The reference to Figure 1 should be to Figure 2.\n\n**Response:**\n**(1)** We appreciate the reviewer's suggestion. A qualitative comparison among MEow, SAC, and SQL in the multi-goal environment is presented in Fig. 2 of the attached PDF file in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp).\n**(2)** We appreciate the reviewer’s attention to detail. The typo and figure reference will be corrected in the final revision.\n\n---\n\n**C4.** **(1)** The limitations of the work are discussed in the appendix, which should be moved to the main text. **(2)** Another unmentioned potential limitation is the sensitivity to the target network smoothing parameter.\n\n**Response:** \n**(1)** We express our gratitude to the reviewer for the suggestions. A discussion of this work's limitations will be incorporated into the main manuscript in the final revision. **(2)** An examination of the sensitivity to the target network smoothing parameter is presented in Fig. 1 of the attachment to the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). This analysis will be included in the paper's final revision.\n\n---\n\n**Questions**\n\n**Q1.** Are deterministic policies used for other algorithms in the evaluation in Figures 3 and 4?\n\n**Response:** During the evaluation phase, MEow, SAC, TD3, DDPG, and PPO employ deterministic policies. SQL, however, does not support deterministic inference and thus utilizes a stochastic policy for inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3ymvShfKe", "reviewer": "Reviewer_S3sW", "summary": "This paper proposes a new framework for Maximum Entropy Reinforcement Learning using Energy-Based Normalizing Flows (EBFlow). The authors argue that this framework offers three main advantages: Firstly, the soft value function can be obtained without estimation. Second, the framework combines policy evaluation and policy improvement in one iteration. Finally, it does not use Monte Carlo methods and interacts with the environment effectively. To achieve this goal, Learnable Reward Shifting (LRS) technique which involves reward shifting for the value function using a reward shifting network and Shifting-Based Clipped Double Q-Learning (SCDQ) technique that employs two reward shifting networks for the value function and utilizes the minimum value function similar to double Q-learning have been proposed.", "questions": "1. Please answer the questions in the weakness section\n\n2. According to the appendix, the proposed method is said to be 2.3 times slower than SAC. Given this difference in speed, a time-wise comparison of the algorithm's performance would be very helpful to understand its efficiency relative to SAC. \n\n3. In Figure 3, SAC and MEow show similar performance in the Mujoco domain, but there is a noticeable difference in performance in the Isaac Gym domain in Figure 4. Could you provide a further analysis on the reasons for the occurrence of this difference? It would provide valuable insight into the strengths and limitations of the proposed method.\n\nMinor Issues\\\nTypo in line 149: \"Jocobian\" should be \"Jacobian.\"\\\nMisreference in section 4.1: It should refer to Fig. 2, not Fig. 1.", "limitations": "They have handled it well.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured, and the overall content is easy to follow and intuitive. The detailed background explanations allow readers to progress through the main text with a solid understanding of the prior work.\n\n2. The experimental domains are diverse, clearly demonstrating the strengths of the proposed algorithm in an easy-to-understand manner. The experiments are well-structured, and the detailed explanations make it easier for readers to follow the setup.\n\n3. The proposed algorithm does not use Monte Carlo-based methods, allowing it to select the proper action for training during each step without waiting for the end of an episode. This enables the policy to interact with the environment more effectively\n\n4. Existing SAC-based algorithms often rely on value function estimation, which can introduce errors. The proposed algorithm can obtain the soft value function without estimation, thus avoiding such errors.", "weaknesses": "1. Despite the fact that the proposed method is the first to apply EBFlow to reinforcement learning (RL), the novelty of the algorithm may appear weak, as there are several existing RL methods based on flow-based models. Besides applying EBFlow to the RL structure, were there are any additional considerations or advantages that were taken into account to enhance its performance or applicability in the context of RL? If such points exist, highlighting them would help to emphasize the contributions of the authors' proposed technique.\n\n2. Introducing new frameworks generally necessitates providing sufficient mathematical proof to validate their efficacy, even for basic techniques or simple conjectures. It would be beneficial if the paper included mathematical proofs demonstrating that the proposed framework converges to the optimal policy. For example, the techniques presented in the paper, namely Learnable Reward Shifting (LRS) and Shifting-Based Clipped Double Q-Learning (SCDQ), have a significant impact on performance. However, the paper does not provide adequate mathematical justification for these techniques. Including such proofs would greatly enhance the confidence in the proposed method.\n\n3. It would be beneficial for the paper to compare its methods with more recent state-of-the-art algorithms. The comparison with SAC, published in 2018, feels a bit outdated. Given that reinforcement learning is a rapidly evolving field, including comparisons with more recent methods would provide a clearer understanding of where this framework stands in the current landscape. Such comparisons would greatly enhance the paper by placing the proposed method in the context of the latest advancements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s time and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Comments**\n\n**C1.** Despite the fact that the proposed method is the first to apply EBFlow to reinforcement learning (RL), the novelty of the algorithm may appear weak, as there are several existing RL methods based on flow-based models. ... If such points exist, highlighting them would help to emphasize the contributions of the authors' proposed technique. \n\n**Response:** As described in Lines 41-53 of Section 1 and Lines 166-174 of Section 3 of our paper, the proposed framework possesses the following two unique features. To the best of our knowledge, none of the existing MaxEnt RL frameworks, including those with flow-based models (e.g., [1-3]), possess the following characteristics:\n\n- Our framework enables the exact calculation of the soft value function, which improves the accuracy of the optimization process.\n- Our framework integrates the policy evaluation step and policy improvement step into a single training step, which streamlines the overall training process.\n\nPrevious actor-critic frameworks without these features may suffer from inaccurate soft value estimation (i.e., Eqs. (6) and (7)) and estimation error when minimizing the policy improvement loss (i.e., Eq. (5)). We substantiate our claim through an extensive set of experiments conducted on the MuJoCo and Omniverse Isaac Gym benchmarks as depicted in Fig. 3 and 4 of the main manuscript.\n\n[1] Haarnoja et al. Latent Space Policies for Hierarchical Reinforcement Learning, ICML 2018.\\\n[2] Mazoure et al. Leveraging exploration in off-policy algorithms via normalizing flows, CoRL 2019.\\\n[3] Ward et al. Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies, ICML Workshop 2019.\n\n---\n\n**C2.** Introducing new frameworks generally necessitates providing sufficient mathematical proof to validate their efficacy, even for basic techniques or simple conjectures. ... However, the paper does not provide adequate mathematical justification for these techniques. Including such proofs would greatly enhance the confidence in the proposed method. \n\n**Response:** We respectfully remind the reviewer that our method is mathematically supported by multiple theories.\n\nOur main methodology is theoretically supported by Proposition 3.1 in Lines 181-183, which verifies the validity of our selections of $Q_\\theta$ and $V_\\theta$. According to Theorem 3 in [1], which is described in Section 2.1, the policy of MEow converges to the optimal policy when the soft Bellman error in Eq. (4) is minimized. \n\nThe theoretical justification for learnable reward shifting (LRS) is provided in Lines 212 to 219. Lines 214 and 215 show that the soft value function can be calculated exactly (Proposition A.3). In Lines 217 and 218, we mathematically demonstrate that the action distribution remains unchanged after applying LRS. The newly defined $Q_\\theta^b$ and $V_\\theta^b$ follow Theorem 3 in [1] and converge to the optimal policy when minimizing the soft Bellman error in Eq. (4). \n\nThe theoretical justification for SCDQ is given in Lines 226 to 233. SCDQ allows for the implementation of clipped double Q-learning [2] without duplicating the policy in MEow. In Eq. (13) and Lines 228-230, we mathematically show how SCDQ prevents the introduction of additional soft value functions. The mechanisms by which clipped double Q-learning reduces training variances and overestimation are analyzed in TD3’s original paper [2]. \n\nWe would also like to emphasize that other techniques and claims in this work are theoretically supported with proofs. The deterministic inference technique is supported by Proposition A.4 in Lines 605-609. The theoretical analysis of the soft value estimation methods is offered in Proposition A.1 and Remark A.2 in Lines 565-568. These technical results support the validity of our method and provide motivation for formulating our approach.\n\n[1] Haarnoja et al. Reinforcement Learning with Deep Energy-Based Policies. ICML 2017.\\\n[2] Fujimoto et al. Addressing Function Approximation Error in Actor-Critic Methods. ICML 2018.\n\n---\n\n**C3.** It would be beneficial for the paper to compare its methods with more recent state-of-the-art algorithms. ... Such comparisons would greatly enhance the paper by placing the proposed method in the context of the latest advancements.\n\n**Response:** A performance comparison between MEow and two latest online RL frameworks, DIPO [1] published in 2023 and S2AC [2] published in 2024, is presented in Table 1 in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). The results show that MEow outperforms DIPO and S2AC with noticeable margins in five MuJoCo environments.\n\n[1] Yang et al. Policy representation via diffusion probability model for reinforcement learning, 2023. \\\n[2] Messaoud et al. S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor-Critic, ICLR 2024.\n\n---\n\n**Questions**\n\n**Q1.** According to the appendix, the proposed method is said to be 2.3 times slower than SAC. Given this difference in speed, a time-wise comparison of the algorithm's performance would be very helpful to understand its efficiency relative to SAC. \n\n**Response:** In response to the reviewer's request, we extended the training time of SAC and provided a timewise comparison between MEow and SAC evaluated on the Hopper-v4 environment in Fig. 3 of the attached PDF file in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp). The results show that MEow converges to a policy that achieves a better average return and exhibits greater stability.\n\n---\n\nThe response to **Q2** is provided in the [global comment](https://openreview.net/forum?id=lhlIUxD5eE&noteId=Vapk6NMOyp).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
This global comment includes additional experimental results and extended discussions addressing the questions raised by reviewers UfJ6 and S3sW.

---

### **Additional Results**

The attached PDF file contains five figures, denoted as **Figs. 1-5**, which encompass the following content:

- **Fig. 1**: A performance comparison of MEow and SAC trained with different target smoothing values ($\tau$).
- **Fig. 2**: The soft value functions and the trajectories generated using MEow, SAC, and SQL on the multi-goal environment.
- **Fig. 3**: The return versus training time comparison of MEow and SAC.
- **Fig. 4**: $V_\theta$ and $b_\theta$ of MEow evaluated on the multi-goal environment.
- **Fig. 5**: The learnable reward shifting values evaluated along a trajectory of states in the Hopper-v4 environment.

The following table compares the performance of MEow and a number of recent online RL methods:

||Hopper|HalfCheetah|Walker|Ant|Humanoid|
|-|-|-|-|-|-|
|MEow|**3332.99** $\pm$ 521.63|**10981.47**$\pm$1812.97|**5526.66** $\pm$ 276.99|**6586.33**$\pm$188.73|**6923.22**$\pm$125.93|
|DIPO [1] ($K$=100)|3123.14$\pm$ 636.23|10472.31$\pm$654.96|4409.18$\pm$469.06|5622.30$\pm$487.09|4878.41$\pm$822.03|
|DIPO [1] ($K$=50)|3214.83$\pm$491.15|9198.20$\pm$1738.25|4199.34$\pm$ 1062.31|4877.41 $\pm$ 1010.35|4513.39$\pm$ 1075.94|
|DIPO [1] ($K$=20)|2511.63 $\pm$ 837.03|9343.69 $\pm$ 986.82|4467.20 $\pm$ 368.13|5288.77 $\pm$ 970.35|4294.79 $\pm$ 1583.48|
|S2AC [2]|<3100|<10000|<3500|<3000|<3500|

**Table 1.** Comparison of the average return between MEow, DIPO [1], and S2AC [2]. The results for DIPO [1] are obtained directly from Table 1 of their original paper. $K$ represents the number of diffusion steps used in the sampling process of [1]. The results for S2AC [2] are derived from Fig. 8 of their original paper.

[1] Yang et al. Policy representation via diffusion probability model for reinforcement learning, 2023.\
[2] Messaoud et al. S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor-Critic, ICLR 2024.

---

### **Reviewer UfJ6 (Cont'd)**

**Q4.** What are the learning dynamics observed during the training process? Are there any notable differences in the convergence behavior compared to traditional actor-critic methods? 

**Response:** In Section A.5.3 of the Appendix, we compare MEow with four different types of actor-critic frameworks. Our observations indicate that MEow converges to policies that achieve higher returns in most of the environments. In particular, the comparison between MEow and FCFA (i.e., an actor-critic framework modeled using two normalizing flows) suggests that the integration of policy improvement and evaluation steps enhances training stability and overall performance. This finding substantiates the primary claim of our paper.

---

**Q5.** How does the proposed framework scale with increasing complexity of the environment or the state-action space? Are there any specific limitations or challenges observed during experimentation? 

**Response:** Both the loss function calculation and the inference process of MEow are scalable with respect to the state-action space. This scalability is attributable to EBFlow's compatibility with existing normalizing flow architectures that support efficient inverse transformation and Jacobian determinant calculation for high-dimensional inputs. These architectures can be utilized in MEow to facilitate efficient training and inference processes with a large action space. For environments with a large state space, states can be encoded using hypernetworks (e.g., MLP in our implementation) to reduce their dimensionality, thus ensuring scalability. Our experiments in high-dimensional environments, such as Humanoid (state dim.: 108; action dim.: 21) and AllegroHand (state dim.: 72; action dim.: 16), support this claim.

---

**Q6.** How robust is the EBFlow framework to different hyperparameters?

**Response:** We appreciate the reviewer’s suggestion. In response, we conducted a performance comparison under different values of $\tau$ (i.e., smoothing target factor in Section 3.3) on five MuJoCo environments. The results of this analysis are presented in Fig. 1 in the attachment of this global comment.

---

### **Reviewer S3sW (Cont'd)**

**Q2.** In Figure 3, SAC and MEow show similar performance in the Mujoco domain, but there is a noticeable difference in performance in the Isaac Gym domain in Figure 4. Could you provide a further analysis on the reasons for the occurrence of this difference? It would provide valuable insight into the strengths and limitations of the proposed method. 

**Response:** The difference in performance between SAC and MEow in the MuJoCo and Isaac Gym environments can be attributed to the varying complexity and nature of tasks in these environments.

In the MuJoCo domain, the five tasks exhibit relative homogeneity, primarily focusing on locomotion. These tasks involve straightforward action outputs, typically torque. On the other hand, the Isaac Gym domain presents a more diverse and complex set of tasks. For instance, the ANYmal robot, unlike HalfCheetah, must navigate in multiple directions and follow specific target velocities. Its action space is different as well, involving outputs related to joint position targets rather than torque. Dexterous manipulations, such as those required in AllegroHand tasks, are widely recognized as challenging in the realm of RL. These tasks entail intricate interactions and contacts between the hand and the objects it manipulates. While SAC performs comparably to MEow in the relatively simple MuJoCo tasks, MEow demonstrates greater robustness and generalizability in the more diverse and complex tasks of the Isaac Gym domain. This highlights the strengths of MEow in handling a wider range of challenging scenarios.

Meta-review: The paper proposes a new MaxEnt Reinforcement Learning (RL) framework using Energy-Based Normalizing Flows (EBFlow). Traditional MaxEnt RL methods typically use actor-critic frameworks and alternate between policy evaluation and policy improvement steps. The proposed framework integrates these steps into a single objective, eliminating the need for Monte Carlo approximation while enabling efficient modeling of multi-modal action distributions. Experiments on the MuJoCo benchmark and Omniverse Isaac Gym environments show superior performance compared to existing MaxEnt RL methods.

**Strengths** 
- The idea of adopting EBFlow for MaxEnt RL is novel.
- The presentation is clear and well-organized, and the method is described in detail.
- The experimental results are convincing.

**Weaknesses**
- The calculation of the determinants is more time-consuming than existing methods like SAC.
Decision: Accept (poster)