Stationary Deep Reinforcement Learning with Quantum K-spin Hamiltonian Equation
Authors: Xiao-Yang Liu
Zechu Li
Shixun Wu
Xiaodong Wang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: deep reinforcement learning
instability
Hamiltonian policy gradient
stationary
quantum K-spin

Abstract: A foundational issue in deep reinforcement learning (DRL) is that \textit{Bellman's optimality equation has multiple fixed points}---failing to return a consistent one. A direct evidence is the instability of existing DRL algorithms, namely, the high variance of cumulative rewards over multiple runs. As a fix of this problem, we propose a quantum K-spin Hamiltonian regularization term (H-term) to help a policy network stably find a \textit{stationary} policy, which represents the lowest energy configuration of a system. First, we make a novel analogy between a Markov Decision Process (MDP) and a \textit{quantum K-spin Ising model} and reformulate the objective function into a quantum K-spin Hamiltonian equation, a functional of policy that measures its energy. Then, we propose a generic actor-critic algorithm that utilizes the H-term to regularize the policy/actor network and provide Hamiltonian policy gradient calculations. Finally, on six challenging MuJoCo tasks over 20 runs, the proposed algorithm reduces the variance of cumulative rewards by $65.2\% \sim 85.6\%$ compared with those of existing algorithms.

Stationary Deep Reinforcement Learning with Quantum K-spin Hamiltonian Equation
Anonymous Author(s)
Affiliation
Address
email
Abstract
A foundational issue in deep reinforcement learning (DRL) is that Bellman' 's op- timality equation has multiple fixed points-failing to return consistent one. direct evidence is the instability of existing DRL algorithms, namely, the high variance of cumulative rewards over multiple runs. As a this problem, we propose quantum K-spin Hamiltonian regularization term (H-term) to help a policy network stably find a stationary policy, which represents the lowest energy configuration of system. First, make novel analogy between Markov Decision Process (MDP) and a quantum K-spin Ising model and reformulate the objective function into quantum K-spin Hamiltonian equation, functional of policy that measures its energy. Then, we propose a generic actor-critic algorithm that utilizes the H-term regularize the policy/actor network and provide Hamilto- nian policy gradient calculations. Finally, on six challenging MuJoCo tasks over 20 runs, the proposed algorithm reduces the variance of cumulative rewards by 65.2% ~ 85.6% compared with those of existing algorithms.
2
6
8
10
12
13
14
Introduction
15
Deep reinforcement learning (DRL) [36] algorithms quite unstable, namely, agents trained with different random seeds may have dramatically different performance. Existing works [ 6 24 reported high variance multiple runs, thus it requires to train tens of agents and then pick the
17
19 best one. Such a high variance puts a challenge on reliability and reproducibility 15 14] limiting the wider adoption in real-world tasks.
21 The difficulty of guarantecing stability lies in a foundational issue that Bellman's optimality equation has multiple fixed points 31 [22] [8] failing to return a consistent one. Consider MDP examples
23 with an terminal state O, as shown in Fig. (we adapt dynamic programming examples
31 into
reinforcement learning settings. Observational experiments given in Section 2.2
25
Shortest path problem (deterministic) in Fig. Ta): At state 1, an agent transits to either state
26 1 or 0 with reward 0 or b, respectively. Assume the value function for state is V(0) 0. The Bellman's 's optimality equation for state 1 is V(1) max(V (MxxVV(1) (1), b). where any V(1) 2 b is feasible solution. If initialize (1) 2 b, a resulting policy is that agent at state always transits back to state 1; otherwise, drives to terminal state (always returns back to itself with reward 0).
27
29
30
Blackmailer's problem (stochastic) in Fig. bb): At state 1, profit maximizing blackmailer demands cash amount (0,, 1]: victim transits state with probability or state 0 with probability a, respectively. At state 0, a vietim always refuses to yield, i.e., V(0) = 0. The 32
Bellman's 's optimality equation for state is V(1) = max, (a+ V where any V(1) > is feasible solution. If initialize Vo(1) > 1, the blackmailer's policy is demanding a - 0 to keep the victim at state 1; otherwise, demanding 1 that drives the victim to terminal state
34
35
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

p 1=1-a
r -C
-||x|]
8x
p
0
a)c
1
Cone
(a) Shortest path problem
(b) Blackmailer's problem
(c) Optimal stopping problem
Figure I: MDP examples where 1. Examples with 7 < 1 are given in Fig.
in Appx.
Optimal stopping problem (terminating policies) in Fig. [e): In space IR? with terminal state of point 0, an agent at point x: moves to either point O with negative reward -c or point az with reward -||=||| respectively, where (0, 1). The Bellman's optimality equation is (xx) max cc, -||=|| + +V(ox) and the optimal policy is to continue inside the sphere of radius (1 a c and stop outside. If add a cone region C within which an agent always receives a reward
37
38
39
40
41
second policy is jumping to point () at any point in region C.
42
The instability problem has been partially addressed, such as ensemble methods g regularization approaches [38 9 and baseline-correction approaches 42] In particular, Generalized Advantage Estimation (GAE) 33 is widely used one that significantly reduces the variance of the advantage function. However, they did NOT fix the foundational issue of Bellman's optimality equation in Fig. I thus the objective function inherently fails to search for a consistent policy. For practical usage, we often expect a DRL algorithm stably converges certain policy independent initialization and noises.
44
45
46
47
48
As fix of the problem, make novel analogy between MDP and quantum K-spin Ising model 50
26 12], and reformulate a reward-maximizatio RI. task into an energy-mininization problem, namely, finding the lowest-energy configuration of quantum spin system. We hypothesize that physically stationary policy would have the lowest energy. Different from our quantum K-spin perspective, several recent papers utilized the (classical) Hamiltonian equation endow RL agents the capability of inductive biases. For example, [21 40
52
53
54
used Hamiltonian mechanics to train an agent that learns and respects conservation laws; applied Hamiltonian Monte Carlo (HMC) simulator to approximate the posterior action probability; and (28 proposed an unbiased estimator for the stochastic Hamiltonian gradient methods for min-max optimization problems.
56
57
58 In this paper, we propose a quantum-inspired K-spin Hamiltonian regularization term (H-term) that helps policy gradient algorithms stably converge physically stationary policy. Our contributions are as follows: 1) we derive novel regularizer, H-ferm, by novel reformulation of the RL's objective K-spin Hamiltonian equation that measures the energy of policy; 2) propose a generic actor-critic algorithm that utilizes the H-term to regularize the policy/actor network and provide Hamiltonian policy gradient calculations challenging MuJoCo tasks over 20 runs, we show that the proposed algorithm reduces the variance of cumulative rewards by 65.2% 85.6% compared with those of existing algorithms.
60
82
63
both deterministic and stochastic cases; and 3) on six
64
85
2
Bellman's Optimality Equation and Multiple Fixed Points
We describe Bellman's optimaility equation and provide observational experiments to verify the existence multiple policies.
68
2.1
Bellman's Optimality Equation
70 A reinforcement learning (RL) (36] agent interacts with an unknown environment and lears an optimal policy that maximizes the cumulative reward. Mathematically, the environment can be formulated as Markov Decision Process (MDP) with the five-tuple AP R, 7). Here S and
71
73 A denote the state and action spaces; P. S > (SS) denotes transition probability function,
where A is probability simplex; R. R:C. S Ax S IR denotes reward function; and (0,

---

Humanoid
Hopper
L
Two-legs
One-leg
Backward
Hopping
Diving
Balancing
HalfCheetah
Ant
Running
Diving
Balancing
Flipping
Running
Flipping
Different policy types for four MuJoCo [39 tasks. Two-legs running, one-leg running, and running backward for Humanoid; running, diving, and balancing for Hopper; running, diving, balancing and flipping for HalfCheetah; running and flipping for Ant. The bold ones indicate physically stationary policies.
Figure 2:
75 denotes a discount factor. The objective is to find an optimal policy x" S - A(A) that maximizes (discounted) expected reward.
78
77
Consider a discrete, finite, discounted MDP with infinite horizon, one can define the Q-value function of state-action pair (s, a) under policy follows
Q" (s)
Σ ,, Ag Se:11)) =,, Ao (I)
where R(Sk, Ak, Sk+11 denotes the immediate reward when taking action A state Sy. and arriving 80 at state S1+1, capital letters denote random variables and lowercase letters denote values. The
81 conventional objective function J(0) of reinforcement learning (36) takes the following form
J(0) A As (So, An)] = Er-- (R(T) P(r[za)],
where is trajectory, i.e., (So, ......). and
85 P(r(mo) (o)) II [8( [82, ag ) (8k).
84
The Bellman equation
36 converts into recursive form as follows
(² Σ P(s' s,a) R(s, a, 57
+
Σ a
stES
a'EAA
= R(ss
+
Σ P(s s, a) Σ (F) (5', sESS
a'EA
as which expresses the expected reward as summation of immediate reward R(s, a) and discounted fu-
86 ture rewards, and the immediate reward R(s, a) is defined as R(s, Eves (P(s' s, a(s,
87
The Bellman's optimality equation
33 is
('( = Σ IPss a, [ + max (d','
(4)
a'
E'ES
88
The optimal policy n² is given by greedy strategy such that T² = arg max; (s)
89
2.2
Observational Experiments for Multiple Policies and Physically Stationary Policy
As pointed out in Fig. [ theoretically there exists multiple policies. Here, perform observa- tional experiments on four challenging MuJoCo tasks 39) namely, Humanoid, Hopper, HalfCheetah, and Ant (details given in Appx. B.1] which typical examples of the locomotion control of robot.
91
33
render the obtained policies over multiple runs and then identify physically stationary ones. We observe various types of moving strategies, shown in Fig. 2 which verifies that multiple policies

---

Table 1: Analogy between MDP and quantum K-spin Ising model.
MDP (Our formulation in
Quantum K-spin Ising Model [26] 12] in 5
State-action pairs
MQ. .... MK -
Spins
Jo. JJK
Optimal policy
Tho Th X TpK-1
Optimal configuration
J3 x a x X OjN
Policy
Tho
Thy X Mag
Configuration
Tio X xj
x ojx
Discounted reward
pK
Density function
Functional policy
HT .n TaK-11
Functional of spins
H .. 5 02x-1)
6H( rn Aag
6H( oig
Stationary condition
Stationary condition
88
80;
95 are very common. For example, the Humanoid agent learns either jumping with a single leg or running with two legs, shown in Fig. 2 (top-left; another interesting example is HalfCheetah, in 97 which an agent can run normally or in a flipped manner, as shown in Fig. (bottom-left) Among
98 the obtained policies, one can easily identify the physically stationary polices that control the robot moving forward with a stable gait (defined as gait that does not lead to fall).
96
3 Reinforcement Learning as Quantum K-spin Hamiltonian Equation
100
First, make novel analogy between the MDP and quantum K-spin Ising model, and reformulate the objective function (2) into a quantum K-spin Hamiltonian equation. Then, we show that this Hamiltonian equation helps fix the issue of multiple fixed points in Fig. [
102
104
3.1
Motivation through Analogy with Quantum K-spin Ising Model
105
Given the multiple policies in Section 2.2 it is critical employ a proper criteria to search for physically stationary policy. In Table we make novel analogy between an MDP and quantum K-spin Ising model [2]
106
107
108
The Hamiltonian equation for a quantum K-spin Ising model
26
measures the energy of a
109
particular configuration of quantum K-spin system and takes the following form
-
Σ Σ3 jo=1
L Lig Jo jje
(5)
110 where N the number of spins in the k-th configuration, 3z #1 are spin variables, and Ljo...s is an energy density function for k nearest spins' configuration (..
112 Analogy in Table [ Starting from an analogy between state-action pair μl (Sx, A&) and
a spin jk. we can map an optimal policy π (12)) E {0,1] in (4) to a single-qubit spin operator
113
114
Je E [-1,1] via π (px) + (Inz Jp2) 1/2, where To (p(x)) denotes the probability of taking
115
action A at state Sg. following policy πy. The energy density function Lj....h can be defined as the
116
discounted reward on path (Ho.... ( of length k, i.c.
Typo.....Pe = x R(UR)= do(so)
II (s (ue,, (obtained via Monte Carlo simulation)
(6)
117 where do())) denotes the distribution of initial state 8g- Analogy to the quantum K-spin Ising model, we can derive functional of policy H(Tmo "......... Tux in the context of MDP, which measures the 119 energy an RI policy. In this case, physically stationary policy satisfies the stationary condition
H (Tar
Tux- /)//T, O.

---

3.2
Reformulation into Quantum K-spin Hamiltonian Equation
122 Inspired by we formally reformulate 22 into a K-spin Hamiltonian equation
(0) _ Ao
[Q
Ao)]
K-1SxA
SxA
lim
Σ
L poe (1OO) -. To (e()))
K K-+00
k=
MO
K
lim
E
Σ
I Iug
K-0
123
Where-K-- the expectation is taken over So do(-), Ao ~ To :), and the density function
pu is given in (6
125
The following blue part will be put into the appendix.
126
Monte Carlo Estimator
30 Consider a general prohabilistic objective F of the form:
F pp( (fx
(8)
127
in which function f of an input variable æ with structural parameters & is evaluated on average with respect to input distribution p(æ; 8) with distributional parameters 0.
Monte Carlo method evaluates the function by first drawing independent samples a(1) ..... ((N) from the distribution pæx; 0), and then computing the average:
130
N
F
Σ s(a!)), where al1) ~ p(x;; for = 1, I,..,,N. N.
(9)
131
132
The Monte Carlo estimator for is
2
N
J(0) = N Σ R(r()), where 70) ~ P()(0) for 1,..., N,
(10)
133
and
T
P(7(0) (T8) (o))
P ( l )a) (101 (sk (.))
(11)
136
The Monte Carlo estimator for is
NO K
A((0)
0) for = I,,.,, N',
(12)
N
4g m
137 and
L Mg ......
18
(2) R(p
(1) do(s)))
II E=
P( (9)
(13)
138
Remark: The above two Monte Carlo estimators quite different in the simulation process. samples a random trajectory by following an environment's stochastic transition and a policy, In contrast, [3) measures a random path's discounted reward (the "energy") without following any policy, and the Hamiltonian equation 7 combinatorially enumerates all possible paths length K the state-action space. In other words, the simulation process of the Hamiltonian term does not rely on any policy. Therefore, the Hamiltonian term a suitable regularizer for both on-policy and off-policy algorithms.
140
142
144
146 This fundamental difference is due the Ising model in which combinatorially enumerates all paths and separates the environment and the policy.

---

148
Physical interpretation: Analogy to quantum S-spin system, (O) in measures the energy of policy π. We hypothesize that the energy of policy favorable criteria, since stationary policy with minimum energy: 1). achieves a relative high reward independent of the initialization; and 2). is robust to interference/noise in the inference stage.
149
150
151
K K-step truncation in practice. Minimizing (7) is NP-hard M. Since E (0, 7x monotonically decreased with look-ahead steps K, therefore, we truncate 7 to finite K terms. One can show that these K terms in 7 is geometric sequence with a truncation error ratio 1 - --² Assuming < -1-66 C, where > 0 small, thus we have the look-ahess steps satisfies K z log, c.
152
153
154
155
1
156
3.3
Revisiting Examples in Fig. 1
157
We elaborate how adding the energy measured by 7 onto each state can help drive to the terminal state (a stationary policy), which fixes the foundational issue of multiple fixed points in Fig. T where
158
159
1. We have (0) = O for the terminal state O.
k== b b= -00 At state 1, the Bellman's ' optimality equation becomes V(1) = max max(V(1) + (1). Independent of the initial value (1), an agent obtains policy that always transits back terminal state 0.
(a) Shortest path problem (deterministic): [[) Σ
161
162
163
(b) Blackmailer's problem (stochastic): H(1) = -00. The Bellman's optimality equation
164
becomes (11 = max, (1 XX for state 1. For any V1(1) < co, the optimal policy becomes a = 1 that drives to the terminal state O.
165
166
(c) Optimal stopping problem (terminating policies): any policy that takes infinite steps will have (xx) -o, since each step number k, there are always trajectories that jump point O with reward -c; and a direct jumping policy will have H (z) = -c. Therefore, adding H (z) to each point x K 0 will lead to a policy of jumping back to point O.
167
168
169
4 Actor-Critic Algorithm with Quantum K-spin Hamiltonian Regularization
170
171
We propose a generic actor-critic algorithm with a H-term, derive two Hamiltonian's 's policy gradient theorems for both deterministic and stochastic cases, and present the Monte Carlo gradient estimation.
172
173
4.1
Stationary Actor-Critic Algorithm with H-term
174
Actor-critic algorithms in reinforcement learning perform a bilevel optimization, namely alternating between approximating value function and optimizing policy. In practice, critic network with parameter & approximates the Q-value function, and an actor network with parameter 0 approximates the policy и, details given in Appx. F However, since the critic's 's update is governed by the Bellman's optimality equation, actor-critic algorithms suffer the multiple fixed points problem.
175
176
178
179
we propose novel H-term for both deterministic and stochastic actor-critic algorithms. Similar to the entropy term in 233, the proposed H-term is an add-on term to regularize the actor network and help it converge to stationary policy. Specifically, the objective functions of actor and critic networks become:
Motivated by Section 3.3
180
182
Actor : max J= Φ)
(1 - s (Q+ (So. Aa)] (0),
8
(14)
Critic: min (0) A
2 (.s-do()).Annn (Q0 A) - y(S, A))²
where target Q-value is y(S,, As) R(S,, Av) + 20 A4+1). and À is temperature parameter. As an interpretation, the second term (H) in the maximization objective function of actor network aims to find minimum energy configuration for the MDP problem, namely, a policy ा that will add a minimum amount of energy to each state's value function (as in Section 3.3
194
185
186
187
New algorithm. In Alg. 1 an agent interacts with an environment and alternatively updates its actor network and critic network. The algorithm has M episodes and each episode consists of a (Monte Carlo) simulation process and learning process (gradient estimation) as follows:
188
190
During the (Monte Carlo) simulation process (lines 5-10 of Alg- D. an agent takes action a; ac- cording to policy Toe (t), t = 0,... ,T 1, generating a trajectory of T' steps/transitions.

---

Algorithm Stationary Actor-Critic Algorithm with H-term
Input: learning rate a, temperature look-ahead step K, and parameters M, B3'
2: Initialize actor network т and critic network Q with parameters 6, o, and replay buffers D1, D2
3: for episode 1.... .M do
4: Initialize state $o
5: for -= 0....
do Select action ae x0(+(
6:
7:
Execute action ot, receive reward rt; and observe new state 81+1
8:
Store a transition (81, 4r Tt 5(+1) in D
9:
end
10:
Store trajectory of length T'in in D2
11:
for g = 1, ,
12:
Randomly sample mini-batch of B transitions {(s; rs, (1-1))= from D1
Randomly sample mini-batch of B' trajectories (of length Y) {r)]5=1 from D2
14:
Update critic network using a conventional method
Update actor network as +- 0 + a (Vo
15:
16:
end
17:
end
Then, these T transitions stored into replay buffer D1, while the full trajectory 7 (Su, Go; To, 81,' 8T ar Pr,1, 8r) stored in replay buffer D2.
193
194
During the learning process (G 2 1 updates in one episode) (lines 11-16 of Alg. I. a mini- and mini-batch of B' trajectories (of (of length K) (T) = (s)) ag: ry. si,,.. x - -1 - )) are sampled from D2 and D2, respectively. The critic network is updated by conventional method, e.g., minimizing the mean squared error (MSE) between an estimated Q-value and a target value. The actor is updated by a Monte Carlo gradient estimator over B transitions and 3' trajectories.
195
batch of B transitions {(s;, 814
196
198
200
Two new hyperparameters. We introduce two hyperparameters: a temperature À >> 0 that is relative weight of the H-term, and look-ahead step K T that defines the horizon of the H-term.
201
202
Implementation of replay buffer D2- After a full trajectory т of length is generated, it is partitioned into T - K trajectories of length K. We rank them according to the cumulative reward and store the top portion, say 80%, into a new replay buffer D. (line 10 of Alg. []. We randomly sample mini-batch of B' trajectories from D2 (line 13 of Alg. compute the H-term.
204
206
4.2
Hamiltonian Policy Gradient and Monte Carlo-based Gradient Estimator
We provide the policy gradient of the quantum K-spin Hamiltonian equation in (7). which are variants of the well-known policy gradient theorem (36). We provide detailed derivations in Appx. D
208
209
Stochastic version. The Hamiltonian stochastic gradient of w.r.t. parameter is
(0 -Epo
Σ K=U
* R(ux). Vo log (TT(( TT( To To(HL))
(15)
210
denote a deterministic policy, while we use Te s to represent that a Gaussian noise (a.k.a, an exploration noise) with standard deviation ó > is added in the exploration process. The Hamiltonian deterministic gradient of (7) w.r.t. parameter is
Deterministic version. Let (.)
212
Σ k. R(px) -Rpx) Vo log (Ho,s(Po) (p1) ......))) (16)
H -E Eggg.....
The quantum K-spin Hamiltonian equation in 7 is a reformulation of (2). We verify the gradient calculation by showing that: when K -> the Hamiltonian stochastic and deterministic policy
213
215 gradient (0) and (0) are equal to the stochastic policy gradient VoJ(O) and
deterministic policy gradient VeJ'(() in [35). respectively.
7

---

R(j)) + .... 3- R((
A(HO) ... ((Hx-1)
R(Ør)
T(PO)
1
T
K
x(pe)
T(UT
R(pe)
Figure 3: Comparison of REINFORCE's (left) and Hamiltonian's 's policy gradient (right).
217 Note that the gradient (0) in 15 and (0) in 16 w.r.t. distributional parameter 0 takes
expectation form. Thus, a Monte Carlo gradient estimator is practically useful. We obtain the Monte Carlo gradient estimator of (0), illustrated in Fig. (right), as follows
219
N'
K
voA((0)= - N'
Vo log (mo(ub) mo(kk]]
(17)
k=
220
As contrast, we provide the Monte Carlo gradient estimator of REINFORCE'S as illustrated in Fig. (left), as follows
policy gradient,
221
N Σ NT
VgJ(0) 1
G. logogne
Vo (o( where Gj-
R(p)).
(18)
222
An interesting observation is that both gradient calculations follow a similar pattern as shown in Fig. REINFORCE's policy gradient (37 in Fig- (1eft) employs estimate of future rewards, while amiltonian policy gradient in Fig. (right) uses trajectories in replay buffer D2.
224
Computational complexity: we measure the computation complexity by the times of computing B and N' = B', since most DRL algorithms use a mini-batch stochastic gradient decent methods. REINFORCE's (37] policy gradient in (8) takes O(BI") computations, while Alg. [ adds (BB + 1)/2) computations in each gradient update step, thus a total complexity of O(BT + B'K(K 1)/2).
226
Va log no (p)). Assume
228
Performance Evaluation
230
We evaluate the proposed H-term from four aspects: 1) increasing cumulative reward, 2) reducing variance, 3) driving to physically stationary policy, and 4) the impact of trajectory length K. All experiments executed NVIDIA DGX-2 server [10]. The server contains A100 GPUs,
232
234
320 GB GPU memory, and 128 CPU cores running at 2.25 GHz.
5.1
Experimental Settings
236
Environments (tasks). We consider six challenging MuJoCo tasks [39 as Section 2.2 The agent learns to control the locomotion of a robot and aims to move forward as quickly as possible. These tasks have high-dimensional continuous state and action spaces, in which there exists multiple locally optimal polices revealed in Section 2.2
238
240
Compared algorithms. To evaluate deterministic and stochastic algorithms, we choose Deep
Deterministic Policy Gradient (DDPG) p7 and Proximal Policy Optimization (PPO) [3). Since the
242
H-term compatible with existing variance reduction techniques, we implement the PPO algorithm with GAE [3]. For a fair comparison, we keep the hyperparameters (listed in Appx. and make sure that obtained results reproduce existing benchmark tests
H the same
244
K3
245
Performance metrics. We employ two performance metrics, the cumulative rewards and variance, while in Section 5.4 we further consider different policies and report the number of convergence. We run each experiment with 20 random seeds and in each run we test 100 episodes.
247
5.2
H-term Increases Cumulative Reward
249
Experience replay crucial in improving performance in terms cumulative reward. The proposed H-term in (17) can be viewed novel experience replay technique for an actor network. Here,

---

HalCChellhh
Ant
Humanoid
DIPG
DOPG DOPO_H
DDPG
DRIGGH
DGPG_L
3K
Cumultt
22
12
1K
2k
4
0.0
0.5
1.0
1.5
2.0
ecamp
Ie5
asomples Swimmer
1e6
>-samoles
147
Hopper
Walker 2D
200
DERS
DOPG
camulutiv rawards
1.0
DOPG, PER
Cumulatoow wwwrrt
DING PER
120
2.0
2.5
2.0
2.5
3.0
3.0
0.5
1.0
l05
Pisampless
Arsamoles
HalfCheetah
Ant
Humanoid
5k-
12k
FPO
PPO
PPO
5k-
#PO,H
48-
PPO
10k
PPO.
41-
3x-
Damutatio
3k
Cumunatvv
2x-
le5
Hopper
Swimmer
Walker2D
5k
200
5k
JPO
170
PPO
4k
HPO.H
Camuat vo renurds
160
PPO.H
Camual un reantss
46.
FPO. .
32
120
Camuat
22
80-
22
14
40
1k
3.0
3.5
2.5
3.0
#sompies
#somplls
arsamolcs
Figure 4: Cumulative rewards vs. #samples for compared DRL algorithms on six MuJoCo tasks.
add compared algorithm, DDPG with Prioritized Experience Replay PER prioritizes experience by the TD error to update a critic network.
32 (DDPG+PER), where
252
In Fig. both DDPG+PER and DDPG+H achieve a substantial improvement of cumulative reward. In particular, DDPG+H achieves the highest cumulative rewards all six tasks, which are comparable to PPO's performance in Fig. It is worthwhile to discuss the advantage of DDPG+H over DDPG+PER. DDPG+PER utilizes priortiized replay strategy obtain a more accurate critie network, however, it is updated via the Bellman equation with the trouble of multiple fixed points. In contrast, the H-term DDPG+H is performed on the actor network. Our results indicate that an experience replay technique actor network may be much more powerful.
254
256
258
5.3
H-term Reduces Variance
The PPO algorithm with GAE is regarded as the state-of state-of-the-ar the-art algorithm in MuJoCo environments. However, it still has a very high variance (the shaded area) after the policies have converged, as shown in Fig. 4 We observe that, at the end of training, the PPO algorithm has a variance of 969.2, 2, 1563.4, 2513.5. 5, 905.3, 60.7, 1290.1 the six tasks, respectively. Such a high variance mainly due to the fact that the agent may converge to a random of multiple policies.
262
264

---

Table 2: Experimental results on six challenging MuJoCo tasks.
Tasks
Policies
PPO
PPO+H (K = 8)
PPO+H (K = 16)
PPO+H (K 24)
running
13
19
20
20
flipping
5
4720.8
5028.4
0
5104.3
0
4995.1 1
HalfCheetah
diving
1.969.2
2211..3
+:228.4
+:383.3
balancing
1
0
0
running
17
20
20
4164
4505.3
4645.6
4662.5
Ant
jumping
0
=1533.4
#253.6
0
#225.4
0
#277.5
flipping
3
two-legs
7
17
16
17
9433.4 4
9670.3
10189.1 1 +683.7 7
9942.2
Humamoid
one-leg
12
3
=2533.5
#49722
+538.4 4
backward
1
0
0
hopping
10
20
20
2659.3
3116.5 +229.44
3300.1
3340.7
diving
8
0
0
Hopper
4.905.3
+184.2
1191.
balancing
2
0
moving
14
110.7
20
130.6
20
132.5
19
132.2
Swimmer
balancing
6
+60.7
+33.5
#31.6
:36.2
walking
16
16
15
5461.7
5819.9
5927.2 2
6089.3
Walker
diving
8
11200.1
#315.6
4
+296.
5
#314.7
balancing
7
0
In Fig. 4 the shaded areas of PPO+H (K 16) are dramatically smaller, i.e., variance of 228.4, 225.4, 4, 683.7, 7, 184.2, 31.6, and 296.8, respectively. The variance has been reduced by 65.2% ~ 85.6%. which verifies the effectiveness of the proposed H-term. In Fig. we also observe that the H-term
287
288
289
can help the DDPG algorithm reduce variances, namely, the variances of DDPG+H are much smaller than those of vanilla DDPG and DDPG+PER. Therefore, we may conclude that the H-term guides the agent to search for stationary policy among multiple feasible ones.
270
272
More experimental results given in Appx. due to the space limit, including the cases of K and K 24, and the curves of the H-term value during the training process. One may verify that the stationary policies have relative lower H-values.
273
275
5.4 H-term Drives to Physically Stationary Policy
278 A key question needs to be answered: is H-term really guiding the agent converge to physically stationary policy? Similar Section 2.2 we perform observational experiments the six MuJoCo tasks and measure the number of convergence to the different policies over 20 runs. As shown in Table the vanilla PPO algorithm converges the physically stationary policy (bold) with 13, 17, 10. 14, and 5 times for the six tasks, while the PPO+H (K = 16) converges to the stationary policy with 20, 20, 16, 20, 20, and 16 times, respectively. From the empirical observation, we find that the PPO gets stuck in locally optimal policies, failing to find consistent one. As expected, PPO+H can converge to the stationary policy with a substantially higher ratio, which verifies the effectiveness of the proposed H-term in finding physically stationary policy.
278
282
283
285
5.5
Impact of Trajectory Length K
We investigate the impact of trajectory length K that defines the horizon of the H-term. Based
287
we foresee that large K means a more accurate estimation of HoH(() but at price of computations.
288
Here, we evaluate the PPO+H with K 8. and set the size of replay buffer D. 1, 000. In Table
2 we observe that the cumulative reward increases and the variance decreases K increases from 8 24, both metrics get worse due to the out-of-memory issue. For K 24, we reduce the replay buffer size to 800 to meet the memory limit. The smaller replay buffer
290
to 16. However, for the case
size hurts the diversity of the trajectories and may lead to small performance drop. This hypothesize
292
verified in Appx. H.2 where all trials use consistent replay buffer size 800.
10

---

Conclusions
294
295
In this paper, we have addressed foundational issue of existing deep reinforcement learning algorithms that Bellman's optimality equation has multiple fixed points. This issue leads to the instability of DRL algorithms, puts a challenge on their reliability and reproducibility, and thus limits the wider adoption in real-world tasks. As a fix of the problem, we propose novel H-term, physically inspired regularizer, by making a novel analogy between a MDP and a quantum K-spin Ising model. Experimentally, we show that the H-term helps DRL algorithms find stationary policy that has the lowest energy system and reduces the variance of cumulative rewards by 65.2% 85.6% compared with those of existing algorithms.
297
298
301
303
For future works, we will explore the potential of directly training a policy network using
in Appx. quantum simulator |25 and quantum reinforcement learning [] It is interesting to
304
305
apply Monte Carlo estimator for unbiased policy gradient calculations. We would like to show that
306
the proposed H-term can help distributional RL algorithms [3] find a stationary policy, since the
307
distributional Bellman optimality operator is not contraction and thus there is also no unique policy.

---

Broader Impact Statement
308
309
In the field of DRL research, this paper points out the issue of multiple fixed points and may attract more attention from various perspectives. In terms of applications, this paper moves toward the reliable and reproducible research by improving the stability of existing DRL algorithms. On the other hand, the obtained stationary policy may have broad practical impact in many real-world application areas, including but not limited to robotics, transportation, and finance.
311
312
313
314
From an interdisciplinary perspective, our approach lies at the intersection of (quantum) Hamiltonian mechanics and DRL, espcially the explicit analogy between an MDP and a quantum K-spin Ising model. We hope that our study will attract more attention to bring together the strengths of both approaches and yield new insights in both fields.
315
316
317
318
In terms of broader societal impact of this work, we do not see any foresccable strongly negative impacts. However, this work essentially makes a tradeoff between the computational resource and stability, which may lead to future works with higher computational cost.
319
320
References
[I] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances Neural Information Processing Systems, 2021.
323
325
[2] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning. In International Conference Machine Learning, pages 176-185. PMLR, 2017.
327
[3] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR,
329
2017.
331
[4] Dimitri Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019.
[5] Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, 549((671): 2017.
333
[6] Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. 2019.
335
77] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34, 2021.
337
339
[8] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without a model. ICLR, 2021.
341
99] Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. In International Conference on Machine Learning, pages 141-11550 PMLR, 2019.
343
[10] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. NVIDIA A100 Tensor Core GPU: Performance and innovation. IEEE Micro, 41(2):29-355 2021.
345
Barry A Cipra. The ising model is np-complete. SIAM News, 33(6):1-3, 2000.
347
[12] Vasil Denchev, Sergio Boixo, Sergei Isakov, Nan Ding, Ryan Babbush, Vadim Smelyanskiy, John Martinis, and Hartmut Neven. What is the computational value of finite-range tunneling? Physical Review X, 6(3):031015, 2016.
349
[13] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 323-1338. PMLR, 2016.
351
12

---

353
[14]
Gabriel Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003. 1881, 2020.
354
355
356
[15] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforce- ment learning. 2019.
357
[16] Vedran Dunjko, Jacob M Taylor, and Hans J Briegel. Quantum-enhanced machine learning. Physical review letters, 11 117(13)130001, 2016.
359
[17] Eliot Kapit Wesley Jones Eric B. Jones, Peter Graf. K-spin Hamiltonian for quantum-resolvvble markov decision processes. ArXiv, abs/2003. 1881, 2020.
381
362
[18] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. ICLR, 2019.
363
384
[19] Evan Greensmith, Peter Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Advances Neural Information Processing Systems, 14, 2001.
365
386
367
[20] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
388
389
370
[21] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
372
[22] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12(1):1-12, 2021.
374
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
375
376
377
[24] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
378
379
380
[25] Eric B Jones, Peter Graf, Eliot Kapit, and Wesley Jones. K-spin Hamiltonian for quantum- resolvable markov decision processes. Quantum Machine Intelligence, 2(2):1-11, 2020.
381
[26] Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. Science, 220(4598):671-66) 1983.
383
[27] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
385
387
[28] Nicolas Loizou, Hugo Berard, Alexia Jolicoeur-Martie Martineau, Pascal Vincent, Simon Lacoste- Julien, and loannis Mitliagkas. Stochastic Hamiltonian gradient methods for smooth games, In International Conference on Machine Learning, pages 6370-6381. PMLR, 2020.
388
389
390
[29] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. ICLR, 2013.
391
382
[30] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning, J. Mach. Learn. Res,, 21(132):1-62, 2020.
393
394
[31] Alexey B Piunovskiy. Examples in Markov Decision Processes, volume 2. World Scientific, 2013.
396
[32] Tom Schaul, John Quan, Joannis Antonoglou, and David Silver. Prioritized experience replay. ICLR, 2016.
13

---

398
[33] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. ICLR, 2016.
399
400
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, ftss/1707.0637,, 2017.
402
[35]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning. pages 387-395. PMLR, 2014.
403
404
405
[36] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
406
407
[37] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
409
410
[38] Pierre Thodoroff, Audrey Durand, Joelle Pineau, and Doina Precup. Temporal regularization for markov decision process, Advances in Neural Information Processing Systems, 31, 2018.
412
[39] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
413
[40] Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. ICLR, 2019.
416
417
[41] Junfeng Wen, Saurabh Kumar, Ramki Gummadi, and Dale Schuurmans. Characterizing the gap between actor-critic and policy gradient. In International Conference on Machine Learning, pages 11101-11111. PMLR, 2021.
419
420
[42] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. ICLR, 2018.
422
423
[43] Duo Xu and Faramarz Fekri, Improving actor-critic reinforcement lcarning via Hamiltonian Monte Carlo method. Deep Reinforcement Learning Workshop at NeurIPS, 2021.
Checklist
425
426
1. For all authors...
427
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
429
(b) Did you describe the limitations of your work? [Yes] More computational cost.
430
(c) Did you discuss any potential negative societal impacts of your work? [Yes] May lead to future works with higher computational cost.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
433
434
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
436
(b) Did you include complete proofs of all theoretical results? [Yes] In the appendix.
3. If you ran experiments..
438
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
439
410
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
441
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We reported the variance over 20 random seeds as shaded
443
area in the figures, and also list it in table.
14

---

445
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
448
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
450
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
451
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
454
455
5. If you used crowdsourcing or conducted research with human subjects...
456
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
457
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
459
480
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15

---

X
r=c
= 1-a
0
|||
r= b 0
p a
1
a)c
Cone C
(a) Shortest path problem
Blackmailer's problem
(c) Optimal stopping problem
Figure 5: Revisiting Fig. for the discounted cases where 7 E (0, 1).
More Examples with Multiple Fixed Points
482
First, consider the discounted formulations of the three examples (shown in Fig. D shown in where E (0, 1). The differences are marked red.
464
Fig.
(a) Shortest path problem (deterministic, discounted case): Given two states 1 and 0, an agent at state 1 transits to either state or with rewards
466
or r = b, respectively. e > (1 7)
At state 0, the value function is (0) O. At state 1, the Bellman's optimality equation is
468
(11 = max c + y (1)), b}. where any ((1) 2 (b c) /ny a solution. If initialize Vo(1) 2
agent obtains policy that always transits back to state 1; otherwise, result policy drives to terminal state
470
(b) Blackmailer's problem (stochastic, discounted case): Different from (a), a profit maximizing blackmaiiee//agent at demands cash amount E (0, 1] (an action), while victim transits to state with probability a or to state with probability
472
473
state 0, victim always refuses to yield to the blackmailer's demand, i.e., V()) = 0. The Bellman's 's optimality equation a) [1)) for state 1, where any V(1) 2
a, respectively.
475
feasible solution. 1, the blackmailer's policy is demanding - 0 at the k-th step to keep the victim stay at state 1, for any & Kg - Llog, cj: and taking a
V(1)= max, {a + 7
initialize [1)
477
to transit to terminal state at the k-th step, for any 2 Ko + 1; otherwise initialize (o(1) = c S 1, the result policy demanding the maximum
478
that drives the victim refusal state 0 (a terminal state).
480
(c) Optimal stopping problem (terminating policies, discounted case): In space IR² with an agent moves either point 0 with negative reward point azr with reward - ||=||| respectively, where E (0, 1). The Bellman's 's optimality equation is ((x) m c, -||=|| + 7 7-V-VVVVaa V(az)} and the optimal policy is to continue inside the sphere of radius (1 a c and to stop outside. If add a cone region C within which an agent always receives a reward -c, C, second policy jumping to point at any point in region C.
terminating state at point 0, point
482
483
485
486
Then, we elaborate how the proposed H-term fixes the problems in Fig.
(a) Shortest path problem (deterministic, discounted case)
488
Assume (1) 2 b and (1 7)b, we have
(1(1) c Vo(1) b
[1(1) c - Vo(1) (1
Vo(1) c + -== Vo(1) 2 (1+7+7")++
k-1
k-
(19)
V((1) Σ
y c x Vo(1) 2
Σ -
2
V*(1 = Σ
-c>b
c
1 1-
16

---

The values of (0) and (1) are as follows:
H 0, H(1) -b
c b)) DC.
(20)
k=
Adding the above H-values state 1 and 0, respectively, we have
V*(1) + H(1 Σ 7 c x
(21)
(*(0) + H())
481
Therefore, (*(1) + H(1) < (0) + (0), independent of the initial value Vo(1). That is, an agent always obtains a policy that drives to terminal state 0 step 1.
493
(b) Blackmailer's problem (stochastic, discounted case)
484
If initialize Vo(1) = c > 1, the blackmailer's policy is demanding a - 0 at the k-th step to keep the transit terminal state
victim stay state 1, for any k S Ko -log, c]: and taking
the k-th step, for any
Kn 1; otherwise initialize Vo(1) S 1, the result policy is demanding the maximum a = 1 that drives the victim to refusal state 0 (a terminal state).
497
The values of (0) and (1) are as follows:
00 k-1
H(0) 0, H(L) ΣΣ
-00.
(22)
499 For arbitrary initial value of Vo (1). Vi(1) = a (1-a) -V6(() + (1)) take maximum Vi(1) = when = 1. Therefore, the policy always drives to terminal state 0 at step 1.
(c) Optimal stopping problem (terminating policies, discounted case)
502
Any policy that takes infinite steps will have
k- Σ rd a² ||]||| + (-c)
H(x) = c
-00
(23)
and direct jumping policy will have H(x) -c. Therefore, the H-term drives to terminating policy.
504
17

---

505
B
Mu.oCo Tasks with Multiple Policies
508
B.1
Description of MuJoCo Taks
507
We selected six challenging robotic locomotion tasks from MuJoCo, namely, Swimmer-v3, Hopper-
508
Walker2D-v3, HaliCheetah-v3, Ant-v3, Humanoid-v3. Table Blists the action space and state space of each task.
509
Table The state and action spaces of six challenging MuJoCo tasks.
Tasks
Agent
Action Space
State Space
Swimmer-v3
Three-line swimming robot
2
Hopper-v3
Two-dimensional one-legged robot
11
Walker2d-v3
Two-dimensional bipedal robot
HalfCheetah-v3
awo-dimensional robot
17
Ant-v3
Four-legged creature
111
Humanoid-v3
Three-dimensional bipedal robot
17
376
510
B.2
Multiple policies in MuJoCo tasks
In the supplementary files, we includes rendered videos of different policies, as given in Table 4
Different policies are obtained over 20 runs of the PPO algorithm. We rendered theses polices and classified them by physical gaits.
518
The policies in bold texts physically stationary.
Table 4: List of video files for different policies.
Task
Different Policies
Video Name
hopping
hopper_ hooper_hopping.oom mp4
Hopper
diving
hopper_diving mp4
standing
popper__tandilne mp4
running
ant_running.nnn mp4
Ant
standing
ant_standing mp4
flipping
ant flipping.mip mp4
walking
walker walker_walking www.wwllkkinmp... mp4
Walker
diving
walker_diving..o
standing
walker_ foakker_ttnning.mppe mp4
two-legs
humanoid two legs. mpp
Humanoid
one-leg
humanoid one mgg
backward
humanoid humanoid_backwwrrdnn bac
running
halfchectah running. .mp
diving
hallchectah diving mpp
HalfCheetah
flipping
halfcheetah_ facchhfffippnnnnnmm
standing swimmer wwimmerrmovinggoo. mp4
halfcheetah. seahhhhsttnnnnn.taeee 1t44
moving
Swimmer
standing
swimmer standing.mpt
18

---

C
Quantum K-Spin Hamiltonian Formulation of Reinforcement Learning
515
518
We provide the detailed steps of reformulating T into a K-spin Hamiltonian equation
(0) As [Q Ab)l
8 R(S An)
Jam0
K-
E Sp Ag. Sh SP/S 1.Ak Aseerm I+ R(Sk, As)]
0=
K-18x; SxA
SxA
k-1
(24)
7h R(px) do(So) =o(po)
II [P(S)_1|A4) To (e(+++))l
No
1=0
K SxA
SxA
A-1
Σ
gh R(ma)) do(So) II P(S
xo((Ho) ... (T((k))
k=
<=0
K
SxA
SxA
Σ
Lpp.....p To(Ho) ... (o(m))
0=
Mk
517 where K + 0o, and the density function is
L = R((x)) do (So)
II
P(S++14))
(25)
1=0
Table 5: Revisiting the analogy between MDP and quantum K-spin Ising model.
MDP (Our formulation in (7)
Quantum K-spin Ising Model [26] 12 in 5
State-action pairs
No. .... PK-1
Spins
J0: ... _JK -1
Optimal policy
T
X Th x ... X π PK-- 140
Quantum field
Jj X o x .... x JHH-1
Cumulative rewards
L MK-1
Density function
Functional policy
H (1)--1)
Functional of spins
HH(mp ang
Stationary condition
Stationary condition
bxy
bd,
19

---

518 D
Derivation Steps for Section 4.2: Hamiltonian's Policy Gradients
We provide the policy gradient of the quantum K-spin Hamiltonian equation in 7 for both stochastic and deterministic cases, which are variants of the well-known policy gradient theorem [36].
520
Theorem (Hamiltonian's 's stochastic policy gradient) The stochastic gradient of the K-spin
522
Hamiltonian equation w.r.t. parameter 0 is
K Σ
(oH -Bpo....PKK-
* R(px) Vo Polog (TT((u)) TT(H2) To(m)))
(26)
Corollary 1. When K - the Hamiltonian' is stochastic policy gradient V VoH in 26 is equal to the stochastic policy gradient (0) in (37),
524
lim VoH(0) = - K-000
(O) = - awyyy IQ (s, a) Vo log (T(s
(27)
Let (-a(-) A denote deterministic policy, while use (1) represent that a Gaussian noise aa. an exploration noise) with standard deviation > 0 is added in the exploration process.
526
Theorem 2. (Hamiltonian's deterministic policy gradient) The deterministic gradient of the K-spin Hamiltonian equation w.r.t. parameter 0 is
528
(0) - -p
Σ
R(px)= -
Vo log (Te,((po) (01)(11) -. Ho,s(m)))
(28)
529
Corollary 2. When K - oc, the Hamiltonian's 's deterministic policy gradient V (oH in 28 is
530
equal the deterministic policy gradient V.J. (0) in 135..
lim K-000
= E VaQ Ro. a0) Vomo(s)
(29)
531
Corollary When the variance of the exploration noise approaches zero, iee, 5 + O, the determin-
istic policy gradient V H (0) () is the limiting case of the stochastic policy gradient V (0),
(oH'(0)
lim VoH(O).
(30)
D.1 Proof of Theorem 1 Hamiltonian's Stochastic Policy Gradient
Proof.
K
SxA
SxA
VoH(O) =
Σ
Σ
Lpp Vo [TT(H))) ... T
k=
Ho
K-1SxA
SxA
Σ
Σ
Lp (TO(PO) To Vo log (po) [
14
K- 1xxA
SxA
Σ
xh R((x) do( (T()))
II
I((S)1)A)) To(H4+1) V log [=o(Ho) (T
k=U
K-1
-E Epo.....i. PK-1
Σ
k. R(ux) V 1olog(mo -. ... [[(m))]]
kmo
(31)
where 14 = (Sx, Au), do(.), A ~ To(Sk.)) ), Sk+1 P(. Se, A&) for k: = 0... K.
20

---

535
D.2
Proof of Corollary 1
Proof.
K-1SxA
SXA
VoH(O)
(
Σ
Σ
L pp Vo (To(pO) no (px)]
Pg
K-18xA
SxA
(6)
Σ
Σ
Te(uo) ... TT ()TO .TT VoTo(u))
Mk
km0 0
i=0
K-1 SxA
SxA
k-1
d II no (1y) j=+t1
@
Σ
? R(NE)= dS())) II (P((+11/))))
Σ
mo(pa)) VOTO(us)
k=
Ag
K-1SxA
SxA
(o(S
To
VOta(us)
πI j=i+1
To
(((ux)
Hk
k
S
SxA
9
(S
Σ
p(So, S1,4)
Σ
(oTo A.).
Σ
plSi, Sk. k 1) TO(UE)) R(NA)
1=0
Sp
S.
A
P4
K-1
A
(S
PSO, S,i) E
(ST
ΣΣ
p(S, - Σ
(T(S"'A') AA)
So
8
i=
A'
8
8
A PSO, S,i)
(So)
Σ
Σ
VoTo(S, A) (S,
So
$
1=
S
S
Σs
Sq (o( E P(So, S,i)
(h)
Σ
do (So)
Σ
P(So S,i)
Σ
VOTo(S, A) (Q(SSA)
dS(S)) P(o,
S
Sp
So
(1)
de (S)
(oTO(S (S,(A
-B (5.) ( (S. A) log no (S,
(32)
536 where p(S, S" i) denotes the probability of state transfer S' steps.
537
We provide detailed explanations step-by-step:
Equality (a) holds by definition.
538
In equality (b), using the chain rule, we take derivative of VolTo(HO) ... =e(uu) with respect to ne (JG), 1,...,h. k.
540
In equality (c), we plug in Lio... Mak in (6).
In equality (d), we insert (S (1) P( (4)) between ne (p4) and nT (p:+1), 1,..., k.
543
equality (e), we split trajectory Mo: . My into two trajectories Po. M4 and pk. Therefore, we can classify all trajectories Po. ... , by μo: Pis H4, and i.
N4 544
"K''KK' K- In equality (f), we reorganize Σk-1 Ei into Σk- Ek The former one first traverses the length & of trajeoctory, and then traverses the i-th step on it. The latter one first traverses the i-th step of a trajectory. and then traverses the length & of it.
545
546
547
In equality (g), calculate the limit of (f) when K approaches oc.
In equality (f), normalize ES do(So) E p(So, S, to be probability distribution.
21

---

550 In equality (i), we remove the constant Σs Es. do(So) Σi=o (S.,, S, and replace the fraction
with d=p (S), the stationary distribution of state S under policy
552
In equality (j), we reformulate (i) as expectation.
D.3 Proof of Theorem 2: Hamiltonian's Deterministie Policy Gradient
555 Proof. Let (:) - A denote deterministic policy, while we use (p)) to represent that Gaussian noise (a.k.a, exploration noise) with standard deviation >> 0 is added in the exploration process. the inference stage, there is no exploration noise, the policy is deterministic, i.e., d and A (S()).)
557
H' (0) - -Es do Qu (So, Ao)
Σx R(Sk, Ax)
Sy 5 (
ko=0
K
(SS P( [+ R(S,, An]]
k=0
K SxA
SxA
h-1
(33)
R(PE) do(So) (e,)(po) II [P(S)_I)m)) 70,0(1111)
k=0
==
K
SxA
SxA
A-1
k R(px) do II ((S (0.(Ho)) ... -
k=0
Pk
i=0
K
SxA
SxA
k=
559 where 00, and
L Lp - dol (So) II (()++))))
(34)
22

---

561
D.4
Proof of Corollary 2
Proof.
K
SXA
SXA
VoH'(TO)
Σ
Σ
(Po Vo (wo) =o(m=)) + VOl =o(po)) =o((u)))
k=0
14g
By
K SxA
SxA
(T(())) ... #o(m))]
K=
K SxA
SxA
k-1 II ((((+1ll1)))
Σ
Vo
7x (-((A)) - do(So)
K=0
t=0
K SxA
SxA
k-1
V
R((s)) do(So)
II
P P(S
Vono(s)
k=0
AAg
Mk
Σ
do (S V VAA. P(1S,.))
Σ xh R(S Ab)
Vono(s)
Sa
$
Σ do(So) An) Vono(5)
So
-R
V Q(So, Ao) Vono(S)]
(35)
where jig- (Sx, AR), do(.), Ax ~ To( ,), Sk+1 P(. Se, As). for h: 0... K.
563 D.5 Proof of Corollary
584
Proof. In Corollary and Corollary
we have
(0) ()),
(36)
(0H(0) VoJ(O).
565
when K oc.
586
35
proved that
VeJ lim VeJ(O), 5-+
(37)
567
where the standard deviation of the Gaussian noise of stochastic policy Ty.
588
Therefore,
H (0) lim ( 5-0
(38)
569
23

---

570 E
Variance Reduction (Newly Added)
571 For the general function in { one simple but effective variance reduction technique is to subtract
572
baseline term as follows:
16 pp(())) [[f(=)) 3) logp(æ;;);
(39)
573 where 3 is the baseline term.
Our reasoning logic:
1). We first briefly describe a high-level idea 19 that adding baseline term, like the proposed H-termm will help reduce the gradient variance.
576
577
2). We sketch the steps to show how the proposed H-term will mathematically reduce the gradient variance, following the framework in Section 5.2 of 20
579
High-level IDEA One generic approach reduce the variance of Monte Carlo estimates is an additive control variate. Suppose we wish to estimate the integral of the function / : - R, and we
580
581
know the value of the integral of another function on the same space @ : X - R.
582
We have
-11))=
o(x)) + / o(x),
(40)
584
and the integral of f(x) ((x) can be estimated. ((x))x))))))))( f(x). meaning that then we have managed to reduce our variance to zero [9]. More generally,
585
Var(ʃ (6) +) Var(J) 2C) Var(ø)-
(41)
If & and f are strongly correlated, so that the covariance term on the right hand side is greater than 586
the variance of Φ, i.e. -2Cvv( Φ) + Var () S 0. then variance reduction has been made over the original estimation problem 19] i.c., Var( - x) s Var(f).
586
589
Our reasoning. Then, we present our reasoning.
Note that the gradient of the new objective function of the actor network in (4) consists of two components, namely (0) and (O). Here, we consider
581
VeJ (0), where À 0 a temperature parameter,
(42)
582
where VoJ in 27 is the above function f(-) and À (oH(O) in (5) is the above function d(-)-
593
The Hamiltonian stochastic gradient in (15) has the optimal value
Σ 7 R(px) Vo log (mo(ua) To((1)) mo(rus)) (43)
(0) lim E +00
594
According to Theorem of 19 that is proved via
we have
1
E Eowdo
[Velog log (T.
Var (V) - A Voll' (0) = Var [o)(
-andg.arTo
Bawdg [Vo log (s
< Var VoJ()],,
(44)
where the second term is positive, and
Bande ( Vo log To To (s)
V Vo
(45)
E Sswd (V, log To a))
596 In Alg. 1 and Alg. 2, we used a general H-term (oH( not the optimal one
43 Next, we
597 provide a general characterization for this case..
According to Theorem 10 of [19, we have
Var (V) A H()) Var (oJ) 2 Eando
(46)
[[Vo log To (o
(s)) Voll* (0
24

---

599
Assume Lipschiz continuity of the graident (0) such that
|1V V (0)l S ( (0) - 2Le,
(47)
600 given K 2 log, with 0,6 0, as pointed out in the end of Section 3.2.
601
Therefore, combining 46 with (48 we obtain that
B aawny log a
Var (VJ((0) [o) Varl Var())
aando
E Bando Vo log no (o( a))
)2(22c)² Egndg [VVo log (s, a))²]
Va)(V))
(48)
when both Vo log (s, a)l and VoJ(()) upper bounded, e.g., Vo lo a)l < C1 and 603 [0) < C2; and we set À properly such that
02 + <
(49)
C2
4L2C7
604
which can be easily satisfied by properly selecting and 2 log,
605
Conclusion:
To up, show that it is easy to achieve Var VoJ VoH(O)) Var(V,,)(0)), which
607
means adding the H-term can lead to smaller variance than that of the conventional gradient.
25

---

Conventional Actor-Critic Algorithms for Deep Reinforcement Learning
608
609
The gradient of
36
VoJ(O) - Σ ds a (S) E Qo A) VoTo(S,
(50)
610
Since Qo in (50) unknown 41 the stationary distribution de unknown), one can plug critic network with parameter 6 as estimator of Qe and obtain
((, - ds (S) E (o(SsA)VoTT((sAA)
(51)
where ds,0 pairs.
R x1 denotes the stationary distribution over the states instead of state-action
(51) is a bi-level optimization problem 77. and natural solution is an iterative algorithm that 615 alternates between estimating Qo with parameter o and improving policy we with parameter Therefore, family of actor-critic algorithms are proposed with following objective functions:
Actor max J= 6) = (1 - 7) Ao (Qp(So, Ao)
1
(52)
Critic max (0, = - - ((Q v(S, A))
617
The gradient of
52p can be estimated as follows
1 N
(0,2(0,0) = Σ Q0(m) Vo Vogoggo(
N
(53)
V voJ((0,0) N (Q(( -((, VoRo(S,A)
618
The parameters & and 0 are updated as follows:
Actor 0 0 + VOJn.
(54)
Critic o+ & a V Jq.
26

---

619 G
Stationary Deterministic Policy Gradient Algorithm with H-term
620 For completeness, we present the details of the deterministic actor-critic algorithm with H-term.
Algorithm 2 Stationary Actor-Critic Algorithm with H-term
1: Input: learning rate a, temperature A, look-ahead step K, and parameters 6, M, T, G, B, B'
2: Initialize actor network 7) and critic network Q with parameters 9, ø, and replay buffers D1, D2
3: for episode = 1................... .M do
4: Initialize state $o
5: for t= 0........ T do
6:
Take action aN no (sx) where (0,
7:
Execute action at, receive reward rt, and observe new state 81+1 Store transition (s2, dr rt, S(+1) in D
8:
9:
end
Store a trajectory of length T in 22
11:
for
1, do
Randomly sample mini-batch of transitions {(s,, a,, rs, ( from D 18" from D2
13:
Randomly sample mini-batch of B' trajectories (of length K) {Ti)
14:
Update critic network using conventional method
Update actor network as + 0 + a (Vo((0)-Vo((r1))
15:
end
17: end
We apply the proposed Hamiltonian equation to regularize the actor network. Specifically, (0)
622
in 7 is added to the actor's objective with weight > 0. The objective functions of actor and critic
623
networks become:
Actor: max (0, = (1 (000:))))) SooBoo 1Q (So, Ao)l (0),
Critic :in JQ( 2
1
ES (.). A= ((o(5, y A))]
(55)
The gradient of (55) is
V0 -11 Σ
ds.. (VVo
(56)
625
S 3
V (0)0(0,0) (s.o(S) ( (S, A) v(S (S, A=)mm(()>>>
(57)
628
To estimate V (0) the Monte Carlo gradient estimator in 17 is used. Therefore, 56 and 57 can be estimated as follows:
627
N'
1 N Σ
(o)
V (s) N
k R( Vo log (T . To
k=
(58)
628
V VoJo(0,0)
N Σ (0 4) (S, A)- A))
(59)
27

---

629 H Experiments: Hyperparameters and More Results
630
H.1
Hyperparameters in Experiments
Table 6: Hyperparameters used for the PPO and PPO H in MuJoCo tasks
Parameters
Values
Optimizer
Adam
Learning rate
3 10
Discount (?)
0.99
GAE parameter
0.95
Number of hidden layers for all networks
3
Number of hidden units per layer
256
Mini-batch size
32
Importance rate H-term (A) Truncation step of H-term (K)
2-3
16
Table Hyperparameters used for the DDPG and DDPG + H in MuJoCo tasks
Parameters
Values Adam
Optimizer
Learning rate
5 10
Target Update Rate (r)
10-3
Discount (7)
0.995
Replay buffer size
10"
Number of hidden layers for all networks
3
Number of hidden units per layer
256
Batch size
64
Importance rate H-term (A) Truncation step of H-term (K)
2-3
16
631
H.2
More Results
632
shows the H-value (average over runs) during the training process, which verified that the trained agents have converged to policies with small H-values.
HalfCheetsh
Ant
Humanoid
H values
Ht values
H valuee
#samples Hopper
esamples Swimmer
esamples WalkerrD
H values
H valuees
Hi vaabice
#samples
#samples
@samples
PPO
PPO H, =
PPO + H, = 16
PPO H. K 24
Figure H values during the training process.
28

---

HalfCheetah
Ant
Humanoid
12k
8
K
K
8
Cumulative rewards
5k-
K
16
16
10k
16
4k
8k
3k
Cumulatiiv
2k
2k-
1k
1k
2k
#samples Hopper
#samples Swimmer
#samples Walker2D
8
K
Cumulatiiv rewards
Curulaaive rewards
Curulative rewards
7.5k
K 24
K 24
6k
24
3k
120
4.5k
2k
3k
1k
0
0.0
0.5
1.0 #samples
1.5
2.0
#samples
1e6
4samples
1e6
1e6
Figure 7: For the proposed PPO+H algorithm, the performance with different K values.
Tasks
Policies
PPO
PPO+H (K 8)
PPO+H (K 16)
PPO+H (K = 24)
running 13
17
20
20
flipping
5
4720.8
4839.5
5001.8
0
4995.1
HalfCheetah
diving
1
+969.2
#392.5
#321.5
0
+383.3
balancing
running
17
20
20
20
4164
4351.6
4553.1
4662.5
Ant
jumping
0
1566..4
4-294.5
4-276.4
4:277.5
flipping
3
0
two-legs
15
17
9433.4
9583.4 .4 7733.4
9873.2
9942.2
Humanoid
one-leg
12
-2513.5
5
+653.7
3
++538.4
backward
hopping
10
18
20
20
2659.3
3014.9
3254.2
3340.7
Hopper
diving
8
1.90.3
=304.7
1246.1
#191.5
balancing
2
0
0
moving
14
110.7
121.5
20
142.3
19
132.2
Swimmer
balancing
6
+
#42.1
0
136.9
1
+36.2
walking
15
5461.7
5794.4
5921.8
6089.3
Walker
diving
+1200.1
+334188
3
+-304.5
5
++314.7
balancing
0
Table 8: Experimental results on six challenging MuloCo tasks.
634 Fig. shows more performance the PPO+H algorithm, for = 8, 16, 24. We run each experiment with 20 random seeds and each test 100 episodes.
696
To verify the hypothesize that smaller replay buffer hurts the performance, we rerun the trials of K 8, 16 with a replay buffer size 800.
29

---

638 I Hamiltonian Policy Network
639
L.1
Hamiltonian Policy Network
Since Hamiltonian equation in 7 is functional of policy me, natural question would be: can we use the Hamiltonian equation replace existing Bellman's equation or the policy gradient's
641
objective function 2
643 As verification, we test the capability of Hamiltonian equation in 7 as loss function to train
644 policy network. The algorithm is first given as follows.
Algorithm 3 Hamiltonian Policy Network
1: Input: learning rate a, look-ahead step K, and parameters M,T, G,
2: Initialize policy network with parameters 0, and replay buffer '
3: for episode 1,... .M do
4: Initialize state 80
5: for 0, T do
6:
Select action d₂ ~ TT(-(s))
7: Execute action a,, receive reward rt, and observe new state 8:4-1
8:
end
9:
Store a trajectory of length T in for 1, G do
10:
Randomly sample mini-bater of trajectories (of length K) (r)) from D
12:
Update pocliy network as 0 0 (oH
13:
end
14:
end
645
In Alg. an agent interacts with an environment and updates its policy network. The algorithm has
M episodes and each episode consists of (Monte Carlo) simulation process and a learning process gradient estimation) as follows:
647
648
During the (Monte Carlo) simulation process (lines 5-9 of Alg. B) an agent takes action a according to policy a T-)+s;;, t = 0, ... T - 1, generating a trajectory of T steps/transitions. Then, the full trajectory (sg, Go. ro. $1, ... 11 11 ST) stored replay buffer D.
649
650
652
During the learning process (G' 2 updates in one episode) (lines 10-12 of Alg. Th. mini-batch of B trajectories (of length K) {T, (s0, ag. ru oi sk --, dxx 11 T from D. respectively. The policy network is updated by a Monte Carlo gradient estimator over B trajectories.
1 /= are sampled
1, -
k
654
656
Implementation of replay buffer D. After a full trajectory of length 7' is generated, is partitioned into 7 K + trajectories of length K. We rank them according to the cumulative reward and store the top portion, say 80%. into a new replay buffer D (line 9 of Alg. 3}. We randomly sample batch B trajectories from (line 11 of Alg. 3) to compute the H-term.
658
659
660
1.2
Frozenlake Task
Environment:
Frozenlake 8 x 8, a game in OpenAl Gym.
682
Rules: As shown in Fig. g (left), the Frozenlake task has 8 states with 4 optional actions move around. The agent needs to go from the start point and find the way to the destination in limited steps. There are 8 holes which can cause the agent to fail the game.
683
664
685
Experiment settings: We take Deep Q-learning (DQN) 29 as our baseline and use the implementa-
tion from the ElegantRL library. We use 4-layer fully connected neural network as the deep policy network both in DQN and DHN. We use the Adam optimizer with a learning rate 1 x 10-3 and batch size 100.
687
689
Evaluation: We evaluate the performance of policy by computing the success rate, in which we use 50 agents to walk 100 steps and compute the rates of agents who successfully arrive the destination.
30

---

Results for the Frozenlake task: Fig. (left) shows the success rate of agents with increasing the number of transitions learned by the network. compared with DQN, DHN has more stable training process. It is easy for DQN to quickly obtain good policy to win the game. But with increasing the number of transitions fed to the network, the performance DQN shows large and frequent shock while the performance of DHN shows the strong stability.
672
674
MB
X
gger
Boo
Jole
X
Mar
Hale
End
Figure 8: The Frozenlake task (left) and Gridworld task (right).
676
L33
Gridworld Task
Environment: Gridworld of size 10 10, game available in code.
678
Rules: As shown in the Fig. (right), the Gridworld has 10 10 states with optional actions to move around. The agent will initialize at random locations and it needs to find the smiley as many as possible which has reward turn. should be noted that there are some endpoints which may cause the agent game over and some transfer-poine points which transfer the agent to certain location.
680
682
Experiment settings and evaluation: Both the experiment settings and evaluation method are the same with that on Frozenlake 8 x game.
683
Results for the Gridworld task: Fig. 9 (rigt) shows the mean reward obtained by the agents with increasing the training time. Compared with DQN, DHN has a faster training process. It only needs
685
massive random parallel samples of trajectories and do not need any policy for guided sampling while DQN needs guided exploration the training process which costs a large time consumption.
687
000
400
Reward
0.4
-230
Mym
DON
DON DV
3003
10000
15000
A03000
25000
JU000
100
150
2513
Training Time (Seconds)
Figure 9: Comparison between the DQN and DHN algorithms. The Frozenlake task (left) and Gridworld task (right).
31


Reviews:
[{"review_id": "df-3O7rWju", "reviewer": "Reviewer_65t2", "summary": "The paper posits that the Bellman optimality operator has multiple fixed-points. It becomes apparent that in defining a discounted MDP, the paper allows discount factors $\\gamma = 1$ in the infinite-horizon setting contrary to conventional wisdom in RL. Arguing that the existence of multiple fixed-points is a key source of high variance in RL, the work draws inspiration from statistical mechanics to regularize actor-critic and policy gradient algorithms, and presents results over PPO and DDPG with reduced variance across seeds and improved average performance. ", "questions": "- In L73, the discount factor is defined as $\\gamma \\in (0, 1]$. In L127, $\\gamma \\in (0, 1)$. Why the difference?\n- L114: there is no summation in (6), so why is this called a _cumulative_ reward?\n- In (7), what does a summation from $\\mu_k$ to $\\mathcal{S} \\times \\mathcal{A}$ mean?\n- L265: if memory budget permits replay buffer size 800 for K=24, for a fair comparison it would make sense to set the buffer size to 800 for K=8 and K=16 too.\n- $R(\\tau)$ should be defined after (2).", "limitations": "As far as I can tell, there is no discussion of limitations, except maybe the complexity trade-off due to the truncation parameter K. I'm curious as to whether there exist MDPs such that Hamiltonian regularization negatively impacts performance. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n\n- In both theory and practice, practitioners typically set $\\gamma < 1$ in infinite horizon settings, in which case the Bellman optimality operator has a unique fixed point due to the monotonicity and contraction properties, and the Banach fixed-point theorem. That being said, there exist some exceptional cases such as those discussed in Ch. 3 of the Bertsekas ADP textbook, where discounting with $\\gamma < 1$ may fail to find the optimum policy and additional restrictions on the space of value functions is necessary (Bertsekas, 2019). However, I think saying that the Bellman optimality operator has multiple fixed points (in bold and italics, multiple times) without making it very clear early on that the setting involves $\\gamma \\in [0, 1]$ rather than $\\gamma \\in [0, 1)$ is misleading. Furthermore, there are lots of other valid sources of variance in deep RL including but not limited to optimization of non-convex/non-stationary objectives, stochastic gradients, reward sparsity, initial conditions, complexity of learner function class, etc. that the wording in this paper neglects. I don't suppose existence of multiple fixed points is a primary concern since practitioners use $\\gamma < 1$ in infinite-horizon (and long-horizon) settings. So, I fail to see the motivation behind the proposed regularization approach and remain deeply skeptical of the soundness of this paper.\n- I found the paper to be rather difficult to read. The paper could use copy editing.\n- The paper claims to fix the multiple fixed-point issue with the Hamiltonian regularization scheme, but only shows the effect of its usage for a few pedagogical examples. But I would expect to see a theorem attached to this claim with a mathematical proof of how the issue is fixed under the proposed framework.\n\nStrengths:\n- Despite the awkwardness of the motivation, story and positioning of the paper, I can see some merit in regularizing the policy in the way that this paper proposes. Indeed, temporal regularization has been shown to serve as a good variance reduction technique when applied to the value function [33]. While [8] proposes a form of temporal regularization on the policy with a control prior, their approach requires having access to some known dynamics, while this paper does not. So, I think a major revision of the paper with better motivation, presentation and discussion of related work has potential.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QMXeup90k36", "reviewer": "Reviewer_JucZ", "summary": "This paper aims to resolve the challenge that Bellman's optimality equation has multiple fixed points, which leads to instability in deriving its solution. To this end, the authors first observe that the evaluation of cumulative reward function can be reformulated into a K-spin Ising model. The authors then propose to use the energy of such an Ising model as a regularizer in the actor-critic algorithm. The authors further conduct multiple experiments on the MuJoCo environment.", "questions": "$\\textbf{Question 1.}$\nIs the regularizer term in (7) equivalent to (2), namely, the corresponding cumulative reward function? Why does adding the cumulative reward function as a regularizer improves the stability of the actor-critic? In fact, by rewriting the log of products of policies into the sum of log policies and reorganizing the sum, I find the gradient equivalent with REINFORCE (with a truncation of trajectory to the $k$-th step). Can I understand the gradient of Hamiltonian as a variant of policy gradient estimation?\n\n$\\textbf{Question 2.}$\nHow does utilizing the Hamiltonian regularizer resolves the raised challenge that Bellman's optimality equation has multiple fixed points? Conjecturing that minimizing energy improves the stability is not sufficiently convincing as the formulation of Hamiltonian in equation (7) seems to be the same as the cumulative reward function. That said, minimizing the energy appears to be the same as maximizing the cumulative reward, which is also the objective of various existing policy gradient approaches. More importantly, the adopted AC-style approach is a policy gradient method, so it does not attempt to solve Bellman's optimality equation directly. In contrast, it improves the cumulative reward function by updating the policy directly.", "limitations": "N.A.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strength.}$\n\nThe quantum K-spin Ising model view of RL is interesting and novel to me. \n\n$\\textbf{Weakness.}$\n\nThe challenge raised by the authors does not seem to be fully addressed by the authors. See Question 2 for the details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "69fOyPd0BmG", "reviewer": "Reviewer_Zi6K", "summary": "The paper first suggests that the Bellman's optimality equation has multiple fixed points. Then a analogy is made between MDP and qunatum K-spin Ising model, and a reformulation of expected return into quantum K-spin Hamiltonian equation is proposed. It is argued that by regularizing the policy to have a stationary Hamiltonian,  the model can 1). achieves a relative high reward independent of the initialization; and 2). is robust to interference/noise in the inference stage, and thus reduce performance variance among random seeds. This idea has been practically implemented by randomly sampling consecutive trajectories from a specific replay buffer and minimizing the Hamiltonian by policy gradient. The experiments on MuJoCo robotic control tasks have shown the effectiveness of the proposed methods using both DDPG and PPO as base algorithms, in terms of slightly higher mean performance and significantly lower variance. Furthermore, the agents converged to the stationary policy with a substantially higher ratio with the proposed method.", "questions": "[Major]\n- Line 48: \"we make a novel analogy between an MDP and a quantum K-spin Ising model\". However, ref [21] proposed to model MDP with K-spin Hamiltonian. What is the difference?\n- The optimal policy function $\\pi^*$ and a general policy $\\pi$ are mixed-up. E.g., line 111,  I understand the optimal policy $\\pi^*(\\mu_k) \\in  $ {0 ,1}   can be mapped to spin operator, which is a common practice in quantum computation. However, how about $\\pi(\\mu_k)$, which is a continuous-value scalar? Also, in Table 1, the optimal policy is analogous to the spin operators, while the Hamiltonian the functional of non-optimal policy. I am a bit confused about this mixing-up.\n- The main motivation is based on that the Bellman’s optimality equation, which is the base of Q-learning-like algorithms such as DDPG, has multiple fixed points. But the algorithm also work well on PPO, which even does not use Bellman equation to learn the value function. Can the authors provide any explanation about why it PPO+H works so well?\n- There is a lack of discussion about related work in deep RL to reduce variance, e.g., https://openreview.net/pdf?id=9xhgmsNVHu\n- My personal opinion is that the analogy is confusing to people without quantum physics background. And in the end, the algorithm used the classic policy gradient algorithm to estimate the Nabla of Hamiltonian. I feel that Algorithm 1 itself is indeed intuitive (heuristic) without introducing quantum physics. My suggestion is to defer some details of the analogy to the appendix (then the authors can have more space  to clearly explain it), and complement the main texts with contents such as related work and ablation studies.\n\n\n[Minor]\n- Line 47: I suggest to add \"often\" or \"sometimes\" because there are cases we want some diversity, e.g., option-critic and DIAYN.\n- Line 87: polices --> policies\n- Fig.4 is not centered.\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n1. The paper touches a relatively important problem in deep RL, namely how to reduce variance of policies with different random seeds.\n2. The proposed method is simple and easy to implement.\n3. The experimental results are good, which show the effectiveness of the proposed methods by reducing variance by 65.2% ~ 85.6%\n4. 3 simple yet motivated examples to show that the Bellman’s optimality equation has multiple fixed points.\n\n[Weakness]\n1. Lack of discussion of related work.\n2. The analogy is hard for a reader without quantum physics background. \n3. Some of the paper's claims need to be further supported.\n4. The paper sometimes mixes-up the optimal policy function $\\pi^*$ and non-optimal policy $\\pi$, making the analogy a bit confusing.\n\nSee below for the details of my concerns.\n\n---------------------- post-rebuttal -----------------\nThe author has resolved most of my conerns and agreed to update the manuscript to address the issues. Correspondingly, I update my score toward acceptance, mainly because the results (Table 2) are appealing with a relatively simple add-on (H term). Nonetheless, I expect the authors in the future to more comprehensively investiagte the motivation using empirical and theoretical analysis to support their claims.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "48QfxQ3EYmM", "reviewer": "Reviewer_UzCH", "summary": "This work proposes to help a policy network stably find a stationary policy by making an analogy between an MDP and a quantum K-spin Ising model. To demonstrate the existence of multiple fixed points of the Bellman Optimality equation, the authors used three examples from dynamic programming. The paper empirically evaluated the performance of the newly proposed method on 6 MuJoCo tasks.", "questions": "1. Rather than an analogy between optimal policy and quantum field, should it be just policy with the quantum field? \n2. How much more computational cost is needed for the additional H term as compared to the baseline methods?\n3. if the main point is to make an analogy with physical systems and MDP, why the quantum k-spin Ising model is specifically chosen?\n4. In the Broader Impace Statement the authors state that 'bring together the strengths of both approaches and yield new insights in both fields'. However, I'm not sure what this can bring for the quantum community?", "limitations": "1. Experiment is limited to the 6 MuJoCo tasks. \n2. The analogy to 'lowest energy' makes me worry that this method only works for physical tasks (humanoid, hopper, etc.).\n3. Lacking theoretical analysis on the resulting new algo.\n4. Maybe can include a comparison of the computational cost of this method versus trainiing ten times (depends on how many would be required to reach the same level of stability) of the original DRL agent.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nInteresting problem and approach. The instability of DRL algorithms is definitely a major concern.\nDerivation seems to be sound.\n\nWeaknesses:\nThe experiment is quite limited (6 MuJoCo tasks)\nNumber of compared baseline is too few & did not compare the performance with any quantum RL/Hamiltonian mechanics method\nThe computational cost may prohibit its ability to scale to more complex problems", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject