Stationary Deep Reinforcement Learning with Quantum K-spin Hamiltonian Equation
Authors: Xiao-Yang Liu
Zechu Li
Shixun Wu
Xiaodong Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Instability is a major issue of deep reinforcement learning (DRL) algorithms --- high variance of cumulative rewards over multiple runs. The instability is mainly caused by the existence of \textit{many local minimas} and worsened by the \textit{multiple fixed points} issue of Bellman's optimality equation. As a fix, we propose a quantum K-spin Hamiltonian regularization term (called \textit{H-term}) to help a policy network converge to a high-quality local minima. First, we take a quantum perspective by modeling a policy as a \textit{K-spin Ising model} and employ a Hamiltonian equation to measure the \textit{energy} of a policy. Then, we derive a novel Hamiltonian policy gradient theorem and design a generic actor-critic algorithm that utilizes the H-term to regularize the policy network. Finally, the proposed method significantly reduces the variance of cumulative rewards by $65.2\% \sim 85.6\%$ on six MuJoCo tasks; achieves an approximation ratio $\leq 1.05$ over $90\%$ test cases and reduces its variance by $60.16\% \sim 94.52\%$ on two combinatorial optimization tasks and two non-convex optimization tasks, compared with those of existing algorithms over $20$ runs, respectively.

Under review as a conference paper at ICLR 2023
STATIONARY DEEP REINFORCEMENT LEARNING WITH QUANTUM K-SPIN HAMILTONIAN EQUATION
Anonymous authors Paper under double-blind review
ABSTRACT
Instability is major issue of deep reinforcement learning (DRL) algorithms high variance of cumulative rewards over multiple runs. The instability is mainly caused by the existence of many local minimas and worsened by the multiple fixed points issue of Bellman's optimality equation. As a fix, we propose a quantum K-spin Hamiltonian regularization term (called H-term) help a policy network converge to high-quality local minima. First, we take a quantum perspective by modeling a policy as K-spin Ising model and employ Hamiltonian equation to measure the energy of policy. Then, we derive novel Hamiltonian policy gradient theorem and design generic actor-critic algorithm that utilizes the H- term to regularize the policy network. Finally, the proposed method significantly reduces the variance of cumulative rewards by 65.2% ~ 85.6% on six MuJoCo tasks; achieves approximation ratio S 1.05 over 90% test cases and reduces its variance by 60. 10.16% ~ 94.52% on two combinatorial optimization tasks and two non-convex optimization tasks, compared with those of existing algorithms over 20 runs, respectively.
INTRODUCTION
Instability a major issue of deep reinforcement learning (DRL) 44 algorithms agents trained with different random seeds may have dramatically different performance. Existing works [ g 16 empirically reported a high variance over multiple runs. Hence, in practice it requires to train tens of agents and pick the best one. Such high variance largely contributes to the RL community's ' dispute of reliability and reproducibility [7 [8], limiting the wider adoption in real- world tasks. The instability issue is mainly caused by the existence of many local minimas and worsened by the multiple fixed points issue of Bellman's 's optimality equation [5] 39]. In Fig. [ adapt dynamic programming examples [ 39 into reinforcement learning settings, while detailed descriptions are given in Appx.
28 31.53 533
Shortest path problem (deterministic) in Fig. 1 aa: two policies, 1) transiting back to state 1; 2)
driving to terminal state O.
Blackmailer's problem (stochastic) in Fig. [b): two policies, 1) demanding a - 0 keep the victim at state 1; 2) demanding a = that drives the victim to terminate state 0.
Optimal stopping problem (terminating policies) in Fig. [c): two polices, 1) continuing inside the sphere of radius (1 - are and stopping outside; 2) jumping to point 0 at any point in region C.
The instability problem has been partially addressed, such as ensemble methods 10) regulariza- approaches [4] 50). In particular, Generalized Advantage Estimation (GAE) 41 is a widely used one that significantly reduces the variance of the advantage function. However, they did NOT fix the issue of local minimas and the multiple fixed points issue of Bellman equation in Fig. [ Existing methods randomly converge to different local minimas. For practical usage, we often expect DRL algorithm stably converges to certain policy independent of initialization and noises.
tion approaches [ (66,, and baseline-correctio
As fix, we propose a quantum K-spin Hamiltonian regularization term (H-term) to help policy network converge to a high-quality local minima. We take a novel quantum perspective by modeling
Without explicit clarifications, both "local minima" and "fixed points" in this paper are referring to policies.

---

Under review as a conference paper at ICLR 2023
x
r
= 1-
r= 0
-|||||
r b
p
(1 - a)c
1
0
r=a
r= -C
Cone
(a) Shortest path problem
(b) Blackmailer's problem
(c) minimal stopping problem
Figure 1: Examples with 7 Examples with ? are given in Fig. 6 of Appx.
policy K-spin Ising model 15 30 and employ a Hamiltonian equation measure the energy of a policy, namely an H-term. We hypothesize that a stationary policy would have a low energy.
In this paper, we propose a quantum K-spin Hamiltonian regularization term (called H-term) to help a policy network converge high-quality local minima. Our contributions can be summarized follows: 1) we take quantum perspective by modeling policy K-spin Ising model and employ a Hamiltonian equation to measure the energy policy, which becomes an add-on term to DRL algorithms; 2) we derive novel Hamiltonian policy gradient theorem and design generic actor-critic algorithm that utilizes the H-term to regularize the policy/actor network; 3) we show that the proposed method significantly reduces the variance of cumulative rewards by 65.2% 85.6% six challenging MuJoCo tasks [47]: achieves an approximation ratio S 1.05 over 90% test cases and reduces the variance of approximation ratio by 60.16% 94.52% on two combinatorial optimization tasks (travelling salesman problem B1, graph maxcut (14) and two non-convex optimization tasks (MIMO beamforming 5G/6G 7 non-convex deep learning classifier 33 compared with those of existing algorithms over 20 runs, respectively.
RELATED WORKS
The existence of many local minimas has been theoretically pointed out in robotic control tasks [16], combinatorial optimization tasks [25] and non-convex optimization tasks 3 [22) Existing solutions can be classified three approaches, ensemble method, regularizer, and Dasline-correcte
The ensemble method 2 10
proposed to reduce the variance by using multiple critic networks to approximate an accurate value function. However, this method will still encounter the multiple fixed points issue of Bellman's optimality equation. Regularization method [11] (46) proposed to guide the updating process of policy network. Adding a regularizer essentially helps find local minima with preferred structure, which cannot help escape from local minimas.
Baselioe-correction approaches [1 50 was used to reduce the bias of monte carlo estimation. In particular, Generalized Advantage Estimation (GAE) [4T] is widely used one that significantly reduces the variance of the advantage function. However, the method is restricted by the accuracy of the baseline, which suffers from the local minimas issue as well.
However, they did NOT fix the issue of many local minimas and the multiple fixed points issue of Bellman equation in Fig. [ In contrast, we propose a physically inspired DRL algorithm that stably converges to a certain policy independent of initialization and noises.
Different from our quantum K-spin perspective, several recent papers utilized the (classical) Hamil- tonian equation to endow RI. agents the capability of inductive biases. For example, [24 48 used
Hamiltonian mechanics to train an agent that learns and respects conservation laws; [5 applied
Hamiltonian Monte Carlo (HMC) simulator to approximate the posterior action probability: and 35 proposed an unbiased estimator for the stochastic Hamiltonian gradient methods for min-max optimization problems.
THE PROBLEM OF MANY LOCAL MINIMAS
First, we show the existence of many local minimas in many tasks. Then, we provide observational experiments to empirically verify the existence of multiple policies.
2

---

Under review as a conference paper at ICLR 2023
Humanoid
Hopper
Two-tegs
One-leg
Backward
Hopping
Diving
Balancing
HalfCheetah
Ant
Running
Diving
Balancing
Flipping
Running
Flipping
Figure 2: Different policies for MuJoCo tasks
47
The bold ones are physically stationary policies.
3.1
EXISTENCE MANY LOCAL MINIMAS
We point out that combinatorial optimization and non-convex optimization have many local minimas.
MuloCo tasks
47
agents randomly converge to policies of different gaits, as shown in Fig.
Travelling salesman problem (TSP) B1 case of 8 cities has 2 local minimas (in Appx. K
Graph max-cut [14] example graph of 20 nodes has 390 local minimas (in Appx. K
MIMO beamforming [7] case of 2 users and 2 antennas has 3 local minimas (in Appx.
Non-convex deep learning classifier 33]: an example problem has 25 local minimas 3
3.2 MULTIPLE POLICIES OF EXISTING DRL ALGORITHMS
We provide observational experiments on four challenging MuJoCo tasks [47], namely, Humanoid, Hopper, HalfCheetah, and Ant (details given in Appx. B.I. which are typical examples of the locomotion control of robot. We render the obtained policies over multiple runs and then identify physically stationary ones. We observe various types of moving strategies, as shown in Fig. 2 which verifies that multiple policies are very common. For example, the Humanoid agent learns either jumping with single leg or running with two legs, as shown in Fig. 2 (top-left); another interesting example is HalfCheetah, in which an agent can run normally or in a flipped manner, as shown in Fig. (bottom-lefft) Among the obtained policies, one can easily identify the physically stationary polices that control the robot moving forward with a stable gait (defined as gait that does not lead to fall).
MODELING POLICY AS K-SPIN ISING MODEL
We take novel quantum perspective by modeling policy as a K-spin Ising model and employ Hamiltonian equation to measure the energy of policy, namely an H-term.
4.1 MOTIVATION
Our modeling policy as a quantum K-spin Ising model is inspired by the simulated annealing algorithms and the analogy in Table Simulated annealing algorithms randomly transit to neighbor solution with probability proportional to the energy gap between the current state and a new state. Here, state can be modelled as spin configuration, and Hamiltonian equation is used to measure the energy of a spin system. Take the graph maxcut problem as an example, a spin is a configuration of nodes, while transition (in terms of state-action pair) is taken according a policy. Using the modelling in Table [] we learn policy network that encodes the transition probability of simulated annealing algorithm.
4.2 QUANTIPYING ENERGY OF POLICY VIA K-SPIN HAMILTONIAN EQUATION
The Hamiltonian equation for a quantum K-spin Ising model particular configuration, which takes the following form
15
30
measures the energy of
K Σ
H -
L jo Jo j
(1)
=1=1
3

---

Under review as a conference paper at ICLR 2023
Table I: Modeling a policy as a quantum K-spin Ising model.
Policy in []
Quantum K-spin lsing model [15
30 in
State-action
(k X A. O..... ...K K
Spins
jt (1....
U.
Policy
Kgn x Tuj
Tpx-1
0.
Contiguration
Go og 8
Optimal policy
Pps Nip
Rpx 10,1)a
Optimal configuration
o x 0 x- 3x
Discounted reward
Lyo
Density function
Energy of policy
(Fppor Fpg-1)
Energy
H(og 188--1
where N the number of spins in the k-th configuration. d. =1 are spin variables, and Lo.... is an energy density function for k nearest spins' configuration jo ...
Modeling in Table Starting from an analogy between a state-action pair μز (Sy. A&) and spin Jk. we can map an optimal policy πt ((ux) E {0. 1} to the optimal single-qubit spin operator ja {[11, 1} via T (p²) <> (1 -u, 1/2, where T(2)) denotes the probability of taking action AR at state Se, following policy æ. The energy density function Ljo...a can be defined as the discounted reward on path (Ho,-.. 11k-1) 1) of length k,
k-1
Lpo.....ue = =/ R(pp)) do(s))
II
P( (pe), (obtained via Monte Carlo simulation) t=
(2)
where (5())) denotes the distribution of initial state sp. Analogy to the quantum K-spin Ising model, can derive energy of RI policy H ( -
We formally express the objective of reinforcement learning (background given in Appx. C into a
K-spin Hamiltonian equation (inspired by (0
K-18xA
SXA Lp No (po) ..(o (3)
(0) A AE 1QT Ao)]
lim
Σ
Σ
Σ
+000
where expectation taken over So ~ do(:), Ao ~ To T :), and Lpo..... is given in
Analogy to quantum K-spin system, H()) in measures random path's discounted reward (the "energy") without following any policy, and the Hamiltonian equation combinatorially enumerates all possible paths of length K over the state-action space. The joint probability distribution, π (po) x π (H1) X... X -(4 is decided by the policy и. monte carlo
Physical interpretation:
simulation The energy of a policy is favorable criteria, since an optimal policy with minimum energy: 1). achieves a relative high reward independent of the initialization; and 2). is robust to interference/noise in the inference stage. In other words, the simulation process of the Hamiltonian term does not rely on any policy. Therefore, the Hamiltonian term is a suitable regularizer for both on-policy and off-policy algorithms.
K-step truncation in practice. Minimizing is NP-hard 13 Since E (0, 1), T monotonically decreases with look-ahead steps K, therefore, we truncate (3) finite K terms. One can show that these K terms in 33 is geometric sequence with truncation eгror ratio 1 - > Assuming C, where > 0 small, thus we have the look-ahead steps K 2 log, c.
1
4.3 H-TERM HELPS CONVERGE TO HIGH-QUALITY LOCAL MINIMA
We elaborate how adding the energy in (3) onto each state can help drive to the terminal state (a stationary policy), which fixes the issue in Section [ We have H()) O for the terminal state O.
(a) Shortest path problem (deterministic), H(1) = - b -o. At state 1, the Bellman's optimality equation becomes V(1) = max(V(1) + (H). Independent of the initial value Vo(1), agent obtains policy that always transits back terminal state 0.
(b) Blackmailer's problem (stochastic): H(1) = -o. The Bellman's optimality equation becomes (1)= max, {a (1-a) (VV (1)) for state For any (1( oo, the optimal policy becomes a = 1 that drives to the terminal state O.
(c) Optimal stopping problem (terminating policies): any policy that takes infinite steps will have Hzz) oc, since each step number k. there are always trajectories that jump point 0 with reward - c; and a direct jumping policy will have H (x) = -c. Therefore, adding H (x) to each point f 0 will lead to policy of jumping back to point 0.
4

---

Under review as a conference paper at ICLR 2023
R(m)) + ... - (((
x(HO) ... ((Hx-1)
R(pT)
T(Po)
1
T
K
a(10)
मt ((pr-1)
R(
Th (R(uk-i)
Figure 3: REINFORCE's policy gradient (left) VS. Hamiltonian's 's policy gradient (right).
STATIONARY DEEP REMENTRCEMENT LEARNING
First, we propose a novel Hamiltonian policy gradient and the corresponding Monte Carlo estimator. Then, we present a stationary actor-critic algorithm with H-term.
5.1 HAMILTONIAN POLICY GRADIENT AND MONTE CARLO-BASED GRADIENT ESTIMATOR
We provide the policy gradient of the quantum K-spin Hamiltonian equation in 3), which are variants of the well-known policy gradient theorem [4]. We provide detailed derivations in Appx. B
Theorem 1. (Stochastic version) The Hamiltonian stochastic gradient of (3) 3 is
K-1 Σ
V (0) -Epp....pxx- PX-1
Vo log (To(Pp) ne (p1) .. To (pk)))
(4)
Let noo (-) S + A denote deterministic policy, while we use (
( to represent that a Gaussian noise (a.k.a, an exploration noise) with standard deviation 5 >> 0 is added in the exploration process.
Theorem 2. (Deterministic version) The Hamiltonian deterministic gradient of 3
K-1
(0) = -Bpp......ppe PK-
Σ * -((()))) Vo log (Mo,)(Ha) To,s(H1) ... (...T. m R=
(5)
The quantum K-spin Hamiltonian equation in (3) is reformulation of 15 We verify the gradient calculation by showing that: when - 0o, the Hamiltonian stochastic and deterministic policy gradient voll (0) and (0) are equal to the stochastic policy gradient VoJ(O) in 45 and deterministic policy gradient (() [43 respectively.
Note that the gradient ()) in (4 and VoH in (5) w.r.t distributional parameter & takes an expectation form. Thus, Monte Carlo gradient estimator is practically useful. We obtain the Monte Carlo gradient estimator of Vo (0), illustrated in Fig. (right), follows
N' Σ
V V (()
1 N
R(p)) V log (TT(()) [+()]]]
(6)
As a contrast, we provide the Monte Carlo gradient estimator of REINFORCE'S illustrated in Fig. 3 (left), follows
policy gradient,
T
V voJ NT Σ
Σ
G
Vo Volog
where G = Σ -
('=+11
An interesting observation is that both gradient calculations follow similar pattern shown in Fig. 3 REINFORCE's policy gradient 45 in Fig. (left) employs estimate of future rewards, while
Hamiltonian policy gradient Fig. (right) uses trajectories in replay buffer Dz.
Computational complexity: we measure the computation complexity by the times of computing one Va log No (p). Assume N B and N' B', since most DRL algorithms use mini-batel stochastic gradient decent methods. REINFORCE's 45] policy gradient in (7) takes OBBT) computations, while Alg. adds O(B + 1)/2) computations in each gradient update step, thus total complexity of O(BT + B' ((K 1)/2).
5.2 STATIONARY ACTOR-CRITIC ALGORITHM WITH H-TERM
Actor-critic algorithms in reinforcement learning perform a bilevel optimization, namely alternating between approximating value function and optimizing policy. In practice, critic network with
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Stationary Actor-Critic Algorithm with H-term
1: Input: learning rate a, temperature A, look-ahead step K, and parameters M, 1"
2: Initialize actor network and critic network Q with parameters 0, o, and replay buffers Dj, D2
3: for episode 1...,............ .M do
4: Initialize state $u
5: for t 0, ...
do
6:
Select action a₃ (0(+(s))
7: Execute action ot, receive reward rt, and observe new state 81+1
8:
Store a transition (st, az, rt, S(+1) in D
9:
end
Store a trajectory of length T in D2
11:
for = 1. do
12:
Randomly sample mini-batch of transitions {(s,, ai Ts, 81- (s=1))=_ from Dj
13:
Randomly sample mini-batch of B' trajectories (of length K) {T, ) from D2
14:
Update critic network using conventional method
15:
Update actor network as <- 0 + -
end
17: end
parameter o approximates the Q-value function, and an actor network with parameter 0 approximates the policy æ, details given in Appx. G However, since the critic's update is governed by the Bellman's optimality equation, actor-critic algorithms suffer the multiple fixed points problem.
Motivated by Section 4.3 algorithms. Similar to the entropy term in (27), the proposed H-term is an add-on term to regularize the actor network and help converge stationary policy. Specifically, the objective functions of actor and critic networks become:
propose novel H-term for both deterministic and stochastic actor-critic
Actor max (,
Φ) & (1 Ao [Q (So, Ao) (0)
Critic min (0,0 Φ) A
2 ( (Qa - USS ())_
where a target Q-value is yS,, Ax) R(S,, Ax) + 7o Ax+1) and > is temperature parameter. As an interpretation, the second term (H) the maximization objective function of actor network aims to find minimum energy configuration for the MDP problem, namely, a policy that will add a minimum amount of energy each state's value function (as in Section 4.3
New algorithm. In Alg. I an agent interacts with an environment and alternatively updates its actor network and critic network. The algorithm has M episodes and each episode consists of (Monte Carlo) simulation process and learning process (gradient estimation) as follows:
During the (Monte Carlo) simulation process (lines 5-10 of Alg. D an agent takes action a; ac- cording to policy (t),,,,, f Then, these transitions are stored into a replay buffer D,, while the full trajectory т (so, ao, To, 81, 277-1,7T111 -T 87) is stored in replay buffer D2.
o.... T - 1, generating a trajectory of T steps/transitions.
During the learning process (G 2 1 updates in one episode) (lines 11-16 of Alg. [.. a mini- batch of B transitions {(si, ai, Ti 81, 1 and mini-batch of B' trajectories (of length K) - rk 8" are sampled from D and D2, respectively. The critic network is updated by conventional method, e.g., minimizing the mean squared error (MSE) between an estimated Q-value and a target value. The actor is updated by a Monte Carlo gradient estimator over B transitions and 3' trajectories.
(7, = (sg ag rg. si,... - 11,
Two new hyperparameters. We introduce two hyperparameters: a temperature > 0 that is a relative weight of the H-term, and look-ahead step K < T that defines the horizon of the H-term.
Implementation of replay buffer D2. After a full trajectory 5 of length is generated, it is partitioned into T - K + trajectories of length K. We rank them according to the cumulative reward and store the top portion, say 80%, into a new replay buffer D., (line 10 of Alg. []. We randomly sample mini-batch of B' trajectories from Dr (line 13 of Alg. to compute the H-term.

---

Under review as a conference paper at ICLR 2023
HallChectah
Ant
Humanold
DRPGG
DOPG DDPG
DEPG
DCPG.
4K
Cumulatie
3k-
Cumuistive
54
2k
4x
1.4
2k-
4
0.0
0.5
1.0
1.3
2.0
s'ammp Hopper
105
Asamples Swimmer
106
Asammles
187
Walker2D
54
200
160
DORG_H
DCRS_
Camulative nenurts
DOPG PER
Cumilthi wordd
DOnG rEll
120
Curuiaaiie
2%
3.5
3.0
3.0
l05
lcd
106
HalfCheelah
Ant
Humanoid
52
12k
PPO
1970
PPO
5k
APO_H
4k
PPO_H
10k
PPO_
42
3k-
Cumudll
3K
Cumuistive
2X
esamples Hopper
asamples
Walker2D
34-
200
IPO
190
PPO FPO, Ht
Cumuuette resverds
4k-
Cumulative nuverds
160
1PO.
3k-
120
2k
80-
14
40-
3.0
d.somples
145
#ssmppes
assmmllss
140
Figure 4: Cumulative rewards vs. #samples for compared DRL algorithms on six MuJoCo tasks.
PERFORMANCE EVALUATION
We evaluate the proposed H-term from four aspects: 1) converging to high-quality local minima, 2) reducing variance, 3) driving stationary policy, and 4) the impact of trajectory length K. All experiments were executed on an NVIDIA DGX-2 server [[2]. The server contains A100 GPUs, 320 GB GPU memory, and 128 CPU cores running 2.25 GHz.
6.1 EXPERIMENTAL SETTINGS
Environments (tasks). We consider six challenging MuJoCo tasks 47], two combinatorial opti- mization tasks ( TSP and graph maxcut described in Appx. K). and two non-convex optimization tasks (MIMO beamforming 5G/6G and non-convex deep learning classifier in Appx. D For the MuJoCo tasks, the agent learns to control the locomotion of a robot and aims to move forward as quickly as possible. For graph maxcut and TSP. the agent learns to find a near-optimal solution. For MIMO beamforming and non-convex deep learning classifier, the agent learns to optimize the objective function. These tasks have high-dimensional continuous state space and action space, in which there exists multiple locally optimal polices revealed in Section 3.1
Compared methods. To evaluate both deterministic and stochastic algorithms, we choose Deep Deterministic Policy Gradient (DDPG) (34] and Proximal Policy Optimization (PPO) [42] for MuJoCo tasks. For TSP, graph maxcut, MIMO beamforming and non-convex deep learning classifier, choose REINFORCE 45
We implement the PPO algorithm with the GAE trick [4T). For
7

---

Under review as a conference paper at ICLR 2023
graph maxcut
bearffrmiig
non-convex classifier
1.0
1.0
1.0
1.0
REINFORCE REINFORCE_ H
REINFORCE
REINFORCE REINFORCE_ H
REINFORCE
0.8
REINFORCE_M H
REINFORCE_H
Frequency
Frequency
Frequency
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
1.09
>1.19
1.01
1.09
>1.19
1.01
1.09
21.19
1.09
11.19
Approximation Ratio
Approximation Ratio
Approximati on Ratio
Appreximation
Figure 5: Frequency of approx. ratio for a) TSP with N 100, b) Graph maxcut with N 100, c) MIMO beamforming with N = 4, and d) Non-convex deep learning classifier. A lower approximation ratio is better.
fair comparison, we keep the hyperparameters (listed in Appx. D the same and make sure that the obtained results reproduce existing benchmark tests [16]. For combinatorial optimization, we use the same datasets from [4] 31].
For MuJoCo tasks, employ two performance metrics, the cumulative rewards and variance, while in Section[6.4 we further consider different policies and report the number of convergence. For TSP, graph maxcut, MIMO beamforming, and nonconvex deep learning classifier, employ the approximation ratio max( Optimal Obj and its variance as the performance Obj Optimal
Performance metrics.
metric. We each experiment with 20 random seeds and in each run we test 100 episodes.
6.2 H- TERM CONVERGES TO A HIGH-QUALITY LOCAL MINIMA
Experience replay is crucial in improving performance in terms of cumulative reward. The proposed H-term in (6) can be viewed as a novel experience replay technique for an actor network. Here, we add compared algorithm, DDPG with Prioritized Experience Replay 40 (DDPG+PER), where PER prioritizes experience by the TD error to update a critic network.
In Fig. 4 both DDPG+PER and DDPG+H achieve a substantial improvement of cumulative reward. In particular, DDPG+H achieves the highest cumulative rewards all six tasks, which are comparable to PPO's performance in Fig. It is worthwhile to discuss the advantage of DDPG+H over DDPG+PER. DDPG+PER utilizes prioritized replay strategy to obtain a more accurate critic network, however, it is updated via the Bellman equation with the trouble of multiple fixed points. In contrast, the H-term in DDPG+H is performed on the actor network. Our results indicate that an experience replay technique actor network may be much more powerful.
In Fig. REINFORCE+H improves the approximation ratio substantially. particular, REIN- FORCE+H achieves approximation ratio 1.05 over 90% test cases in TSP, graph maxcut, MIMO beamforming, and non-convex deep learning classifier. Specifically, H-term achieves an approximation ratio 1.01 (near-optimal) over 90% tests MIMO beamforming task. Our results indicate H-term helps the policies converge to high-quality local minimas over multiple runs.
6.3 H-TERM REDUCES VARIANCE
The PPO algorithm with GAE is regarded as the state-of-the algorithm in MuJoCo environments. However, it still has a very high variance (the shaded area) after the policies have converged, as shown in Fig. We observe that, at the end of training, the PPO algorithm has a variance of 969.2. 22 1563.4, 2513.5. 905.3, 60.7, 1290.1 in the six tasks, respectively. Such a high variance is mainly due to the fact that the agent may converge to a random one of multiple policies.
In Fig- 4 the shaded areas of PPO+H (K = 16) are dramatically smaller, i.e., variance of 228.4, 225.4, 683.7, 184.2, 31.6, and 296.8, respectively. The variance has been reduced by 65.2% 85.6%, which verifies the effectiveness of the proposed H-term. In Fig. 4 we also observe that the H-term can help the DDPG algorithm reduce variances, namely, the variances of DDPG+H are much smaller than those of vanilla DDPG and DDPG+PER. Therefore, may conclude that the H-term guides the agent to search for a stationary policy among multiple feasible ones.
In Fig. []] the variance of approximation ratios of REINFORCE+H (K 16) are substantially smaller. The variance has been reduced by 60.16% ~ 94.52%. The results indicate that the H-term guides policies high-quality minimas over multiple runs.
8

---

Under review as a conference paper at ICLR 2023
Table Experimental results on six challenging MuloCo tasks.
Tasks
Policies
PPO
PPO+1I (K K
8)
PPO+1I (K 16)
PPO+II = 24)
running
13
19
20
20
flipping
5
4720.8
5028.4
0
5104.3
0
4995.1
HalfChectah
diving
1
#969.2
=211.3
0
#228.4
0
#383.3
balancing
1
0
0
running
17
20
20
20
4164
4505.3
1645.6
1662.5
Ant
jumping
+1563.4
+253.66
0
+225.4
+277.5
flipping
3
0
0
two-legs
7
17
16
17
9433.4
9670.3
10189.1
9942.2
Humanoid
one-leg
12
12513.5
1497.2
4
1683.7
3
1:588.4
backward
1
0
0
hopping
10
20
20
2659.3
3116.5
3300.1
3340.7
Hopper
diving
8
+905.3
=289.4
0
11842
0)
1991.
balancing
2
0
0
moving
14
110.7
20
130.6
20
132.5
19
132.2
Swimmer
balancing
6
=60.7
:33.5
0
+31.6
1
#36.
walking
16
15
5461.7
5819.9
5927.2
6089.3
Walker
diving
8
12200.1
#315.6
4
2296
5
#314.7
balancing 7
2
0
0
More experimental results are given in Appx. [uue to the space limit, including the cases K and K 24, and the H-term value during the training process. One may verify that the stationary policies have relative lower H-values.
6.4 H-TERM DRIVES TO PHYSICALLY STATIONARY POLICY
key question needs be answered: is H-term really guiding the agent converge physically
stationary policy? Similar to Section 3.1 we perform observational experiments MuloCo tasks and measure the number of convergences to different policies over 20 runs. As shown in Table [2] the vanilla PPO algorithm converges to the physically stationary policy (bold) with 13, 17, 7, 10, 14, and 5 times for the six tasks, while the PPO+H (K = 16) converges to the stationary policy with 20, 20, 16, 20, 20, and 16 times, respectively. From the empirical observation, we find that the PPO gets stuck in locally optimal policies, failing to find a consistent one. As expected, PPO+H can converge to the stationary policy with a substantially higher ratio, which verifies the effectiveness of the proposed H-term in finding a physically stationary policy.
6.5 IMPACT OF TRAJECTORY LENGTH K
We investigate the impact of trajectory length K. From 6 we know that large K means more accurate estimation of H (f) but at price of computations. Here, we evaluate PPO+H with K 8, 16 and set the size of replay buffer D: to 000. In Table 2 we observe that the cumulative reward increases and the variance decreases as K increases from to 16. However, for the case K 24, both metrics get worse due to the out-of-memory issue and we reduce the replay buffer size to 800. The smaller replay buffer size hurts the diversity of the trajectories and may lead to performance drop. Appx. I.2 provides results for replay buffer size 800.
CONCLUSIONS
In this paper, we have addressed the foundational issue of many local minimas. This issue leads to the instability of DRL algorithms, puts a challenge on their reliability and reproducibility, and thus limits the wider adoption of DRL algorithms real-world tasks. As fix of the problem, propose physically inspired regularizer by modeling policy quantum K-spin Ising model. Experimental results show that the H-term helps DRL algorithms converge
high-quality local minima, reduce the variance of cumulative rewards by 65.2% ~ 85.6% on six MuJoCo tasks, achieve an approximation ratio 1.05 over 90% test cases and reduce its variance by 60. 60.16% 94.52% on two combinatorial tasks and two non-convex optimization tasks, compared with those of existing algorithms over 20 runs, respectively.
For future works, we will explore the potential of directly training a policy network using Appx. J quantum simulator [29] and quantum reinforcement learning [19] It is interesting to apply Monte Carlo estimator for unbiased policy gradient calculations. We would like to show that the proposed H-term can help distributional RL algorithms [4] find a stationary policy, since the distributional Bellman optimality operator is not contraction and thus there is also unique policy.
in

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances Neural Information Processing Systems, 2021.
[2] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning. In International Conference Machine Learning. pages 175-185. PMLR, 2017.
[3] Peter Auer, Mark Herbster, and Manfred KK Warmuth. Exponentially many local minima for single neurons. Advances neural information processing systems, 8, 1995.
[4] Marc G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International Conference Machine Learning, pages 449-458 PMLR, 2017.
[5] Dimitri Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019.
[6] Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rehentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, 549(7671) (1177761)):199220) 2017.
[7] Emil Björnson, Eduard Jorswieck, al. Optimal resource allocation in coordinated multi-cell systems. Foundations and Trends@ in Communications and Information Theory, 9(2-3): 113- 381, 2013.
[8] Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the reliability reinforcement learning algorithms. 2019.
[9] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34, 2021.
[10] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without a model. ICLR, 2021.
[11] Richard Cheng. Abhinav Verma, Gabor Orosz, Swarat Chaudhuri. Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. International Conference on Machine Learning, pages 1 1141-1150. PMLR, 2019.
[12] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. NVIDIA A100 Tensor Core GPU: Performance and innovation. IEEE Micro, 41(2):29 25, 2021.
[13] Barry Cipra. The ising model np-complete. SIAM News, 33(6): 2000.
[14] Hanjun Dai, Elias Khalil, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
[15] Vasil Denchev, Sergio Boixo, Sergei Isakov, Nan Ding, Ryan Babbush, Vadim Smelyanskiy. John Martinis, and Hartmut Neven. What is the computational value of finite-range tunneling? Physical Review ((3):031015, 2016.
[16] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329-1338. PMLR, 2016.
[17] Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint (rXiv:2003.1188) 2020.
[18] Gabriel Dulac-Arnodd, Daniel Mankowitz, and Todd Hester. Challenges real-world reinforce- ment leamning 2019.
10

---

Under review as a conference paper at ICLR 2023
[19] Vedran Dunjko, Jacob M Taylor, and Hans Briegel. Quantum-enhaneee machine learning. Physical review letters, 117(13)113050,, 2016.
[20] Eliot Kapit Wesley Jones Eric B. Jones, Peter Graf. K-spin Hamiltonian for quantum-resolvable markov decision processes. ArXiv, abs/2003. 1881, 2020.
[21] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. ICLR, 2019.
[22] Evan Greensmith, Peter Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Advances Neural Information Processing Systems, 14, 2001.
[23] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
[24] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks, Advances in Neural Information Processing Systems, 32, 2019.
[25] Don A Grundel, Pavio A Krokhmal, Carlos AS Oliveira, and Panos Pardalos. On the number of local minima for the multidimensional assignment problem. Journal of Combinatorial Optimization, 13(1):1-18, 2007.
[26] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12(1):1-12) 2021.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
(28) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[29] Eric B Jones, Peter Graf, Eliot Kapit, and Wesley Jones. K-spin Hamiltonian for quantum- resolvable markov decision processes. Quantum Machine Intelligence, 2(2):1-11, 2020. 202.
[30] Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario Vecchi. Optimization by simulated annealing. Science, 220(4598):671-600 1-600, 1983.
[31] Wouter Kool, Herke Van Hoof, and Max Welling. Attention, learn solve routing problems! 2018.
arXiv preprint
[32] Eugene I. Lawler and David E Wood, Branch-and-bound methods: A survey. Operations research, 14(4):699-719, 1966.
[33] Ke Li and Jitendra Malik. Learning optimize. arXiv preprint arXiv:1606.01883 2016.
[34] Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez. Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
[35] Nicolas Loizou, Hugo Berard, Alexia Jolicoeur-Martinng Martineau, Pascal Vincent, Simon Lacoste- Julien, and loannis Mitliagkas. Stochastic Hamiltonian gradient methods for smooth games, In International Conference on Machine Learning, pages 6370-6381. PMLR, 2020.
[36] Rich Caruana Matthew and Matthew Mullin. Estimating the number of local minima in big, nasty search spaces. In In Proceedings of IJCAI-99 Workshop on Statistical Machine Learning for Scage-Scale Optimization. Citeseer, 1999.
[37] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. ICI.R, 2013.
11

---

Under review as a conference paper at ICLR 2023
[38] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(132):1-22 2020.
[39] Alexey B Piunovskiy. Examples in Markov Decision Processes, volume 2. World Scientific, 2013.
[40] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. ICLR, 2016.
[41] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. ICLR, 2016.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 0as/1707706347, 2017.
[43] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning. pages 387-395. PMLR, 2014.
[44] Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[45] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[46] Pierre Thodoroff, Audrey Durand, Joelle Pineau, and Doina Precup. Temporal regularization for markov decision process, Advances in Neural Information Processing Systems, 31, 2018.
[47] Emanuel Todorov. Tom Erez, and Yuval Tassa. MuJoCo: A physies engine for model-basee based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[48] Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. ICLR, 2019.
[49] Junfeng Wen, Saurabh Kumar, Ramki Gummadi, and Dale Schuurmans. Characterizing the gap between actor-critic and policy gradient. In International Conference on Machine Learning, pages 11101-11111111 PMLR, 2021.
[50] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. ICLR, 2018.
[51] Duo Xu and Faramarz Fekri. Improving actor-critic reinforcement learning via Hamiltonian Monte Carlo method. Deep Reinforcement Learning Workshop at NeurIPS, 2021.
[52] Xuchen You and Xiaodi Wu. Exponentially many local minima in quantum neural networks. In International Conference on Machine Learning, pages 12144-12155. PMLR, 2021.
[53] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 161 1.01578, 2016.
12


Reviews:
[{"review_id": "ct5mjCmmUvA", "reviewer": "Reviewer_947x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a new regularization term for deep reinforcement learning which is inspired by the energy term from quantum physics. In general, the regularization term aims to maximize the adjusted likelihood function over the existing trajectories, which is opposite to the optimization goal of policy gradient which aims to maximize the expected reward in the future. Experiment results suggest that such an approach can improve the trained policy with better value and smaller variances.", "summary_of_the_review": "This work proposes a new regularization term for policy optimization on deep reinforcement learning which seems a better choice than existing approaches on Mujoco dataset. However, I feel the demonstration of several examples in this paper is hard to understand, and the discussion of the limit of the proposed method is not well-discussed. Therefore, I recommend a reject to this paper. ", "strength_and_weaknesses": "Pros:\n- The presentation is easy to follow.\n- The experiment results are comprehensive.\n- The proposed method is easy to implement and has the potential to be applied to more tasks. \n\nCons:\n- Some demonstrations are hard to understand. For instance, In Sec 4.3, I do not understand the Bellman equation used in (a-c). Why there does not exist the discount factor $\\gamma$? If the authors want to discuss the case $\\gamma=1$, they should not simply adopt the original Bellman equation, but the revised one for infinite-horizon MDPs, like (5) in [1]. \n\n[1] Jaksch, Thomas, Ronald Ortner, and Peter Auer. \"Near-optimal Regret Bounds for Reinforcement Learning.\" Journal of Machine Learning Research 11 (2010): 1563-1600.\n\n- Potential drawbacks of the proposed algorithm are not discussed. My understanding about why the proposed regularization term can help the policy have a smaller variance and better value is that it sacrifices the ability to explore. The regularization term $H$ forces the policy to follow the existing samples, especially the samples in the early steps $h\\ll H$. Although such a strategy may work well on Mujoco dataset where the exploration is not crucial, I doubt such a strategy will work well on some harder instances, such as the combination lock example in [2]. \n\n[2] Agarwal, Alekh, et al. \"Pc-pg: Policy cover directed exploration for provable policy gradient learning.\" Advances in neural information processing systems 33 (2020): 13399-13412.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X-QDXwoK6Y", "reviewer": "Reviewer_6uJh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a regularization term (namely H-term) for stabilizing the training of DRL. The inspiration of the H-terms comes from the K-spin Ising model with energy minimization principle. Experiments on PPO and DDPG for MuJoCo show variance reduction of the proposed method.", "summary_of_the_review": "The paper investigates an interesting problem about stabilizing DRL training. But the proposed H-term is not well justified from a theoretical perspective. The performance improvement of different MuJoCo tasks seem to be marginal. Experiments testify the reduced variances of policy performance but not policy gradient estimation. I would also suggest showing clear evidence of variance reduction of the gradients on small games with the ground truth gradient available.", "strength_and_weaknesses": "Strength:\n\nStabilizing the training of DRL is an important topic. The idea from Ising model for minimizing the energy in policy optimization is novel. \n\nWeakness:\n\nThe relationship of the Hamiltonian policy gradient (HPG) and the REINFORCE policy gradient (RPG) is less clear. From the method description it looks like the HPG is an alternative format for estimating the gradient of objective in RPG, however in the end the H-term is only a regularizer.\n\nSince the variance estimation is important in the paper, I would also suggest to report the results with stratified bootstrap confidence intervals [1].\n\nThe proof of variance reduction using H-term is not convincing. In appendix F.2 the paper gives a theoretical justification of the variance reduction optimizing the policy with an additional H-term. However, it looks like the proof is merely about showing the strong correlation of $H$ and $J$ thus reducing the total variance when $\\nabla_J(\\theta)$ is subtracted by $\\lambda \\nabla H(\\theta)$ with a small $\\lambda$. If this is the case, simply letting $H=J$ with $\\lambda<1$ can reduce the variance, but with no help to the policy gradient optimization. \n\nK-step truncation is less clear. In the paragraph in Sec. 4.2 about truncation of K, what is the truncation error about and what is $\\epsilon$ here, and why does it simply want $\\gamma^K\\ge \\epsilon$?\n\nTypos:\nLine 631 in appendix should refer to `Section 4.2`\nParagraph Physical interpretation in Section 4.2, ‘monte carlo simulation’.\n\n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4evI2MJ89Km", "reviewer": "Reviewer_arYu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new policy gradient method, which can potentially reduce the variance of the policy gradient estimator.", "summary_of_the_review": "I don’t think the newly proposed policy gradient estimator has any connections to the K-spin Ising Model. Meanwhile, I suspect the newly proposed policy gradient estimator is identical to the standard one and the only difference is in the empirical estimation of the cumulative reward.", "strength_and_weaknesses": "### Weaknesses:\n* I feel the motivation for the paper is not so clear. I don’t see any superiority to reinterpret the reinforcement learning problem in this K-spin Ising Model framework. It only covers a very small subset of the Markov Decision Process problem (e.g. the optimal action should be on a lattice), and there are no quantum related things here.\n* I think the authors do not have a clear understanding of statistical mechanics. At least, the term “Hamiltonian equations” is created by the authors themselves. We only have Hamilton’s equations of motions (or Hamiltonian mechanics). In fact, the authors only use the Hamiltonian of the K-spin Ising model. I did expect there are some more motivations from statistical mechanics, e.g. we need to develop specific update rules to reserve the Hamiltonian for certain purposes. However, if I understand correctly, what the authors do do not have any relations with this K-spin Ising Model framework.\n* This new policy gradient estimator seems closely related to other estimators, as $\\nabla_{\\theta} \\log (\\Pi_i \\pi_{\\theta}(a_i |s_i)) = \\sum_i \\nabla_{\\theta} \\log \\pi_{\\theta} (a_i|s_i)$. I even suspect by this decomposition the proposed policy gradient estimator is identical to the original one and the only difference between the newly proposed estimator and the original policy gradient estimator is how to estimate the cumulative reward (by the critic or by roll-out). The authors should provide more discussions on this.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject