Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper first suggests that the Bellman's optimality equation has multiple fixed points. Then a analogy is made between MDP and qunatum K-spin Ising model, and a reformulation of expected return into quantum K-spin Hamiltonian equation is proposed. It is argued that by regularizing the policy to have a stationary Hamiltonian,  the model can 1). achieves a relative high reward independent of the initialization; and 2). is robust to interference/noise in the inference stage, and thus reduce performance variance among random seeds. This idea has been practically implemented by randomly sampling consecutive trajectories from a specific replay buffer and minimizing the Hamiltonian by policy gradient. The experiments on MuJoCo robotic control tasks have shown the effectiveness of the proposed methods using both DDPG and PPO as base algorithms, in terms of slightly higher mean performance and significantly lower variance. Furthermore, the agents converged to the stationary policy with a substantially higher ratio with the proposed method.
strengths_and_weaknesses: [Strength]
1. The paper touches a relatively important problem in deep RL, namely how to reduce variance of policies with different random seeds.
2. The proposed method is simple and easy to implement.
3. The experimental results are good, which show the effectiveness of the proposed methods by reducing variance by 65.2% ~ 85.6%
4. 3 simple yet motivated examples to show that the Bellman’s optimality equation has multiple fixed points.

[Weakness]
1. Lack of discussion of related work.
2. The analogy is hard for a reader without quantum physics background. 
3. Some of the paper's claims need to be further supported.
4. The paper sometimes mixes-up the optimal policy function $\pi^*$ and non-optimal policy $\pi$, making the analogy a bit confusing.

See below for the details of my concerns.

---------------------- post-rebuttal -----------------
The author has resolved most of my conerns and agreed to update the manuscript to address the issues. Correspondingly, I update my score toward acceptance, mainly because the results (Table 2) are appealing with a relatively simple add-on (H term). Nonetheless, I expect the authors in the future to more comprehensively investiagte the motivation using empirical and theoretical analysis to support their claims.


questions: [Major]
- Line 48: "we make a novel analogy between an MDP and a quantum K-spin Ising model". However, ref [21] proposed to model MDP with K-spin Hamiltonian. What is the difference?
- The optimal policy function $\pi^*$ and a general policy $\pi$ are mixed-up. E.g., line 111,  I understand the optimal policy $\pi^*(\mu_k) \in  $ {0 ,1}   can be mapped to spin operator, which is a common practice in quantum computation. However, how about $\pi(\mu_k)$, which is a continuous-value scalar? Also, in Table 1, the optimal policy is analogous to the spin operators, while the Hamiltonian the functional of non-optimal policy. I am a bit confused about this mixing-up.
- The main motivation is based on that the Bellman’s optimality equation, which is the base of Q-learning-like algorithms such as DDPG, has multiple fixed points. But the algorithm also work well on PPO, which even does not use Bellman equation to learn the value function. Can the authors provide any explanation about why it PPO+H works so well?
- There is a lack of discussion about related work in deep RL to reduce variance, e.g., https://openreview.net/pdf?id=9xhgmsNVHu
- My personal opinion is that the analogy is confusing to people without quantum physics background. And in the end, the algorithm used the classic policy gradient algorithm to estimate the Nabla of Hamiltonian. I feel that Algorithm 1 itself is indeed intuitive (heuristic) without introducing quantum physics. My suggestion is to defer some details of the analogy to the appendix (then the authors can have more space  to clearly explain it), and complement the main texts with contents such as related work and ablation studies.


[Minor]
- Line 47: I suggest to add "often" or "sometimes" because there are cases we want some diversity, e.g., option-critic and DIAYN.
- Line 87: polices --> policies
- Fig.4 is not centered.

limitations: N/A
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper posits that the Bellman optimality operator has multiple fixed-points. It becomes apparent that in defining a discounted MDP, the paper allows discount factors $\gamma = 1$ in the infinite-horizon setting contrary to conventional wisdom in RL. Arguing that the existence of multiple fixed-points is a key source of high variance in RL, the work draws inspiration from statistical mechanics to regularize actor-critic and policy gradient algorithms, and presents results over PPO and DDPG with reduced variance across seeds and improved average performance. 
strengths_and_weaknesses: Weaknesses:

- In both theory and practice, practitioners typically set $\gamma < 1$ in infinite horizon settings, in which case the Bellman optimality operator has a unique fixed point due to the monotonicity and contraction properties, and the Banach fixed-point theorem. That being said, there exist some exceptional cases such as those discussed in Ch. 3 of the Bertsekas ADP textbook, where discounting with $\gamma < 1$ may fail to find the optimum policy and additional restrictions on the space of value functions is necessary (Bertsekas, 2019). However, I think saying that the Bellman optimality operator has multiple fixed points (in bold and italics, multiple times) without making it very clear early on that the setting involves $\gamma \in [0, 1]$ rather than $\gamma \in [0, 1)$ is misleading. Furthermore, there are lots of other valid sources of variance in deep RL including but not limited to optimization of non-convex/non-stationary objectives, stochastic gradients, reward sparsity, initial conditions, complexity of learner function class, etc. that the wording in this paper neglects. I don't suppose existence of multiple fixed points is a primary concern since practitioners use $\gamma < 1$ in infinite-horizon (and long-horizon) settings. So, I fail to see the motivation behind the proposed regularization approach and remain deeply skeptical of the soundness of this paper.
- I found the paper to be rather difficult to read. The paper could use copy editing.
- The paper claims to fix the multiple fixed-point issue with the Hamiltonian regularization scheme, but only shows the effect of its usage for a few pedagogical examples. But I would expect to see a theorem attached to this claim with a mathematical proof of how the issue is fixed under the proposed framework.

Strengths:
- Despite the awkwardness of the motivation, story and positioning of the paper, I can see some merit in regularizing the policy in the way that this paper proposes. Indeed, temporal regularization has been shown to serve as a good variance reduction technique when applied to the value function [33]. While [8] proposes a form of temporal regularization on the policy with a control prior, their approach requires having access to some known dynamics, while this paper does not. So, I think a major revision of the paper with better motivation, presentation and discussion of related work has potential.
questions: - In L73, the discount factor is defined as $\gamma \in (0, 1]$. In L127, $\gamma \in (0, 1)$. Why the difference?
- L114: there is no summation in (6), so why is this called a _cumulative_ reward?
- In (7), what does a summation from $\mu_k$ to $\mathcal{S} \times \mathcal{A}$ mean?
- L265: if memory budget permits replay buffer size 800 for K=24, for a fair comparison it would make sense to set the buffer size to 800 for K=8 and K=16 too.
- $R(\tau)$ should be defined after (2).
limitations: As far as I can tell, there is no discussion of limitations, except maybe the complexity trade-off due to the truncation parameter K. I'm curious as to whether there exist MDPs such that Hamiltonian regularization negatively impacts performance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes to help a policy network stably find a stationary policy by making an analogy between an MDP and a quantum K-spin Ising model. To demonstrate the existence of multiple fixed points of the Bellman Optimality equation, the authors used three examples from dynamic programming. The paper empirically evaluated the performance of the newly proposed method on 6 MuJoCo tasks.
strengths_and_weaknesses: Strengths:
Interesting problem and approach. The instability of DRL algorithms is definitely a major concern.
Derivation seems to be sound.

Weaknesses:
The experiment is quite limited (6 MuJoCo tasks)
Number of compared baseline is too few & did not compare the performance with any quantum RL/Hamiltonian mechanics method
The computational cost may prohibit its ability to scale to more complex problems
questions: 1. Rather than an analogy between optimal policy and quantum field, should it be just policy with the quantum field? 
2. How much more computational cost is needed for the additional H term as compared to the baseline methods?
3. if the main point is to make an analogy with physical systems and MDP, why the quantum k-spin Ising model is specifically chosen?
4. In the Broader Impace Statement the authors state that 'bring together the strengths of both approaches and yield new insights in both fields'. However, I'm not sure what this can bring for the quantum community?
limitations: 1. Experiment is limited to the 6 MuJoCo tasks. 
2. The analogy to 'lowest energy' makes me worry that this method only works for physical tasks (humanoid, hopper, etc.).
3. Lacking theoretical analysis on the resulting new algo.
4. Maybe can include a comparison of the computational cost of this method versus trainiing ten times (depends on how many would be required to reach the same level of stability) of the original DRL agent.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to resolve the challenge that Bellman's optimality equation has multiple fixed points, which leads to instability in deriving its solution. To this end, the authors first observe that the evaluation of cumulative reward function can be reformulated into a K-spin Ising model. The authors then propose to use the energy of such an Ising model as a regularizer in the actor-critic algorithm. The authors further conduct multiple experiments on the MuJoCo environment.
strengths_and_weaknesses: $\textbf{Strength.}$

The quantum K-spin Ising model view of RL is interesting and novel to me. 

$\textbf{Weakness.}$

The challenge raised by the authors does not seem to be fully addressed by the authors. See Question 2 for the details.

questions: $\textbf{Question 1.}$
Is the regularizer term in (7) equivalent to (2), namely, the corresponding cumulative reward function? Why does adding the cumulative reward function as a regularizer improves the stability of the actor-critic? In fact, by rewriting the log of products of policies into the sum of log policies and reorganizing the sum, I find the gradient equivalent with REINFORCE (with a truncation of trajectory to the $k$-th step). Can I understand the gradient of Hamiltonian as a variant of policy gradient estimation?

$\textbf{Question 2.}$
How does utilizing the Hamiltonian regularizer resolves the raised challenge that Bellman's optimality equation has multiple fixed points? Conjecturing that minimizing energy improves the stability is not sufficiently convincing as the formulation of Hamiltonian in equation (7) seems to be the same as the cumulative reward function. That said, minimizing the energy appears to be the same as maximizing the cumulative reward, which is also the objective of various existing policy gradient approaches. More importantly, the adopted AC-style approach is a policy gradient method, so it does not attempt to solve Bellman's optimality equation directly. In contrast, it improves the cumulative reward function by updating the policy directly.
limitations: N.A.
review_title: null

Meta-review: metareview: The paper proposes to add a regularisation term H to RL algorithms in order to work around issues caused by the multiple fixed points of the Bellman’s optimality equation. The added H term is inspired by quantum field theory, specifically the K-spin Ising model.
All reviewers thought this was an interesting idea, but by the end of the review period, there remained some problems with this paper. Indeed, this paper is not a theory paper, and there is no mathematical proof that the added H term does accomplish the stated goal of variance reduction. This leaves us with empirical evidence. Unfortunately, as was pointed out by reviewers, "Experiment is limited to the 6 MuJoCo tasks", which is not enough to convince that the algorithm should generally work. Finally, many reviewers were confused by the claim that PPO solves the Bellman Optimality Equation. By the end of the review, not all reviewers were convinced this problem had been resolved. This point should be clarified, and it would be better for the paper to go through a new round of reviews before being accepted for publication.
confidence: Certain
Decision: Reject