Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a regularization term (namely H-term) for stabilizing the training of DRL. The inspiration of the H-terms comes from the K-spin Ising model with energy minimization principle. Experiments on PPO and DDPG for MuJoCo show variance reduction of the proposed method.
strength_and_weaknesses: Strength:

Stabilizing the training of DRL is an important topic. The idea from Ising model for minimizing the energy in policy optimization is novel. 

Weakness:

The relationship of the Hamiltonian policy gradient (HPG) and the REINFORCE policy gradient (RPG) is less clear. From the method description it looks like the HPG is an alternative format for estimating the gradient of objective in RPG, however in the end the H-term is only a regularizer.

Since the variance estimation is important in the paper, I would also suggest to report the results with stratified bootstrap confidence intervals [1].

The proof of variance reduction using H-term is not convincing. In appendix F.2 the paper gives a theoretical justification of the variance reduction optimizing the policy with an additional H-term. However, it looks like the proof is merely about showing the strong correlation of $H$ and $J$ thus reducing the total variance when $\nabla_J(\theta)$ is subtracted by $\lambda \nabla H(\theta)$ with a small $\lambda$. If this is the case, simply letting $H=J$ with $\lambda<1$ can reduce the variance, but with no help to the policy gradient optimization. 

K-step truncation is less clear. In the paragraph in Sec. 4.2 about truncation of K, what is the truncation error about and what is $\epsilon$ here, and why does it simply want $\gamma^K\ge \epsilon$?

Typos:
Line 631 in appendix should refer to `Section 4.2`
Paragraph Physical interpretation in Section 4.2, ‘monte carlo simulation’.


[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

clarity,_quality,_novelty_and_reproducibility: The writing and notations of the paper need to be heavily polished. There are lots of misleading notations in the current version. For example, in Eq. (3) $H(\theta)$ the $\theta$ is parameters of the policy, while in the above paragraph the argument of $H$ is the chain of policies, and later in Sec. 4.3 the arguments of $H$ become the state indices. In Sec. 5.1 computational complexity, does the ‘computations’ here actually mean the gradient computation through the whole network in the backpropagation? It’s very vague to say the number of computations. 

In Section 6.1 the performance metrics, the ‘Optimal’ and ‘obj’ are not explained.

The idea is novel.

The code is given and should be reproducible.

summary_of_the_review: The paper investigates an interesting problem about stabilizing DRL training. But the proposed H-term is not well justified from a theoretical perspective. The performance improvement of different MuJoCo tasks seem to be marginal. Experiments testify the reduced variances of policy performance but not policy gradient estimation. I would also suggest showing clear evidence of variance reduction of the gradients on small games with the ground truth gradient available.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new policy gradient method, which can potentially reduce the variance of the policy gradient estimator.
strength_and_weaknesses: ### Weaknesses:
* I feel the motivation for the paper is not so clear. I don’t see any superiority to reinterpret the reinforcement learning problem in this K-spin Ising Model framework. It only covers a very small subset of the Markov Decision Process problem (e.g. the optimal action should be on a lattice), and there are no quantum related things here.
* I think the authors do not have a clear understanding of statistical mechanics. At least, the term “Hamiltonian equations” is created by the authors themselves. We only have Hamilton’s equations of motions (or Hamiltonian mechanics). In fact, the authors only use the Hamiltonian of the K-spin Ising model. I did expect there are some more motivations from statistical mechanics, e.g. we need to develop specific update rules to reserve the Hamiltonian for certain purposes. However, if I understand correctly, what the authors do do not have any relations with this K-spin Ising Model framework.
* This new policy gradient estimator seems closely related to other estimators, as $\nabla_{\theta} \log (\Pi_i \pi_{\theta}(a_i |s_i)) = \sum_i \nabla_{\theta} \log \pi_{\theta} (a_i|s_i)$. I even suspect by this decomposition the proposed policy gradient estimator is identical to the original one and the only difference between the newly proposed estimator and the original policy gradient estimator is how to estimate the cumulative reward (by the critic or by roll-out). The authors should provide more discussions on this.

clarity,_quality,_novelty_and_reproducibility: The paper is overall poorly written and hard to follow. Meanwhile, I believe the proposed method is not so novel.
summary_of_the_review: I don’t think the newly proposed policy gradient estimator has any connections to the K-spin Ising Model. Meanwhile, I suspect the newly proposed policy gradient estimator is identical to the standard one and the only difference is in the empirical estimation of the cumulative reward.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies a new regularization term for deep reinforcement learning which is inspired by the energy term from quantum physics. In general, the regularization term aims to maximize the adjusted likelihood function over the existing trajectories, which is opposite to the optimization goal of policy gradient which aims to maximize the expected reward in the future. Experiment results suggest that such an approach can improve the trained policy with better value and smaller variances.
strength_and_weaknesses: Pros:
- The presentation is easy to follow.
- The experiment results are comprehensive.
- The proposed method is easy to implement and has the potential to be applied to more tasks. 

Cons:
- Some demonstrations are hard to understand. For instance, In Sec 4.3, I do not understand the Bellman equation used in (a-c). Why there does not exist the discount factor $\gamma$? If the authors want to discuss the case $\gamma=1$, they should not simply adopt the original Bellman equation, but the revised one for infinite-horizon MDPs, like (5) in [1]. 

[1] Jaksch, Thomas, Ronald Ortner, and Peter Auer. "Near-optimal Regret Bounds for Reinforcement Learning." Journal of Machine Learning Research 11 (2010): 1563-1600.

- Potential drawbacks of the proposed algorithm are not discussed. My understanding about why the proposed regularization term can help the policy have a smaller variance and better value is that it sacrifices the ability to explore. The regularization term $H$ forces the policy to follow the existing samples, especially the samples in the early steps $h\ll H$. Although such a strategy may work well on Mujoco dataset where the exploration is not crucial, I doubt such a strategy will work well on some harder instances, such as the combination lock example in [2]. 

[2] Agarwal, Alekh, et al. "Pc-pg: Policy cover directed exploration for provable policy gradient learning." Advances in neural information processing systems 33 (2020): 13399-13412.
clarity,_quality,_novelty_and_reproducibility: This work studies a new regularization term for deep reinforcement learning. It is different from the existing regularization term, for instance, the entropy of the policy, which aims to enhance the exploration ability of the policy. It seems that the authors' strategy works well on several benchmarks such as MuJOCO. Therefore, I think this work is novel. 
summary_of_the_review: This work proposes a new regularization term for policy optimization on deep reinforcement learning which seems a better choice than existing approaches on Mujoco dataset. However, I feel the demonstration of several examples in this paper is hard to understand, and the discussion of the limit of the proposed method is not well-discussed. Therefore, I recommend a reject to this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to address the instability issue of Deep Reinforcement Learning (DRL) and proposes a quantum K-spin Hamiltonian regularization term (called H-term) to help a policy network converge to a high-quality local minima. This new regularization term is inspired from the K-spin Ising model with energy minimization principle. Authors perform experiments by adding H-terms to both PPO and DDPG on MuJoCo tasks, which show that the proposed methods (1) are more likely to find better solutions, (2) have lower variance.

While the paper presents a new interesting idea, reviewers all have difficulty understanding the deep connections between deep reinforcement learning and K-spin Ising model. The theoretical arguments/derivations presented in the paper are not convincing. Multiple reviewers also pointed out the potential connection between the proposed methods with REINFORCE policy gradient, which is not clearly specified in the paper. We thus recommend rejection.
justification_for_why_not_higher_score: The main idea of the paper is not very convincing. A important connection to prior algorithm is not well discussed.
Decision: Reject