Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper demonstrates the use of a quantum annealing machine to solve a free-energy based reinforcement learning problem. Experimental results are demonstrated on a toy gridworld task, where if I understand correctly it does better than a DQN and a method based on RBM-free-energy approximation (Sallans and Hinton, 2004)

Clarity: The paper is very hard to read. It seems to be targeted towards a physics/quantum hardware crowd rather than a machine learning audience. I think most readers, even those very familiar with probabilistic models and RL, would find reading the paper difficult due to jargon/terminology and poorly explained concepts. The paper would need a major rewrite to be of interest to the ML community.

Relevance: RL, probabilistic models, and function approximators are all relevant topics. However, the focus of the paper seems to be on parts (like hardware aspects) that are not particularly relevant to the ML community. I have a hard time imagining follow-up work on this, given that the experiments are run on a toy task and require specialized hardware (so they would be extremely difficult to reproduce/improve upon).

Soundness: I can't judge the technical soundness as it is mostly outside my expertise. I wonder if the main algorithm the work is based on (Crawford et al, 2016) has been peer-reviewed (citation appears to be a preprint, and couldn't find a conference/journal version).

Significance: It's good to know that the the quantum annealing machine can be used for RL. However, the method of choice (free-energy based Q-function approximation) seems a bit exotic, and the experimental results are extremely underwhelming (5x3 gridworld).
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: There is no scientific consensus on whether quantum annealers such as the D-Wave 2000Q that use the transverse-field Ising models yield any gains over classical methods (c.f. https://arxiv.org/abs/1703.00622). However, it is an exciting research area and this paper is an interesting demonstration of the feasibility of using quantum annealers for reinforcement learning. 

This paper builds on Crawford et al. (2016), an unpublished preprint, who develop a quantum Boltzmann machine reinforcement learning algorithm (QBM-RL). A QBM consists of adding a transverse field term to the RBM Hamiltonian (negative log likelihood), but the benefits of this for unsupervised tasks are unclear (c.f. https://arxiv.org/abs/1601.02036, another unpublished preprint). QBM-RL consists of using a QBM to model the state-action variables: it is an undirected graphical model whose visible nodes are clamped to observed state-action pairs. The hidden nodes model dependencies between states and actions, and the weights of the model are updated to maximize the free energy or Q function (value of the state-action pair).

The authors extend QBM-RL to work with quantum annealers such as the D-Wave 2000Q, which has a specific bipartite graph structure and requires special consideration because it can only yield samples of hidden variables in a fixed basis. To overcome this, the authors develop a Suzuki-Trotter expansion and call it 'replica stacking', where a classical Hamiltonian in one dimension higher is used to approximate the quantum Hamiltonian. This enables the use of quantum annealers. The authors compare their method to standard baselines in a grid world environment.

Overall, I do not want to criticize the work. It is an interesting proof of concept. But given the high price of quantum annealers, limited applicability of the technique, and unclear benefits of the authors' method, I do not think it is relevant to this specific conference. It may be better suited to a workshop specific to quantum machine learning methods. 
=======================================
+ please add an algorithm box for your method. It deviates significantly from QBM-RL. For example, something like: (1) init weights of boltzmann machine randomly (2) sample c_eff ~ C from the pool of configurations sampled from the transverse-field Ising model using a quantum annealer with chimera graph (3) using the samples, calculate effective classical hamiltonian used to approximate the quantum system (4) use the weight update rules derived from Bellman equations (spell out the rules). 

+ moving the details of sampling into the appendix would help; they are not important for understanding the main ingredients of your method

There are so many moving parts in your system, and someone without a physics background will struggle to understand it. Clarifying the algorithm in terms familiar to machine learning researchers will go a long way toward helping people understand your method. 

+ the benefits of your method is unclear - it looks like the method works, but doesn't outperform the others. this is fine, but it is better to be straightforward about this and bill it as a 'proof of concept' 

+ perhaps consider rebranding the paper as something like 'RL using replica stacking for sampling from quantum boltzmann machines with quantum annealers'. Elucidating why replica stacking is a crucial contribution of your work would be helpful, and could be of broad interest in the machine learning community. Right now it is too dense to be useful for the average person without a physics background: what difficulties are intrinsic to a quantum Hamiltonian? What is the intuition behind the Suzuki-Trotter decomposition you develop? What is the 'quantum' Boltzmann machine in machine learning terms (hidden-hidden connections in an undirected graphical model!)? What is replica-stacking in graphical model terms (this would be a great ML contribution in its own right!)? Really spelling these things out in detail (or in the appendix) would help
==========================================
1) eq 14 is malformed

2) references are not well-formatted

3) need factor of 1/2 to avoid double counting in sums over nearest neighbors (please be precise)
review_title: replica stacking for quantum annealers used for RL

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is easy to read  for a physicist, but I am not sure how useful it would be for ICLR... it is not clear for me it there is an interest for quantum problems in this conference. This is something I will let to the Area Chair to deceede. Other than this, the paper is interesting, certainly correct, and provides a nice perspective on the future of learning with quantum computers. I like the  quantum "boltzmann machine" problems. 

I feel, however, but it might be a bit far from the main interest of the conference.

Comments:

* What the authors called "Free energy-based reinforcement learning" seems to me just the minimization / maximiation of the free energy. This is simply maximum likelihood applied to the free energy and I think that calling it "reinforcement learning" is not only wrong, but also is very confusing, given this is usually reserved to an entirely different learning process.

* While i liked the introduction of the quantum Boltzmann machine, I would be happy to learn what they can do? Are these useful, for instance, to study correlated fermions/bosons? The paper does not explain why one should be concerns with these devices.

* The fact that the simulation on a classical computer agrees with the one on a quantum computer is promising, but I would say that this shows that, so far, there is not yet a clear advantage in using a quantum computer. This might change, but in the mean time, what is the benefits for the ICLR community?

review_title: Interesting but maybe far from the main topic of the conference

Meta-review: All three reviewers agreed that the paper was an interesting, giving a demonstration of what quantum computer could achieve. However, they all also felt that the topic was outside the main interests of the conference and better suited to other venues, e.g. a quantum computation workshop. The AC agrees with them. Thus unfortunately, the paper cannot be accepted.
Decision: Reject