Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: I thank the authors for their response. I think the empirical results are encouraging and the idea is reasonably novel, so will keep the score as is.

--- 
The paper discusses a parametrized quantum computations as a policy trained via policy gradient. The policy is parametrized with a quantum computer whereas parameter updates are done by a classical computer. Situations where the quantum policy can have advantage over classical ones are discussed and evaluated. In PQC-generated environments, PQC-policies have a significant advantage over classical ones. 
main_review: Originality: The approach taken is generally a standard one taken with quantum machine learning -- a quantum circuit is parametrized and optimized via a classical update algorithm. Since gradients of the quantum circuit can be evaluated using e.g. the parameter shift trick, it is not entirely surprising for a policy gradient algorithm to be developed. 

The paper mentioned some recent related work on this front, but it would also be interesting to discuss older related works (https://arxiv.org/abs/0810.3828, https://arxiv.org/abs/1401.4997, https://arxiv.org/abs/1612.05695). One contribution is the introduction of softmax-PQC which clearly outperforms raw-PQC policies.

Applicability: The PQC-constructed environment is largely an MDP-extension of Liu et al., and it does not seem that the action in the PQC-cliffwalk affects the trajectory other than episode termination. The PQC policy is limited to discrete actions and a comparison with classical algorithms is not included.

Clarity: the paper is written very well and is about readable as it can get for an audience that knows ML but not quantum computing. 

Minor questions and or comments:
- How does the TensorFlow quantum environment simulate quantum noise (or does it)? I would suppose a larger noise would hinder learning progress even though policy gradient should be able to learn with more episodes.
- It seems to me that the major advantage of PQC is the ability to learn a map for DLP -- are there other more realistic examples where PQC policies are needed?
- It seems that the softmax-PQC is not limited to reinforcement learning. Do you expect it to achieve similar improvements in supervised learning?
limitations_and_societal_impact: Yes.

No negative societal impact is discussed, but I don't think it is necessary for this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The article titled ‘Parametrized Quantum Policies for Reinforcement Learning’ presents a hybrid quantum-classical learning model, which the author claims that can be effectively trained to solve several standard benchmarking environments. The learning model is based on parametrized quantum circuits (PQC), where the parameters can be adjusted during the training process. By defining RAW- and SOFTMAX-PQC, the author shows the detailed learning methods corresponding each of the cases. The work also displays the result of this two methods, which we can see a moderate learning process is happening. In the end, the author also argues the ability of parametrized quantum circuits to solve certain learning tasks that are intractable to classical models. 

main_review: The article is written nicely and the it is easy to follow the construction. However, I think there are two major drawbacks in this work. Firstly, in the RAW-PQC design, the policy is output of the quantum circuit. The probabilities of different strings are first evaluated and the an action is sampled from this distribution. This is redundant in my opinion because a quantum circuit is already a good sampler. Even though the author proves it is efficient to sample in this case, one can just use the measurement result each time as the sample of the distribution. So I do not see an obvious reason why the author need to evaluate the probability distribution first and then do the sampling. Secondly, in the SOFTMAX-PQC design, the policy is determined by the weighted mean value in line 112. It seems nice but there is a fatal drawback here. With this design, it may happen that the classical optimizer can optimize the weight \omega to get whatever probability distribution, even with the quantum circuit always producing the same output at all times. In this sense, the information that the quantum circuit is supposed to convey is absorbed into the classical parameters \omega.

limitations_and_societal_impact: As for the limitations and societal impact, the author argues that there exist RL environments where PQC policies can provide a learning advantage over standard classical policies. However, it looks like the argument is mostly change of narratives based on previous results and the contribution is rare.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: ## Summary 

The paper proposes a policy-based training framework with variational quantum circuits (VQC). Although the VQC-based deep reinforcement learning algorithm has been first studied in [20] and [23], this paper further addresses a policy-based optimization instead of a value-based method (e.g., DQN) in [20]. 

The introduction and related work parts are good covering most of the important references in the field. 

The paper further discusses two variants of the parameterized quantum circuit (PQC), raw-PQC and softmax-PQC to compared with a DNN-based RL method in three DRL environments from OpenAI gym. 

- Noted I am not sure how fair is the selection DNN-based RL method in terms of model capacity according to the current version. 

The authors further discuss some points included of (1) Efficient policy sampling and (2) quantum advantages. 

Overall, the paper is well-written and motivated. The major weakness is on the baseline selection and some scalable discussion on quantum advantages. 

- baseline selection

The works in [22] using a continuous quantum Deep RL method, called Quantum DDPG. What is the advantage between the proposed softmax PQC compared with Quantum DDPG?

- scalable discussion

I just notice that the value-based method paper [20] uses a simple tangible radio RL environment to discuss the trade-off between parameter complexity (Figure 11 in [20]) versus different classical and quantum RL methods. 

If the author could provide experimental supports or discussion versus that existing analysis on the QA on parameters, the discussion could be more comprehensive. 



main_review: ## Pros

- The paper provides a policy-based method for quantum deep reinforcement learning.

- The discussion and related works are of good quality. 

- The results show the capability of Quantum DRL working on new challenges environment as Mountains Car (initial with negative values) and Acrobot-v1.

## Points to Improve

- The quantum advantage needs some empirical analysis [Ref 1] or learning complexity analysis (PAC theory) to have a full claim. 
Please refer to the potential scalable discussion in the summary to have parameter complexity improvement. 

- The DNN policies in Figure 5 are associated with the same amount of trainable parameters or? This would need more clarification. List or provide a parameter table for each baseline would be a solution. 


## Addtional Disccusion 

- There is less discussion on the full checklist, such as potential energy consumption (CPU-simulated or using QPU batch directly) corresponding to the NeurIPS policy. 

- If the proposed method could also benefit from a decentralized framework as the recent studies [Ref 2] in a hybrid classical-quantum A2C policy? 


Overall, the paper is around the borderline acceptance threshold. 

I give my recommendation to borderline acceptance considering its pros, weaknesses, and detailed supplementary report. I would consider modifying my score based on the additional baseline studies and discussion in a revised version.




Ref 1. A Grover-search based quantum learning scheme for classification, New Journal of Physics, 2021

Ref 2. Decentralizing feature extraction with quantum convolutional neural network for automatic speech recognition, IEEE ICASSP 2021

Ref 3. Quantum-enhanced machine learning, Physical review letters 2016
limitations_and_societal_impact: - The checklist is not fully satisfied on 1. (c) Did you discuss any potential negative social impacts of your work? [No]

The authors may consider adding a board impact section in their revised supplementary.
review_title: null

Meta-review: The authors present a quantum reinforcement learning regime that utilizes a parameterized quantum circuit. The paper is clearly written. There are a few concerns with the scientific contributions and side points from the reviewers, especially why the PQC plays an essential role in the learning model. However, the authors have replied explicitly the concerns in the discussion and the reviewers and the authors have reached consensus. The authors also explain in detail on possible applications of the work and the questions on the complexity analysis. In short, I would recommend this work to be accepted. 

Decision: Accept (Poster)