Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose to augment the observation with heuristic information in deep reinforcement learning.
Because the heuristic information might not always help the agent, they are learning two policies at the same time.
One is using the heuristic information while the other is not.
They performed experiments with ablation studies showing that their approach was learning faster.

strengths_and_weaknesses: Strength:
The paper is easy to understand and well written.

Weaknesses:
The approach is a bit ad hoc. There is no real justification for picking one or the other policy.

In Fig. 5 DIAYN w/ z and EBE-AC(0.2) w/ DIAYN are actually quite close.
I suspect that the small gain of EBE-AC(0.2) w/ DIAYN is actually due to the fact that some information in z impedes the learning.
For instance, if I have not grasped the object yet f_distance(object, target position) and f_fingers will not help to make a decision.
Accordingly, once the object is already grasped, f_distance(gripper, object) is not helping.

Two missing baselines would be to learn two policies with SAC both without z and both with z. To see if the gain is not coming from a higher expressive power (2 policies).

summary_of_recommendation: I am not convinced by the method because it somehow tries to tackle an ill-defined problem.
Using this method, it is assumed that the information in z is not always useful. In this case, it would be easier to detect which information of z is useful at which times and condition the policy on that. Rather than learning two policies at the same time, it seems easier to learn a succession of policies. For example, one to catch the object with a well-defined z, then another to move it to the target with another z definition.

issues: Some effort for the readability of the curves should be made.

Typos: 
-line 105 are induced 
-line 151 need 
-line 240 maintain 
-line 252 improve
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for guiding exploration during reinforcement learning using binary and continuous heuristics.
strengths_and_weaknesses: Strengths
-- Incorporating human knowledge into learning is an important area of research
-- The authors have included significant formal detail about their approach

Weaknesses

-- The related work section is highly inadequate.  
-- The paper was very difficult to read and needs significant proofreading
-- The evaluation fails to provide strong justification for the proposed method

Suggestions for Improvement

-- Include other methods that incorporate human knowledge into learning as baselines
-- Validate the method across more than task (ideally including real-world robot experiments as well as simulation)
-- Provide significantly more context to relate the contribution to prior work and the state-of-the-art

summary_of_recommendation: The general idea of using heuristics to speed learning (especially in robotics) is good, but the paper needs significant revisions to make it easier to follow, better situate the work relative to prior work, and more fully demonstrate that the approach improves over prior work.
issues: -- Please discuss the proposed work in more detail in relation to the works cited. What are the contributions of the proposed work? How does it advance the state-of-the-art?
-- The notation in the algorithmic sections needs to be revised; the p(m=[s|z]) notation in equations 5 and 7 is particularly strange
-- From equation 5, it seems that this is using an epsilon-greedy approach to exploration (guided by the heuristic) versus exploitation (guided by the current best policy).  If this is true, please just say that.
-- Consider moving some of the formalization from section 4 to the appendix (especially background information not critical to your contribution) and providing more detail on the heuristics used in the experiments 
-- The charts in Figure 3 are very difficult to read and interpret
-- It would be helpful to give an intuition for what you're doing in section 4, especially where and how the heuristics come in; right now it's very difficult to follow
-- The entire first paragraph can be removed
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this paper, the authors propose the "exploration-bounded exploration actor-critic" (EBE-AC) algorithm, a deep RL algorithm that can optimize a stochastic policy through the use of heuristic information. Such heuristic information includes prior human knowledge about physical relationships (e.g., between a robot and objects). The authors implement and evaluate their approach on a robotic manipulation task, Pick-and-Lift.
strengths_and_weaknesses: PROS:
* The paper is well written and clearly structured
* The math is easy to follow
* The idea is interesting
* The experimental evaluation is sound and it includes the relevant baselines
* All the hyper-parameters and details are provided in the supplementary material for reproducing the experiments

CONS:
* This is not really a negative point, but it would be interesting to see another application of the proposed algorithm, eventually in a different domain.
* Plots could be improved for readability.
summary_of_recommendation: The paper is well written and well structured. The idea is clearly described by means of mathematical formulations, and the experimental evaluation is convincing. All the details for reproducing the experiments are provided in the supplementary material.
issues: * Can you point out some different domain/application examples?
* Improved plots would be good
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an information theoretic objective function to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. The objective is then reformulated to two soft Q functions and all parameters are optimized in an actor critic way including two temperature parameters which control the degree of the maximum entropy and mutual information. In experimental results, the proposed method outperforms the state of the art soft actor critic algorithms like SAC and DIAYN in a pick-and-lift robot simulator.

strengths_and_weaknesses: Pros:
    1. The information-theoretic objective takes human heuristic information into account which can guide exploration into unknown or a more reasonable region.
    2. There are two hyperparameters called temperature in the objective function. One encourages the actions to have rich exploration, the other one can encourage or bound exploration automatically. A dual problem is formulated to optimize the two temperature parameters.
    3. The experimental result is based on a pick-and-lift robot simulator. The performance of the proposed method outperforms existing soft actor critic method in both performance and speed of convergence.
Cons:
    1. The most interesting part of this paper is the temperature parameter which is optimized automatically. It would be great if the author can show the evolution of the temperature parameters during the training phase and explain why such results happen.
    2. Because this approach used step-based exploration rather than trajectory exploration. So it’s questionable whether there is temporally consistency in the robot motions, which is a big issue to apply this method on real robot tasks.
summary_of_recommendation: The paper presents an information theoretic objective function to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. And two temperature parameters as well as the policy parameters are optimized automatically to encourage or bound the agent’s exploration automatically. Overall, this paper is marginally above the acceptance threshold.


### Post Rebuttal Comment: 
I thank the reviewers for their comments. I keep my evaluation as is because the paper is missing real world experiments.
issues: 1. The experiments are not based on real robot systems, it might be better to transfer the existing result to a real robot system.
2. Figure 3-4 needs some improvement. It’s hard to tell the settings of each curve. And the legend should be consistent with all plots.
review_title: null

Meta-review: The paper proposes a new approach for policy optimization. The main contribution lying at the information theoretic objective function that targets to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. It is suggested that there should be more details on the choice of  the heuristics used in the experiments. And it would be great if there are additional validation results  across more tasks or discussions on the use of the proposed formulation for different domain/application examples.

After the rebuttal: The authors have made great effort in providing additional results and clarifying concerns from the reviewers. There is one critic that the proposed method has addressed an ill-defined problem, which I think can be an interesting direction for the authors to look at for future researches. On the other side, the proposed method still makes sense and is accompanied with a good set of experimental results that demonstrate some of the claims. 
Decision: Accept (Poster)