Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new algorithm for integrating off-the-shelf automatic symbolic planners with online RL for efficient exploration in the sparse reward setting. The authors propose to itteratively refine a symbolic representation that a planner uses to predict paths towards a goal, which gets rolled out in the environment. It is theoretically proven that the proposed algorithm terminates, provided some conditions on the planner are satisfied. The authors show results comparing their method to Q-learning and show an improvement in the outlined domains. 
strength_and_weaknesses: Strengths:

The paper presents the problem setting and background in a very rigorous manner, and the theoretical guarantees are relevant to the proposed task. The paper includes code for reproduction.

Weaknesses:

The paper is lacking in thorough experimentation and results discussion. Additionally, a major limitation comes from the constraint to deterministic MDPs (which the authors acknowledge). Additionally, the paper assumes human input for annotation of the actions, which begs the question why the baseline is a Q-learning model, as opposed to something that similarly uses human guidance, i.e. imitation learning. Additionally, the use of online RL, as opposed to the more prevalent offline RL setting within the symbolic representations, is not well justified. It would be good to compare against other existing symbolic planning methods as well. 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and clearly written. The presented approach is novel although perhaps of limited practicality. The appendix contains sufficient implementation details for reproducibility.
summary_of_the_review: The paper is clear and well written with good theoretical justification, however it is questionable if the proposed approach could scale to more complex and realistic domains (such as the kitchen domain from Danfei Xu, Roberto Martín-Martín, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Regression Planning Networks). I would be willing to increase my score if more detailed experiments are provided.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to learn a new exploration technique to learn a symbolic model of an environment. This model is then used to speed up goal directed exploration. The proposed technique is then evaluated against Q-learning with epsilon-greedy and R-max as well as a hierarchical RL method. Experiments are done on blockworld and minigrid and show that using the optimistic symbolic model improves results over the baselines
strength_and_weaknesses: While the paper proposes an interesting technique to solve the specifics problems presented in the paper it is unclear how applicable the proposed technique are to other problems.
For example the paper chooses to ignore the challenge of stochasticity in exploration in reinforcement which is one of the main cause of the difficulty of exploration in reinforcement learning
"We specifically chose a setting, which forefronts
the problems related to exploration, while placing less emphasis on other aspects of an RL problem".

This paper also relies on human input and a goal directed planner, those two assumptions do not seem realistic to solve more general problems.

I also do not understand why the paper decides to introduce and focus on blockworld. The paper does not properly justify why this is an interesting problem and we should care about solving as opposed to the larger number of existing reinforcement learning environments.

It appears that the paper is using a tabular version of Q-learning ? In that case can the proposed method scale to problems that require function approximation? In that setting it would be interesting to use more recent algorithms as baselines such as RND and/or NoveID.


clarity,_quality,_novelty_and_reproducibility: Overall I found that the paper was really difficult to follow.
Section 3.1 spends a lot of time to introduce the blacksworld environment even though the information presented is not particularly interesting and could be pushed to the appendix.

"In fact, possibly the only effective methods in the mainstream RL repertoire we can use are curiosity driven or intrinsic reward based methods" 
Posterior sampling methods such as Thompson sampling could be used.

"For the symbolic model, we will be using an untyped normalized PDDL task model Helmert (2009)."
Normalized PDDL were not introduced.
I found section 3.3 really hard to understand, a plot / figure of the symbolic model would have been helpful to help the reader.

"The basis of our approach is an observation that every deterministic MDP has a precise symbolic
representation." what does precise means?
summary_of_the_review: Overall while the proposed idea is interesting I do not think the paper present a compelling evidence that their method can be useful to improve exploration in most environments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the issue of requiring to access a complete and correct symbolic model when combining symbolic methods with reinforcement learning (RL). To address the issue, this work proposes a new method for learning optimistic symbolic approximations of the underlying model in an online fashion. Besides, theoretical results are provided to further analyze the proposed algorithm. Finally, experiments were conducted on three traditional planning domains (Blocks, Elevator, Gripper) and one traditional RL benchmark (Minigrid), with results demonstrating its effectiveness in terms of solvability, the average time, and the number of samples.
strength_and_weaknesses: Strengths:
- Symbolic representation has been shown promising in guiding RL for the efficient exploration, explainability, and human-machine interaction. This is an interesting work targets to learning a symbolic representation for effective exploration within RL, particularly under the sparse reward setting. It would be intriguing to both the communities of symbolic planning and RL.


Weaknesses:
- The presentation of the paper could be improved. For example, how the symbolic model in this work is learned in an online fashion? As mentioned in the related work, I notice some of them allowing to learn the symbolic representation in an online setting, what's the difference between the proposed method and the existing work? How human guidance/ advice interact with this symbolic representation in practice? There lacks more technical details. For the pseudocode of algorithm 1, it would be more clear if the procedures, such as PruneModel, UpdateModel, can be referred to a more specific steps/ equations/ definitions. Besides, I am wondering how does this learned representation interact with RL since I didn't find too many details about this in the paper. Also, how does this learned representation help RL w.r.t. the effective exploration? Does it aim to reduce the exploration space by using prior knowledge or guide RL to explore more meaningful states?
- Current empirical results could not well support the claims on addressing the problem of effective exploration and speeding up the learning process with human minimal input. Comparing with Q-learning with the proposed method (w/wo lifting) in Table 1 looks unfair to me. For the planning domains, there should be included some planning baseline in addition to the proposed method. Particularly, I am quite curious why hierarchical RL fails in Minigrid domain? Even if hierarchical RL doesn't work, it at least has a better performance than flat Q-learning. Can you elaborate more on this?


Minors:
- ’goal‘ on pp.3 should be 'goal'.

clarity,_quality,_novelty_and_reproducibility: - This paper could be improved by re-organizing the papers and showing more technical details as I mentioned in the main Weaknesses. Besides, there are some questions could be clarified.
- Novelty: this work provides some insights in the symbolic method side. However, it's unclear how it will help the RL side.

summary_of_the_review: According to my comments in both the main Weaknesses and the section of Clarity, Quality, Novelty, I feel this is a paper where reasons to reject outweigh reasons to accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine human input in the form of a set of predicates and objects that they associate with a given problem, and realising the fact that for every deterministic MDPs there is a symbolic representation that they dub optimistic representation that reaches the goal state. They provide  the theoretical underpinnings necessary to refine and update a model that will solve the task when combined with a complete planner.


strength_and_weaknesses: Strengths: This paper is clearly written and provides an interesting approach to Exploration by lifting the action representation to a symbolic level and combining exploration with planning. 
The approach seems very novel to the best of my knowledge.
Weaknesses: the experimental section is lacking in comparison with other HRL approaches and how is Q learning performing in the tasks, how far does it go or where does it fail? It would also help understand the significance of the contribution if the results were given in terms of samples as well.

clarity,_quality,_novelty_and_reproducibility: The paper introduces notation on an inverse mapping function that did not seem to be used later on, where was this lifting of the symbolic model being used? And how exactly is it being learned?
Suggestions:
The mapping between states and labels C(s) is applied to actions as well although it is defined over states can the authors clarify? What does it mean C(a) is a predicate true or false in the MDP ? when performing an action? But this is also dependent on the state where it is being executed /equivalent initial and seq of actions in a deterministic env.
The delete_{i+1} =del^a_{i} U (C(s’)\C(s)) reads equals to the del set union with the set of true predicates in (s’) set difference set of true predicates in s? It does not seem to match the text description that the set of predicates that are false in s’. It would help if the authors could clarify on the meaning of the operator \.
How is the model being pruned exactly? And how sensitive is the approach to the planner?
How sensitive is the optimistic representation to the size of the sets? Is the min_add max_del faster?
Typos:
-bee -> been pg 9

summary_of_the_review: This paper proposes a novel way of doing exploration using symbolic reasoning and combines it with planning. The main contributions seem to be in the model update by leveraging human input to formulate preconditions, deletion and addition of predicates for every state, and actions and using update rules to guarantee the existence of a path from start to goal conditions as long as a complete planner is in place. The experimental section only compares the proposed model with and without symbolic model lifting with Q learning in a time budget constraints. The paper would benefit if results would also be presented in terms of samples. Also no HRL baseline is provided, which would seem a more fair comparison.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an idea for using symbolic model learning to aid in exploration to find an optimal plan in goal-directed sequential decision making. The idea is to start with an overcomplete/optimistic model estimate, and then to refine/prune it as experience is gathered. Results are presented in several classical planning domains (e.g. blocksworld) as well as symbolic forms of RL-focused domains (e.g. minigrid, kitchen), showing that the proposed method outperforms various RL methods (Q-learning, HRL) and a symbolic baseline. While the central idea behind the paper is quite interesting and worthy of attention, the paper in its current form is not ready for publication at ICLR. While I and the reviewers appreciated the addition of the kitchen experiments and symbolic baseline, there was agreement that (1) the paper was difficult to understand, and would not be replicable as written; (2) the Q-learning baseline was inappropriate as it is not a method for exploration---something like Go-Explore (Ecoffet et al. 2021) would be more appropriate; and (3) a central claim of the paper is that the method is applicable to RL, but the paper does not actually demonstrate this empirically (e.g. by training an RL agent on the data generated by the symbolic exploration). As such, I recommend rejection.
justification_for_why_not_higher_score: While an interesting idea, the paper is not written clearly enough for the method to be useful to the community. It also has a number of other limitations, such as inappropriate baselines and unsupported claims.
Decision: Reject