Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Knowledge-grounded RL (KGRL) for the purpose of incorporating, reusing, recomposing and generalizing external knowledge in RL tasks. Taking a unified form of external knowledge as an external knowledge policy, this work proposes an actor model that adaptively weighs or activates different external knowledge policies and a learning inner policy. This is realized by a typical self-attention mechanism, with a state-dependent query, inner/external policy-dependent keys. Query, key embeddings and the inner policy are all learned in an end-to-end fashion from standard RL policy signals/gradients. The proposed approach is empirical evaluated in MiniGrid and OpenAI-Robotics benchmark, demonstrating the supriority of the proposed in learning efficiency and generalization ability, given a few pre-defined external knowledge policies.
strength_and_weaknesses: $\textbf{Strengths:}$
+ The writing and presentation of the proposed method is almost clear.
+ The related work includes multiple related domains.
+ The experiments are conducted from multiple aspects.

&nbsp;

$\textbf{Weaknesses:}$
- The major issue is on the central notion of this paper, i.e., Knowledge or KGRL. I think it is improper and kind of misleading to use the notion of knowledge through the paper. After reading the paper, I recommend the authors to use the notion e.g., Policy Reuse/Transfer, which is more accurate. 
  - The authors emphasize many times in this paper like the claim ‘all knowledge can be arbitrarily recomposed, rearranged, and reused anytime in the learning and inference stages’. However, the concrete form of external knowledge considered is the external knowledge policy.
  - In other words, acctually the main content of this work is on how to incorporate, reuse and transfer existing policies in RL. For KGRL at least in my opinion, the key part is how different (or several common) forms of external knowledge are represented, after which how to use the knowledge is to be considered. However, this part is not included in this work and in the experiments, the external knowledge is implemented by the pre-defined policies.
  - Given the external knowledge policies, the proposed method in Section 4 is not novel to me. The self-attention mechanism and the end-to-end update are straightforward.
- The experiments provide few impressive results, especially when the pre-defined external knowledge policies are strong and the environments are not challenging.
- Important related work is missing. For example, Knowledge Guided Policy Network (KoGuN) [1] propose a similar idea, where suboptimal external knowledge is represented by trainable module based on fuzzy logic and then is incorporated into PPO policy network serving as a refine module.

&nbsp;

Reference:

[1] Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, Yan Zheng:
KoGuN. Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge. IJCAI 2020



&nbsp;

$\textbf{Questions:}$

1) The authors claim “current learning-from-demonstration and knowledge-reuse approaches lack the flexibility to rearrange and recompose different demonstrations or knowledge, so they cannot dynamically adapt to environmental changes”. Do the author mean the ability of zero-shot adaptation or generalization? To my knowledge, policy reuse and policy transfer are well-studied domain in this direction. Can the authors justify this claim?
2) The authors claim “These two research directions do not enable incorporating external guidance from different sources into learning, so their transferred knowledge is limited to the one learned among similar environments” and “KGRL allows an agent to follow and reuse an arbitrary set of knowledge obtained from different sources in a new task, and these sources can be very different from the new task”. What do the ‘different sources’ mean? External knowledge in different forms? It is confusing to me since this paper does not address how arbitrary forms of external knowledge are transformed exactly into the external knowledge policies. Instead, the external knowledge policies are given in the setting considered in this work.
3) Using a state-dependent key for the inner policy makes sense to me. However, I recommend the authors to add a KGRL variant without state-dependent key as an additional baseline in the experiments for more complete evaluation and abalation.
4) I also have concern on the claim “we disentangle the decision component (the query) and the knowledge representations (the keys)”. Indeed, they are separated but they are updated in an entangled form regarding the policy gradients.
5) I think there is a straightforward alternative to the gumble-softmax way to acting with the inner policy and multiple external policies. Concretely, to sample an action for each policy (with reparameterization) and then to mix them according to a softmax distribution of $w$. Why is this way not considered? Or what is the drawback of this alternative way?
6) Are the external knowledge policies used in the experimenst stochastic or deterministic?
7) It is strange to see the very small error bars in Figure 2a and 2b (in fact, I cannot tell if there is a error bar in Figure 2b).
8) In Figure 2e, why do PPO-KGRL (KG3) and PPO-KGRL (KG1+KG2) perform worse than PPO while PPO-KGRL (all) outperforms PPO?
9) In Section 5.2, it seems that KG1 is already optimal in Reach task and KG1 + KG2 is optimal in Pick-and-Place. Can the authors provide more explanation on this point? Similarly, KG1 + KG2 + KG3 is also optimal in 5x5 environments since the view range is also 5x5.
10) The authors mention “indicating that a KGRL agent learns to not only follow the guidance but also mimic its strategy. This imitation process allows an agent to develop an inner policy that outperforms external strategies”. Can the authors provide more explanation on how such imitation happen under KGRL and why imitation can be better than the external policies?

clarity,_quality,_novelty_and_reproducibility: The writing and presentation of the proposed method is almost clear. The central notion (i.e., knowledge) is kind of misleading and improper. The proposed method is lack of novelty. A few key claims and statements are not well supported or explained.

For reproductibility, some details on implementation and experiments are provided in the appendix. Some key details of the pre-defined external knowledge policies are missing (e.g., $\epsilon$ in Section 5.2). The source codes are not provided.

summary_of_the_review: According to my detailed review above, I think this paper is clearly below the acceptance threshold mainly due to the improper claims on knowledge, the lack of novelty of the proposed method and insufficient experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper defines a knowledge-ground reinforcement learning (KGRL) problem that an agent learns to follow external guidelines and develop its own policy. It also provides a realization of this problem by using an attention-based actor model which can switch between either a learnable internal policy or external knowledge. Experiments on MiniGrid and OpenAI-Robotics show the KGRL agent's effectiveness.
strength_and_weaknesses: Strength:
1. The idea that agents learn to follow external guidelines and develop their own policies makes sense.
2. The related work description is very detailed.

Weaknesses:
1. As mentioned in Section 4.1, "We learn a key for each external knowledge mapping...", how to learn these keys? I did not find any descriptions.
2. In Figure 1, $k_in$ is generated by the Inner Actor, which can be learned during the interaction. However, there is no input about $k_gi$. Is it missing? Or please give some more description about my misunderstanding.
3. In the paper, PPO and SAC are used as the baseline algorithm respectively, and I hope that more baseline algorithms can be combined to illustrate the generality.
4. At the beginning of Section 5, this paper claims to answer the question "(1) Can our KGRL agent learn effective new knowledge by referring to an arbitrary set of external guidance?". Does the term 'new knowledge' simply refer to the stringing together of old knowledge? If so, the use of the word "new" is not appropriate, and if not, could you please give some experimental examples?
5. Since Atari games often appear in reinforcement learning algorithm articles as a basic verification environment, it is hoped that this paper can also do a brief algorithm verification in this type of environment.
clarity,_quality,_novelty_and_reproducibility: While the main idea of this work is straightforward and the English writing is OK, many important details of the proposed method are missing, making many parts of this work difficult to follow. 
summary_of_the_review: I like the main idea behind this work, and the proposed actor model architecture looks reasonable. However, since many important details of the proposed method are missing, and there is no source code provided to check how the proposed model is implemented, I thus can not recommend acceptance of this work in its current form. I strongly encourage the authors to re-organize the manuscript's content and re-write the whole paper for better understanding by the audiences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for incorporating background knowledge into reinforcement learning, Knowledge-Grounded RL. The approach uses an embedding-based attention-mechanism model that learns to attend to external and internal knowledge based on observations. The proposed framework is agnostic to the policy training algorithm, and was demonstrated using PPO and SAC algorithms. 

The proposed framework is evaluated on several tasks from two benchmarks: navigation tasks from the MiniGrid environment and pick-and-place manipulation tasks from OpenAI-Robotics benchmark. In both sets of tasks, the proposed framework superior or competitive performance in terms of sample efficiency. 
strength_and_weaknesses: Strengths

The proposed framework demonstrates superior performance in terms sample efficiency on tasks used and does so with different policy training algorithms. 

Weaknesses

The paper misses related work such as Kimura, Daiki, et al. "Reinforcement learning with external knowledge by using logical neural networks." arXiv preprint arXiv:2103.02363 (2021) and Murugesan, Keerthiram, et al. "Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines." AAAI. 2021.

Text-based RL benchmarks such as those used in the above works (e.g., Textworld, Zork, etc) are good candidates for this work. The proposed work would be much strengthened it also used these benchmarks and compared to SOTA methods there.

Although the proposed framework is described generally, the external knowledge used in the experiments is task specific and could be learned and re-used by existing transfer techniques. Why shouldn't curriculum learning be equally successful in this setting? The experiments seem to be demonstrating a way of recombining/re-using previously learned knowledge, which could be learned internally only using e.g., some curriculum. 

The implementation details are very general. I don't think it would be easy to reproduce the results. For example, details are provided (Appendix) about how to build embeddings for external knowledge but it's not clear how that external knowledge is encoded. This is supposed to be done by the external knowledge mappings but not details are provided for any of the experiments. 

Lastly, it's not clear what the internal policy is learning. It seems to be that the internal policy is learning to explore the environment and relies on external knowledge for task execution. So, in my opinion, the internal policy is not learning the policy to complete the tasks but the whole model is learning how to use external knowledge without building any internal knowledge for reuse later.

Also, I'm not sure the learned policy is interpretable. As shown in the experiments, we can see when the agent chooses to use which piece of knowledge but the individual pieces of knowledge are not interpretable in and of themselves, which is generally the issue of interpretability.  
clarity,_quality,_novelty_and_reproducibility: The work is described clearly and is of good quality. The novelty is lacking as several literature work on reinforcement learning with external knowledge is missing (see comments above in Strengths and Weaknesses). Not enough details is provided to reproduce the experiments. For example, it's not clear how the external knowledge is mapped into executable actions. It's also not clear how to integrate other forms of knowledge in general. E.g., domain rules that could be represented in formal logic or even text. 
summary_of_the_review: The paper address an important aspect of AI: how to incorporate external knowledge into a learning agent. It proposes an embedding-based actor model that learns to attend to different sources of knowledge and evaluates this on several benchmarks. However, it misses important pieces of the literature on the same topic and doesn't evaluate against SOTA in the space of reinforcement learning with external knowledge. It also misses important benchmarks in the area.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces an agent architecture intended to facilitate the reuse of existing knowledge, called ‘knowledge grounded RL’. The knowledge grounding consists of a dictionary with randomly initialized learnable keys paired with hard-coded policies (these could possibly also be pre-trained, in principle) which is available to the agent. There is one additional policy in the dictionary which is not hard-coded or pre-trained, but trained online. Action selection happens by attending over this dictionary.

The proposed architecture is evaluated on six MiniGrid and two robotics tasks, with a specific set of hard-coded policies for each environment. With the right set of pre-baked policies, the proposed architecture outperforms the chosen baselines on both environments.

strength_and_weaknesses: - Weakness: experimental evaluation. The MiniGrid environment is clearly a toy one, which might be suited for a proof of concept, but is not sufficient for a serious evaluation. The two robotics tasks are quite elementary as well; more complex tasks exist, and would be very natural applications for the proposed method - a very simple example being stacking blocks.
- Weakness: the method as it is used in the paper relies heavily on hand-crafted pre-baked policies. The paper does not show how agents learn policies that can be transferred to agents learning other tasks.
- Strength: the related literature section is fairly comprehensive and covers a good number of related topics.
- Strength: the proposed method is explained clearly.
clarity,_quality,_novelty_and_reproducibility: - Clarity: As stated in the ‘strengths’ section, the proposed method is explained well, and the relevant literature section makes clear where the paper is placed relative to the rest of the field. The title (which is used throughout the paper) is a bit misleading, in the sense that the ‘knowledge’ used to ‘ground’ the RL in fact consists of pre-baked policies. While it is technically possible to argue that this is a form of grounding RL in knowledge, it would be more accurate to phrase it in terms of something along the lines of ‘sub-policy re-use’.
- Quality: As stated in the weaknesses, there are major gaps in both the usefulness of the method that is demonstrated, and the scope of the experimental evaluation. The idea of reusing policies has merit in principle, but it is also difficult to execute well, and this paper does not reach the bar of a convincing demonstration. To start with, the authors don’t address the question of how to pick pre-trained policies for a particular task. They don't show how to automate a system based on re-using policies, how to create diversity in the learned policies, or how to deal with the probably very large dictionary of policies that agents would have to learn to navigate. As it is, the paper does nothing to convince the reader that the proposed method would go beyond toy tasks where it is easy to provide pre-baked policies.
- Novelty: while the HRL and options literature is referenced, there is a degree of disconnect between the claims about the proposed method made in the section discussing HRL/options, and the experiments and implementation that is actually presented. Paraphrasing, the claim is made that re-used policies in KGRL can be much more varied than skills in HRL or options. That might in theory be true, but in practice the pre-baked policies provided to the agents in the paper are very good examples of skills or options.
- Reproducibility: no concerns.
summary_of_the_review: The idea of re-using existing knowledge, or more specifically, of policies trained or defined previous to the agent's initialization, has merit. However, the implementation and evaluation of that idea in this paper falls well short of the standard for novelty and quality for an academic contribution. The method relies too much on hand-crafted solutions, and the evaluation tasks are too simple and too few and unvaried. I recommend rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an architecture that can leverage external knowledge to make better decisions in RL problems. The architecture implements an attention mechanism that dynamically chooses to attend to an internal knowledge mapping or external ones. It is trained end-to-end via the Gumbel-softmax trick. Experiments are conducted on grid-world and robotic tasks. 
strength_and_weaknesses: Strengths:
* Incorporating external knowledge into RL policy is an important problem. 
* The method outperforms proposed baselines.
* The paper is easy to read.

Weaknesses:
* The experimental settings and results do not support claims made by the authors. First, the author "knowledge" as "*can be any commonsense, suggestion, guideline, principle, specification, rule, restriction, or previously learned knowledge described in formal or informal language*", but the knowledge in the experiments is simply commands or high-level options. The authors only construct instruction-following tasks rather than knowledge-incorporating tasks, which are much broader in scope and more difficult to solve. 
* The proposed method is not novel. Attentional architecture is the de-facto choice for building instruction-following agents [1]. The authors simply train the agent end-to-end without proposing any specific techniques that help the policy incorporate the knowledge effectively. Knowledge such as constraints or environment specifications may require more special treatments than simple attention mechanisms [2,3]. 
* There are other ways to design the attention mechanism but the authors do not explain why they choose the presented one. They also do not compare with alternative designs in the experiments. For example, instead of mixing the policies, I can mix at internal representation level h = sum(w_i * h_i), where each h_i is the representation of a piece of knowledge. Then I construct a single policy based on the mixed representation. This avoids having to perform the gumbel-softmax trick. 

[1] https://arxiv.org/pdf/1711.07280.pdf
[2] https://arxiv.org/pdf/2010.05150.pdf
[3] https://arxiv.org/pdf/2101.07393.pdf




clarity,_quality,_novelty_and_reproducibility: The paper does not present a novel idea. The experiments do not reflect the broad range of problems claimed to be solved by the authors. The authors do not consider alternative architectures and justify why the proposed design was chosen. 
summary_of_the_review: The merits of the paper are marginal. I do think the idea presented would be significant for the community. I am learning towards rejecting the paper. 

=====After Rebutal=====

I have read the authors' response and decided to keep my current score. My main point of rejection is that the paper over-claims the generality of their methods while not having experiments to back up and avoiding comparing with related work. The definition of knowledge, which can be "commonsense, suggestion, guideline, principle, specification, rule, or restriction, described as a knowledge mapping" is very vague without defining what the mapping actual is (e.g. how is that policy related to the optimal policy?). I suggest the authors revise the scope of their claims and provide a more specific specification of their setting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of integrating knowledge bases with reinforcement learning. The problem is formulated as doing RL by learning to index into a knowledge base of pre-trained policies (+ an online learned policy), using an attention based mechanism. The approach is validated on MiniGrid and robotics environments. 

The approach is clearly written and there was no confusion about the proposed approach. However, there was a strong consensus that the environments considered were too simplistic to establish the generality of this approach. There are a lot of assumptions made in this paper about how the policies are hand constructed and the dimensionality of the knowledge base needed for generality. Without scaling up the approach to more realistic environments, it will be difficult to build upon this work and clearly establish claims. 
justification_for_why_not_higher_score: The experiments in this paper are limited due to the simplicity of the task and environments. There was a consensus about this with all reviewers. The use of toy environments like MiniGrid and simple control tasks are well suitable to establish the efficacy of the approach but does not provide any insights or path for scaling. They are good to demonstrate proof of concept but insufficient to establish and build upon for more complicated and real world problem domains. The level of compositionality required to solve the tasks should be expanded to make the paper and experiments stronger. 

The paper also relies heavily on manually crafted policies. It is not clear how to learn or extract them from experience or learning from other tasks. The paper should have evaluated more robust and general policies to better assess the proposed method's capabilities and limitations.
Decision: Reject