Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is interested in improving the performance of natural language generation tasks that require external knowledge (which are often called knowledge-intensive tasks). In particular, the paper focuses on how it can leverage existing language model (e.g. GPT-2, BART) without changing its architecture and injecting external knowledge into it. To do so, the paper proposes Knowledge Infused Decoding (KID), which alters the decoding probability at each time step (in vocab space) so that it favors decoding certain words, which are obtained by retrieving a relevant document from a corpus. In order to trains such decoder, the authors also add a KL divergence to the loss that prevents the decoding distribution not to be too different from the original Language Model's decoding distribution. When KID is used in conjunction with GPT2-M or BART-L, they show clear improvements, and in some datasets, they achieve higher accuracy than the state of the art. The paper also shows that KID outperforms Retrieval Augmented Generation (Lewis et al., 2020) which uses the retrieved documents directly into the input (whereas KID uses it indirectly to constrain the decoding in a soft way).
main_review: Strengths:
- The proposed method can be considered as a new approach for constrained decoding, but rather than doing it in a hard way, the proposed method gives a soft constraint by rewarding a set of target words.
- The experiments were done in multiple datasets and the proposed method consistently shows a better result than its baselines. It is especially encouraging because the proposed method is a plug-and-play module that can be applied to any language model in practice.

Weaknesses:
- The paper is overall difficult to understand, and I had to re-read it multiple times to make sense of it. This might have caused several misunderstandings.
- The paper does not compare against vanilla (non-graph) constrained decoding which limits the output space to the words in the retrieved corpus. This will be an important baseline that can show the advantage of having such graph for constrained decoding.

Questions:
- I am confused with the motivation behind formulating the loss with RL. The paper discusses the disadvantages of vanilla decoding, e.g. exposure bias of teacher forcing. However, the proposed loss function with reward at every time step is also biased. The proposed loss seems to me more like an MLE with some regularization/modification.
summary_of_the_review: I think the paper's result is great and I agree with the overall motivation behind the paper. However, many parts of the paper are unclear that I will need further clarification to correctly assess this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a text decoding strategy for knowledge-intensive text generation tasks with the goal to infuse external knowledge based information at each time-step of decoding process via efficient document retrieval, creation of local and fast global knowledge access data structure and the interaction guided decoding method. The decoding method uses RL based policy gradient for token selection. The method is evaluated using diverse types of knowledge-intensive tasks such as abstractive question answering, logic centric writing and dialog generation. This method works with existing decoder only or encoder-decoder LMs as a plug and play method and outperforms existing knowledge aware task optimized models and correlates well with human evaluation. 
main_review: Strengths: 
- This method does not require re-training or fine-tuning LMs with knowledge based objectives. This is a plug and play decoding strategy that competes with other sampling based or beam based decoding strategies with weighted decoding methods. 
- This method can 'dynamically' infuse external knowledge into each step of LM decoding. This is an advantage over static knowledge retrieval based methods that retrieve documents only once and suffers from problems with long range generation. 
- The probability mass of tokens at each step of token generation can be informed directly by the concepts and relations extracted via the entity-relation extraction methods; this token level reinforcement helps with overcoming sparse rewards generally popular with sequence level reward induction (via evaluation metrics such as BLEU) 
- Results show that larger LMs benefits Knowledge infused decoding(KID) method and outperforms Beam search and sampling decoding. In addition, KID brings more gain for non-finetuned LMs than fine-tuned ones, potentially because the fine-tuning LM is already fit onto the new domain thus less knowledge infusion is needed.


Weaknesses:
- The paper lacks details of the text decoding inference time and comparison with other decoding methods (sampling) including knowledge based decoding methods. Figure 2(a) and 2(b) do show time latency; however there is no comparison or discussion of this in the paper. 
- The paper presents a text decoding method for knowledge infusion; however, it does not compare this knowledge infused decoding with other non-RL based decoding techniques such as penalized sampling of words (Keskar et al. 2019), decoding with distributional constraints (https://arxiv.org/pdf/1809.01215.pdf) or other plug and play decoding methods such as PPML (https://arxiv.org/abs/1912.02164). 
- The ablation studies described in the paper do not make explicit the contribution of the RL based decoding technique. 
- The trie data structure is created using open IE tools and reference previous literature. However, some more analysis about the compression of memory would be interesting to understand (with availability of longer sequence handling transformers, how important is the role of compression and trie based external memory?)
- It is not clear how dynamic is the local memory; Figure 1 is not well-described in the paper. At time-step t or t+1, it seems that the local memory is not going to change (extracted from context); what are the dynamics of local memory in this model?


summary_of_the_review: This paper presents a novel plug and play text decoding method that allows improved text generation for knowledge intensive tasks via reinforcement learning based technique. The method allows several new future investigations in this direction and is very interesting (addressing exposure bias problems with LM tuning). There's work on knowledge retrieval, data structures for knowledge encoding and text decoding with state of the art integrated methods. The results are also superior compared to many existing decoding and knowledge aware decoding methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper deals with the well-known but important problem of language models -- enhancing language model with (external) knowledge. To that end, the authors present Knowledge Infused Decoding (KID), a novel decoding algorithm for generative LMs, which infuses external knowledge into each step of the LM decoding. This method maintains a local knowledge memory based on the current context and continuously updates the local memory via reinforcement learning to guide each step of language generation. As a result, it can integrate knowledge on the fly and perform well on various knowledge-intensive tasks such as abstractive QA, logic-centric writing, and dialogue generation.
main_review: ## Strengths

- Proposed method outperforms baselines on various knowledge-intensive tasks.
- Proposed objective can be a good loss term for explicitly modeling knowledge in LM.
- Considering the performance gap between KID and RAG (or BART+DPR), dynamic knowledge infusing will be a good alternative to static knowledge grounding.

## Weaknesses

- Knowledge-graph construction and graph traversal are not well explained. Can you give an example knowledge trie and explain to me how does the graph traversal work? How would the queried results look like?
- Optimization looks somewhat tricky. Which learning rate is used for Adam? How did you select K = 3 steps for policy update? What would happen if we set K < 3 or K > 3?
- Can you show me example sentences generated by KID and compare them with RAG-generated sentences?
- In Table 5, can you also measure naturalness or grammaticality? It is obvious that KID will improve knowledge grounding, but not sure how much it will break the syntax of generated sentences.
- Can you also add FiD-BART as one of the main baseline methods?
summary_of_the_review: The proposed method is fairly sound and shows good empirical results. However, some details are missing (e.g., knowledge graph construction and traversal), and hyperparameter selection is not well explained (e.g., learning rate and the number of optimization steps). It is also unclear how much KID will harm the original LM in terms of naturalness and grammaticality. Additional human evaluation or generation examples can remedy these issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Knowledge Infused Decoding (KID), an RL-based approach for grounded decoding by conditioning on external knowledge. For every new example:
- retrieve the relevant passages from Wikipedia (using dense passage retrieval DPR)
- construct a knowledge trie of triples extracted from the passages and initialize a local trie that tracks all the nouns/verbs mentioned
- Generate tokens conditioned on both prior context and by retrieving triples from the knowledge trie, used as demonstrations for policy gradient
This approach is model-agnostic and can be used to fine-tune both encoder-decoder and decoder-only models.

Experiments are divided into two parts:  
- evaluating the decoding method, on six NLG datasets -- outperforming beam search and (top-k/nucleus) sampling
- evaluating the knowledge infusion, on four abstractive QA datasets -- outperforming a number of approaches including retrieval-augmented, graph neural network, and knowledge graph methods.

Ablation studies are presented for swapping out the retriever, different hyperparameters for KID and the size of the underlying LM.

main_review: Strengths:  
This approach is interesting and very relevant to the growing work on retrieval- and memory-augmented models for grounded generation.  
Since it is a new decoding approach, it is model and task agnostic, and can be used as a plug-in replacement for existing decoding approaches without changing the architecture and for grounding each generated token in external knowledge. The knowledge graph could be further used to study provenance of generated tokens.  
Experiments are conducted on a variety of knowledge-intensive tasks/datasets and KID clearly does outperform vanilla beam search and sampling which are not grounded.

Weaknesses:  
There is a lack of clarity and details which makes it hard to follow some explanations and results; choice of baselines seems a bit unclear -- digging into these points through the following questions to allow for discussion.
1. Since probability mass is shifted in favor of retrieved demonstrations at every time step, how is the model generating words that don’t appear in those demonstrations? Eg: stop words for fluency. What is happening in those cases? 
2. During constrained decoding, are the concepts always the subjects (keys) in the triples? If so, then is the local trie reconstructing portions of the external trie? If not, how is the local trie used to traverse the external trie?
3. What is the “in-house split dev set”? No official test sets are used, right? The number for RAG on MS MARCO in Table 2 (40.8 R-L) appears to match the one from the original paper for test scores (https://arxiv.org/abs/2005.11401 their table 2). But the dev set scores in their table 6 seem to be higher (57.2 R-L). It’s a bit confusing. Could you please clarify which subsets were used for each of the evaluations and what is the fair comparison? 
4. Are the beam and sampling baselines in Table 1 conditioning on external knowledge? If not, then aren’t the baselines far too weak for a fair comparison and should include some external knowledge? The published SotA numbers appear to paint a very different picture and that seems to call the claims into question. On a related note, why not also compare against other constrained decoding baselines eg: lexically constrained decoding (https://arxiv.org/abs/2010.12723)?
5. Could you please elaborate on the runtime details? Providing a breakdown for the different steps might even be useful. How does this compare to vanilla decoding methods? How does it compare to RAG-token/kNN-LM/other related approaches?
6. Fusion-in-Decoder (https://arxiv.org/abs/2007.01282v2) seems like a very important baseline for knowledge infusion. Was there a reason to not include it?


Some additional comments:  
1. RAG-token (https://arxiv.org/abs/2005.11401) supports token-level retrieval, so does kNN-LM (https://arxiv.org/abs/1911.00172). Discussion in the intro and some other places appears to misrepresent the former and leave out the latter. It might also be worth highlighting that RAG-Sequence is used for all the comparisons.
2. Would it be possible to add a table somewhere (perhaps the appendix) that provides the dataset sizes and other relevant stats?
3. Would it be possible to add some qualitative examples? It would be useful to see, for instance, what a single timestep’s retrieved demonstrations, local trie etc. look like.

=============================

Update: Thanks to the authors for their detailed response. While the responses helped to clarify a lot of details, there remain some issues particularly regarding the clarity of the paper itself. That being said, the community stands to benefit from ideas presented. Recommending acceptance.

Some suggestions for revisions:  
- Not convinced that claiming the win over beam search/sampling is a notable contribution since a large body of work has already shown that injecting knowledge is useful. Would suggest the authors consider reframing this -- it’s fine to include the experimental results, but the argument and claims need to at least be thought out.  
- Probably update the text to reflect what changes have been made to MS Marco results so it’s clear what the evaluation procedure is. 
- Adding all the response material to the appendix doesn’t help the clarity of the paper itself, might help to coherently revise sections.    
- Need to clarify the discussion on runtime. It’s a bit tedious to parse.  

summary_of_the_review: Updated to weak accept after author response.

Recommending a weak reject (5) pending the discussion on concerns raised in the main review (questions 3,4,5,6) regarding baselines and results. These seem important enough to clarify before updating the recommendation.
review_title: null

Meta-review: The paper introduces a novel decoding algorithm that allows to dynamically integrate external knowledge with generative 
LMs. The proposed technique is plug-and-play, it does not require re-training or fine-tuning LMs with knowledge based objectives. 
The author report a series of experiments on several datasets and tasks showing improvements over competitive baselines and in some 
cases above SOTA. The authors have addressed the reviewers' queries and added experimental results (additional baselines) as
well as clarifications of their approach (e.g., Figure A1 in the Appendix). I think the topic  (e.g., constrained LM decoding) is of general interest to the ICLR community and the approach compelling.
Decision: Accept (Poster)