Knowledge Infused Decoding
Authors: Ruibo Liu
Guoqing Zheng
Shashank Gupta
Radhika Gaonkar
Chongyang Gao
Soroush Vosoughi
Milad Shokouhi
Ahmed Hassan Awadallah
Venue: ICLR 2022 Poster (2022)
Keywords: natural language
decoding
reinforcement learning
knowledge integration
generation

Abstract: Pre-trained language models (LMs) have been shown to memorize a substantial amount of knowledge from the pre-training corpora; however, they are still limited in recalling factually correct knowledge given a certain context. Hence. they tend to suffer from counterfactual or hallucinatory generation when used in knowledge-intensive natural language generation (NLG) tasks. Recent remedies to this problem focus on modifying either the pre-training or task fine-tuning objectives to incorporate knowledge, which normally require additional costly training or architecture modification of LMs for practical applications.

We present Knowledge Infused Decoding (KID)---a novel decoding algorithm for generative LMs, which dynamically infuses external knowledge into each step of the LM decoding. Specifically, we maintain a local knowledge memory based on the current context, interacting with a dynamically created external knowledge trie, and continuously update the local memory as a knowledge-aware constraint to guide decoding via reinforcement learning. On six diverse knowledge-intensive NLG tasks, task-agnostic LMs (e.g., GPT-2 and BART) armed with KID outperform many task-optimized state-of-the-art models, and show particularly strong performance in few-shot scenarios over seven related knowledge-infusion techniques. Human evaluation confirms KID's ability to generate more relevant and factual language for the input context when compared with multiple baselines. Finally, KID also alleviates exposure bias and provides stable generation quality when generating longer sequences.

Published as a conference paper at ICLR 2022
KNOWLEDGE INFUSED DECODING
Ruibo Liu", Guoqing Zheng Shashank Gupta Radhika Gaonkar',
Chongyang Gao Soroush Vosoughi Milad Shokouhi Ahmed Hassan Awadallah
Dartmouth College, "Microooft, NNorthwestern University
Iruibo. liu.gr, soroush. soroush.vosought. w &dartmouth.edi
(zheng, shagup, ragaonka, milads, hassanam @microsoft.com
cygaolu. wygaolu..orttwwsserrncco
ABSTRACT
Pre-trained language models (LMs) have been shown to memorize a substan- tial amount of knowledge from the pre- tr-inriinnng corpora; however, they are still limited recalling factually correct knowledge given a certain context. Hence, they tend to suffer from counterfactual hallucinatory generation when used in knowledge-intensing natural language generation (NLG) tasks, Recent remedies to this problem focus on modifying either the pre-training or task fine-tuning objec- tives to incorporate knowledge, which normally require additional costly training or architecture modification of LMs for practical applications.
We present Knowledge Infused Decoding (KID)-a novel decoding algorithm for generative LMs, which dynamically infuses external knowledge into each step of the LM decoding. Specifically, we maintain a local knowledge memory based on the current context, interacting with a dynamically created external knowledge trie, and continuously update the local memory as knowledge-aware constraint guide decoding via reinforcement learning. On six diverse knowledge- intensive NLG tasks, task agnostic LMs (e.g., GPT-2 and BART) armed with KID outperform many task-optimized state-of-the-arr models, and show particularly strong performance in few-shot scenarios over seven related Enowledgeiinfusion techniques. Human evaluation confirms KID's ability to generate more relevant and factual language for the input context when compared with multiple basclines. Finally, KID also alleviates exposure bias and provides stable generation quality when generating longer sequences. Code for KID is available at https: gi github. com/ microsoft/KKI
INTRODUCTION
Pre-trained language models (LMs) have been shown to capture rich semantic and syntactic features, demonstrated by the state-of-the-art performance on many generation tasks such as abstractive summarization (Zhang et al., 2020b; Chu & Liu, 2019) and dialogue generation (Roller et al., 2021; Zhang et al., 2020c). Their generations. however, can be quite limited in scenarios requiring knowledge theyy frequently struggle with issues such as being easily misguided by phrase co- occurrence (e.g.. "Talk? Birds can talk." "), failing to handle negation (e.g., "The theory of relativity was not developed by Einstein." ") (Kassner & Schütze, 2020), and being unable to compare common- sense concepts, such as time (Qin et al., 2021) and digits (Talmor et al., 2020).
To enhance the performance of LMs on knowledge-intensive NLG tasks' prior studies have proposed to re-train LMs with knowledge-awe aware objectives (Zhou et al., 2021; Xiong et al., 2020; Zhang et al., 2019; Khandelwal et al., 2020) or add special architectures to encode knowledge (Bosselut et al., 2019; Logan et al., 2019; Peters et al., 2019b) from external resources (e.g., knowledge graphs such as CONCEPTNET (Speer et al., 2017) and ATOMIC (Sap et al., 2019)). These methods, though yielding impressive results on many downstream tasks, can be computationally expensive. More importantly.
Wee define knowledge-nttensive NI.G tasks those whose input context alone does not provide complete knowledge for legitimate and plausible generation.

---

Published as a conference paper at ICLR 2022
knowledge implicitly parameterized in LM architectures is difficult to revise and expand (Lewis et al., 2020b), and wrong generations are hard to diagnose due to lack of interpretation (Talmor et al., 2020), which heavily limits their real-world applications.
More recent retrieval-basee models try to tackle these problems by augmenting inputs with retrieved knowledge evidence (Lee al., 2019; Guu et al., 2020). For example, RAG (Lewis et al., 2020b) lever- ages parameeraeec memory to access extensive knowledge (in the form of unstructured documents), and jointly fine-tunes a parametric LM (i.e., BART (Lewis al., 2020a)) to enable knowledge-awee generation. A key limitation of such methods is that they retrieve documents only once while ground- ing them in the imput static context, and thus cannot support the dynamic nature of the context as new tokens are generated. The static knowledge becomes major problem for tasks where longer and abstractive generation is expected, such as open-ended story generation (Mostafazadeh et al., 2016), multi-turn dialogues (Zhao et al., 2020), and conversation summarization (Gliwa et al., 2019). Moreover, in a recent study, Krishna et al. (2021) replaced the knowledge retriever in RAG with a random retriever and found little difference in the resulting performance long-form QA task named ELI5 (Fan et al., 2019b), indicating the model may not be actually grounding its text generation to the retrieved documents.
To address these limitations, in this work, we present a novel decoding algorithm KID, aiming to better infuse knowledge into generation in dynamic manner. Instead of solely relying the static knowledge retrieved at beginning, during each step of LM decoding, KID dynamically searches promising continuation from retrieved knowledge, to guide the current step generation. Specifically, KID maintains local knowledge memory, interacts it with a knowledge trie dynamically created from retrieved supporting documents, and updates the local memory to guide the generation. The key intuition behind KID is that existing LM pre-training objectives are usually defined at the token level yet do not explicitly model concept-centie knowledge (Xiong et al., 2020) thus motivating us to reshape the probability mass at each step decoding towards the distribution of entities in knowledge.
knowledge-awe constraint
The contribution of this work is three-fold: First, we introduce KID as a model and task agnostic decoding method that integrates knowledge on the fly and can be applied to various knowledge- intensive tasks with different generative LMs. Second, from docoding perspective, six knowledge- intensive NLG tasks, GPT2 (Radford et al., 2019) and BART (Lewis et al., 2020a) equipped with KID significantly outperform conventional beam search sampling decoding by a large margin. Third, baselines which
from a knowledge infusion perspective, unlike seven strong require either additional retraining or special architecture modifications, KID leverages knowledge more effectively as a light- knowledge infusion solution. Additionally, in few-shot scenarios KID significantly improves over them, demonstrating its generalization ability in low-resource and domain shifting regimes.
RELATED WORK
We briefly review existing work enhancing LMs with external knowledge and representative decoding algorithms for generation.
Enhancing Language Model with Knowledge. Large language models implicitly encode knowledge in their parameters but with limits (Petroni et al., 2019; Kassner & Schütze, 2020; Lin et al., 2020a). Several architectures and objective functions have been proposed to explicitly encode external knowledge (Sun et al., 2020; Logan et al., 2019; Roberts et al., 2020; Levine et al., 2020), or to augment LM pre-training data with retrieved knowledge (Lewis et al., 2020b; Guu et al., 2020; Lee et al., 2019). However, Talmor et al. (2020) notes that the reasoning ability of such LMs is strongly tied to the context seen during pre-training and thus hard to generalize to new domains. Built on LMs, KID incorporates extra knowledge from external resources (Wikipedia) and thus shows strong performance in knowledge-intensive NLG tasks.
Better Decoding Algorithm. Two common strategies dominate the decoding algorithms used by most generative models: beam search which maximizes likelihood in a local horizon (due to finite beam size), and sampling decoding (e.g., top-k sampling (Fan et al., 2018; Holtzman et al., 2018)) which relies on randomness. Holtzman et al. (2020) find beam search often produces generic and repetitive generation, while top-k sampling tends to pick unlikely tokens which creates incoherent and
2

---

Published as a conference paper at ICLR 2022
unrelated sequences. Existing attempts to mitigate these problems include reranking (Adiwardana et al., 2020; Shao et al., 2017), adding control signals (Zhang et al., 2018; Xing et al., 2017), and self-adaptive truncation (Welleck et al., 2020; Peters et al., 2019a). None of these decoding algorithms consider integrating knowledge in the generation process. Reflective decoding (West et al., 2021) and DeLorean (Qin et al., 2020) two recent decoding algorithms that focus abductive commonsense reasoning. Reflective decoding in particular has the potential to be extended to other knowledge-intensive tasks. We compare it with KID in our experiments.
KNOWLEDGE INFUSED DECODING
We detail the implementation of KID in this section. As shown in Figure 1, KID comprises of three steps: retrieving relevant knowledge ($3.1), constructing external and local knowledge memory ($3.2), and guiding current step decoding under the constraint of the knowledge trie ($3.3).
3.1 RETRIEVING EXTENSIVE KNOWLEDGE FROM WIKIPEDIA
The first step of KID is to retrieve several context-relevane documents to ground the following generation. We use DPR (Karpukhin et al., 2020) our general-purpose knowledge retriever, which projects contexts and relevant documents to a 768-dim shared embedding space using a bi-encoder network (i.e., two independent BERTs (Devlin et al., 2019)). Here, the documents refer to the 100- word chunks of Wikipedia passages released by RAG (Lewis et al., 2020b), a total of 21M documents as our knowledge source 2. We pre-load the weights from the latest checkpoint of DPR (March 2021), as it improves retrieval performance by using mined negative samples and contrastive learning, which is also suggested by Jernite (2020). During retrieval, we perform a maximum inner- -nner-product search (MIPS) with faiss" accelerated by GPU (Johnson et al., 2019). Formally, we retrieve most relevant document 2[ []] E Z for context Foontext as:
[[......] ={
t/top (BERT (Feentex) BERT(2))3
where BERT(-) means the vectors encoded by BERT. The number of retrieved documents k is task-specific hyper-parameter-aa discuss its impact on performance in $4.3.
3.2 CONSTRUCTING EXTERNAL AND LOCAL KNOWLEDGE MEMORIES
We convert multi-document input retrieved from previous step into compressed knowledge memories, in order to 1) allow relevant knowledge to be easily identified, 2) reduce the memory footprint of the knowledge, whose size grows linearly with the number of retrieved documents.
Following design choice of previous successful methods (Bosselut et al., 2021; Huang et al., 2020; Fan et al., 2019a), we adopt co-reference resolution and open information extraction (OpenIE) (Stanovsky et al., 2018) to convert plain text into triplet form', For example, knowledge statement like "Iceland a Nordic island country in the North Atlantic Ocean and is the most sparsely populated country in Europe." " will be converted to subject-relation-bbe
triplets such as (subj:Iceland, rel:is, obj:Nordic island country). (subj:lceland, rel:is, objt sparsely populated country in Europe). etc. To account for overlapping elements from these triplets, we use prefix tree (namely Knowledge Trie Geas) to store and organize the extracted triplets, by setting the subject each triplet as the key in the Geu- Note that unlike common character-level dictionary tries (Chen et al., 2020; Germann et al., 2009), Gest each triplet is stored token unit as our goal is to efficiently query and traverse the knowledge triplets stored in it.
A tree structure encoding knowledge is appealing to knowledge intensive NLG tasks, since 1) the non-cyclic structure helps reduce repetitions in generations, and 2) querying prefix tree can be efficiently completed in constant time (O([rconext)) which does not involve any costly traversal on the graph (Ji et al., 2020; Zhang et al., 2020a), regardless of the scale of grounding knowledge (normally | <
k Wiki docs)). We also maintain a local memory G (a first-in-first-out list)
"The faiss project can be found here: ht https: / //////gttts//tt com/ facebookresearchh faiss.
3 As an empirical trick also remove the stop words in the documents as they seldom carry knowledge.
3

---

Published as a conference paper at ICLR 2022
Multi-hop Imagination
Relevant Docs
Knowledge Trie
RL Policy Gradient
Retriever
root
[......k
21....
Local Memory
y/
query
Peontext
+
Step 1. Knowledge Retrieval
Step 2. Memory Construction
Step 3. Interaction Guided Decoding
Figure I: Overview of our KID decoding algorithm. For given context Fcontext> first retrieve A most relevant Wikipedia documents =[2....kk with a knowledge retriever (Step 1), and then convert them into compressed knowledge trie Gext (Step 2). Meanwhile, the local memory Gjoe which is first-in-first-out list will keep track of entities in current context, and in the final step (Step 3), it will continuously query the knowledge trie with max number of hops m Current step LM decoding will be guided by the query results with policy gradient, to generate new token y:-
that records all the mentioned entities in current context, to focus on concept-centric knowledge (Zhou et al., 2021; Lin et al., 2020b). More details about how we construct and query these knowledge memories can be found in $A.2 of Appendix.
3.3 KNOWLEDGE TRIE CONSTRAINED DECODING VIA POLICY GRADIENT
Background. Current generative LMs are trained to maximize the probability of generating ground- truth tokens at each decoding step. Assuming ,ं. {vi ..., yT) is the ground-truth output sequence for given context Feontexts the MLE objective minimizes the following loss:
T
JMLE
Σ 1=1
* log p(vi wii,.., V Feontext)
In knowledge-intensive NLG tasks, however, it is reported that the MLE training goal does not explicitly model knowledge, and thus the LM often produces counterfactual generation by surface- level misguidance (Zhou et al., 2021; Petroni et al., 2019). Furthermore, the teacher-forcing algorithm used by MLE training leads to the exposure bias problem (Ranzato al., 2016), as the LM has access to ground truth sequence up to the next token during training. but does not have such signal during testing. which causes accumulated errors when generating longer sequence (Paulus et al., 2018). Both problems heavily limit the performance of popular LMs on diverse knowledge-intensing NLG tasks. One remedy is to learn a generation policy that not only maximizes knowledge correctness but also alleviates exposure bias in longer generation, which can be made possible with policy gradient in reinforcement learning (RL).
Knowledge Trie Constrained Decoding. To formulate NLG as RL problem, define the state as the generated tokens before & (i.c., Vet), and the action as the current step output token (i.c., a, yt). The softmax output the language modeling head, i.c., a categorical distribution Pt over the entire vocabulary, is considered as the policy n, for picking token y/ (action a,) given the state s, = Yer (Guo al., 2021; Liu et al., 2020). Note that p. (i.e., TT)) could be from either a pre- trained LM LM fine-tuned with task data. While conventional sampling decoding and beam search pick next token directly from Pt, here we look to inject knowledge to adjust pt to guide decoding.
For current input context Feontext at step I, we first take its concept mentions in local memory (ci, .... } C G and then query the knowledge trie Ges with these concepts by max hops of hmax- The union of queried results of all hops V; {vi, v2.... vnn) for i = 1,.., Amax then serve as knowledge demonstrations for current step generation. To put probability mass on tokens aligning with these demonstrations, we compute the (-th step knowledge gain r, as the total log probability of
all retrieved tokens from Gen under decoding policy ne as r2 = Ei Σnev, [o]] log Te, where [[-] is the indicator function that will output an one-hot vector with in the coordinate of token
4

---

Published as a conference paper at ICLR 2022
in the vocabulary and O's elsewhere. With r,, we define the t-th step reward JRI., trajectories induced by ne as:
Te. (ag(s)) (ag (3) (KL(TellT)),
JRL..
Br-
(3)
where nं is the desired policy (a vector initialized from mT) produce tokens approaching knowledge demonstrations, and the KL penalty term (L((,lITT?) is to avoid the updated policy drifts too much away from the original one, also known as trust region constraint (Schulman al., 2017; 2015). Note that we use off-policy sampling to collect trajectories, with an importance weight / to calibrate the knowledge gain rt. which can stabilize the optimization (Munos et al., 2016).
Algorithm shows how we obtain the updated policy through policy gradient. We set dynamically to control the KL penalty within the reward function. The target divergence tunes the strength of knowledge infusion-smmalerr a means less infusion while larger provides more space for policy gradient (We set a 0.02 across all tasks). To generate the actual token for step t. we pick from the updated policy ne with sampling de- coding. The token should conform to the knowledge demonstrations, since its corresponding hidden states have shifted towards V due to policy gradient. We empirically choose K 3 for good per- formance in most cases.
Algorithm 1: -oietronst Policy Gradient
for / .... do
Collect samples (s.]s:) by vanilla policy m. Compute reward JRL. by Eq. (3);
Compute updated policy
arg max;; JRL, by taking K steps of
SGD (via Adam);
if ( 2 2a then
3.+1 23,;
else if (EKKL(1 (T. |T s 2 then
3++1 3,/2;
end
Generate token y/ with updated policy n,;; end
Prior approaches have also explored to set sequence-level metrics such as BLEU (Papineni et al., 2002) as the reward to directly optimize the generation quality (Li et al., 2016; Paulus et al., 2018). However, many studies report such sparse reward will cause low efficiency optimization (i.e., J, 0. VI. < T) (Guo et al., 2021). Our trie-constrained policy gradient method seemingly mitigates this problem by using an immediate reward (JRL) at each step with reasonable approximation. Recent lff- RL for NLG work show promising results when using data itself (rather than metrics) to estimate rewards (Pang & He, 2021; Jaques et al., 2020) our design of knowledge trie echoes their findings. The gold data distribution memorized in trie is treated as guidance to reshape each step probability distribution, and thus brings benefits on alleviating exposure bias during long generation.
EXPERIMENTS
4.1 EXPERIMENTS SETTINGS
We consider three diverse types of knowledge-intensive tasks for evaluation (statistics see $A.1):
Abstractive Question Answering. We study Abstractive QA, which requires the model to generate free-form answers to the questions. We choose long-form QA task ELI15) (Fan al., 2019b) and MSMARCO NLG task v (Nguyen et al., 2016) as two commonsense QA tasks whose questions can be mostly answered by referring to Wikipedia passages. We also use two extra QA tasks PIQA (Bisk et al., 2020) and PubMedQA (Jin et al., 2019) whose questions require domain- specific knowledge to answer (i.e., physical interaction knowledge for PIQA, and medical knowledge for PubMedQA). We calculate BLEU-I and ROUGE-L scores to be able to compare directly with related work (Lewis et al., 2020b; Krishna et al., 2021).
Logic-centric Writing, We also investigate whether KID can benefit NLG tasks that do not have an explicit query form for certain knowledge (i.c., with specific questions, like QA tasks). We study ROCC story ending generation (Mostafazadeh et al., 2016), which requires generating legitimate ending given a four-sentence context, and (NLG (Bhagavatula et al., 2020) which
5

---

Published as a conference paper at ICLR 2022
Table 1: Benchmark results six diverse knowledge-intensing tasks. Compared with beam search (Beam) and sampling decoding (Sample), KID decoding improves the generation quality in general (by at most 15%). For each LM, we report their performance zero-shot setting (*), and that of being fine-tuned (FT). We color the next-best performance. We also annotate the performance reported by published state-of-the
those results of KID that achieve 5% improvement over
models to date (- means missing official reports).
EL.15
MSMARCO
ROC
oNLG
WoW
Mu'uul
Existing Method
B-1
R-L
B-1
R-L
B-1
R-L.
B-1
R-L
F-1
R-L
MRRg
R-L
GPT2-M [34SM]*
16.1
28.8
30.1
12.0
13.9
13.2
15.0
9.3
10.8
29.7
7.1
FT + Beam
22.9
24.6
44.1
50.6
26.4
20.6
19.4
25.7
10.3
12.1
47.9
11.3
FT + Sampling
23.8
25.4
46.2
51.2
26.0
20.0
18.9
25.1
12.6
11.9
51.6
20.3
FT + KID
27.9
26.6
47.4
53.9
28.1
21.2
21.7
26.9
16.4
15.9
53,3
22.4
BART-L [406M]
21.4
20.6
19.1
19.7
12.5
16.7
24.6
28.0
8.1
8.5
35.1
12.9
FT + Beam
25.6
24.7
44.5
50.5
22.3
20.2
31.9
34.1
10.9
11.9
49.2
20.6
FT + Sampling
25.8
25_1
48.4
53.5
24.4
20.9
30.5
33.6
12.2
15.0
53.7
20.4
FT + KID
27.4
26.3
51.9
56.9
26.5
21.3
33.4
35.6
15.7
16.6
54.5
22.7
Published SotA
26.2
57.2
26.3
20.8
45.0
13.5
15.5
requires generating reasonable hypothesis given two observations. We follow related work (Zhou et al., 2021; Guan et al., 2020) in using BLEU-1 and ROUGE-L as evaluation metrics.
Dialogue Generation. Chitchat dialogues are normally multi-turn discussions over a variety of topies or concepts, which often involve topical and factual knowledge (Petroni et al., 2021). We study two dialogue datasets that require knowledge grounding: Wizard of Wikipedia (WoW) (Dinan et al., 2019), where the speaker in the conversation must ground their utterances in Wikipedia passages, and Mu'Tual (Cui et al., 2020), where utterances have to be a logically-coheeen continuation of the given multi-turn context. We follow existing work and the leaderboard of WoW in using F- F-ROOGG for WoW and MRR/ROUGE-L for MuTual evaluations.
We take two representative language models to demonstrate the effectiveness of KID: 1) GPT2- medium (Radford et al., 2019) which is an auto-regressive LM, and 2) BART-argge (Lewis et al., 2020a) which is a text-to-text LMM We tune the hyperparameters based on the models' performance on an in-house split dev set, and report the results that were best on the official dev set".
4.2 MAIN RESULTS ON DECODING PERFORMANCE
Comparison with off-the-shelf decoding methods. Table compares the results of GPT2-medium and BART-large six diverse NLG tasks with beam search (Beam), sampling (Sampling), and our proposed KID decoding algorithm. Compared with the other two commonly used decoding algo- rithms, knowledge-guide
KID achieves better results for all the cases, with significant improvements (with v-value p < 0.01) over the next- best decoding strategy (e.g.. 4.1 absolute increase for BLEU-1 in ELI5 (GPT2)). We also notice that KID brings greater improvements models - above 5% improvement 12 for text2text language model (BrRT--arre). The reason could be that the reinforcement learning objective of KID is more similar to the MLE objective of GPT2 than the denoising objective of BART (Pang & He, 2021; Guo et al., 2021). Compared with task-specific state-of-the-art models task-agnostic LMs armed with KID can beat SOTA results on three different tasks (ELI5, ROC, WoW), which is not possible when beam search and sampling decoding used. This interesting observation demonstrates huge potential in inference- time optimization, which we believe is worth exploring further.
nuto-regressive language out 12 metrics for GPT2-medium, in contrast to 5 out of
'For sampling decoding. we run experiments with all combinations of top-p (p E [0, (0,0., 0.1,1..... 1]) and top-k (k E [0, 10. 100]). while for beam search, we sweep the number of heams from to 10. With the updated decoding policy, KID uses sampling decoding (with similar search get optimum p and k) pick actual tokens.
'Published SotA models to date (October 2021): I): ELI5 (RAG: 2020b), MSMARCO (RAG: 2020b), CCCCKKKnledgeeeaaaeeee GPT2; 2020), aNLG (T5; 2019), WoW (BART+DPR; 2021), and MuTual (Human Performance; 2020). The information mainly comes from corresponding leaderboards.

---

Published as a conference paper at ICLR 2022
Table 2: Performance of six related works Wiki-answerable ELI5 and MSMARCO, and out-of-
domain PIQA and PubMedQA QA tasks. We report ROUGE-L score with 10% and 100% of the training data. As model-agnostic method, KID shows particularly strong performance in few-shot scenarios, which can better help LMs transfer to new domain with minimum training data.
ELIS
MSMARCO
PIQA
PubMedQA
Method Available FT Data
10%
100%
10%
100%
10%
100%
10%
100%
GPT2 Knowledge (KG Triplets Post-train; 2020)
9.3
15.7
22.3
42.1
7.5
16.2
3.6
8.4
GPT2 COMeT Emb. (KG Embedding Fuse; 2020)
13.4
17.3
30.3
44.6
6.4
16.5
4.5
5.8
RAG (Wiki Retrieval Augmented; 2020b)
5.7
21.4
25,4
57.2
3.2
17.3
1.2
6.7
FiD-T5 (Iwo-step Retrieval + Seq2Seq; 2021b)
3.9
18.1
23.7
53.1
4.5
17.4
1.3
4.5
QA-GNN (GNN Attention; 2021)
6.2
19.5
21.3
50.5
6.3
19.1
7.8
11.7
ReFlective (Forward + Backward LMs; 2021)
8.7
18.2
23.5
44.7
5.9
16.7
4.1
9.2
Ours: KID (with GPT2-medium)
15.2
26.6
32.3
53.9
10.5
18.4
9.9
11.5
Comparison with existing knowledge-infusion methods. Besides evaluating KID from pure decoding perspective, we also compare KID with existing methods of integrating knowledge for knowledge-intensee intresivee NLG tasks. In addition to ELI5 and MSMARCO, also evaluate two extra QA tasks: PIQA and PubMedQA (discussed in $4.1), where answers are considered to be fully covered by Wiki knowledge. Besides RAG (Lewis al., 2020b), we consider several competitive prior methods incorporating knowledge including a) GPT2 + Knowledge (Guan et al., 2020), which post-trains GPT2 on converee augmentation data (e.g.. (helium, is, gas) "helium is gas. "); b) GPT2 COMeT Embeddings, which fuses knowledge-aware embeddings (e.g.. from CoMET (Bosselut et al., 2019); c) FiD-T5 (Izacard Grave, 202 b;),, which concatenates retrieved grounding documents with context new input; d) QA-GNN (Yasunaga et al., 2021), which traverses a graph neural network; e) Reflective decoding (West et al., 2021), which relies on forward and backward LMs to encode bi-directional context for generation.
As shown in Table 2 (columns with 100% training data), KID outperforms all other methods in ELI5 (with 5.2 ROUGE-L points improvements over the second-best method (RAG)), and achieves competitive results requiring neither specific model architecture nor additional training to infuse knowledge. We also evaluate few-shot setting, where only 10% of task training data is available to mimic domains for which ample training data is unavailable or difficult to acquire in practice. This setting also tests method's ability transfer to concepts or events. Also shown in Table 2, KID with a LM similar in size to the baselines (GPT2- medium) achieves best few-shot performance in all four tasks, including PIQA and PubMedQA. Our experiments find baseline methods tend to generate off-topic and hallucinatory answers when the expected answer length is long (e.g., ELIS and PIQA). RAG shows limited performance in few-shot scenarios, due to its static knowledge retrieved by initial context cannot be generalized to newly generated tokens. KID, instead, dynamically searches references for grounding, thus showing more agility in unfamiliar context. Comparison with more baselines can be found in $A.3 of Appendix.
domain and to generalize unseen entities,
4.3 ABLATION STUDIES FOR THE BEST PERFORMING KID
How to choose retriever? We experiment with replacing the default DPR document retriever of KID, with popular retrievers including the TF-IDF retriever from DrQA (Chen et al., 2017), and the Wiki retriever used in BLINK (Wu et al.. 2020). We also experiment with a random retriever baseline that retrieves documents randomly given the context. We choose two tasks ELIS and WoW that provide ground-truth knowledge provenance, which are also the only two KILT tasks requiring long and abstractive generation (Petroni et al., 2021). Following KILT benchmarking metrics, we use precision @ (Prec @ I) to measure the top-1 retrieval accuracy, and ROUGE-L (R-L) to evaluate generation quality. We also consider directly measuring how much knowledge in the ground-truth evidence (provenance) appear in the actual generation. We compute knowledge Coverage (Cov) (used in Guan et al. (2020)) which the 1 I-gram overlap between the triplets of provenance and the generation, to measure the extent to which our generation is actually using the retrieved knowledge.
7

---

Published as a conference paper at ICLR 2022
26.6
27.0
(a)
5.5
5.0
26.58
(b)
3.6
(c)
26.54
3.2
25.4
* Performance
3.5
Huroon Ratines
25.0
Time
3.0
26.46
* Performance
2.4
26.42
2.0
Reflective
24.2
2.0
Sampling
23.8
1.5
26.34
10
20
30
26.30
0.8
80 100
120
180 200 220 240 260 Sequence Length
140
160
Number of Retrieved Docs
Number of Hops
Figure 2: Impact of hyperparameters on KID's ELI5 performance when (a) more documents are retrieved, and (b) more hops taken when querying the knowledge trie. (c) Average human ratings different-length sequences generated by KID, sampling, beam search, and reflective decoding. KID generation has more stable quality across lengths by restraining exposure bias.
Table 3 shows the corresponding results. For reference, RAG obtains nearly equivalent performance (R-L and Cov) with random retriever and the DPR which indicates its generation mainly relies on the tne-tuned BART and ignores retrieved knowledge (also observed in Krishna et al. (2021)). In contrast, KID with the default DPR retriever outperforms all other retrievers and RAG variants in retrieval accuracy (Prec @ 1), knowledge coverage (Cov) and final generation quality (R-L). We observe high correlation between (Cov, R-L) and Prec I, indicating that a good knowledge retrieval is essential for mowledge-intensive NLG tasks.
Table 3: A closer comparison of BART-L with KID and
Table The performance (R-L) on ELI5 of LMs with different sizes (similar architecture). Vanilla LMs (*) benefit more with KID than the fine-tuned ones (FT). (The absolute gain over the best is annotated.)
RAG (Lewis et al., 2020b) which also leverages retrieved Wikipedia passages as knowledge. We switch between differ- ent retrievers to study its impact on retrieval accuracy (Prec@ I), generation quality (R-L), and knowledge coverage (Cov).
ELI5
WoW
ELIS
Prec@1 1
R-L
Cov
Prec@1
R-L
Cov
Model
Beam
Sample
KID
RAG w/. Random
3.4
21.3
2.62
15.6
10.4
9.12
RAG w/. DPR PT
16.7
21.4
2.68
26.3
11.8
11.8
GPT2-M*
16.0
16.1
20.2 44.1
GPT2-M
24.6
25.4
26.6
KID w/. Random
3.4
16.5
1.34
15.6
9.5
7.79
KIDD w/. TE-DF
11.0
20.9
3.88
41.9
15.4
12.8
GPT3-1.3B*
21.7
22.0
24.5 25.
KID w/. BLINK
8.9
21.5
2.87
24.3
10.0
8.31
GPT3-1.3B
24.9
25.5
26.6
KID w/. DPR (default)
17.3
26.3
4.39
45.5
16.6
15.7
GPT3-2.7B*
22.8
24.6
26.7 2.1
How much knowledge do we need? We also study the impact of number of documents (k) we retrieve and number of hops (hmma) we use to query the knowledge trie, two factors that determine how much knowledge we use ground the generation. As shown in Figure (a) and (b), for the example task ELI5, we find the generation performance measured by ROUGE-L does not benefit from simply more retrieved documents- an optimum & is through empirical observation, and similarly, 1max brings best performance. A larger & might risk retrieving less relevant Wiki documents and a larger hop Amax with deeper traverse through the knowledge trie tends to bring in off-topic knowledge. We also plot the average time consumed for generating each sentence reference (the second yxaxis in Figure 2 (a) and (b)), which demonstrate the best performing k = 5 and Amma = 4 achieve reasonable trade-off between generation quality and decoding speed.
In Figure 2 (c), we quantify the exposure bias problem through human judgements. We first sample 200 ELI5 test set questions and generate answers of various lengths {80, 100, .... 260> (260 is the average sequence length in training set) with beam search, sampling, reflective (West et al., 2021), and KID. We then ask humans to rate these generations with 7-point Likert scoring (Joshi et al., 2015) how likely the generated text is natural sentence. Each generation receives at least 15 ratings. We observe that both beam search and sampling methods suffer from the exposure bias problem since
RRAG fine-tunes the query encoder of DPR with BART, which differs the off-the-shelf DPR used in KID. Wee use beam size for heam search, and top p 0.9 and i 20 for sampling decoding, as they yield hest. ROUGE-L score during automatic evaluation.
8

---

Published as a conference paper at ICLR 2022
Table 5: Human assessments of generation in terms of Relevance, Factuality, and Grammaticality
on 7-point Likert scale. We run paired sample t-test comparing human references (Gold) with
beam search (BM) with beam size 5, sampling (SP) with top p = 0.9 and k = 20, reflective (RFLC)
decoding, and our KID generation. p value describes the significance of difference from Gold. (*
corresponds to p-value< 0.05 and ** to 0.01.)
FLJ55
aNLG
WoW
Gold
BM
SP
RFLC
KID
Gold
BM
SP
RFLC
KID
Gold
BM
SP
RFLC
KID
Relevance Mean
5.14
4.51
4.97
4.73
5.07
5.42
5.22
5.10
5.27
5.36
4.62
4.68
4.44
4.35
4.57
p-value
-00**
.10
.03*
.30
.14
.02
.17
.23
.12
.10
.06
.30
Factuality Mean
4.95
4.37
4.61
4.23
4.87
5.35
5.17
5.19
5.25
5.30
4.72
4.20
4.38
4.41
4.53
p-value
.14
.00*
.00*
244
.05
.06
.30
.41
.00**
.06*
.15
.29
Mean
5.66
5.52
5.54
5.07.
5.50
5.40
5.23
5.34
4.97
5.27
4.53
4.48
4.40
4.14
4.33
Fluency
p-value
.09
.02
.07
.15
.4*
.16
.18
.10
.08
their ratings deteriorate as the length grows. Reflective decoding exhibits similar trend since stills relies on MLE training. KID, instead, dynamically infuses global knowledge with LM predictions at each step and thus can mitigate exposure bias by imitating non-local demonstrations.
Does the size of LM matter? We run experiments with different sizes of LMs that have the similar architecture (GPT2-medium, and GPT3 with 1.3B, and 2.7B parameters') ). Table 4 shows that overall larger LMs benefits all decoding methods with KID consistently outperforming Beam search and sampling decoding. In addition, KID brings more gain for non non-finetuned LMs than fine-tuned ones, potentially because the fine-tuning LM is already fit onto the new domain thus less knowledge infusion needed. Interestingly with KID, vanilla GPT3-2.7H outperforms its 1.3B fine-tuned counterpart, which is especially meaningful since ever-large foundation models (Bommasani et al., 2021) are expensive to fine-tune effectively on common hardware settings (Liu et al., 2021).
4.4 HUMAN EVALUATION
We recruited 300 MTTurk participants to manually examine generated outputs of several decoding algorithm in terms of relevance, factuality, and fluency. Each participant was asked to review five sample generations without revealing their source. We used paired samples t-tests examine the difference between human references and other decoding methods generation. As Table 5 shows, KID generates similar quality sequences human references without significant differences across all three tasks, while in ELI5, beam search and reflective decoding generation are rated significantly low in both relevance and factuality, partially due to exposure bias in longer generation. There is no significant difference in grammaticality between KID and vanilla decoding methods. The exact questions we asked participants can be found in $A.5 of Appendix.
5 CONCLUSIONS AND FUTURE WORK
In this work, we proposed KID, a novel decoding algorithm which in each step of decoding dy- namically fetches knowledge and guides token decoding via interaction between a local knowledge memory and constructed external knowledge trie. Given its decoding nature, KID doesn't 't require architecture changes to existing LMs and is applicable to various generative LMs. Evaluations on eight wewolelde-nntessiie NI.G tasks demonstrated the effectiveness of KID and that its generations are aligned with human references well. In addition, KID is also found to alleviate exposure bias and maintain quality for long generations.
Future work of KID could study more efficient data structures to accelerate knowledge fetching and integration. One could also investigate combining KID with prompt based generations to further boost the performance, especially in the few-shot settings. Another direction is to further study the integration of KID with full size foundation models, like GPT-3 175B to understand KID's potential.
WWe adopt the public implementation of GPT3 GPT-Neo (gi thub. com www.uuthrrlll /gpprr/llg/ppete.rro

---

Published as a conference paper at ICLR 2022
ETHICS AND REPRODUCIBILITY STATEMENT
The goal of KID is to provide general-purpose knowledge infusing decoding algorithm by leveraging retrieved Wikipedia documents. Still, the generation of KID can be affected by certain biases from the LM is based on though those biases may be partially mitigated by the knowledge demonstrations (Liu et al., 2022; Rae et al., 2021). Another major ethical consideration is that KID can mimic undesirable attributes of the target knowledge source documents that could be non- contemporary and do not represent current norms and practices-and KID has scheme diagnose these problems (Lewis et al., 2020b). Furthermore, our experiments and analysis are done in English, and therefore we do not claim that our findings will generalize across all languages. although our framework has potential to be extended to other languages with necessary modifications.
For reproducibility, evaluations of KID and baseline methods are all conducted on public NLG data sets. We compare results from published papers and public leaderboards. Code and scripts for reproducing KID is available on GitHub at https: / figituub www.//ccooooo///ee./oooo
REFERENCES
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open- domain chatbot. ArXiv preprint, 0ass2001.099971 2020. URL https: /ata/aivivvo/// org/ /abs/ 2001.09977.
Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. Generating more interesting responses in neural conversation models with distributional constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3970-3980, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 1/D18-1411 URL https: //a //aclaaaae taclanthology hrwtchology..rr/D1eee 1431.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: http:///eeeeeeeeeeeeeee/eeeeeeee net/ forum?ide ByylvIHKDE
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439, AAAI Press, 2020. URL https: / /awaiaaaaa...//aaa//aaaa/aaaaaa. org/ o js/ iinex...x.. php/ AAAI wall//articll wartaceeeiiiiiiiieeeeeeeaaeeeeit 6239.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the op- portunities and risks of foundation models. ArXiv preprint, abs/2108.07258, 2021. URL fttps:://ariis org/ aas///b///// 2108 07258.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4779, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1470. URL https: / aaclantoooogyy.....o wwwplloggoor.gg..ge wttpp/9--/7/-7/
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. In AAAI, pp. 4923-4931, 2021. URL
https://ojs aaai. org/ iinex.idxxi//iii php/ http://///aarticeeeeeeeeeee/////
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1877 Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/v 1/117-1171.
URL https: / ac anthology.
org/P17-117.,
10

---

Published as a conference paper at ICLR 2022
Pinzhen Chen, Nikolay Bogoychev, Kenneth Heafield, and Faheem Kirefu. Parallel sentence mining by constrained decoding. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pp. 1672-1678, Online, 2020. Association for Computational Linguistics. doi: 10. 000//3/v1//220.00 acl- 152. URL ht : / anthol wwg//0200000......o ac -ma in. 152.
Eric Chu and Peter J. Liu. Meansum: A neural model for unsupervised summarization. In Kamalika Chaudhuri and Ruslan Salakbutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1223-1232. PMLR, 2019. URL http:://groceedingg mlr.. press / / chul 9b. html.
abstractive
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. MuTual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Annual Meeting the Association for Computa- tional Linguistics, pp. 1406-1416, Online, 2020. Association for Computational Linguistics. doi: 10.18663//1/2022.acc main.130. URL https: acl httpa/caaatttooooooooooooo org 2020 aac-malllaal 130.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino. Jason Yosinski, and Rosanne Liu. Plug and play language models: simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: /otp::s///peeneeeee. net/ vet/fozze wtt////oummddaoeeeee Hee.:mmmddddllleeeBBBBB
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, 2019, Association for Computational Linguisties. doi: 10. 10.18653/1//19--44233 URL https://
aclanthology org/N19-
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powert conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpps://opeerrrviies net/ www.teernnnteeeee i 2e11r1111 11173iRqKm.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P18. 1082 URL https:t/p///.... /eat://aalaaann /aallaaaanollooooooeo www.g...rg/1.gg1 -1002.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 41 4186-4196, Hong Kong, China, 2019a. Association for Computational Linguistics. doi: 10.18653/w1/D19 1428. URL https: / aac.anaaaannooyyyc.ccoc. wtgg//1g 1428.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558-3567, Florence, Italy, 2019b. Association for Computational Linguistics. doi: N..11655//1/P1911 1346. URL https: / a antho logy- org/p1 9- 1344
Ulrich Germann, Eric Joanis, and Samuel Larkin. Tightly packed tries: How to fit large models into memory, and make them load fast, too. In Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (SETQA-NLP 2009), pp. 31- 39, Boulder, Colorado, 2009. Association for Computational Linguistics. URL https:/ aclanthology org/ 909-10-101 1109.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotater annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop New Frontiers in Summarization, pp. 70-79, Hong Kong, China, 2019. Association

---

Published as a conference paper at ICLR 2022
for Computational Linguistics. doi: 10. 10.18633/1/D19-54099 URL htLhttss///cclaattooooyy oxq/D19-5400
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. A knowledge-enhaneed pretraining model for commonsense story generation. Transactions of the Association for
Computational Linguisties, 8:93-108, 2020. doi: 10.1 (a166//talla_000000 URL https:
s/aclantholl wrt.oorgg20000 ttal-1.7.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. Text generation with efficient (soft) q-learning. ArXiv preprint, abs/2106.07704, 2021. URL https: http://axxvvv...../.. abs/ 2106. 07704.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3929 3938. PMLR, 2020. URL http:/ hatee://prroceee proceediiggg. hll.ppres/vlle he1111////2/00 2uu2 html.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638-1649 Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/w1/P18- 1152. URL https: / aacl antho 1 ogy. wrgg/r//11.... P1 1152..
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: http::///pperevveeee ew.
net/forum? wiw.:mididdyyyyyy33000 rygGQyrFv..
Luyang Huang. Lingfei Wu, and Lu Wang. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 1107, Online, 2020. Association for Computational Linguistics. doi: 10. 118866/////220... aaain2..cllmminn4455 main.457. URL https: htt////aaaaaatttaaaaeaa org/ 457.
2020. acl-mallaaaaa.
ma maain
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https:/ /ttttps://openreveee.. n forum? Dirum?idNNEE z 6wysdb.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, 2021b. URL https:
Association for Computational Linguistics. doi: 10. 122653//120021.eaclllm // //aclanthology org/ 2021. eace-main..................m
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985 4003, Online, 2020. Association for Computational Linguis- tics. doi: 10. 2ttp://///220..mmpmmmiiial emr.ppmmain m.... 327.
.rg. 2000.. URL htl.tttss///ccaataaaat
Yacine Jernite. Explain anything like i'm five: A model for open domain long form question answering, 2020. URL pentts::/yjjnntt http///eeeeeeeeeeeeeeeeee githee..itt........... io /o/lfqa html.
Haozhe Ji, Pei Ke, Shaohan Huang. Furu Wei, Xiaoyan Zhu, and Minlie Huang. Language generation with multi-hop reasoning on commonsense knowledge graph. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pp. 725-736, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18653/v 10.18653/v120220.cmm 1/2200.e120002mlppppp. www.mmmnnpppnms..mmmmmmm 54, URL https: / //cl/ antho ogy. orgg/020 emnlp i n.. 54.
12

---

Published as a conference paper at ICLR 2022
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.-IJCNIP), pp. 2567-2577, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10. 101169//1/ URL htt.t://///// //acl /tt.::::://alaanhoooog...o org/D19- 1259..
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019, URL https:://eeexppl feee .org/ aaaasttsracc
Document/873305
Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4):396, 2015.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-676677 6781, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18653/11/2220 emnlp- maip- URL ht tttpps:. : /aalaacannnt/ aclanthologe org / 2020. emnlp 5500
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 781 1-7818, Online, 2020. Association for Computational Linguistics. doi: 10. 20.18653/1/2020..o actp:220000.02mmaaaaaaaa 69.. URL https: http///aaaanthooooooo org/ 202 acl-main. 698.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL A Conditional Transformer Language Model for Controllable Generation. ArXiv preprint, 1909.0588,, 2019. URL
abs/ 1909. 05858.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representationss ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https::/
net/ forum?id HklBj 1K1B:CEKVH. CEKvH.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form question an- swering. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 4940-4957, Online, 2021, Association for Computational Linguistics. doi: https: https////aaaaaatttoaogae org 2022..211122 naa.la maa in.. 393.
URL
Kenton Lee, Ming-Wei. Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 6086-606668880 6096, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.118533v1//19-111 1612. URL https: http:///ccaaattooooooo htwphehhoooooooooii--eeee org/p19-1611.....1
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4656-4661 Online, 2020. Association for Computational Linguistics. doi: 10. 00.18855/v1/2020.cm main..223325 URL https: //acaaaaaa aalllanhhoolyyy org/ 2020 acl.aaclmmaaa 423.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeene pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 7871-7888 1-8800 Online, 2020a. Association for Computational Linguistics. doi: 10. 10.18653/v1/200..cm main.703. URL https: / aacllel antho wowy.oloo..ooooooooooooooooo...o org/ 2020. acl -ma-mallla-mmm in.i 703.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NI.P tasks. In
13

---

Published as a conference paper at ICLR 2022
Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b. URL hRt.tttps:/ http://////oceoeeooeeeeecee neurips. wttccc/pppeee wppppr//2///aaaa/eeepppppp 6b4 02493230205f7888 010.977000112202555778662----000
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192- 1192-1202, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/DI6-- 1127. URL https: / /at///aaaanntttoooooee
org/D16 1DDD-112..21
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6862-6868, Online, 2020a. Association for Computational Linguisties. doi: 10. 18653/v1/2020. Lmnlp-main.555 URL htttps://aiiaanoologg... org/2020
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for gener- ative commonsense reasoning. In Findings of the Association for Computational Linguis- tics: EMNLP 2020, pp. 1823-1840. Online, 2020b. Association for Computational Linguistics. doi: 10.18653/v 1/2020. 1/2000.inniingseeeee findings--enll.165. URL https: http://acaaaatoocoooooooooooccoa org/2020. findi Findings-ee meiiing--mmmm emn 1 p 65.
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9031 1041,, Online, 2020. Association for Computational Linguistics. doi: 10. 101138861/22/ 1/2020. emnlp- mapp-mmin.7770 726. URL ht tps /t///alaantttt aalaattonotooooyyoooooooo org / 20000. emnlp-mnmmmmmm mwa.p.mmai....mmmmm...mm..m.m... 726.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviat- ing political bias in language models. Artificial Intelligence, 304:103654, 2022. iSSN 0004-3702. doi: htts://doi.org// 2ttp://00..0011//..ttt2101011110 com/
URL https: http://wwwwwccneeeeeeeeeceoe sec.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. ArXiv preprint, abs/2103.10385, 2021. URL https:/ http://xxxvvvrro./.e org/abs/
2103.1038
Robert Logan. Nelson Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5962-5971, Florence, Italy, 2019, Association for Computational Linguistics. doi: 10. 18853//11/P19115900 URL https: /t/////aanthlll.oorrrgo// www.ooyyoorg/1111/.0011
Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han. Constrained abstractive summarization: Preserving factual consistency with constrained generation. ArXiv preprint, abs/2010.12723, 2020. URL
https://arxiv.cr org/ abs 20100000002 12723.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association ConprstooooooCooooooooottttioooo Computational Linguistics: Human Language Technologies, pp. 839-848 849, San Diego, California, 2016. Association for Computational Linguistics. doi: 10. 1...866//1/N11-1099 URL https://///........ / / aac antho ogy- org/Nil 6- 1098.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off- policy reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016. Barcelona, Spain, pp. 1046 1054, 2016. URL https: /ttptt:::///ocoeeeeo pttp://tceeeee.ccccoccccpcc...se newrips......... ccp pttpar/pps.ccppppp///00 hash/ c. 9229999999 5 2bd1 848884855555 5b3dddd----ooodo Abstr--tbtttttt.ooo html.
14

---

Published as a conference paper at ICLR 2022
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016. URL http: /ttt::://ceurviee ceur- wwo.
/tt//Voo-11111 11oooo11711-00 C/CCCCCCCCC _0162016662 p_ppppppppp____ pdf.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: http:///pppezeeeeeeeeee net forum? id=Rovx-u0HHuu.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002, Association for Computational Linguisties. doi: 10.3115/1073083110733 1073315.. URL https: /aaaaaaaannnooooyo.oooooo
org/PO2-1049
Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive sum- marization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018, URL http://pppereeeeeeeeeeeee...ee.e net/ / Forum?id=HKaClen
Ben Peters, Vlad Niculae, and André T. Martins. Sparse seuueece-to-sequenee models. In Proceedings of the 57th Annuul Meeting of the Association for Computational Linguisticx, pp. 1519, Florence, Italy, 2019a. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1146. URL https: / /aa/aalaaattt ccl.aahooooooyyyyyyooooooooooooe - org 2rgg/1992 P19-1146,
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 43-54, Hong Kong, China, 2019b. Association for Computational Linguistics. doi: 10.18653/v1/D199 1005. URL https: / /aa/accaannhhoo
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 2463-2473, Hong Kong, China, 2019, Association for Computational Linguistics. doi: 10.18653/v1/D19- 1250. URL https: / aa//acaannn www.cnnnnthooyyy..o www.looggyrrrggg// D19-125..
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. main.200 URL 2021. Association for Computational Linguistics. doi: 10.18653/v w.t.2l.aaaclllleeaeaaaaaaaa https: / /acllalle anthology org/ 2021 www.naallllaam mma in. 200.
Online,
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 794-805, Online, 2020. Association for Computational Linguistics. doi: 10. 2...16655w122222.mmmmmmmmmaa //acl antho ogy. orgg 2020. emn 1-maa
URL https:
ma i in. 58.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. TI- MEDIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Con- ference on Natural Language Processing (Volume Long Pupers), pp. 7066-7076, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 1ac.222.....oooo55555044 URL https: https////aaaaatooogo..ooo otg/20211 acl- 549.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl Blog. 1(8):9, 2019.
15

---

Published as a conference paper at ICLR 2022
URL http:p///waaaaao.....o.oooootraa
lanquage-modelle mdd.....d.......................
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112. 12.111..144. 1446,, 2021. URL https: a arxiv...... www.iivorg..oo.....ooo org/ alos/ /oss//111. 2212. 11111.114.....
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv preprint, abs/1910. 10683, 2019. URL https: aaw.a.iivvoooo www.ii.orrg/bbaa. 910. 10683.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: pett:://axxiae /tta/aaxvv.ooo/ abs httpa/ass/1111111 06732....7..
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5416,442,, Online, 2020. Associa- tion for Computational Linguistics. doi: 10. 10.18653/v1 (0/65////2220...oo// http://..0220....1mmm./ maammlpmainn43... URL https:
//aclantho 1 ogy. org/ 2020. emn 1a-mllppmalee main........ 437.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 300-325, Online, 2021, Asso- ciation for Computational Linguistics. doi: 10.18653/1/22211.comm manelmmii..24, //aclanthology org/ 2021. wacl-main..........
URL https:
Maarten Sap. Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. ATOMIC: an atlas of machine com- monsense for if-then reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligncee Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pp. 3027-3035. AAAI Press, 2019. URL https:///// / boo.ooorroooooooooo 10. 1609 /aaiiaaaa.........aa v33101.
doi: 10. (//6///a/../3311....... 33C13027.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897.MMLR.. JMgR.org, 2015. URL http::/tp/// wwwppreeeinngs...m.om.........se mwr.press//v3 sww.umaallamaaacmccccc/c.a 5 html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347, 2017. URL https:t////// /ttps///axxvvoooo abs/1707 06347.
Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2210-2219, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1
URL https: /t//caaannnttt www.a/hwwooooo..ooo111112220 org/D17-12355
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Satinder P. Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-First AAAI Conference California, USA, pp. 1444-4451 AAAI Press, 2017. URL http: aaai. org/ocs indexx. php/AAA AAATI wttp://papppa///ee 149777
Artificial Intelligence, February 4-9, 2017, San Francisco,
16

---

Published as a conference paper at ICLR 2022
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pp. 885- 895, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v N18-1081. URL https: /tt////lllcc wwathhoooog... .rg /N1 8110
Yu Sun, Shuohuan Wang. Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8968- 8975. AAAI Press, 2020. URL htt.t://////// /aaai. capaaiorgg org/ojs index. php/ / AAAI AAAA/arrille/ vi wiee//K 6428.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. Mp.ppcs-oon what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758, 2020. doi: 10.1162/tac__ a 00342. URL https: htapps//accaae hat/aaaaaanntoooooooaaa wwooor///002 tacl-1 -1. 48.
Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. Neural relation extraction for knowledge base enrichment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 229-240, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 1///5/ 1023. URL https: : / /at/aacaanntttt anlaalohhhooyyy wtt///9-111/ 1023.
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Consistency of a recurrent language model with respect incomplete decoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5553-5568,
Online, 2020. Association for Computational Linguistics. doi: 10. URL https::tp///occ...t /ccl/aacccccc www.ccaanntooooogoo org/ 2020. emnlp-ma innaiin... 4
Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena D. Hwang, and Yejin Choi. Reflective decoding: Beyond unidirectional generation with off-the-shelf language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1435-14555 1450, Online, 2021. Association for Computational Linguistics. doi: 10. 121133/v1/222... acl- 1 14. URL https::pp........p. : / ac onnhollgy.........o........o.oot org/ 2222222211 accl -lng. 114.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing (EMNLP), pp. 6397-6407, Online, 2020. Association for Computational Linguistics. doi: 10.11863/11/202000oo www.:000..mmmpmmmii..... 519. URL https: //aclant Taclanthologyy...o org/ 2020. emnlp-main 519.
Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 219-228, Hong Kong, China, 2019, Association for Computational Linguistics. doi: 10.18653/ 1021. URL https: / ac www.caannhooogoooooo www.oooooooooogo.eooooe 1021.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. Topic aware neural response generation. In Satinder P. Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 3351-3357, AAAI Press, 2017. URL http: /aaai.aa......... owg//ocs....oooooooooooooooooooe index. php Chpp/AAA// AAAAAAAAAAA.....A http//pppp//iiiiii e445663.
Wenhan Xiong. Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: Weakly supervised knowledge- goowledg-preerrinee language model. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:tt////o// /twp///veeeevvee/eee////eeeveeee ew. net forum? d=BJ1 zm 64 L....
17

---

Published as a conference paper at ICLR 2022
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 535-546, Online, 2021. As- sociation for Computational Linguistics. doi: 10.18653/v1/203 naacl-main URL https:
//aclanthology org/ 2021. cineel-mmann... main.45.
Houyu Zhang. Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. Grounded conversation gen- eration as guided traverses in commonsense knowledge graphs. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2031-2043, Online, 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl main. 184. URL https://aclanthhee
org/ 2020. acl acl-maalllnna... 184.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of
Proceedings of Machine Learning Research, pp. 1 11328-11339. PMLR, 2020b. URL http: //proceedings mlr press /veess/vlll /vs1s/v1///////// htap1///zhagggaa0e html.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2204-2213, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1205. URL https: htttt://cclattttteee http///aaantoooo.orooooooo wwt/p18-121100000
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT Large-scale generative pre-training for conver- sational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 270-278, Online, 2020c. Asso- URL https:
ciation for Computational Linguistics. doi: 10. //acl antho 1 htngy.org/20000 acl-demos..... 30.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.1160116000 N0.11853//1//19111 139. URL htttps://aa antho logy. org/P19- 1t////991133
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. Knowledge- grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3377-3390, Online, 2020. Association for Computational Linguistics. doi: 10. URL https: : a anthology org/ 2020 emnlp-mat 272.
Wangchunshu Zhou, Dong- Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. Pre-training text-to-text transformers for concept-centric common sense. In 9th International Conference on Learning Representations, ICLR 1021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://t//etttt..tt http://///////vvvie/eeee eww net. Forum? a HYL2..
18

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DATASETS STATISTICS
We chose eight knowledge intensive NLG tasks to evaluate KID. Here we present the dataset statistics of these tasks in Table AI. The links these datasets can be found in $4.1.
Table A1: The dataset statistics of the eight knowledge-intensive NLG tasks we evaluate for KID.
Split
ELIS
MSMARCO
PIQA
PubMedQA
ROC
oNLG
WoW
MuTual
Train
272,764
153,725 12,467
16,115
800
52,665
50.481
63,734
7,088
Dev 1,507
1,838
100
1,571
7,252
3,054
886
Test 600
12,467
3,000
100
4,081
14,313
2,944
886
A.2 DETAILS ON KNOWLEDGE TRIE CONSTRUCTION AND QUERY
A.2.1 KNOWLEDGE TRIE CONSTRUCTION
In this section, we detail the process of constructing the knowledge trie and how we query for the knowledge a dynamic fashion. In Figure A1 (a), for a given question (say from the EL15 dataset), the DPR retriever (Karpukhin et al., 2020) would retrieve k documents (the effect of choosing different k on performance has been discussed in $4.3; we use k 3 here for simplicity) from 21M 100-token Wiki documents as the grounding passages for the question. We then use co-reference resolution to replace the pronouns with their referents (colored in red), normalize the text (e.g.. removing links, lower-casing, etc.), and pass them through the OpenlE (Stanovsky et al., 2018) to obtain knowledge triplets (the use of OpenIE can be also seen in related prior work (Trisedya et al., 2019; Wu et al., 2019; Fan et al., 2019a)). The end nodes of the extracted triplets (i.e., the subj and obj) serve as the key-value pairs when they are stored in the external knowledge trie (Ge and the relations between nodes are translated to the edges. We use the stems of the tokens as the keys in Gex (e.g.. "driving" + "drive"), "), in order to compress duplicated nodes in a concept-centric manner.
During knowledge trie construction, besides the one-step key-value pairs from the triplets, also consider multiple-step relations by looking for key mentions in the values of previous keys (e.g.. value: "influence of drugs"
next-hop key: "drug"). Such iterative procedure will end when there are no more keys appearing in the values. Specifically, we use depth-first search (DFS) record the maximum depth of the child branch for each key node, and also memorize the corresponding values (and its next-hop key) on the branch to facilitate further query.
A.2.2 DYNAMIC QUERYING FOR KNOWLEDGE
Since retrieving knowledge from millions of documents is slow (even with GPU acceleration). the knowledge trie described above is constructed off-line. We pre-compute and store this knowledge trie on the disk, and build the local knowledge memory on the fly. The local knowledge memory is simply a -ot (FIFO) list which continuously stores newly generated entities (Gree initialized with entities in the input question), whose length will be limited by Wmax (we set Wmax = Amux empirically, where Amax is the max query hops). In Figure A1 (b) we show a query example with the query word "driving" in the local memory. The query word is firstly converted to its stem (i.e., "drive"), and then its demonstrations are retrieved with the key "drive".
We show the whole generation loop of KID in Algorithm 2. At each step of the decoding, we first collect knowledge demonstrations by querying Gon with each entity in Glee as key. The collected demonstrations (a list of tokens) will then serve as target word space guide current step generation (has been detailed in $3.3). The local knowledge memory is updated only when a new entity is generated. This is because the non-entity words (such as stop words, digits, etc.) a) are rarely the keys in the Gext (which are subj in OpenIE triplets), and b) rarely lead to meaningful next-step constraints (it's hard to answer what's proper continuation of "the").
Since the external knowledge trie is constructed off-line, the query time mainly depends on the number of query words in the local knowledge memory (with max length wax)). and the maximum number of
19

---

Published as a conference paper at ICLR 2022
(a) ELI5 Question: Does marijuana impair driving obility?
Gooc marijuana, driving, ability
some studies to have a negative effect on driving ability. The British Medical Journal indicated that "drivers who consume cannabis within three hours of driving nearly twiee likely cause a yehicle collision as those twhet drivers are not under the influence of drugs or alcohol". In "Cannubis and driving: review of the literuture and commentury" the United Kingdom's Department for Transport reviewed data on cannabis and driving. finding 'Cannabis impairs driving behaviour. However, this impairment is mediated in that subjects under canmabis treatment appear to perceive that they indeed impaired. Where they can compensate, they
Paragraph
ID: 20644819
alcohol concentration performed using three methods blood. breath, or urine. For enforcement purposes, breath is the preferred method, since results are available almost instantancously. Drug testing sereens are typically performed scientific laboratories that the results will be admissible in evidence at trial. Due to the overwhelming number of impairing substances that
Paragraph
ID: 16441766
alcohol, drugs are elassified into different categories for detection purposes. Drug impaired drivers still show impairment during the battery of standardized field sobricty tests, there additional Classification program is
to help detect drug impaired driving. The Drug Evaluation and
for life, Driving under the influence of drugs "to an extent as to be incupable of having proper control" illegal. The Police may require suspects undergo objective (at the police station) Impairment Test. Suspects who fail the test shall surrender tthiir) suspects who fail the test driving licence for 24 hours. Regarding common drugs, the Road Safety Council reminds drivers to check the side effects before driving. The law prohibits driving with any concentration of illicit drugs: heroin, cocaine, ketamine, methamphetaming cannabis and MDMMA Drug impaired driving Drug- iaap-iidered driving, in the context of its logal definition, is the act driving
Paragraph
ID: 16441788
Gooc
marijuana, driving, ability... ...
cannabis, vehicle, collision, influence, drugs, impaired, drivers, testing. screens, driving,
(b)
cannabis
Gext
vehicle collision
testing screens
driving behaviour
drive
influence of drugs
drug
impaired drivers
impair
subjects under cannabls
Amax
Figure Al: A sample of KID's retrieved documents for the ELI5 question "Does marijuana impair driving ability?" and its corresponding knowledge trie. (a) The top three relevant documents retrieved by DPR. We annotate the triplets end nodes (subj and obj) picked by OpenlE for knowledge trie construction in green. (b) The partially observed knowledge trie when query "driving" in the current local knowledge memory (blue) by umax hops. We perform co-reference resolution to replace pronouns with actual entities in the documents, and use the stems of the tokens as the query words and keys in the external knowledge tric. The retrieved demonstratinns (values in the trie; purple) in multiple hops will then serve as guidance for current step decoding, after split into single tokens.
hops we query the external knowledge trie (Imaa) with, which is approximately Wmax * hmax in total. We claim this actually constant time complexity because hmax and Wmax are fixed hyper-parameters (normally 1-5, as discussed $4.3) and it will not scale up with sequence length or the number of grounding documents retrieved.
A.3 MORE COMPARISON WITH IMPROVED DECODING ALGORIHTS
Besides the baselines compare in the paper, there are other inference-time decoding methods that focus on improving generation quality in different perspectives, such as diversity (Baheti et al., 2018). and attributes controlling (Dathathri et al., 2020), and entity appearance (Mao et al., 2020)".
We cannot find its official implementation, and thus we do not include in the following comparison.
20

---

Published as a conference paper at ICLR 2022
Algorithm 2: The Generation Loop of KID)
Input: constructed knowledge trie Geu, local knowledge memory Give input context q, max query hop Mmax. max sequence length L.
Output: generated tokens (I1, T2,,,,, F.).
Goe entities in q;
while current sequence length < T. do
Knowledge demonstrations D - []: for each entity e Gise do
L Query Gext with by Amax hops to collect demonstrations, and store in D.
Use D to guide current step decoding by Algorithm I, and generate token zt.
if a is entity then Append Ia to Glses and slim Glee by window size Wax
L
return (x1, T2, T2,,.,, x.)
We also notice some methods that are training-ting time optimization, such as CTRL (Keskar et al., 2019), which requires re-training language models conditioned on a set of static control words. and iusion-in-decoder (FiD) (Izacard & Grave, 2021b;a), which concatenates additional retrieved grounding sentences with input text, and fine-tunes text2text language models (e.g., T5 (Roberts et al., 2020)). KID differs from all these methods, since 1) KID aims to improve knowledge awareness during decoding, though we also find KID could mitigate exposure bias, and 2) KID can function as standalone module requiring neither specific model architecture language model (as shown $4.3, the experiments with vanilla GPT3-2.7B) In the following part, we compare KID with all above methods except CTRL since requires costly language model retraining on each new domain and how to pick control words for dynamic knowledge seems ambiguous. We also prepare naive version of KID (Little KID) which uses plain list instead of graph structure (the knowledge trie) to store knowledge, where we directly use all the end nodes of extracted triplets from the documents as constraints without any dynamic query. We run experiments on QA tasks as FiD was trained on, and discuss their differences on performance, deployment, and efficiency (time/space).
re- training or fine-tuning the
A.3.1 PERFORMANCE ON KNOWLEDDGE AWARENESS
In Table A2 we present the comparison results about performance on knowledge awareness. Not surprisingly, FiD is the strongest baseline among others, as it also explicitly leverages retrieved documents to ground its generation. Compared with KID, we find the performance of FiD is relatively limited when the NLG tasks requires longer and creative generation. For example, EL15, whose average reference length is around 220 tokens, and FiD tends generate generic and repeated sequence after about 80 tokens from our observation, potentially because though FiD fuses knowledge during training, there is no scheme to guarantee the knowledge parameterized in the model can be expressed during inference. Diverse Decoding, and PPLM show mediocre performance in these tasks because no knowledge-aware object was considered, but relatively well on longer generation tasks (e.g., ELIS), which seems demonstrate the advantage of inference-ting time optimization methods (as KID). Little KID is not able to dynamically query knowledge when new context is generated, and thus performs not good as KID, especially in longer generation (e.g., ELI5)..
A.3.2 COMPARISON OF TIME AND SPACE CONSUMPTION
The time and space cost of running KID could be a matter of concern especially when we consider real-world applications. Table A3, we compare the time complexity KID with other baselines, including vanilla beam search and sampling decoding. We provide breakdown for different stages: retrieving knowledge (retrieving), training the generative model (training). and the actual generation (inference). Beam search and sampling decoding only have noticeable cost when there is sorting process filterng the candidate tokens (common in beam search and top-p or top-k sampling), which will often bring ( log k) cost (O(1) when pure random sampling without any sorting). Diverse
21

---

Published as a conference paper at ICLR 2022
Table A2: Compare KID with additional baselines. Note that only FiD (Izacard & Grave, b;a) is explicitly optimized for knowledge awareness. Diverse Decoding (Baheti et al., 2018) aims to improve generation diversity by constraining decoding distribution with topic and semantic similarity, PPLM (Dathathri et al., 2020) uses static set of tokens as global constraints to do conditional generation. KID differs from all these methods as its knowledge awareness and dynamic nature.
Little KID is naive implementation of KID that uses plain list instead of trie to store knowledge.
ELI5
MSMARCO
PIQA
PubMedQA
Method
B-1
R-L
B-1
R-L
B-1
R-L
B-1
R-L
FiD-T5 (base)
15.3
11.3
45.3
47.3
12.9
11.9
4.0
4.4
FiD-T5 (large)
15.7
18.1
51.9
53.1
14.8
17.4
4.0
4.5
Diverse Decoding
4.6
15.7
27.5
22.9
11.3
15.5
3.5
3.8
PPI.M
11.9
15.0
23.4
26.6
10.3
11.9
3.8
4.2
Little KID (ref))
12.1
14.3
40.2
46.3
12.2
13.5
5.2
6.6
Ours: KIDGPT2
27.9
26.6
47.4
53.9
17.7
18.4
9.7
11.5
Decoding, PPLM, and KID are similar as all are inference-time optimization (not necessarily need model fine-tuning, denoted "LM FT -"), but KID requires additional off-line knowledge trie construction (nearly the same time cost as RAG and FiD). During inference, KID takes constant time for query (as discussed in $A.2.2), and several iterations policy gradient (by constant number of steps) to guide the generation. We also analyze the space cost of KID and other baselines (Table A4). The difference between KID and RAG FiD is the external and local knowledge memory, which is relatively small (normally < 10Mb) from our observation. Other decoding methods do not have the ability infuse knowledge, and thus do not bring additional space cost.
Table A3: Time complexity analysis of KID and other baselines. We study the time consumption during knowledge retrieving, model training, and inference (i.c., generation). KID has comparable time efficiency as RAG and FiD, but outperforms them in knowledge infusing (discussed in the $4.2). Note that KID can function standalone module operating only at inference time, rather than rely on a specific model fine-tuning or pre- training together (such as RAG and FiD).
Method / Time
Retrieving
Training
Inference
Beam Search
O k), # of beams
Sampling
0(1) or O(k log k), k= top--
RAG
O #ddoos) #does 10
LM FT
2 0(1)
FiD-T5 (base)
0 (dos),, #docs = 100
LM FT
2 0(1)
FiD-T5 (large)
0 0 docs), #oocs z 100
LM FT
2 0(1)
Diverse Decoding
LM FT /
Two Neural Models Prediction + O(1)
PPLM
LM FT
O O##oonroo words * step)
Ours: KID
#doocs) + DFS, #does = 5 LM FT/-
( = O(1)
A.4 SAMPLE GENERATIONS
We list several generation samples of KID and RAG. In general, the generated texts of both methods are highly readable; however, we find RAG's generation tends to off-topic and not coherent with current on-going concepts flow when the generation long (e.g., EL.5, "smoking inside the vehicle"), or the context is multi-fold and complicated (e.g.. multi-turn dialogue WoW, "do some exercise") these are legit continuation we only consider immediate context. KID is superior especially in longer generation, which seems to echo findings that KID can mitigate exposure bias.
We show some generation samples of KID and RAG in Table A5.
A.5 ACTUAL QUESTIONS WE ASK HUMAN ANNOTATORS
We asked participants about: 1) Relevance (i.e., "What's 's the relevance level between the generated text and the given context?" Answer is from 1-totally not relevant to 7-totally relevant), 2) Factuality
22

---

Published as a conference paper at ICLR 2022
Table A4: Memory footprint of KID and other baselines. Similar to RAG and FilD, KID requires pre-store the grounding documents on disk (and corresponding dense vectors). In addition, KID builds knowledge trie off-line (less than 10Mb for all the tasks we studied), and a local knowledge memory (a limited length FIFO list), to enable knowledge infusion during generation. Knowledge?
Method Space
External Resources Model
Beam Search
Sampling
RAG
13.4G Docs + 64.6G Index / BART
FiD-T5 (base)
13.4G Does 64.6G Index / T5
FiD-TS (large)
13.4G Docs + 64.6G Index / T5
Diverse Decoding
Two Models 2.8G in Total LM BoW or Classifier LM
PPLM
Ours: KID
13.4G Does + 64.6G Index + Gine + Gen / LLM
Table A5: Sample generation of KID and RAG on ELI5 (QA), ROC (Logic-centric Writing), and WoW (Dialogue) with fine-tuned language models.
ELI5
Question
Does marijuana impair driving ability?
Marijuana does impair your ability to drive, but you avoid this by not being asleep together in class together and using napkin for balance.. One way avoid discomfort would be smoking inside the vehicle. That being said, smoking in public can do the same thing smoking outside.
RAG
Marijuan does impair your ability to drive because cannabis is a depressant drug, which means it slows reaction times and information processing.
KID
A person who has been using cannabis is more likely to cause a vehicle collision, but may still think that they can drive safely.
ROC
Keith lived in an area that had occasional carthquakes. But he thought keeping an first aid kit was unnccessary.
Context
The next day, he cut his hand while cutting food.
And he had to drive himself to hospital.
RAG
Keith quickly became worried and sad.
KID
Keith eventually learned first Aid at clinic instead of building his own kit.
WoW
f: would you like some tea or coffee ?
m no thank you it 's very late now, that will keep me awake the whole night.
f: then what about some water ?
Utterance
yes, please.
f: do n't work too late since you were not in good health. you should be careful with your health.
RAG
m: thank you. i would sleep earlier and do some exercise in the morning
KID
m thank you for your concern. could have some water please.
(i.e., "What's 's the factuality level of the generated text?" Answer is from 1-totally not factual to 7-very well factual), and 3) Grammaticality (i.c., "How much the text similar to seman-generated text?" Answer is from 1-not similar at all to 7-very much similar).
23


Reviews:
[{"review_id": "rBJgbzDTL4W", "reviewer": "Reviewer_RDPE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method can be considered as a new approach for constrained decoding, but rather than doing it in a hard way, the proposed method gives a soft constraint by rewarding a set of target words.\n- The experiments were done in multiple datasets and the proposed method consistently shows a better result than its baselines. It is especially encouraging because the proposed method is a plug-and-play module that can be applied to any language model in practice.\n\nWeaknesses:\n- The paper is overall difficult to understand, and I had to re-read it multiple times to make sense of it. This might have caused several misunderstandings.\n- The paper does not compare against vanilla (non-graph) constrained decoding which limits the output space to the words in the retrieved corpus. This will be an important baseline that can show the advantage of having such graph for constrained decoding.\n\nQuestions:\n- I am confused with the motivation behind formulating the loss with RL. The paper discusses the disadvantages of vanilla decoding, e.g. exposure bias of teacher forcing. However, the proposed loss function with reward at every time step is also biased. The proposed loss seems to me more like an MLE with some regularization/modification.", "summary_of_the_paper": "The paper is interested in improving the performance of natural language generation tasks that require external knowledge (which are often called knowledge-intensive tasks). In particular, the paper focuses on how it can leverage existing language model (e.g. GPT-2, BART) without changing its architecture and injecting external knowledge into it. To do so, the paper proposes Knowledge Infused Decoding (KID), which alters the decoding probability at each time step (in vocab space) so that it favors decoding certain words, which are obtained by retrieving a relevant document from a corpus. In order to trains such decoder, the authors also add a KL divergence to the loss that prevents the decoding distribution not to be too different from the original Language Model's decoding distribution. When KID is used in conjunction with GPT2-M or BART-L, they show clear improvements, and in some datasets, they achieve higher accuracy than the state of the art. The paper also shows that KID outperforms Retrieval Augmented Generation (Lewis et al., 2020) which uses the retrieved documents directly into the input (whereas KID uses it indirectly to constrain the decoding in a soft way).", "summary_of_the_review": "I think the paper's result is great and I agree with the overall motivation behind the paper. However, many parts of the paper are unclear that I will need further clarification to correctly assess this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nWQw0MUQ5ZU", "reviewer": "Reviewer_wRJi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- This method does not require re-training or fine-tuning LMs with knowledge based objectives. This is a plug and play decoding strategy that competes with other sampling based or beam based decoding strategies with weighted decoding methods. \n- This method can 'dynamically' infuse external knowledge into each step of LM decoding. This is an advantage over static knowledge retrieval based methods that retrieve documents only once and suffers from problems with long range generation. \n- The probability mass of tokens at each step of token generation can be informed directly by the concepts and relations extracted via the entity-relation extraction methods; this token level reinforcement helps with overcoming sparse rewards generally popular with sequence level reward induction (via evaluation metrics such as BLEU) \n- Results show that larger LMs benefits Knowledge infused decoding(KID) method and outperforms Beam search and sampling decoding. In addition, KID brings more gain for non-finetuned LMs than fine-tuned ones, potentially because the fine-tuning LM is already fit onto the new domain thus less knowledge infusion is needed.\n\n\nWeaknesses:\n- The paper lacks details of the text decoding inference time and comparison with other decoding methods (sampling) including knowledge based decoding methods. Figure 2(a) and 2(b) do show time latency; however there is no comparison or discussion of this in the paper. \n- The paper presents a text decoding method for knowledge infusion; however, it does not compare this knowledge infused decoding with other non-RL based decoding techniques such as penalized sampling of words (Keskar et al. 2019), decoding with distributional constraints (https://arxiv.org/pdf/1809.01215.pdf) or other plug and play decoding methods such as PPML (https://arxiv.org/abs/1912.02164). \n- The ablation studies described in the paper do not make explicit the contribution of the RL based decoding technique. \n- The trie data structure is created using open IE tools and reference previous literature. However, some more analysis about the compression of memory would be interesting to understand (with availability of longer sequence handling transformers, how important is the role of compression and trie based external memory?)\n- It is not clear how dynamic is the local memory; Figure 1 is not well-described in the paper. At time-step t or t+1, it seems that the local memory is not going to change (extracted from context); what are the dynamics of local memory in this model?\n\n", "summary_of_the_paper": "This paper presents a text decoding strategy for knowledge-intensive text generation tasks with the goal to infuse external knowledge based information at each time-step of decoding process via efficient document retrieval, creation of local and fast global knowledge access data structure and the interaction guided decoding method. The decoding method uses RL based policy gradient for token selection. The method is evaluated using diverse types of knowledge-intensive tasks such as abstractive question answering, logic centric writing and dialog generation. This method works with existing decoder only or encoder-decoder LMs as a plug and play method and outperforms existing knowledge aware task optimized models and correlates well with human evaluation. ", "summary_of_the_review": "This paper presents a novel plug and play text decoding method that allows improved text generation for knowledge intensive tasks via reinforcement learning based technique. The method allows several new future investigations in this direction and is very interesting (addressing exposure bias problems with LM tuning). There's work on knowledge retrieval, data structures for knowledge encoding and text decoding with state of the art integrated methods. The results are also superior compared to many existing decoding and knowledge aware decoding methods.    ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_7bGnpLGYlE", "reviewer": "Reviewer_zdRT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Proposed method outperforms baselines on various knowledge-intensive tasks.\n- Proposed objective can be a good loss term for explicitly modeling knowledge in LM.\n- Considering the performance gap between KID and RAG (or BART+DPR), dynamic knowledge infusing will be a good alternative to static knowledge grounding.\n\n## Weaknesses\n\n- Knowledge-graph construction and graph traversal are not well explained. Can you give an example knowledge trie and explain to me how does the graph traversal work? How would the queried results look like?\n- Optimization looks somewhat tricky. Which learning rate is used for Adam? How did you select K = 3 steps for policy update? What would happen if we set K < 3 or K > 3?\n- Can you show me example sentences generated by KID and compare them with RAG-generated sentences?\n- In Table 5, can you also measure naturalness or grammaticality? It is obvious that KID will improve knowledge grounding, but not sure how much it will break the syntax of generated sentences.\n- Can you also add FiD-BART as one of the main baseline methods?", "summary_of_the_paper": "This paper deals with the well-known but important problem of language models -- enhancing language model with (external) knowledge. To that end, the authors present Knowledge Infused Decoding (KID), a novel decoding algorithm for generative LMs, which infuses external knowledge into each step of the LM decoding. This method maintains a local knowledge memory based on the current context and continuously updates the local memory via reinforcement learning to guide each step of language generation. As a result, it can integrate knowledge on the fly and perform well on various knowledge-intensive tasks such as abstractive QA, logic-centric writing, and dialogue generation.", "summary_of_the_review": "The proposed method is fairly sound and shows good empirical results. However, some details are missing (e.g., knowledge graph construction and traversal), and hyperparameter selection is not well explained (e.g., learning rate and the number of optimization steps). It is also unclear how much KID will harm the original LM in terms of naturalness and grammaticality. Additional human evaluation or generation examples can remedy these issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4dSd6VPNkC4", "reviewer": "Reviewer_Kmpi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  \nThis approach is interesting and very relevant to the growing work on retrieval- and memory-augmented models for grounded generation.  \nSince it is a new decoding approach, it is model and task agnostic, and can be used as a plug-in replacement for existing decoding approaches without changing the architecture and for grounding each generated token in external knowledge. The knowledge graph could be further used to study provenance of generated tokens.  \nExperiments are conducted on a variety of knowledge-intensive tasks/datasets and KID clearly does outperform vanilla beam search and sampling which are not grounded.\n\nWeaknesses:  \nThere is a lack of clarity and details which makes it hard to follow some explanations and results; choice of baselines seems a bit unclear -- digging into these points through the following questions to allow for discussion.\n1. Since probability mass is shifted in favor of retrieved demonstrations at every time step, how is the model generating words that don’t appear in those demonstrations? Eg: stop words for fluency. What is happening in those cases? \n2. During constrained decoding, are the concepts always the subjects (keys) in the triples? If so, then is the local trie reconstructing portions of the external trie? If not, how is the local trie used to traverse the external trie?\n3. What is the “in-house split dev set”? No official test sets are used, right? The number for RAG on MS MARCO in Table 2 (40.8 R-L) appears to match the one from the original paper for test scores (https://arxiv.org/abs/2005.11401 their table 2). But the dev set scores in their table 6 seem to be higher (57.2 R-L). It’s a bit confusing. Could you please clarify which subsets were used for each of the evaluations and what is the fair comparison? \n4. Are the beam and sampling baselines in Table 1 conditioning on external knowledge? If not, then aren’t the baselines far too weak for a fair comparison and should include some external knowledge? The published SotA numbers appear to paint a very different picture and that seems to call the claims into question. On a related note, why not also compare against other constrained decoding baselines eg: lexically constrained decoding (https://arxiv.org/abs/2010.12723)?\n5. Could you please elaborate on the runtime details? Providing a breakdown for the different steps might even be useful. How does this compare to vanilla decoding methods? How does it compare to RAG-token/kNN-LM/other related approaches?\n6. Fusion-in-Decoder (https://arxiv.org/abs/2007.01282v2) seems like a very important baseline for knowledge infusion. Was there a reason to not include it?\n\n\nSome additional comments:  \n1. RAG-token (https://arxiv.org/abs/2005.11401) supports token-level retrieval, so does kNN-LM (https://arxiv.org/abs/1911.00172). Discussion in the intro and some other places appears to misrepresent the former and leave out the latter. It might also be worth highlighting that RAG-Sequence is used for all the comparisons.\n2. Would it be possible to add a table somewhere (perhaps the appendix) that provides the dataset sizes and other relevant stats?\n3. Would it be possible to add some qualitative examples? It would be useful to see, for instance, what a single timestep’s retrieved demonstrations, local trie etc. look like.\n\n=============================\n\nUpdate: Thanks to the authors for their detailed response. While the responses helped to clarify a lot of details, there remain some issues particularly regarding the clarity of the paper itself. That being said, the community stands to benefit from ideas presented. Recommending acceptance.\n\nSome suggestions for revisions:  \n- Not convinced that claiming the win over beam search/sampling is a notable contribution since a large body of work has already shown that injecting knowledge is useful. Would suggest the authors consider reframing this -- it’s fine to include the experimental results, but the argument and claims need to at least be thought out.  \n- Probably update the text to reflect what changes have been made to MS Marco results so it’s clear what the evaluation procedure is. \n- Adding all the response material to the appendix doesn’t help the clarity of the paper itself, might help to coherently revise sections.    \n- Need to clarify the discussion on runtime. It’s a bit tedious to parse.  \n", "summary_of_the_paper": "This paper presents Knowledge Infused Decoding (KID), an RL-based approach for grounded decoding by conditioning on external knowledge. For every new example:\n- retrieve the relevant passages from Wikipedia (using dense passage retrieval DPR)\n- construct a knowledge trie of triples extracted from the passages and initialize a local trie that tracks all the nouns/verbs mentioned\n- Generate tokens conditioned on both prior context and by retrieving triples from the knowledge trie, used as demonstrations for policy gradient\nThis approach is model-agnostic and can be used to fine-tune both encoder-decoder and decoder-only models.\n\nExperiments are divided into two parts:  \n- evaluating the decoding method, on six NLG datasets -- outperforming beam search and (top-k/nucleus) sampling\n- evaluating the knowledge infusion, on four abstractive QA datasets -- outperforming a number of approaches including retrieval-augmented, graph neural network, and knowledge graph methods.\n\nAblation studies are presented for swapping out the retriever, different hyperparameters for KID and the size of the underlying LM.\n", "summary_of_the_review": "Updated to weak accept after author response.\n\nRecommending a weak reject (5) pending the discussion on concerns raised in the main review (questions 3,4,5,6) regarding baselines and results. These seem important enough to clarify before updating the recommendation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel decoding algorithm that allows to dynamically integrate external knowledge with generative 
LMs. The proposed technique is plug-and-play, it does not require re-training or fine-tuning LMs with knowledge based objectives. 
The author report a series of experiments on several datasets and tasks showing improvements over competitive baselines and in some 
cases above SOTA. The authors have addressed the reviewers' queries and added experimental results (additional baselines) as
well as clarifications of their approach (e.g., Figure A1 in the Appendix). I think the topic  (e.g., constrained LM decoding) is of general interest to the ICLR community and the approach compelling.
Decision: Accept (Poster)