SlowGuess/NAACL_2021_OCR huggingface dataset:
Adaptable and Interpretable Neural Memory Over Symbolic Knowledge
Pat Verga*, Haitian Sun*, Livio Baldini Soares, William W. Cohen
{patverga, haitiansun, liviobs, wcohen}  $@$  google.com
Past research has demonstrated that large neural language models (LMs) encode surprising amounts of factual information: however, augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. To address this problem, we develop a neural LM that includes an interpretable neuro-symbolic KB in the form of a "fact memory". Each element of the fact memory is formed from a triple of vectors, where each vector corresponds to a KB entity or relation. Our LM improves performance on knowledge-intensive question-answering tasks, sometimes dramatically, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art open-book model, despite using  $5\%$  of the parameters. Most interestingly, we demonstrate that the model can be modified, without any re-training, by updating the fact memory.
Neural language models (LMs) (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2019) that have been pre-trained by self-supervision on large corpora contain rich knowledge about the syntax and semantics of natural language (Tenney et al., 2019), and are the basis of much recent work in NLP. Pre-trained LMs also contain large amounts of factual knowledge about the world (Petroni et al., 2019; Roberts et al., 2020; Brown et al., 2020). However, while large LMs can be coerced to answer factual queries, they still lack many of the properties that knowledge bases (KBs) typically have. In particular, it is difficult to distinguish answers produced by memorizing factual statements in the pre-training corpus from lower-precision answers produced by linguistic generalization (Poerner et al., 2019). It is also difficult to add or remove factual information without retraining the LM, an expensive pro
cess<sup>1</sup>. The difficulty of updating knowledge in neural LMs contrasts with symbolic KBs, where it is very easy to add or modify triples, and is a major disadvantage of using a LM "as a KB"—as in many domains (news, product reviews, scientific publications, etc) the set of known facts changes frequently. Symbolic KBs thus remain practically important (Google, 2012; Dong, 2017), especially for NLP applications where text is hard to automatically process (e.g., scientific, technical, or legal) or tasks rich in information that exists only in structured form (e.g., technical specifications of a new product, where no product page or review text discussing it yet exists).
Motivated by this, past work has sought to combine the benefits of neural LMs with the large, broad-coverage KBs that now exist (Bollacker et al., 2008; Auer et al., 2007; Vrandecic and Krötzsch, 2014). This paper continues this research program with a new knowledge-augmented LM called Fact Injected Language Model (FILM). FILM is a masked LM, where masks can be filled either from the token vocabulary or an entity vocabulary. The vector representation of each entity in a KB is jointly learned alongside other parameters of a Transformer LM, and stored in a separate entity memory. FILM also includes a fact memory where each element is derived from a triple of vectors, representing a KB entity or relation. Since these triples are defined compositionally from (representations of) entities and relations, they have an interpretable symbolic meaning: e.g., if  $\mathbf{e}_{\mathrm{mtv}}$  is the vector representation of KB entity "Mountain View, CA" and  $\mathbf{e}_{\mathrm{google}}$  and  $\mathbf{r}_{\mathrm{hq}}$  similarly correspond to "Google Inc" and the relation "headquartered in", these vectors can be used to construct a memory element  $f(\mathbf{e}_{\mathrm{google}}, \mathbf{r}_{\mathrm{hq}}, \mathbf{e}_{\mathrm{mtv}})$  for the KB assertion "Google, Inc is headquartered in Mountain
<sup>1</sup>Models large enough to achieve good factual coverage require extreme amounts of compute, and the largest neural LMs now cost millions of dollars to train (Brown et al., 2020).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3678-3691 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Fact Injected Language Model architecture. The model takes a piece of text (a question during fine-tuning or arbitrary text during pre-training) and first contextually encodes it with an entity enriched transformer. FILM uses the contextually encoded MASK token as a query to the fact memory. In this case, the contextual query chooses the fact key (Charles Darwin, born_in) which returns the set of values {United Kingdom} (The value set can be multiple entity objects such as the case from calling the key [United Kingdom, has_city]). The returned object representation is incorporated back into the context in order to make the final prediction. Note that the entity representations in the facts (both in keys and values) are shared with the entity memory. The portion within the dashed line follows the procedure from Févry et al. (2020).
View, CA". This means that the fact memory can be easily extended with new facts.
In analysis on four benchmark question answering datasets we show that FILM improves significantly, and sometimes dramatically, over several strong baselines (e.g. BART (Lewis et al., 2019) and T5 (Raffel et al., 2019)) and this improvement is even larger when removing train-test overlap. In one setting of WebQuestionsSP, we outperform the next best performing model (RAG (Lewis et al., 2020a)) by 27 points despite using only  $5\%$  of the number of parameters.
Most interestingly, we demonstrate that FILM models can be updated without any re-training, by modifying the fact memory. Specifically, in §4.1, we show we can inject new fact memories at inference time, enabling FILM to correctly answer questions about pairs of entities that were never observed in the training (either during pre-training or fine-tuning). In §4.2 we also evaluate updating the model by inserting contra-positive facts that contradict facts mentioned in the pretraining data, and we show that FILM can correctly answer novel questions in this scenario as well. To summarize, this paper's contributions are:
1. We propose a neural LM for knowledge-intensive question-answering tasks that incor
porates a symbolic fact memory.
2. We outperform most baselines on several benchmark open-domain QA datasets, and dramatically if test-train overlap in the datasets are removed.
3. We show FILM can easily adapt to newly injected and modified facts without retraining.
2 Fact Injected Language Model Model
The Fact Injected Language Model (FILM) model (see Figure 1) extends the Transformer (Vaswani et al., 2017) architecture of BERT (Devlin et al., 2019) with additional entity and facts memories. These memories store semantic information which can later be retrieved and incorporated into the representations of the transformer. Similar to the approach in Févry et al. (2020), entity embeddings will (ideally) store information about the textual contexts in which that entity appears, and by inference, the entity's semantic properties. The fact memory encodes triples from a symbolic KB, constructed compositionally from the learned embeddings of the entities that comprise it and implemented as a key-value memory which is used to retrieve entities given their KB properties. This combination results in a neural LM which learns to
access information from a symbolic KB.
We represent a Knowledge Base  $\mathcal{K}$  as a set of triples  $(s,r,o)$  where  $s,o\in \mathcal{E}$  are the subject and object entities and  $r\in \mathcal{R}$  is the relation, where  $\mathcal{E}$  and  $\mathcal{R}$  are pre-defined vocabularies of entities and relations. A text corpus  $\mathcal{C}$  is a collection of paragraphs2  $\{p_1,\ldots ,p_{|C|}\}$ . Let  $\mathcal{M}$  be the set of entity mentions in the corpus  $\mathcal{C}$ . A mention  $m_i$  is encoded as  $(e_m,s_m^p,t_m^p)$ , indicating entity  $e_m$  is mentioned in paragraph  $p$  starting at token position  $s_m^p$  and ending at  $t_m^p$ . We will usually drop the superscript  $p$  and use  $s_m$  and  $t_m$  for brevity.
2.2 Input
The input to our model is a piece of text; either a question during fine tuning (see §A.2.2) or a paragraph in pre-training (see §A.2.1). Pretraining is formulated as a cloze-type Question Answering (QA) task: given a paragraph  $p = \{w_1, \dots, w_{|p|}\}$  with mentions  $\{m_1, \dots, m_n\}$ , we sample a single mention  $m_i$  to act as the cloze answer and replace all tokens of  $m_i$  with [MASK] tokens. The entity in  $\mathcal{E}$  named by the masked entity is the answer to the cloze question  $q$  ('United Kingdon' in the example input of Figure 1). Mentions in the paragraph other than  $m$  are referred to below as context mentions. In the following sections we describe how our model learns to jointly link context entities (\$2.3) and predict answer entities (\$2.5).
2.3 Entity Memory
Our entity memory  $\mathbf{E} \in \mathbb{R}^{|\mathcal{E}| \times d_e}$  is a matrix containing a vector for each entity in  $\mathcal{E}$  and trained as an entity-masked LM. The model input is a text span containing unlinked entity mentions with known boundaries<sup>3</sup>. Mentions are masked with some probability. Our entity memory follows Entity as Experts (EaE) (Févry et al., 2020) which interleaves standard Transformer (Vaswani et al., 2017) layers with layers that accesses the entity memory<sup>4</sup>.
Given a piece of text  $q = \{w_{1},\dots ,w_{|q|}\}$  the contextual embedding  $\mathbf{h}_i^{(l)}$  is the output at the  $i$ 'th
token of the  $l$ th intermediate transformer layer. These contextual embeddings are used to compute query vectors that interface with the entity memory.
For each context mention  $m_{i} = (e_{m_{i}}, s_{m_{i}}, t_{m_{i}})$  in q, we form a query vector to access the Entity memory by concatenating the context embeddings for the mention  $m_{i}$ 's start and end tokens,  $\mathbf{h}_{s_{m_i}}^{(l)}$  and  $\mathbf{h}_{t_{m_i}}^{(l)}$  and projecting them into the entity embedding space. We use this query to compute attention weights over the full entity vocabulary and produce an attention-weighted sum of entity embeddings  $\mathbf{u}_{m_i}^l$ . The result is then projected back to the dimension of the  $j$ -indexed contextual token embeddings, and added to what would have been the input to the next layer of the Transformer:
$$
\mathbf {h} _ {m _ {i}} ^ {(l)} = \mathbf {W} _ {e} ^ {T} \left[ \mathbf {h} _ {s _ {m _ {i}}} ^ {(l)}; \mathbf {h} _ {t _ {m _ {i}}} ^ {(l)} \right] \tag {1}
$$
$$
\mathbf {u} _ {m _ {i}} ^ {(l)} = \operatorname {s o f t m a x} \left(\mathbf {h} _ {m _ {i}} ^ {(l)}, \mathbf {E}\right) \times \mathbf {E} \tag {2}
$$
$$
\tilde {\mathbf {h}} _ {j} ^ {(l + 1)} = \mathbf {h} _ {j} ^ {(l)} + \mathbf {W} _ {2} ^ {T} \mathbf {u} _ {m _ {i}} ^ {(l)}, s _ {m _ {i}} <   j <   t _ {m _ {i}} \tag {3}
$$
After the final transformer layer  $T$ ,  $\mathbf{h}_{m_i}^{(T)}$  is used to predict the context entities  $\hat{e_{m_i}}$  and produce a loss with  $\mathbb{I}_{e_{m_i}}$ , the one-hot label of entity  $e_{m_i}$ . Following Févry et al. (2020), we supervise the entity access for the intermediate query vector in Eq. 1.
$$
\hat {e} _ {m _ {i}} = \operatorname {a r g m a x} _ {e _ {i} \in \mathcal {E}} \left(\mathbf {c} _ {m _ {i}} ^ {T} \mathbf {e} _ {i}\right)
$$
$$
\operatorname {l o s s} _ {\mathrm {c t x}} = \operatorname {c r o s s \_ e n t r o p y} (\operatorname {s o f t m a x} \left(\mathbf {c} _ {m _ {i}}, \mathbf {E}\right), \mathbb {I} _ {e _ {m _ {i}}})
$$
$$
\operatorname {l o s s} _ {\text {e n t}} = \operatorname {c r o s s} _ {\text {e n t r o p y}} (\operatorname {s o f t m a x} \left(\mathbf {h} _ {m _ {i}} ^ {(l)}, \mathbf {E}\right), \mathbb {I} _ {e _ {m _ {i}}})
$$
2.4 Fact Memory
FILM contains a second fact memory, populated by triples from the knowledge base  $\mathcal{K}$ , as shown on the right side of Figure  $1^5$ . The fact memory shares its on entity representations with the entity memory embeddings in  $\mathbf{E}$ , but each element of the fact memory corresponds to a symbolic substructure, namely a key-value pair  $((s,r),\{o_1,\ldots ,o_n\})$ . The key  $(s,r)$  is a (subject entity, relation) pair, and the corresponding value  $\{o_1,\dots,o_n\}$  is the list of object entities associated with  $s$  and  $r$ , i.e.  $(s,r,o_i)\in \mathcal{K}$  for  $i = \{1,\dots,n\}$ . Conceptually, KB triples with the same subject entity and relation are grouped into a single element. We call the subject and relation pair  $a_{j} = (s,r)\in A$  a head pair and the list of objects  $b_{j} = \{o_{1},\dots,o_{n}\} \in B$  a tail set<sup>6</sup>.
2Although we use the term paragraph here, in our experiments we use spans of 128 tokens, which need not follow paragraph boundaries.
3Févry et al. (2020) also showed the model is capable of learning to predict these boundaries. For simplicity, in this work we assume they are given.
4We follow the implementation of Fevry et al. (2020) and have a single entity memory access between the fourth and fifth transformer layers.
In our experiments we use a single fact memory access after the final (12th) transformer layer.
<sup>6</sup>The size of the tail set  $b_{j}$  can be large for a popular head pair  $(s, r)$ . In such cases, we randomly select a few tails and drop the rest of them. The maximum size of the tail set is 32 in the experiments in this paper.
In more detail, we encode a head pair  $a_{j} = (s,r)\in A$  by concatenating embeddings for the subject entity and relation, and then projecting them linearly to a new head-pair embedding space. More precisely, let  $\mathbf{E}\in \mathbb{R}^{|\mathcal{E}|\times d_e}$  be the entity embeddings trained in §2.3, and  $\mathbf{R}\in \mathbb{R}^{|\mathcal{R}|\times d_r}$  be embeddings of relations  $\mathcal{R}$  in the knowledge base  $\kappa$ . We encode a head pair  $a$  as:
$$
\mathbf {a} _ {j} = \mathbf {W} _ {a} ^ {T} \left[ \mathbf {s}; \mathbf {r} \right] \in \mathbb {R} ^ {d _ {a}}
$$
where  $\mathbf{s} \in \mathbf{E}$  and  $\mathbf{r} \in \mathbf{R}$  are the embeddings of subject  $s$  and relation  $r$ , and  $\mathbf{W}_a$  is a learned linear transformation matrix. We let  $\mathbf{A} \in \mathbb{R}^{|A| \times d_a}$  denote the embedding matrix of all head pairs.
Let the answer for  $q$  be denoted  $e_{\mathrm{ans}}$ , and its masked mention  $m_{\mathrm{ans}} = (e_{\mathrm{ans}}, s_{\mathrm{ans}}, t_{\mathrm{ans}})$ . For a masked mention  $m_{\mathrm{ans}}$ , define a query vector to access the fact memory as:
$$
\mathbf {v} _ {m _ {\text {a n s}}} = \mathbf {W} _ {f} ^ {T} \left[ \mathbf {h} _ {s _ {\text {a n s}}} ^ {(T)}; \mathbf {h} _ {t _ {\text {a n s}}} ^ {(T)} \right] \tag {4}
$$
where  $\mathbf{h}_{s_{\mathrm{ans}}}^{(T)}$  and  $\mathbf{h}_{t_{\mathrm{ans}}}^{(T)}$  are the contextual embeddings for the start and end tokens of the mention  $m_{\mathrm{ans}}$  and  $\mathbf{W}_f$  is the linear transformation matrix into the embedding space of head pairs  $\mathbf{A}$ .
Head pairs in  $A$  are scored by the query vector  $\mathbf{v}_{m_{\mathrm{ans}}}$  and the top  $k$  head pairs with the largest inner product are retrieved. This retrieval process on the fact memory is distantly supervised. We define a head pair to be a distantly supervised positive example  $a_{\mathrm{ds}} = (s, r)$  for a passage if its subject entity  $s$  is named by a context mention  $m_i$  and the masked entity  $e_{\mathrm{ans}}$  is an element of the corresponding tail set, i.e.  $e_{\mathrm{ans}} \in b_{\mathrm{ds}}$ . When no distantly supervised positive example exists for a passage, it is trained to retrieve a special "null" fact comprised of the  $s_{\mathrm{null}}$  head entity and  $r_{\mathrm{null}}$  relation: i.e.  $a_{\mathrm{ds}} = (s_{\mathrm{null}}, r_{\mathrm{null}})$  and its tail set is empty. This distant supervision is encoded by a loss function:
$$
\begin{array}{c} \mathrm {T O P} _ {k} (\mathbf {v} _ {m _ {\mathrm {a n s}}}, \mathbf {A}) = \operatorname {a r g m a x} _ {k, j \in \{1, \dots , | A | \}} \mathbf {a} _ {j} ^ {T} \mathbf {v} _ {m _ {\mathrm {a n s}}} \\ \mathrm {l o s s} _ {\mathrm {f a c t}} = \mathrm {c r o s s} _ {-} \text {e n t r o p y} (\mathrm {s o f t m a x} (\mathbf {v} _ {m _ {\mathrm {a n s}}}, \mathbf {A}), \mathbb {I} _ {a _ {\mathrm {d s}}}) \end{array}
$$
The result of this query is that the tail sets associated with the top  $k$  scored head pairs, i.e.  $\{b_j|j\in \mathrm{TOP}_k(\mathbf{v},\mathbf{A})\}$ , are retrieved from the fact memory.
2.5 Integrating Knowledge and Context
Next, tail sets retrieved from the fact memory are aggregated. Recall that a tail set  $b_{j}$  returned from the fact memory is the set of entities  $\{o_1,\dots,o_n\}$
s.t.  $(s,r,o_i)\in \mathcal{K}$  for  $i\in \{1,\ldots ,n\}$  with the associated  $a_{j} = (s,r)$ . Let  $\mathbf{o}_i\in \mathbf{E}$  be the embedding of entity  $o_i$ . We encode the returned tail set  $\mathbf{b}_j$  as a weighted centroid of the embeddings of entities in the tail set  $b_{j}$ .
$$
\mathbf {b} _ {j} = \sum_ {o _ {i} \in b _ {j}} \alpha_ {i} \mathbf {0} _ {i} \in \mathbb {R} ^ {d _ {e}}
$$
where  $\alpha_{i}$  is a context-dependent weight of the object entity  $o_i$ . To compute the weights  $\alpha_{i}$ , we use a process similar to Eq. 4: we compute a second query vector  $\mathbf{z}_{m_{\mathrm{ans}}}$  to score the entities inside the tail set  $b_{j}$ , and the weights  $\alpha_{i}$  are the softmax of the inner products between the query vector  $\mathbf{z}_{m_{\mathrm{ans}}}$  and the embeddings of entities in the tail set  $b_{j}$ .
$$
\mathbf {z} _ {m _ {\mathrm {a n s}}} = \mathbf {W} _ {b} ^ {T} \left[ \mathbf {h} _ {s _ {\mathrm {a n s}}} ^ {(T)}; \mathbf {h} _ {t _ {\mathrm {a n s}}} ^ {(T)} \right] \tag {5}
$$
$$
\alpha_ {i} = \frac {\exp \left(\mathbf {o} _ {i} ^ {T} \mathbf {z} _ {m _ {\mathrm {a n s}}}\right)}{\sum_ {o _ {l} \in b _ {j}} \exp \left(\mathbf {o} _ {l} ^ {T} \mathbf {z} _ {m _ {\mathrm {a n s}}}\right)} \tag {6}
$$
where  $\mathbf{W}_b$  is a transformation matrix distinct from  $\mathbf{W}_e$  in Eq. 1 and  $\mathbf{W}_f$  in Eq. 4. The top  $k$  tail sets  $\mathbf{b}_j$  are further aggregated using weights  $\beta_j$ , which are the softmax of the retrieval (inner product) scores of the top  $k$  head pairs  $a_j$ . This leads to a single vector  $\mathbf{f}_{m_{\mathrm{ans}}}$  that we call the knowledge embedding for the masked mention  $m_{\mathrm{ans}}$ .
$$
\mathbf {f} _ {m _ {\mathrm {a n s}}} = \sum_ {j \in \operatorname {T O P} _ {k} (\mathbf {v} _ {m _ {\mathrm {a n s}}}, \mathbf {A})} \beta_ {j} \mathbf {b} _ {j} \tag {7}
$$
$$
\beta_ {j} = \frac {\exp \left(\mathbf {a} _ {j} ^ {T} \mathbf {v} _ {m _ {\text {a n s}}}\right)}{\sum_ {t \in \mathrm {T O P} _ {k} \left(\mathbf {v} _ {m _ {\text {a n s}}}, \mathbf {A}\right)} \exp \left(\mathbf {a} _ {t} ^ {T} \mathbf {v} _ {m _ {\text {a n s}}}\right)} \tag {8}
$$
Intuitively  $\mathbf{f}_{m_{\mathrm{ans}}}$  is the result of retrieving a set of entities from the fact memory. The last step is to integrate this retrieved set into the Transformer's contextual embeddings. Of course, KBs are often incomplete, and especially during pre-training, it might be necessary for the model to ignore the result of retrieval, if no suitable triple appears in the KB. To model this, the final step in the integration process is to construct an integrated query  $\mathbf{q}_{m_{\mathrm{ans}}}$  with a learnable mixing weight  $\lambda$ . Algorithmically,  $\lambda$  is computed as the probability of retrieving a special "null" head  $a_{\mathrm{null}}$  from the fact memory, i.e. whether an oracle head pair exists in the knowledge base.  $\mathbf{q}_{m_{\mathrm{ans}}}$  is used to predict the masked entity.
$$
\mathbf {q} _ {m _ {\mathrm {a n s}}} = \lambda \cdot \mathbf {c} _ {m _ {\mathrm {a n s}}} + (1 - \lambda) \cdot \mathbf {f} _ {m _ {\mathrm {a n s}}}, \lambda = P (a _ {\mathrm {n u l l}})
$$
$$
\hat {e} _ {\mathrm {a n s}} = \operatorname {a r g m a x} _ {e _ {i} \in \mathcal {E}} (\mathbf {q} _ {m _ {\mathrm {a n s}}} ^ {T} \mathbf {e} _ {i})
$$
lossans=cross_entropy(softmax(qmans,E),Ieans)
Model	P@1
K-Adapter †	29.1
BERT-Large †	33.9
BERT-KNN ‡	38.7
EaE	38.6
FILM	44.2
Table 1: LAMA TREx Precision@1. † copied from Wang et al. (2020a), ‡ copied from Kassner and Schütze (2020)
The final loss is the sum of the individual losses (See §A.2.1 and §A.2.2 for additional details.)
lossfinal  $=$  lossent  $^+$  lossctx  $^+$  lossfact  $^+$  lossans
The primary focus of this work is investigating the incorporating of new symbolic knowledge by injecting new facts without retraining (§4.1) and updating stale facts (§4.2). However, we first validate the efficacy of our model on standard splits of widely used knowledge-intensive benchmarks against many state-of-the-art systems (§3.3), as well as two subsets of these benchmarks restricted to examples answerable with wikidata (§3.4) and examples filtered for train/test overlap (§3.5).
We evaluate on four knowledge intensive tasks<sup>7</sup>.
WebQuestionsSP is an Open-domain Question Answering dataset containing 4737 natural language questions linked to corresponding Freebase entities and relations (Yih et al., 2015) derived from WebQuestions(Berant et al., 2013).
LAMA TREx is a set of fact-related cloze questions. Since we are interested in entity prediction models, we restrict our LAMA investigations to TREx, which has answers linked to Wikidata.
TriviaQA (open) contains questions scraped from quiz-league websites (Joshi et al., 2017). We use the open splits following Lee et al. (2019).
FreebaseQA is an Open-domain QA dataset derived from TriviaQA and other trivia resources (See Jiang et al. (2019) for full details). Every answer can be resolved to at least one Freebase entity and each question contains at least one entity.
T5 (Raffel et al., 2019) and BART (Lewis et al., 2019) are large text-to-text transformers.
Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) is a two stage retrieve and read model.
Retrieval Augmented Generation (RAG) (Lewis et al., 2020a) and Fusion in Decoder (FID) (Izacard and Grave, 2020) use DPR retrieval, followed
by generative decoders based on BART and T5 respectively. FID is the current state-of-the-art on the open domain setting of TriviaQA.
K-Adapter (Wang et al., 2020a) and Bert-KNN (Kassner and Schütze, 2020) are recent BERT extensions that perform at or near state-of-the-art on the LAMA benchmark.
Entities-as-Experts (EaE) (Févry et al., 2020) is discussed in §2.3. Our EaE models are trained using the same hyperparameters and optimization settings as FILM.
3.2.1 Open vs Closed Book models
Generally, open book models refer to 'retrieve and read' pipelines (Chen et al., 2017) which, given a query, 1) retrieve relevant passages from a corpus, 2) separately re-encode the passages conditioned on the question and then 3) produce an answer. Conversely, closed book models answer questions directly from their parameters without additional processing of source materials. We consider FILM and EaE closed-book models as they do not retrieve and re-encode any source text, and instead attend to parameterized query-independent memories.
3.3 Results in Convention Settings