MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (2024)

Abstract: Leveraging external knowledge is crucial for achieving high performance in knowledgeintensive tasks, such as question answering. The retrieve-and-read approach is widely adopted for integrating external knowledge into a language model. However, this approach suffers from increased computational cost and latency due to the long context length, which grows proportionally with the number of retrieved knowledge. Furthermore, existing retrieval-augmented models typically retrieve information from a single type of knowledge source, limiting their scalability to diverse knowledge sources with varying structures. In this work, we introduce an efficient memoryaugmented transformer, called MATTER, designed to retrieve relevant knowledge from multiple heterogeneous knowledge sources. Specifically, our model retrieves and reads from both unstructured sources (paragraphs) and semistructured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed. Furthermore, MATTER achieves competitive results compared to conventional retrieve-and-read models while having 100x throughput during inference.

heading: 1 Introduction
text: Retrieval-augmented models have enhanced performance in various natural language processing tasks, such as open-domain question answering (Izacard et al., 2020; Lewis et al., 2020b). These models leverage a two-step process: an initial retrieval phase to gather relevant information from a knowledge source (retrieve), followed by a reading or comprehension phase to generate responses based on the retrieved context and input (read).
∗This work was done while Dongkyu was a summer intern at Amazon AGI
However, the strong performance of retrievalaugmented QA models is offset by a substantial drawback of high inference latency (Wu et al., 2022; Chen et al., 2023). (Wu et al., 2022; Izacard et al., 2020) demonstrate that even with relatively small reader models, such as T5-base, retrieveand-read QA models struggle to process more than 10 questions per second. Recent studies attribute the problem to the increase in context length for a reader to condition on (de Jong et al., 2023; Hofstätter et al., 2023; Wu et al., 2022). For instance, the Fusion-in-Decoder model (Izacard et al., 2020) retrieves 100 documents, each comprising of 250 tokens, resulting in the reader model attending to 25,000 tokens during answer generation; this poses a significant bottleneck during inference.
To address this limitation, (Wu et al., 2022) transforms the retrieved texts as neural memories. A neural memory is an efficient way of storing knowledge with a fixed length latent representation (Khandelwal et al., 2020; Cai et al., 2021). As a result, memory-augmented QA models generate an answer conditioned on retrieved neural memories, rather than retrieved raw text. This approach shortens the context length, enabling memory-augmented models to respond to several hundred questions per second (Wu et al., 2022). While this improves the throughput compared to conventional retrieval-augmented models, there is still room for improvement in performance.
Another critical drawback of existing retrievalaugmented models, both conventional and memorybased approaches, is their narrow focus on a single type of knowledge source, either QA pairs or Wikipedia articles (Lewis et al., 2021; Wu et al., 2022; Chen et al., 2023). External knowledge sources come in various formats, such as unstructured, semi-structured, and structured, each with its own merits and use cases. For instance, unstructured data, like a Wikipedia paragraph, is easily accessible and often covers a broad range of topics.
16110
However, it may suffer from noise and a lack of precision. In contrast, semi-structured knowledge, such as a question-answer pair (QA), is concise and clear but can be challenging to gather. Previous retrieve-and-read approaches have mainly utilized a single type of knowledge source, and this limited focus results in reduced scope and coverage of knowledge. A potential approach to overcome this limitation is by synchronizing the knowledge structure, such as converting a Wikipedia paragraph into QA pairs using a QA generation pipeline, as demonstrated in (Chen et al., 2023). However, this format transformation process comes with significant computational cost and the potential risk of introducing noise or corrupting knowledge during the transformation.
To address these limitations, we introduce MATTER, a novel memory-augmented QA model designed to retrieve information from diverse knowledge sources. Unlike existing retrieval models, MATTER retrieves from multiple heterogeneous knowledge sources. This allows our model to maintain a comprehensive and type-agnostic knowledge index, enabling retrieval and conditioning on a broader range of knowledge snippets in response to questions. Moreover, our model cross-encodes a given question and retrieved neural memories, ensuring a comprehensive understanding of input and context. With this efficient cross-encoding capability and access to heterogeneous knowledge sources, our model significantly outperforms existing efficient QA models in both zero-shot and fine-tuned settings. Furthermore, it achieves a remarkable 100x throughput improvement over raw text-augmented QA models like FiD (Izacard et al., 2020), while maintaining competitive performance. Overall, our approach strikes a balance between speed and performance demonstrated over popular QA benchmarks and supported by in-depth analysis.

heading: 2 Related Work
text: Question answering is a central task in the NLP community, and various approaches have emerged to address it, falling into three main categories. Firstly, closed-book approaches like T5 (Raffel et al., 2020) and BART (Lewis et al., 2020a) have gained prominence relying solely on the input question and parametric knowledge to generate an answer. These QA models remain suboptimal as they rely exclusively on parametric knowledge (Roberts
et al., 2020). The second category comprises retriever-only models, where a retriever fetches a relevant QA pair based on the input question, often with a reranker to enhance QA performance (Lewis et al., 2021; Karpukhin et al., 2020; Seonwoo et al., 2022). This approach restricts the knowledge source to QA pairs and relies on strong overlap between the candidate question and QA pairs in the index.
The third category is the retrieve-and-read pipeline models, which have become the standard for building QA models with strong performance (Izacard et al., 2020; Yu et al., 2022; Lewis et al., 2020b). This method involves a retriever fetching pertinent knowledge, followed by a separate reader model that generates answers based on the acquired context. However, it’s worth noting that while these retrieve-and-read models excel in performance and combat hallucination, they are notorious for their slow inference speed (Hofstätter et al., 2023; de Jong et al., 2023; Wu et al., 2022). To address this issue, various strategies have been proposed, such as (Guan et al., 2021) and (Wu et al., 2020), that dynamically determine which retriever results to read, aiming to reduce computational overhead during inference. Additionally, (de Jong et al., 2023) eliminates cross-attention from most decoder layers and incorporates multiquery attention. EMAT (Wu et al., 2022) takes a novel approach by having the retriever fetch neural memories, significantly accelerating the answer generation process.

heading: 3 Approach

heading: 3.1 Task Definition
text: Let D be a QA dataset consisting of n questionanswer pairs (D = {(q, a)ni=1}), where q and a denote a question and an answer respectively. In this work, we are interested in open-domain question answering, in which a model has access to a single or multiple knowledge sources. In this setting, a model conditions on both an input question and additional contexts retrieved from a knowledge source when generating an answer.
P (a|q, C; θ), where C = top-k(P (·|q;K;ϕ))
θ and ϕ denote reader and retriever1 parameters respectively, and C is the k retrieved knowledge records from a knowledge source, K.
1this work assumes a dense retriever.

heading: 3.2 Overview
text: Our work sheds light on two aspects in opendomain QA: inference latency and extension to multiple knowledge sources with different types. Our model achieves fast inference by incorporating neural memory into the framework. Unlike previous retrieve-and-read approaches, our model does not read retrieved knowledge represented in text. Instead, it conditions on the neural memory representations of the knowledge, retrieved using an off-the-shelf retriever. Most importantly, our model can incorporate memories from different types of knowledge sources, enriching context information and lifting the restriction on the format of knowledge. Our model is a retrieve-and-read approach with memory, and thus we discuss the reader, retriever, and neural memory in the following sections.

heading: 3.3 Reader
text: We utilize the T5-base (Raffel et al., 2020) encoderdecoder model for a reader.

heading: 3.3.1 Encoder
text: Encoding a Question A question is fed to the encoder as in standard practice, yet the core difference is that the question representations are mapped with only the first j layers of the encoder.
Hq = ENC1:j(tq(q); θenc) (1)
tq is a template for formatting an input question, i.e. “Question: <q> Answer:”, which is fur-
ther described in Appendix E. ENC1:j indicates the first j encoder layers. Hq indicates the jth layer question representations with length |q|, Hq = {hq1, hq2, · · · , hq|q|}, where |q| is equivalent to the length of the question.
Cross Encoding Question and Memories After obtaining the question representation and topk memories with a retriever, the encoder crossencodes the latent representations in the remaining encoder layers to create H̃ .
H̃ = ENCj+1:L([M : Hq]; θenc),
where M = [m∗1 : m ∗ 2 : · · · : m∗k]
(2)
[:] indicates concatenation and m∗i indicates a neural memory of varying knowledge formats. We concatenate the retrieved memories M and the latent question representation Hq on the j-th layer, and the remaining encoder layers cross-encode the representations. H̃ is the final encoder representations mapped by the remaining layers, from j + 1 to L encoder layers.

heading: 3.3.2 Decoder
text: The cross-encoded representations hold both a question and retrieved context information. The decoder then creates a prediction based on the fused representations in an auto-regressive fashion. Suppose ât is an answer prediction at time step t, then
ât = DEC(H̃, â<t; θdec) (3)
where DEC is a decoder parameterized with learnable parameters θdec.

heading: 3.4 Neural Memory
text: Our system uses a neural representation of knowledge, and thus we utilize a model to map raw text knowledge into latent representations. To be specific, we utilize the same encoder that is used to encode questions, with the only difference being that a memory is represented with only a fixed length of latent vectors.
m∗ = Hc ∗ 1:l, where H c∗ = ENC1:j(t∗(c∗); θenc)
Here c∗ denotes knowledge with any type of structure, which is formatted with a knowledge-typespecific template. The first j layers of the encoder are utilized to obtain latent representations of the knowledge, denoted as Hc ∗ . Then, the first l vectors of the latent representation are taken as the memory representation of the knowledge. Therefore, knowledge is represented with only l latent vectors, reducing the sequence length.
Distinct from existing memory-based approaches, such as EMAT (Wu et al., 2022), our framework does not take structure of knowledge into account when mapping to a memory; for instance, Wu et al. (2022) build question and answer memory separately, yet we simply view a QA pair as a single sequence and map it as a single memory. This approach lifts restrictions on the format of knowledge that can be stored as memories. Therefore, our framework can maintain multiple and varying types of knowledge sources, which is hardly feasible with existing methods.

heading: 3.5 Retriever
text: Given a question, a retriever maps the question to a single latent representation, and the latent vector is used to search for relevant knowledge in a knowledge pool, stored as (key, value) pairs. Specifically, the search is done in two steps as in Figure 1: 1) find the index of relevant knowledge snippets through similarity matching of the given question and keys of the knowledge pool with maximum inner product search (MIPS), and 2) retrieve corresponding neural memory knowledge of the top-k indices and pass on the memories for cross-encoding as in Equation 2. The typical item being retrieved in a text-augmented QA model is raw text knowledge (Izacard et al., 2020; Yu et al., 2022; Lewis et al., 2020b; Chen et al., 2023), whereas our retriever fetches knowledge in the form of a neural
memory. Furthermore, a distinction from existing memory-based methods (Wu et al., 2022; Chen et al., 2023) is that our framework utilizes an offthe-shelf retriever, thereby eliminating retriever training which leads to several benefits such as extensiblity which we discuss later in detail. Lastly and most importantly, our retriever retrieves from multiple heterogeneous knowledge sources, one sharp contrast from existing QA models.

heading: 3.5.1 Training
text: Memory Learning Loss Our model conditions on retrieved memories during inference for efficiency, and hence, a memory representation is expected to hold salient encoded information of the retrieved knowledge snippet. Therefore, we introduce an auto-encoding loss, L∗ae. L∗ae = − logP (c∗|m∗; θ) (4) θ is a set of learnable parameters of the reader model. The objective is to train the model to reconstruct the original knowledge c∗ from its memory representation m∗.
Memory-Augmented Generation Loss The model is trained to utilize retrieved memories when generating an answer, and thus we define memoryaugmented generation loss, Lg, as follows.
Lg = − logP (a|q,M ; θ) (5) M is a set of memories retrieved.
MATTER-QA and MATTER-QA/PRG The proposed framework leverages multiple heterogeneous knowledge sources with varying structures. As a result, we propose two model variants: 1) MATTER-QA, the proposed model with QA knowledge source similar to existing memorybased models, and 2) MATTER-QA/PRG, the proposed QA model with multiple heterogeneous knowledge sources, namely QA pairs (QA) and Wikipedia paragraphs (PRG).
The loss for MATTER-QA is as follows:
L(θ) = λgLg + λaeLqaae (6) where λi is a hyper-parameter to balance the two loss components.
For MATTER-QA/PRG, an extra loss term is added due to the additional knowledge source.
L(θ) = λgLg + λqaaeLqaae + λprgae Lprgae (7) Auto-encoding loss on paragraphs, denoted as Lprgae , is added to Equation 6 to handle the additional unstructured knowledge source.

heading: 3.6 Discussion on Model Framework
text: Shared Encoder Utilizing separate encoders for encoding questions and knowledge is a viable option, yet cross-encoding is the core deciding factor why a single encoder is used for both tasks. We take the nature of self-attention into consideration; self-attention is known to attend to similar representations with the use of dot-product. By sharing parameters of the encoder, the latent question representations and related knowledge in neural memories are likely to share similar representations, and hence they are likely to attend to each other during cross-encoding. Empirical results show this to be true as a single encoder approach outperforms separate encoder approach by a meaningful margin.
Attention Complexity With the neural memory module, attention complexity significantly drops, leading to a significant boost in speed and less GPU memory2. Numerous studies have found that a decoder takes up majority of time during inference, specifically due to cross-attention (Izacard et al., 2020; Hofstätter et al., 2023); a decoder attends to encoder representations at every time step and at every decoder layer. Our approach greatly reduces the length of encoder representations with the introduction of a memory module. For instance, encoder representations with the popular FiD (Izacard et al., 2020) model have the length of O(k|c|), where k is the number of retrievals and |c| is a length of knowledge. FiD uses 100 Wikipedia documents, with each document consisting of 250 tokens, resulting in an encoded representation of length 25,000. On the other hand, with the same k and memory size set to 2, our approach results in 200 latent vectors in our encoded representation, which is only 0.8% of tokens used by FiD. The cross-attention has a linear complexity, and thus the time complexity of the proposed model shrinks down proportional to the reduced amount. Lastly, our approach utilizes less GPU memory, as fewer latent vectors from the encoder are stored for computation. The reduced time and space complexity improve the inference speed and GPU usage, respectively.
Benefits of the Off-the-Shelf Retrieval There are several benefits that come from the introduction of an off-the-shelf retriever, of which, the first is faster training. Recent retriever augmented QA models (Wu et al., 2022; Chen et al., 2023) jointly train retriever and reader models, which re-
2or CPU memory
quire considerable amount of computation and time. Specifically, as training progresses, the knowledge index as a whole is periodically refreshed with new vector representations as the retriever is updated with hard negative sampling. A large knowledge base, such as PAQ (Lewis et al., 2021), includes millions of knowledge records. Hence, updating such a large search index introduces significant slowdown in training time. With an off-the-shelf retriever, retriever training and index updates are eliminated, reducing training time and computation cost noticeably.
Furthermore, a plug-and-play approach becomes feasible. With an off-the-shelf retriever, one can switch out different retrievers to trade-off speed and performance based on the use-case at hand. For instance, we show in later sections that our reader model can be combined with a smaller retriever, doubling inference throughput for a small drop in performance.
A Single Retrieval with Multiple Heterogeneous Knowledge Sources As this work deals with multiple knowledge sources with varying structures, the simplest way is to utilize multiple retrievers, one for each knowledge source. However, this approach costs time and resources; multiple retrievers map a question to its own space, and each retriever retrieves from its designated knowledge pool. In this work, we mitigate the limitation and utilize a single off-the-shelf retriever model, hence reducing the cost. The intuition is from the recent finding that a well-trained retriever can be used as an universal retriever for varying structures (Baek et al., 2023). This aspect of ours has largely reduced the inference time and computation cost that otherwise would have been required to maintain multiple retrievers.

heading: 4 Experiment

heading: 4.1 Model
text: MATTER-QA and MATTER-QA/PRG are based on the T5-base model (Raffel et al., 2020). Following prior works (Chen et al., 2023; Wu et al., 2022), we train them in two phases: pre-fine-tuning and fine-tuning. During pre-fine-tuning, our models are trained on the PAQ-L1 dataset (Lewis et al., 2021), which is a subset of the PAQ dataset consisting of 14.1 million QA pairs. After pre-fine-tuning, the models are further trained on the corresponding downstream dataset. Both memory learning loss
and memory-augmented generation loss are used in both pre-fine-tuning and fine-tuning. Model and training hyperparameters are reported in Appendix A for reproducibility.

heading: 4.2 Knowledge Source
text: For our experiment, we employ two knowledge sources:
Semi-Structured Knowledge (QA Pairs) We use the PAQ dataset, which contains 64.9 million question-and-answer pairs (Lewis et al., 2021).
Unstructured Knowledge (Plain Text) We leverage Wikipedia paragraphs, the same set used in previous works3 (Izacard et al., 2020; Karpukhin et al., 2020; de Jong et al., 2023). This unstructured knowledge pool comprises 21 million paragraphs, with details provided in Appendix D.

heading: 4.3 Dataset
text: We test the proposed framework on the three popular open-domain QA datasets, namely TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), and Web Questions (Berant et al., 2013).

heading: 4.4 Baselines & Metrics
text: We compare our proposed approach with four categories of QA models: closed-book, retriever-only, retrieve-and-read, and memory-based QA models. In the closed-book category, we utilize variants of the T5 model. For retriever-only models, we compare with RePAQ models of varying sizes (Lewis et al., 2021) and a retriever coupled with a a reranker. In the retrieve-and-read category, we consider RAG (Lewis et al., 2020b), FiD (Izacard et al., 2020), and QAMAT (Chen et al., 2023), which are arguably the most popular approaches. Finally, for models with neural memory, we compare with EMAT (Wu et al., 2022). For models that require index search, we have used faiss library with HNSW index as in (Lewis et al., 2021). We have run all of the models on a single V100 machine.
To measure QA performance, we report Exact Match (EM). We also report the number of questions processed per second to measure inference speed, which helps assess throughput. For models that retrieve from a knowledge source, we provide
3https://dl.fbaipublicfiles.com/dpr/wikipedia_ split/psgs_w100.tsv.gz
details about the type of knowledge source and the number of retrievals (k). Additionally, we report model parameter count and CPU RAM usage for memory-based approaches.

heading: 4.5 Experiment Result
text: We evaluate our approach in two settings: zeroshot (after pre-fine-tuning) and fine-tuned (after fine-tuning) stages. The results are shown in Table 1. In the zero-shot setting, both MATTER-QA and MATTER-QA/PRG significantly outperform QAMAT and EMAT across all datasets. For example, the MATTER-QA model achieves 47.5 EM on TQA in the zero-shot setting, while EMAT scores 32.4 and QAMAT scores 34.1. The performance gap becomes more pronounced with the addition of retrieval from unstructured knowledge pool. When the proposed model conditions on both retrieved QA pairs and Wikipedia paragraphs, the EM score reaches 51.6 on TQA dataset. Both of the proposed models outperform previous approaches by more than 10 EM points. Furthermore, it is worth noting that our approach in the zero-shot setting achieves stronger performance than the fine-tuned baselines of EMAT and QAMAT on TQA and NQ datasets.
In the fine-tuned setting, MATTER achieves competitive performance. The closed-book QA models have low latency but exhibit inferior QA performance. Retrieval-only models demonstrate fast inference speed and competitive EM scores, and coupling them with a reranker can boost performance at the expense of inference speed, presenting a strong baseline. FiD and RAG models exhibit the strongest QA performance across all the datasets; nevertheless, their inference speed is by far the slowest among all the baselines. EMAT, a memory-based approach, overcomes the high latency problem, but the EM scores still have a significant gap compared to those of the retrieve-andread models. Our models strike a good balance in terms of QA performance and speed trade-off. For example, MATTER-QA and MATTER-QA/PRG achieve EM scores of 51.2 and 56.0 respectively on TQA dataset, which are comparable or even better than the scores of RAG model, while being at least 10x faster. Furthermore, MATTER-QA outperforms existing efficient approaches, EMAT and RePAQ, across all datasets in accuracy, while achieving competitive throughput of 284 questions per second. With access to multiple heterogeneous knowledge sources, MATTER-QA/PRG model out-
performs MATTER-QA in terms of performance while maintaining a slightly slower, yet competitive throughput.
Our proposed approach offers the added benefit of utilizing an off-the-shelf retriever. Our reader can be easily coupled with a smaller retriever in a plug-and-play fashion, denoted as MATTER-QA (Fast) in Table 1. The fast version can process 463 questions per second, which is approximately twice as fast as the MATTER-QA model, albeit with a small drop in performance. This clearly demonstrates the flexibility to use different retrievers for varying needs, highlighting one of the core advantages of having an off-the-shelf retriever. Detailed inference speed metrics for each module are provided in Appendix C.

heading: 5 Analysis
text: Retrieval vs Reader in Model Performance With the superior results achieved by our proposed model on various QA benchmarks, one natural question arises: are the gains in EM score solely attributable to using an off-the-shelf retriever? In fact, this is not the case; our strong reader models bring such superior EM scores. For instance, the EMAT retriever achieves an EM score of 43.3 on the TQA dataset, while the EM score with the full EMAT model is 44.4, a gain of 1.1 points through reading the retrieved memories. On the other hand, a sig-
nificant improvement is observed with our reader model: our off-the-shelf retriever model achieves an EM score of 40.0, and when coupled with a reader model that cross-encodes retrieved memories, we achieve a score of 56.0, marking an absolute EM score improvement of 16.0 points. While our retriever is inferior to that of EMAT, the EM score of the full retrieve-read model surpasses that of EMAT by a significant margin. This clearly demonstrates that the retriever is not the core contributor to the performance gains; rather, it is the proposed memory-augmented model that brings a substantial improvement.
Scalability to Varying k In this section, we analyze the scalability of model performance with varying values of k. As shown in Table 3, in the zero-shot setting, we observe a meaningful improvement by increasing k beyond 10, even though our models are trained with k set to 10. For example, the EM score increases by 0.9 for the QA-only
model and 1.9 for the QA/PRG model when k is increased to 20. However, we find that in zero-shot settings, increasing k does not always result in better performance, as evident in the scores when k is set to 30. After fine-tuning, on the other hand, a larger k leads to better QA performance, with the performance linearly increasing with increasing k. This demonstrates that our models are capable of answering questions with varying number of retrieved contexts, both in zero-shot settings and after fine-tuning.
Furthermore, we conducted experiments with various combinations of heterogeneous knowledge sources. Our model, MATTER-QA/PRG, has the flexibility to condition on different proportions of these sources. Notably, we observed that our model achieves its highest performance when leveraging knowledge retrieved from both sources simultaneously. While using only a single type of knowledge still outperforms several baselines, it’s worth noting that combining information from both sources yields significant improvements over strong baselines, reaching an impressive score of approximately 56 in TriviaQA. This demonstrates the advantage of integrating multiple knowledge sources to enrich the contextual information available to our model.
A Closer View on Model Conditioning on Heterogeneous Knowledge Base We examine what the model focuses on in the retrieved context when generating an answer. In detail, we naively assume that our model has conditioned on specific knowledge when the predicted answer is present within that knowledge. In Table 4, we observe that our
models employ both retrieved QA pairs and paragraphs during inference. MATTER-QA utilizes both questions and answers in making predictions, while MATTER-QA/PRG equally uses questions, answers, and paragraphs.
An interesting finding is that our models perform exceedingly well when relevant knowledge is provided in the context, i.e., when the answer is present in the retrieved context. When the groundtruth answer is present in the retriever results, our QA model achieves an impressive EM score of 84.7, while our QA/PRG model reaches 74.0. Additionally, our model demonstrates its ability to selectively condition on the retrieved knowledge; the proposed models condition on the retrieved knowledge when the similarity of the retriever results is high. On the other hand, when the retrieved results have low similarity scores to the input question, our model still generates a correct answer that is not present in the retrieved results 15% of the time. These key findings reveal that our memory-augmented model can selectively and effectively extract answers from the retrieved knowledge. Moreover, this suggests that our model’s performance can be further enhanced when paired with a superior retriever.

heading: 6 Conclusion
text: In this paper, we propose an efficient memoryaugmented question answering model with multiple heterogeneous knowledge sources. The proposed QA framework is able to retrieve from and condition on retrieved knowledge from multiple sources with varying formats and achieves remarkable performance in popular QA benchmarks, while having high throughput.
Limitations
In this paper, we experiment with two types of knowledge sources, unstructured and semistructured knowledge sources. Our model can be extended to retrieve from structured knowledge sources, namely knowledge graphs. As knowledge graphs are also a viable source of knowledge, we leave it as a future work to incorporate such knowledge source to enrich the knowledge pool.
Ethics Statement
Most question answering models, including the proposed model, may cause hallucination potentially leading to misinformation. Preventing such issues calls for careful attention and one possible mitigation is to adopt a thresholding approach. In this paper, we demonstrate that an appropriate retriever result is likely to lead to a correct and hallucinationfree answer. Combining this insight with the finding that retriever models are well-calibrated (Lewis et al., 2021), the confidence score (similarity score) of a retriever can be used as a meaningful proxy for evaluating retrieved results. Hence, one can choose to generate an answer from the reader only if the similarity scores from the retrieval model is above a certain threshold reducing the risk of hallucination.

heading: B Discussion on Heterogeneous Knowledge Sources
text: One natural question with heterogeneous knowledge sources is “are we giving advantage to our model by giving access to more knowledge, compared to the other baselines?”. The answer is “No”. The QA pairs, PAQ dataset, are generated from the Wikipedia paragraphs, and hence, the coverage and subject of each knowledge source are the same.
C Inference Speed by Each Module
Our model performs inference with low latency, and we present the speed of each module of the proposed model with a close view in Table 5. We find that a large portion of the inference is spent on the MIPS operation and decoder-side. As our model can be switched with an off-the-shelf retriever with a smaller hidden dimension, the total inference time can be greatly reduced; our model with RePAQ-256 retriever runs twice as faster compared to our model with mpnet-base model with reasonable decrease in QA performance as seen in Table 1.

heading: D Preprocessing Knowledge
text: Our framework maps knowledge into 2 latent representations, and hence the small number of vectors may not fully capture salient information of a long paragraph. In this sense, we split a paragraph into a set, where each item is a two sentence long utterance.

heading: E Templates
text: Here, we show the three templates used in the experiments. For input question, the template is as follows:
tq(q) = Question: $q Answer:
q denotes an input question. For a question-answer pair, the below template is used.
tqa(q, a) = <spe1><spe2> Question: $q Answer: $a
<spei> indicates a special token. In this paper, we take the first two representations, and hence we prepend two special tokens in the template for simplicity.
Lastly, for a Wikipedia paragraph, the Wikipedia title and paragraph are used.
tprg(t, p) = <spe3><spe4> Title : $t Content: $p
t and p denote Wikipedia title and corresponding content. Note that the special tokens are different from those in the QA template, and we differentiate different knowledge types simply with special tokens.

heading: F Case Study
text: Table 6 shows an example output of MATTERQA/PRG on a NQ test sample.

heading: Future Work
text: As knowledge graphs are also a viable source of knowledge, we leave it as a future work to incorporate such knowledge source to enrich the knowledge pool