Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models
Authors: Shangbin Feng
Weijia Shi
Yuyang Bai
Vidhisha Balachandran
Tianxing He
Yulia Tsvetkov
Venue: ICLR 2024 oral (2024)
Keywords: large language models
black-box language models
modular and collaborative knowledge
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: By design, large language models (LLMs) are static general-purpose models, expensive to retrain or update frequently. As they are increasingly adopted for knowledge-intensive tasks, it becomes evident that these design choices lead to failures to generate factual, relevant, and up-to-date knowledge. To this end, we propose Knowledge Card, a modular framework to plug in new factual and relevant knowledge into general-purpose LLMs. We first introduce knowledge cards---specialized language models trained on corpora from specific domains and sources. Knowledge cards serve as parametric repositories that are selected at inference time to generate background knowledge for the base LLM. We then propose three content selectors to dynamically select and retain information in documents generated by knowledge cards, specifically controlling for relevance, brevity, and factuality of outputs. Finally, we propose two complementary integration approaches to augment the base LLM with the (relevant, factual) knowledge curated from the specialized LMs. Through extensive experiments, we demonstrate that Knowledge Card achieves state-of-the-art performance on six benchmark datasets. Ultimately, Knowledge Card framework enables dynamic synthesis and updates of knowledge from diverse domains. Its modularity will ensure that relevant knowledge can be continuously updated through the collective efforts of the research community.

Published as a conference paper at ICLR 2024
KNOWLEDGE CARD: FILLING LLMs' KNOWLEDGE GAPS WITH PLUG-IN SPECIALIZED LANGUAGE MODELS
Shangbin Feng'
Weijia Shi'
Yuyang Bai?
Vidhisha Balachandran" Tianxing He"
Yulia Tsvetkov
University of Washington Xi''n Jiaotong University shangbinécs. washington.en. edu
'Carnegie Mellon University
ABSTRACT
By design, large language models (LLMs) are static general-purpose models, expensive to retrain update frequently. As they are increasingly adopted for knowledge-intensivee tasks, it becomes evident that these design choices lead to failures to generate factual, relevant, and up-to-date knowledge. To this end, we propose KNOWLEDGE CARD, a modular framework to plug in new factual and relevant knowledge into general-pon purpose LLMs. We first introduce knowledge cards-specialized language models trained on corpora from specific domains and sources. Knowledge cards serve as parametric repositories that selected at inference time to generate background knowledge for the base LLM. We then propose three content selectors dynamically select and retain information documents generated by knowledge cards, specifically controlling for relevance, brevity, and factuality of outputs. Finally, we propose two complementary integra- tion approaches to augment the base LLM with the (relevant, factual) knowledge curated from the specialized LMs. Through extensive experiments, we demonstrate that K KNOWLEDDGE CARD achieves state-of-the-ard performance on six benchmark datasets. Ultimately, KNOWLEDGE CARD framework enables dynamic synthesis and updates of knowledge from diverse domains. Its modularity will ensure that relevant knowledge can be continuously updated through the collective efforts of the research community.
INTRODUCTION
Large language models (LLMs) have demonstrated an impressive ability to encode world knowledge in model parameters (Petroni al., 2019 Roberts et al. 2020). However, they still face various challenges in knowledge-intensive tasks and contexts: they suffer from hallucination (Kryšciáski et al.
2020; Pagnoni et al.] 2021 Ji al. 2023). struggle to encode long-tail facts (Kandpal al. 2023 Mallen et al. 2023). ) and could not be casily updated with new and emerging knowledge (De Cao ct al. 2021] Hase et al. 202T). Existing works propose addressing these limitations through retrieval augmentation or generated knowledge prompting. Retrieval-augmente
LMs (Guu et al., 2020
Borgeaud et al. [022]] Shi et al., (223) employ retrieval systems to fetch relevant documents from a general and fixed retrieval corpus (e.g.. Wikipedia the Pile (Gao et al.) (0200), leveraging external knowledge from non- non- parametric sources to aid LLM generation. Generated knowledge prompting
approaches (Shin et al. 2020 Liu et al. 2022a Sun et al.) 2022) prompt LLMs to incorporate and generate contextual documents to encourage knowledge-awer aware generation.
While the two lines of work have achieved some success, these existing systems struggle to reflect two key properties of knowledge. Knowledge is modular (Stuckenschmidt et al. 2009): it is an "archipelago" rather than single "continent", encapsulating information that exists in diversified forms, domains, sources, perspectives, and more. The lack of knowledge modularity has made gener- alization new domains and targeted updates of knowledge stored in LMs difficult. Knowledge is collaborative (Cayzer 2004]: LLMs should be able to represent and incorporate diverse and evolving knowledge, from multi-faceter sources and perspectives, while enabling collaborative contribution from various stakeholders. Community-driven knowledge could aggregate new knowledge from domain experts and enable the development of specialized I.LMs, tailored to specific industries or applications. That being said, existing approaches and systems did not employ modular collab- orative knowledge sources that enable the plug-and-pley ann-pply updates and contributions from various stakeholders. While approaches such retrieval augmentation could be extended for modularity,
R Resources are available http:://ghhhiiccomo//mssnnnnnnne Card

---

Published as a conference paper at ICLR 2024
they are hardly compatible with the current landscape of model sharing (Wolf et al. 2019) and do not facilitate community-driver efforts to fill in LLMs' knowledge gaps.
To this end, we propose KNOWLEDGE CARD, novel framework to empower general-purpose purpose LLMs with modular and collaboratively-sorre knowledge through the integration of smaller, but specialized language models. As increasing amount of powerful LLMs released behind API calls, not directly accessible, and are prohibitively expensive to train or adapt, KNOWLEDGE CARD specifically focuses on augmenting black-box LLMs to enrich their knowledge capabilities. We first curate specialized LMs, knowledge cards, trained on corpora from diverse sources and domains to serve as modular knowledge repositories ($2.1). Compared to existing approaches, knowledge cards enable flexible and targeted information access, searching over domains, and employing private and personalized knowledge sources. These specialized L.Ms are later prompted to generate background information to support general-purpone L.MMs. We then propose three levels of knowledge selectors to dynamically select and refine generated documents and control for topic relevance, document brevity, and knowledge factuality ($2.2). Finally, we propose bottom-up and top-down-twe two approaches to empower knowledge cards into the LLM) 22.3) Specifically, the bottom-up approach starts by prompting all knowledge cards to generate multiple documents, then performs selection with the three knowledge selectors, while concatenating the final knowledge paragraph with the query for LLM generation. While the bottom-up approach uniquely enables multi-domain knowledge synthesis, it also presents the risk of presenting irrelevant information to LLM in contexts where external information is not needed. This motivates us to propose the top-down approach, where the general purpose LLM itself decides whether external knowledge is necessary for the given query, then relevant knowledge cards are selectively activated for knowledge integration; this process is repeated until the general-purposs. LLM has enough confidence to generate a response.
LLMs by integrating outputs from specialized LMs (i.e. fie..plgggig in
Extensive experiments demonstrate that KNOWLEDGE CARD outperforms vanilla LLMs, retrieval- augmented LMMs, and generated prompting approaches on three tasks across six datasets. For general-purpose knowledge QA, KNOWLEEDGE CARD improves Codex performance by 6.6% on MMLU and even outperforms the 3-times larger Tlan-PaLM For misinformation analysis that tests multi-domain knowledge integration, KNOWLEDGE CARD outperforms all baseline approaches by at least 15.8% and 10.0% balanced accuracy scores on two- and four-way classification settings. In the third task, to evaluate the ability to update the knowledge of general-purpose LLMs we curate MIDTERMOA, a QA dataset focusing on the 2022 U.S. midterm elections while the knowledge cutoff of LLMs is generally 2021 or earlier. Experiments demonstrate that KNOWLEDGE CARD outperforms all baselines by at least 55.6% on exact match scores, showcasing the ability for temporal knowledge update while only adding one knowledge card trained on midterm election news with 100x fewer parameters than the ceneral-purpose LLM. Our findings demonstrate the potential of filling in ceneral-purpose purpose L.LMs by integrating modular and collaborative knowledge from small, independently trained, and specialized LMs. We envision KNOWLEDGE CARD as an initiative to encourage LM developers to collaborate in expanding the knowledge of large language models while reducing the carbon footprint from retraining gigantic LMs from scratch.
the knowledge gaps of general-
METHODOLOGY
We introduce KNOWLEDGE CARD, novel framework to empower general-purpose LLMs with modular and collaborative knowledge (Figure D]. We train various knowledge cards, LMs trained on specialized knowledge corpora from diversified domains and sourees ($2. We then them to produce background knowledge for the ceneral-purposs. LLMs while employing three knowledge selectors ensure quality in knowledge synthesis ($2.2). Finally, we propose bottom-up and top- down, two approaches to condition the L.M on the content sourced from knowledge cards and post-processed using the knowledge selectors 22.3)
2.1
KNOWLEDGE CARDS
While existing approaches rely on one fixed source of knowledge to improve LLMs (one retrieval corpus (uu et al. 2020; Borgeaud et al. 2022 Shi et al. 2023), one knowledge graph (Wang et al.)
2021 Zhang et al. 2021 1 Feng et al.] 2023c), or one pretrained LLM itself (Shin et al. 2020: Liu et al. 2022a Sun et al. (2022))
hypothesize that since knowledge is modular, general purposs LLMs should be augmented with modular plug-and aan--pay knowledge repositories that allow users to collaboratively add, remove, edit, or update information. In addition, different communities might
2

---

Published as a conference paper at ICLR 2024
purposs perenul-pugess
peneral
LUM
beghh
Who is the sentor sehator of Tem Brady's sirihplac??
Do you need more Information? You or No:
Glanne Felestele
Yes
auto selection
ckp se selection
Clrve fetrraii, veneer
of
lrowledge
Choese aninternaton source from the folkoiigg:: sporti tirmolica Parater.
Krnwle-gp foes Bradh retured to Maleo,
intrred hererce 1120
per
?
Brady
froms
snice servve aeaky's
Cannee fatrels Cabbeil
sparts
Arswann
Caloea
Fart
Trrtin Sclecio
K
documents
gereral-purpoon
?
Fdiecane Sobote
phags
Sectallty
Milict andec catar
Darne Fonoteele
Bottom-Up
Top Down
Figure I: Overview of KNOWLEDGE CARD. We train knowledge cards on various knowledge domains and employ three knowledge selectors for quality control. We propose bottom-up and top-down to integrate knowledge synthesis (bottom-up) and proactively secking external knowledge (top-down).
purpose LLMs with modular and specialized LMs for multi-domain
have different definitions and requirements for knowledge, Wikipedia factoids, biomedical literature, mathematical formulae, and commonsense knowledge graphs are all valuable knowledge components in various contexts, thus LLMs should be able to represent and incorporate knowledge contributed by stakeholders across
domains and industries.
To this end, we propose curate knowledge cards, specialized LMs that are much smaller than black-box LLMs, trained on diversified knowledge corpora from a wide range of domains and sources. Concretely, we obtain knowledge cards C [c1, .., Cn checkpoint and further trained on a specific knowledge corpora D; with the causal language modeling objective. Given query to the LLM, these knowledge cards are selectively activated and used with prompted generation. Formally, given query specialized LM e defines mapping c(q) q - dg where q used as prompt to generate a continuation the knowledge document dg which are later prepended into the context of general-punge purpose LLMs through various mechanisms ($2.3).
each starting from existing LM
In this way, the modularity of knowledge is demonstrated through the effortless addition, removal, or selective activation of various knowledge cards during the LLM generation process. Similarly, the collaborative nature knowledge is reflected by enabling individuals to contribute trained knowledge cards on their desired knowledge source to KNOWLEDGE Card, expanding the knowledge of general-purpose purpose LLMs through community-drive efforts.
2.2 KNOWLEDGE SELECTORS
While it is possible to directly adopt d as relevant knowledge, we identify three key challenges in the successful integration of knowledge cards and general-purpose LLMs: relevance, brevity, and factuality. We design three respective selectors to control for such factors.
Relevance Selector While we expect knowledge cards to generate background information that is relevant and helpful to the query q. LMs sometimes deviate from the query (Holtzman et al.) 2019). Furthermore, only a handful of knowledge cards would be relevant for a given query. To this end, we propose select and retain knowledge documents based on relevance. Concretely, given a of m generated documents {d..... dm and the query we aim to retain the top-k k relevant documents and discard irrelevant information. We adopt separate encoder-bare sequence to a feature vector and cosine similarity simn( .) to measure relevance. Formally, we retain d, if i E top-k, (sim(enc(d, enc(q))) where top-k the top-k argmax operation.
LM enc(-) that maps a token
Pruning Selector Existing works mostly integrate one piece of external knowledge into LLMs (Sun et al. 2022 Shi et al. 2023). while tasks requiring integration of multiple domains of information,
3

---

Published as a conference paper at ICLR 2024
such as misinformation detection (Karimi et al. 2018) and multi-hop QA (Nishida et al. 2019), not well supported by existing paradigms. To effectively incorporate generated documents from multiple LMs while fitting into the LLM context length limit, we propose to prune knowledge documents. Formally, given m documents (dl.-. ......................... dm J. we adopt a pruning model prune()), .). operationalized most simply as summarization system (Zhang al. 2020: Liu et al. 2022b), to obtain the condensed versions separately {d,,- J. This pruning method allows for the integration into the main
LLM of information from multiple domains while preserving space for in-context learning.
Factuality Selector Language models are prone to hallucination (Ji et al. 2023) and the knowledge cards are no exception. Given a set of m pruned knowledge documents {dh,... dm }, their original versions {dj, d I, and the query q. we filter out the non-factual knowledge and retain & documents. Specifically, we evaluate the factuality of knowledge documents with two measures.
We first evaluate summarization factuality, ensuring that the pruned version d, factually captures the important points in the original d, Concretely, we adopt factuality evaluation models (Kryšciński ct al. 2020 Feng et al. 2023a) as a scoring function sum-fact(:; +), where each knowledge document
is assigned a summarization factuality score tuom = sum-fact(( (d d) E [0. 1].
We then propose to evaluate whether the generated knowledge document is well-supported by real-world knowledge through aerrivallaugmeented fact checking. Specifically, given a knowledge document d, we retrieve k documents from retrieval corpus t1, t,...., tk. then employ fact-checking model (Schuster et al., 2222)) as scoring function fact-check(..). We then assign a fact-checked factuality score to each d based on the retrieved document that most supports d. formally fuet max fact-chock(d t.)) E [0, 1]. We then average the summarization factuality score and the fact-checking score for each document to obtain sd.
While it is straightforward to greedily select & knowledge documents with the highest 8a scores, new and more recent knowledge might not be well-supported by existing fact-checking tools. As result, we propose top-k factuality sampling to allow for flexibility while remaining stringent towards knowledge documents that are clearly wrong. Formally, first obtain D the set of knowledge documents with the top-k factuality scores where > is a hyperparameter. We then define a sampling probability distribution over all knowledge documents:
eexp(sa)) (o EDE exp(sa,), if d, E Db:
p(d, ())
0.
if d, & 'DA
We sample l knowledge documents from (di, dm) with probabilities (p(d, gq.. ... p dm 4)}. In this way, knowledge documents with very low factuality scores are strictly removed while flexibility is built in through sampling from the knowledge with factuality scores near the top.
2.3 KNOWLEDDGE INTEGRATION
After defining the modular components in KNOWLEDGE CARD (a pureral-purposeeeee LLM, knowledge cards, and knowledge selectors), we propose two approaches, bottom-up and top-down, to integrate the
L.M with external knowledge sources, which are selected outputs of knowledge cards. Specifically, bottom-up activates all available knowledge cards at once and employs the three knowledge selectors to control for knowledge quality. Bottom-up enables multi-domain knowledge synthesis across all available sources, but these might occasionally introduce irrelevant information which may adversely impact LLM inference. We additionally propose a top-down approach, in which the LLM proactively seeks external information from selected knowledge cards. top-down is advantageous in tasks and domains where external knowledge is not always necessary.
Bottom-Up Approach Bottom-up starts by prompting available knowledge cards, then progres- sively goes through the three knowledge selectors, and these outputs are incorporated into the LLM via the prompt context. Formally, given n knowledge cards C .c...... c, and the query q. we generate n documents with each knowledge card through temperature sampling (Holtzman al., 2019) to obtain (di,.... d. We first apply the relevance selector to retain 72 most relevant documents (di.... ,dng J. then conduct knowledge pruning through the pruning selector (d,,... J. and finally leverage the factuality selector to obtain ny high-quality knowledge documents (dh,'.. ,d
4

---

Published as a conference paper at ICLR 2024
The final prompt for the LLM is a concatenation of knowledge documents and the query, formally ["Knowledge: I dj d2 ... dn, q] where || denotes concatenation. We expect the bottom- up approach to be strong in multi-domain knowledge synthesis since multiple knowledge cards could be activated at once to provide background knowledge from diverse perspectives. In addition, hyperparameters ni, n2, and ng enable fine-grained control over the knowledge synthesis process.
Top-Down Approach In bottom-up, assume that every query would benefit from external knowledge generated by knowledge cards. However, this could introduce unnecessary information in the LLM's prompt context (Zhao al. (023). Following Kadavath et al. (2022), who showed that LLMs possess preliminary abilities to identify their inherent knowledge limitations, we propose the top-down approach, putting the LLM in charge to iteratively identify whether external knowledge is needed and selectively activate relevant knowledge cards through various strategies.
Concretely, for the n knowledge cards C- {c1, ). also ask the knowledge card contributors to submit a textual description of LMs S = {81.-.. 8n such as "biomedical literature", "college calculus", or "commonsense knowledge graph". We first ask the LLM yes/no question to determine whether external knowledge is needed for the given query Q. specifically "Do you need more information? (Yes or No)". We encourage better-calibrated answers to the yes/no question through in-context learning (Wei et al.; Press et al. 2022): specifically, we introduce a set of in-context learning examples that encompass two distinct categories of questions posed to the LLM. The first category consists of questions that the LLM is capable of answering accurately without the need for any extra information. For these questions, the response to the query "Do you need more information (Yes or No)?" is "No." The second category comprises questions that the LLM cannot answer correctly without the provision of additional information. In this case, the corresponding output label for the query is "Yes." In this way, we prompt the L.M to learn to request external knowledge through in-context learning; we analyze the effectiveness of this approach in Section 5 If the LLM answers "No", we directly prompt the LLM to generate based on the query, without resorting to knowledge cards. If the LLM requests external knowledge by answering "Yes", we employ two strategies (Algoithm 2 to select relevant knowledge card and generate background knowledge.
Automatic Selection (AUTO) We further prompt the LLM with "What kind of information do you need?" and select one knowledge card based on its response rq. Concretely, we identify which LM description {s) the corresponding LM to generate multiple knowledge documents, then select one with the highest factuality score based on the factuality selector (.2) to obtain d.
s,, } is most relevant to rg with the relevance selector ($2.)) and activate
Explicit Selection (EXP) Alternatively, we ask the LLM to directly select one knowledge card by prompting with "Choose information source from the following: s....., S,,", If the LLM responds with 8;, we activate the corresponding knowledge card e; to generate multiple knowledge documents and select one with the factuality selector ($2. to obtain d.
Upon obtaining the document, we append " "Knowledge: d' to the LLM context. We then iteratively ask "Do you need more information? (Yes or NoJ" again, repeat the above process, until the LLM answers "No" and generates a knowledgeiinfornet when external knowledge not always necessary. In this way, the top-down approach enables LLMs to take charge in identifying their inherent knowledge limitations and seeking help from external knowledge cards proactively. We provide prompt examples in Tables and the Appendix.
response. We expect top-down to perform better
3 EXPERIMENT SETTINGS
Implementation For knowledge cards, we use OPT-1.3B (Zhang et al.
2022 as the starting point
and separately train 25 specialized LMS on wide range of knowledge sources and domains, including corpora in the Pile (Gao et al. 2020). branch-train-merge (Li et al. 2022), knowledge graphs (Speer al. 2017 West et al. [2022] Vrandečié Krötzsch 2014 Pellissier Tanon et al. 2020 Feng et al. 2021 Zhang et 2021), news and social media (Liu et al. [022c Feng et al. 2023b). and more.
(Appendix E) We use MPNet (Song al., (020) as the encoder the relevance selector, Pegasus
(Zhang et al. 2020) as the summarization model in the pruning selector, the WikiSearch API as the
retrieval system in the factuality selector, and FactKB (Feng et al., (023a) and VitaminC (Schuster) ct al. 2)21) as the summarization and fact-checking factuality scoring functions. We use Codex (CONI-DAVINCI-002) (Chen et al. 202 as the default, general-purpose, black-box LLM.
5

---

Published as a conference paper at ICLR 2024
Type
Model
Human.
Sucial
STEM
Other
All
Modd
BAce
Mal'
MoF
Vanilla LM
Cosex
PaLM
14.2
71.0
81.0
76.9
57.3
55.6
69.6
70.1
68.3
60.3
Vanilla LM
Cobex
65.6
51.0
52.8
44.0
FLAN PALM
ATLAS
52.8
72.2
Retrieval
RePluu LSR
REPaLO
78.8
78.8
68.5
67.8
57.5
5588
58.4
53.0
Retrieval
REPLUI LSR
79.7
72.1
73.5
71.6
Generate
RECTEATTION
Gonurate
GKP RECITATTIO
76.9
21.
74.5
78.1
99.0
99.5
74.0
71.4
20.0
71.9
Borroes-Ui
59.8
87.3
7066
67.3
KNOWLEDGE CARD
Tor Down AUTO
Barronss 12P
77-7
77.2
76.9
76.7
57.9
99.2
73.0
72.2
70.7
72.0
KNOWLEDGE CARD
Tox DowN AUTO
ToR-DOwN EXP
91.3
56.4
75.7
56.0
63.0
60.4
65.5
50.2
Toi-DowN EXP
88.9
74.3
71.8
Table 2: Performance on misinformation de- tection. BAcc and MaF are balanced accu- racy and macro F1. bottom-up performs best due to multi-domain knowledge integration.
Table 1: Model performance on the MMLU Bench- mark. KNOWLEDGE CARD improves Codex by at least 3.5% while top-down outperforms all baselines.
Type
Model
Open-Book
2-way
Multipie-Cholce
4-way
Tasks and Datasets 1) For general- purppss QA. we adopt MMLU (Hendrycks et al. 2020), a multiple-choice QA dataset covering
Vanilla I.M
CoDEX
55.1
57.9
90.9
RuPLUs
44.8
85.7
6288
tasks in humamities, STEM, social sciences, and oth- ers. Following previous works (Si et al. 2022
Retrieval
REPLOG 1.RR
37.2
86.9
65.3
SLET AL.
52.1
54.5
$4.7
61.4
GKP
45.0
46.9
89.1
53.5
Generate
RECITATION
44.4
46.4
89.3
52.3
Shi et al. 2023), adopt 5-shot in-context learning setting. 2) To evaluate multi-domain knowledge synthesis,
GRTR
Borrrow-Ei
81.6
64.5
adopt misinformation detection, since news articles often encompass facts and opinions at the intersection of differ- ent domains and perspectives. We leverage the widely adopted LUN misinformation detection dataset (Rashkin et al. 2017) with both 2-way and 4-way classification settings. All models are evaluated based on 16-shot in-context learning. 3) To evaluate temporal knowledge update, we curate MIDTERM a QA benchmark focusing on the 2022 U.S. midterm elections since the knowledge cutoff of black-box LLMs is often 2021 or earlier. MIDTERMQA presents three evaluation datasets and settings: open-book, 2-way, and 4-way multiple choice. 5-shot in-context learning is adopted to evaluate KNOWLEDGE CARD and baselines. We did not consider existing temporal QA datasets (Jang al.) 2021 Dhingra et al. 2022 Kasai et al. 2022 since they do not focus on any specific event or knowledge domain.
KNOWLEDGE
Top-Dow>
89.5
Tor-DowN EXP
75.3
73.7
91.9
67.6
Table 3: Performance on MidtermQA. KNOWL- EDGE CARD successfully updates the knowledge of Codex by adding single knowledge card.
Baselines We compare KNOWLEDGE CARD with wide range of baseline methods in three categories. 1) vanilla black-box LLMs: Codex (Chen et al. (021). PaLM (Chowdhery et al. 2022). and Flan-PaLM (Chung al.)
2022); 2) generated knowledge prompting approaches: GKP (Liu et al.)
2022a), recitation (Sun et al. 2022), GRTR (Yu et al. 2022) (Note that we apply these methods to the same LLM Codex (Chen et al. 202T for fair comparison); 3) retrievll-augmenter language
models: Atlas (Izacard et al. 2022). Si ct al. 20222 RePlug, and RePlug LSR (Shi et al. 2023).
4 RESULTS
For general- purposs knowledge QA, use the MMLU benchmark (Hendrycks et al. 2020). As shown in Table 1, all three configurations of KNOWLEDGE CARD significantly improve vanilla Codex. Among them, the top-down approach with explicit selection performs best, improving
MMLU
Codex by 6.6% overall accuracy. Concurrently, top-down approaches surpass all baselines, including FlaM-PaLM with a few hundred billion more parameters. These results suggest that we present an purpose LLMs better in knowledge-intensive contexts. In addition, top-down generally outperforms bottom-ut pup likely because MMLU contains math-related questions that do not necessitate external knowledge. This observation suggests that top-down approaches are better at tasks where external knowledge is not always necessary.
effective approach for making
Misinformation Detection To examine whether KNOWLEDGE CARD successfully integrates multi- faceted knowledge from diversified sources, we adopt the LUN misinformation dataset (Rashkin et al.) 2017) with two- and four-way classification settings. Table 2 demonstrates that KNOWLEDGE CARD significantly improves Codex by at least 31.7% and 19.4% in balanced accuracy scores for both settings. In addition, bottom-u up outperforms both variants of top-down, thanks to its methodology to jointly activate knowledge cards from various domains and enable multi-domain knowledge synthesis.

---

Published as a conference paper at ICLR 2024
95.0
ADC
+2.0
full model
w/o relevance
11.0
92.5
w/o pruning
factuality ity
BechCoraus
hubRed
MD5
hows
+ Widisodi
Doancen ACE
11.18.
+2.0
3.3
87.5
-0.6
1.0
85.0
2.5
ACC
Macro F1
Figure 2: Performance on misinformation de- tection when each knowledge card is separately added. KNOWLEDGG CARD enables modular patching of LLMS while in-domain knowledge cards help the most.
Figure 3: Ablation study of the three knowledge selectors on misinformation detection. While the three selectors all contribute to model perfor- mance, the factuality selector is most crucial.
Mi To examine whether KNOWLEDGE CARD could update the parametric knowledge of LLMs. train additional knowledge card news articles regarding the 2022 U.S. midterm elections and plug it into KNOWLEDGE CARD. We present model performance on MidtermQA in Table which demonstrates that KNOWLEDGE CARD substantially outperforms all baselines in the open-book setting by as much as 57.3% in exact match scores (EM). This indicates that one knowledge card with 1.3B parameters successfully updates the parametric knowledge the 175B Codex through KNOWLEDGE CARD. In addition, top-down outperforms bottom-up. indicating that the selective activation of knowledge cards is better when there is a specific knowledge card tied to the task domain. KNOWLEDGE CARD also outperforms SI ET AL. (Codex Contriever) that uses the same midterm election news as retrieval corpora. addition, generated knowledge prompting approaches (GKP, recitation, GRTR) underperform vanilla Codex, showing that probing LLMs for explicit knowledge is counterproductive when internal LLM knowledge is outdated or wrong.
ANALYSIS
Patching LLM Knowledge When general-purper LLMs struggle at tasks due to knowledge limitations, KNOWLEDGE CARD could serve as efficient approach to patch LLM weaknesses by adding specialized language models. To this end, we evaluate the change in performance when five knowledge cards separately added augment Codex with the top-down approach. Results in Figure 2 demonstrate diguur@[emonntrateee that patching the LLM with all five LMs leads to various levels of performance gains on misinformation detection, while the most in-domain LMs (Wikipedia and news) lead to greater improvements. This suggests that when LLMs perform poorly on mnowledge-intenivg tasks, an additional knowledge card trained on in-domain corpora could help with KNOWLEDGE CARD.
Model
Two-Way BAcc
Four- Four-Way BAcc
Knowledge Selector Study In Section 2.2 we pro-
MaF
MaF
pose three levels of knowledge selectors to control for various factors and ensure knowledge quality. We conduct ablation studies to remove each knowledge
REPLUG
78.8
67.8
55.8
$3.0
REPLUG LSR
78.8
68.5
57.5
54.4
BOTTOM-UR UP
90.0
87.0
65.3
63.3
selector in the bottom-up approach and re-evaluate on misinformation detection. Figure demonstrates that while all three knowledge selectors are helpful, the factuality selector contributes most to model per- formance and thus plays crucial role in ensuring the quality of generated knowledge documents.
ToP-DOwN AUTO
80.7
70.9
60.1
56.8
Top-DOwN EXP
80.6
70.0
59.7
56.5
Table 4: KNOWLEDGE CARD outperforms retrieval LM REPLUG in the Wikipedia-only setting, suggesting that modular LMSs present a better knowledge repository than retrieval.
Retrieval vs. Specialized LMs In order to assess the effectiveness of modular specialized LMs as compared non-parametric sources like retrieval, we exclusively use the Wikipedia LM in ENOWLEDGE CARD and compare with the state-of-the-a retrieval LM REPLUG that also uses Wikipedia as the retrieval knowledge source. Table 4 demon- strates that KNOWLEDGE CARD outperforms REPLUG on both settings of misinformation detection, suggesting that knowledge cards present better knowledge repository. Note that KNOWLEDGE CARD is also compatible with multiple knowledge formats (e.g. retrieval and search engine) while they could be complementary (Appendix(A) A)).
Knowledge Stream Analysis In bottom-up cuptom-upp.oo three hyperparameters ($.3) govern the "knowledge stream" from knowledge cards to the general-purpose L.Mss. Specifically, ny controls how many
7

---

Published as a conference paper at ICLR 2024
85-
+56
0.90
0.90
0.90
ACC
Macro
70
duvici
0.85
0.85
0.85
75
126
2
4
10 12
3
5
n1 (n2 5,
n3 = n2 (n1
n3 = 3)
n3 [n] = 3, n2
=
tarto botiom wulnrend Wi esprard
Figure 4: Investigating the impact of n1, 712, and n3, which Figure KNOWLEDGE CARD govern the knowledge stream from modular knowledge cards is compatible with other LLMs, to general-pus purposee LLMs. These hyperparameters enable fine- specifically TEXT-DAVINCI-003 grained control over the knowledge synthesis process.
and GPT-3.5-TURE
documents each LM generates, n2 controls how many are retained after the three knowledge selectors, and fi controls how many are put into the context of LLMs.. We investigate these control measures and report performance in Figure 4 It is illustrated that: 1) n& has marginal impact, suggesting that knowledge cards generate largely homogeneous knowledge even with temperature sampling (Caccia al. 2018); 2) larger n2 leads to performance drops, suggesting that the three knowledge selectors ensure knowledge quality: 3) n3 = 1, where only one knowledge document is adopted at a
time (as in previous works (Sun al. 2022; Shi et al. )023)) is worse than larger values, showing the advantage of multi-domain knowledge synthesis uniquely enabled by KNOWLEEDGE CARD.
LLM Compatibility
While we follow previous works (unn et al. Shi et al. 2023) and adopt Codex the default black-box
E-way
2022
1,290
198
314
171
LLM. KNOWLEDGE CARD is compatible with different models. We
additionally evaluate KNOWLEDGE CARD with two other LLMs. TEXT-DAVINCI-OR and
1.140
1.576
and present results in Fig- Both bottom-up and top-down consistently improve different
kagr: Z-wway
wAp:: 4-kwy
ure 5
1.004
42
364
68
LLMs across various datasets and evaluation metrics.
1.642
311
1,718
849
Yes/No in Top-Down top-down ($2.3) we begin by asking LLMs if they might need external knowledge for the given query and adopt in-context examples to encourage well-calibrated answers. We illustrate LLM responses along with the correctness of their answer in Figure 6 The vast majority of queries are mapped to the "yes, correct" and "no. correct" categories, suggesting that LLMs have preliminary abilities to "know what they know" and seek external information necessary. However, this ability is far from perfect, evident in the nogllgglie category of "no, incorrect", suggesting that prompting L.MMs to ac- knowledge knowledge limitations requires further research (Kadavath et al. 2022, Zhao et al. 2003). while new approaches abstain could be easily integrated into KNOWLEDGE CARD. In addition, the "yes, incorrect" categories suggest that specialized LMs occasionally fail to provide enough information. These confusion matrices provide fine enainna error analysis and guidance as to whether the gerproe
Figure 6: Confusion matri- ces of yes/no and correctness in top-down, enabling fine- grained error analysis.
purpose LLM, the yes/no question, or knowledge cards require further improvements.
Ruce
Codexx
KNOWLEDGE CARD
Qualitative Analysis We curated MIDTERMQA evaluate whether KNOWLEDGE CARD enables effi- cient knowledge update. We examine the 88 races where the incumbent was not re-elected: Codex an- swered out of the 88 questions correctly, while bottom-up and top-down with automatic and explicit selection answered 63, 77, and 42 correctly. Table shows that Codex states the incumbents would win again 2022, while KNOWLEDGE CARD success- fully updates LLMs with 100x more parameters.
AL, senate
Doug Joncs X
Katic Beitt /
PA,
Bah Casey
John Fetternan
CA,
Mike Thampson X
Kevin Kiley /f
IN, 2nd
Jackie Walorski
Jim Banks
NV, governor
Steve Sisolak
Joe Lombardo
Table 5: While vanilla Codex falsely claims that these incumbents won again in the 2022 elections, KNOWLEDGE CARD successfully updates the knowledge of black-box L.LMs.
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
Retrieval- Augmenttd Language Models
Augmenting language models with retrieval has ad-
vanced the -fate-of-the-art in open-domain QA (Guu et al.) 2020 Izacard et al. [ Lewis et al.)
2020 et al.
2022),
text classification (Zhao et al.) 2023), and language modeling (Hu et al. [2022]
Borgeaud et al.
2022 ei al. 2023). The retrieval system could be integrated into encoder-decoder
(Izacard et al.)
2022) and decoder-only models (Borgeaud et al. 2022 Shi et al. 2022 Rubin et al.)
2022). or leveraged to interpolate the next token probability distributions (Khandelwal et al. 2019
Zhong al. 2022). Recent advances incorporated frozen (Mallen et al. 2023 SS et al. [002]] Khattab
et al., 2022) and trainable retrievers (Shi et al. 2023) as well as search engines (Press et al. 2022)
to augment L.LMs. Compared to retrieval models and search engines, KNOWLEDDGE CARD enables flexible information seeking, searching over knowledge domains, and employing private knowledge sources. In addition, these works often leverage only one retrieval corpora and assume that it's "omniscient" while suffering from various issues such as domain coverage and knowledge update. In contrast, we propose to reflect the modularity and community-dree driven nature of knowledge by integrating plug- andu-aad-pue -lay knowledge cards with general-purpose purpose LLMs.
Generated Knowledge Prompting LMs acquire knowledge through training on gargantuan textual corpora (Petroni et al. 2019 Dhingra et al., 2022 He et al. 202T). Generated knowledge prompting (Liu et al. 2022a) is one of the early approaches to tap into the parametric knowledge of LLMs by prompting them to generate background information and re-using it for QA. Related works also propose to use LM parametric knowledge for retrieval (Tay et al.) (022), answer commonsense questions with self-talk (Shwartz et al., 22220) generate queries (Wang et al., 2022; Zhuang et al.)
2022) or token sequences (Bevilacqua et al. 20022) for document augmentation. In addition, recitation-
augmented language models (Sun et al. 2022) propose to augment QA examples with diversified
knowledge recitations, while (Yu et al., 2022) shows that generated knowledge is, under certain cireumstances, better than retrieval. However, this line of work assumes that the encoded knowledge in LLM parameters is all we need, while LLM knowledge suffers from hallucination (Ji et al. 2023) struggles to encode long-tail facts (Mallen et al.) 2023), and can not be efficiently updated (De Cao et al. 2021). While recent works propose to edit LLM knowledge (Meng al. 2022, Hernandez et al. 2023). they are hardly compatible with black-box LLMs. In addition, parametric knowledge in LLMs far from modular and collaborative, while LMs should be able to incorporate knowledge contributed by all stakeholders in L.MM research and applications. To this end, we propose KNOWLEDGE CARD as a community-driven initiative to empower general-purpose LLMs with modular and collaborative knowledge through the sharing and re-using of knowledge cards.
Modular LMs Mixture-of-Expert (MoE) (Masoudnia & Ebrahimpour 2014) aims to activate one expert based on the imput instance, which has been adopted in language model research (Gururangan
al. 2022: Roller al. 2021 Lewis et al. 202T; Kudugunta al. 202T Preiffer et al. (022)
Adapters are also proposed for task transfer and serramtter-eficieee
(Houlsby et al. 2019
Přeiffer et al.] [020; Zaken et al. [022). In addition, parameter averaging (Matena & Raffel
2022
McMahan et al. 2017 [zmailov et 20118 Wortsman et al.) 2022: Li et al.) |022 Gururangan et al.
2023), model fusion DDon-Yehiye et al. 2022] Borzunov et al. 20222, continual learning (Jang et al.
2021 Qin et al. 2022 Ke ct al. [022 Qin al.) (023), and other collaborative approaches (Köpf ct al. 2023; Sha 2023; Luo et al. 2023) have also shed light on the possibility of distributed LM training. However, existing modular LMs mostly operate in the white-box setting, i.e. assuming access to the model parameters, token probabilities, and more. Since the most prominent LLMS only released behind API calls, we propose KNOWLEDGE CARD with the aim of empowering black-box general-purpose LLMs with community-driven and collaborative knowledge.
7 CONCLUSION
We propose KNOWLEDGE CARD. a novel framework to empower purproe LLMs with modular and collaborative knowledge. We first present knowledge cards, specialized LMs trained on various domains and sources of knowledge, and propose three knowledge selectors to ensure knowledge quality. We then propose bottom-up and top-down approaches to integrate knowledge cards with general-purport external information when necessary. Extensive experiments demonstrate that KNOWLEDGE CARD outperforms vanilla LLMs,, retrieval LMs, and generated knowledge prompting approaches across three tasks and six datasets, showcasing its ability integrate multiple sources of information, efficiently update LLM's knowledge, and more. We envision KNOWLEDGE CARD as a community- driven initiative to empower general general-purpose LLMs with modular and collaborative knowledge.
LLMs to enable multi-domain knowledge synthesis and grounding in

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the reviewers, the area chair, members of T'svetshp. and the UW NLP Group for their feed- back. This research is supported in part by the Office the Director of National Intelligence (ODNI) Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200004. This material is also funded by the DARPA Grant under Contract No. HR001 120C0124. We also gratefully acknowledge support from NSF CAREER Grant No. (I2214,,39, NSF Grants No. IIS2125201 IIS2203097, and the Alfred P. Sloan Foundation Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily rep- resenting the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Sharegpt. https://sharegatt com/ 2023. Accessed: 2023-09-27.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3554-3565, 2021.
Eugene Bagdasaryan and Vitaly Shmatikov. Spinning language models: Risks of propaganda-as-a- service and countermeasures. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 769-786.
IEEE, 2022.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Information Processing Systems, 1.331668-3168. 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptis Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Workshop on Broadening Research Collaborations 2022, 2022.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In International Conference Learning Representations, 2018.
Steve Cayzer. Semantic blogging and decentralized knowledge management. Communications of the ACM, 47(12):47-52, 2004.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3000 2013.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1IX.::2117.03344 2021.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. Kgpt: mhowledge-rooundee pre-training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8635-8648, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
10

---

Published as a conference paper at ICLR 2024
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling arXiv preprint arXiv:2210.11416. 2022.
language models.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, 2021.
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273. 2022
Shachar Yon-Ye, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, arXiv preprint arXiv:2212.01378 2022.
Shangbin Feng, Zilong Chen, Wenqian Zhang, Qingyao Li, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. Kgap: Knowledge graph augmented political perspective detection in news media. arXiv preprint (tti:2108.03800 2021.
Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang, Zilong Chen, Binchi Zhang, Qinghua Zheng, Wenqian Zhang, Zhenyu Lei, Shujie Yang, et al. Twibot-22:: Towards graph- based twitter bot detection. In Thitt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. FactKB: Generaliz- able factuality evaluation using language models enhanced with factual knowledge. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Em- pirical Methods in Natural Language Processing, pp. 933-952, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. http:////2233..mmmmmmo/ wwwppogy..rgg.....o.ooo emn ma 59
URL
Shangbin Feng. Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11737-11762, July 2023b.
Shangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei, and Yulia Tsvetkov. KALM: Knowledge- aware integration of local, document, and global contexts for long document understanding. In Proceedings the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2116-2138, 2138, July 2023c.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling- arXiv preprini urXiv:2101.000 00000707 2020.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 5557-5576, 2022.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177. 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR. 2020.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zoritsaa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654. 2021.
11

---

Published as a conference paper at ICLR 2024
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. Analyz- ing the forgetting problem in pretrain-finetume finetuning of open-domaan dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1121-115 133, April 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Dan Hendryeks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review. In Thirty-fifth Conference on Neural Information Processing Systems Dalasets and Benchmarky Track (Round 1), 2021.
Evan Hernandez, Belinda Z Li, and Jacob Andreas. Measuring and manipulating knowledge repre- sentations in language models. arXiv preprint urXiv:2304.00744 2023.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-27999 PMLR, 2019.
Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Lonng Pupers), pp. 754-763, 2021.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt-guided task-aware image captioning, arXiv preprint arXiv:2211.09699 2022.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint orXiv:2208.0322 2022.
Izmailov, AG Wilson, D Podoprikhin, Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885, 2018.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, KIM Gyeonghun, Stan- ley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations, 2021.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know, arXiv preprint arXiv:2207.05222 2022.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn -ail knowledge. In International Conference on Machine Learning, pp. 15696-15707 PMLR, 2023.
Hamid Karimi, Proteck Roy. Sari Saba-Sadiy Sadiya, and Jiliang Tang. Multi-source multi-class fake news detection. In Proceedings of the 27th international conference on computational linguistics, pp. 1546-15577 2018
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What's the answer right now? arXiv preprint arXiv:2207.13322. 2022.
12

---

Published as a conference paper at ICLR 2024
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference on Learning Represen- tations, 2022.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2019.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. cammonstratessaarh--reeeee Composing retrieval and language models for knowledge-intensiin nlp. arXiv preprint arXiv:2212.14022 2022.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, al. Openassistant conversations-ddmccrraiiii large language model alignment. arXiv preprint arXiv:2304.07322 2023.
Wojeiech Kryšcińsski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 9332-9346, 2020.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, 2021.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning. pp. 62744. PMLR, 2021.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented genera- tion for knowledge-intensivy nlp tasks. Advances in Neural Information Processing Systems, 33: 2020.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyr. merge Embarrassingly parallel training of expert language models. In First Workshop on Interpolation Regularizers and Beyond at NeurlPS 2022, 2022.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannanch Hajishirzi. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3154-3169, 2022a.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2890-2903, 2022b.
Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp. and Lu Wang. Politics: Pretraining with same-story article comparison for ideology prediction and stance detection. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1354-1374, 2022c.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. S2ore: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969-4983, 2020.
Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Augmented large language models with parametric knowledge guiding. arXiv preprint 237.07700 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, 2023.
13

---

Published as a conference paper at ICLR 2024
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey. The Artificial Intelligence Review, 42(2):275, 2014.
Michael S Matena and Colin A Raffel. Merging models with wesher- averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication--fficienn learning of deep networks from decentralized data. In Artificial intelli- gence and statisticy, pp. 1273-1282. PMLR, 2017.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2022.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1097-2118, July 2023.
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. Answering while summarizing: Multi-task learning for multi-hop qa with evidence extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2335-2345 2019.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4812-4829) 2021.
Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. Yago 4: reason-able knowledge base. In The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings 17, pp. 583-596. Springer, 2020.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419 3448, 2022.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-24733 2019.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654-7673, 2020.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3479-3495, 2022.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Elle: Efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2789-2810 10, 2022.
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Recyclable tuning for continual pre-training. arXiv preprint arXiv:2305.08700 08702, 2023.
14

---

Published as a conference paper at ICLR 2024
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2019.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 2931-2937, 2017.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418 (118-542,, 2020.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 14411555--1566, 2021.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2671, 2022.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304. 14406, 2023.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference Learning Representations, 2019.
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin e! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 624-643, 2021.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retreeval-augmenned black-box language models. arXiv preprint arXiv:2301.12652, 2023.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, 2020.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langugge Processing (EMNLP), pp. 4615-4629, 2020.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to reliable. In The Eleventh International Conference on Learning Representations, 2022.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33: 16857-16867, 2020.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: general knowledge. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
An open multilingual graph of
Heiner Stuckenschmidt, Christine Parent, and Stefano Spaccapietra. Modular ontologies: concepts, theories and techniques for knowledge modularization, volume 5445. Springer, 2009.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations, 2022.
15

---

Published as a conference paper at ICLR 2024
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. Advances Neural Information Processing Systems, 35:21831 2022.
Denny Vrandečié and Markus Krötzsch. Wikidata: free collaborative knowledgebase. Communica- tions of the ACM. 57(10):78 85, 2014.
Heng Wang, Wenqian Zhang, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Qinghua Zheng, and Minman Luo. Detecting spoilers in movie reviews with external movie knowledge and user networks. arXiv preprini arXiv:2304.114111 2023,
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Cheng- min Chi, Guoshuai Zhao, Zheng Liu, et al. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems, 35:25600-2555 25614, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Lc, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4602 4625, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-an natural language processing. arXiv preprint arXiv::9100003: 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Loper Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-233333 23998. PMLR, 2022.
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. The Eleventh International Conference on Learning Representations, 2022.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models.. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. International Conference on Machine Learning, pp. 11328-11333 PMIR, 2020.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language models. arXiv preprint arXiv:2205.0000 2022.
16

---

Published as a conference paper at ICLR 2024
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Perey Liang, Christopher D Man- ning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models. International conference on learning representations. 2021.
Xinran Zhao, Hongming Zhang, Xiaoman Pan, Wenlin Yao, Dong Yu, and Jianshu Chen. Thrust: Adaptively propels large language models with external knowledge. arXiv preprint arXiv:2307.10442. 2023.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5657-5673, 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. Bridging the gap between indexing and retrieval for differentiable search index with query generation, arXiv preprint arXiv:2206. 10128, 2022.
17

---

Published as a conference paper at ICLR 2024
A DISCUSSION
Modularity at every turn. All components in KNOWLEDGE CARD modular and easily substi- tuted with future state-of-the-art. 1) While Codex is the default LLM in the experiments, KNOWL- EDGE CARD also works with TEXI-DAVINCI-003 and GPT-3.5-TURBO ($5) and could be easily adapted to future LLMs. 2) If better models for embedding space similarity, abstractive summa- rization, and fact-checking are developed, the three knowledge selectors ($2.2) could be seamlessly updated. 3) When new knowledge, information, and domains emerge, more knowledge cards could be trained and uploaded a model-sharing infrastructure (Wolf et al. 2020) by any member of the machine learning community and adopted to improve general-purpose LLMs.
User-centric LLM adaptation. When general-purpose LLMs are released, everyone uses the same LLM with the same API calls, while real-world users have heterogeneous use cases and expectations that require personalization (Salemi et al. 2023). For example, grade school students might expect LLMs to be absolutely factual about knowledge and information in common textbooks, NLP researchers might expect LLMs to have a basic understanding of current NLP research, cooking amateurs might expect LLMs to understand the basic recipes and cuisines for different occasions, and more. As a result, KNOWLEDGE CARD presents a preliminary approach by letting the user select and activate knowledge cards to empower LLMs with different skill sets and domain expertise.
Compatible with diversified forms of knowledge. By default, KNOWLEDGE CARD employs language models trained on varying domains and corpora as modular knowledge sources. In addition, KNOWLEDGE CARD is also compatible with 1) retrieval systems, where the retrieved text could similarly go through the three knowledge selectors and enrich LLM context, while retrieval corpora are harder to share and use than modular language models; 2) knowledge graphs, when combined with various proposals to construct natural language corpora out of symbolic knowledge bases (Agarwal ct al. 2021 Chen et al., 2020 Feng et al.) (023a). which is already included in our prototype; 3) search engines, where content on the web could also be integrated into the black-box L.LMs through KNOWLEDGE CARD. Such flexibility and compatibility are possible since KNOWLEDGE CARD conducts knowledge integration through natural language. Compared to retrieval models, using language models as knowledge sources enables flexible information seeking (rather than rigid token exact match), searching over knowledge domains, and employing private knowledge sources.
Knowledge cards heterogeneity. While existing modular LM proposals often require modular sub-models to be of the same size and architecture for parameter averaging and model fusion (Li eet al., (022). the knowledge cards in this work could be fully heterogeneous. 1) Different knowledge cards could have different sizes. While OPT-1.3B is adopted as the default architecture in this work, other sizes of OPT, from 125M all the way up tens of billions, could all be used to initialize knowledge cards. In addition, 2) knowledge cards could have different model architectures. Since the integration of level, any language generation models could be adopted as knowledge cards. These two levels of heterogeneity allow for flexibility in knowledge card training: larger and more capable models could be trained on large corpora and extensive knowledge domains by compute-richh individuals, while smaller knowledge cards trained on small and dedicated domains by computationally underprivileged researchers could also help improve black-box L.MMs, democratizing LLM research.
LLMs and modular knowledge cards happens the natural language
Knowledge cards hierarchy. We believe that knowledge cards could reflect the hierarchical nature of knowledge. If KNOWLEEDGE CARD is adopted for general question answering, then a general biomedical knowledge card trained on PubMed corpus would suffice. However, if KNOWLEDGE CARD is adopted for more fine-grained use cases, the "biomedical" domain could be further divided into sub-domains and one knowledge card could be trained for each. Similar divisions could be applied to sub-fields in NLP research, political news in different countries, and more.
Combining bottom-up and top-down. One straightforward way to combine the two knowledge integration approaches would be: in each step of top-down, the LLM proposes multiple knowledge cards as candidates, then employs the bottom-up approach with the pool of these knowledge cards for knowledge generation. We leave further explorations to future work.
18

---

Published as a conference paper at ICLR 2024
B LIMITATIONS
Knowledge cards are not perfect knowledge generators. While knowledge cards could be of any size model architecture, we used OPT-1.3B, relatively small language model to initialize knowl- edge cards trained on different domains and sources. As result, not all of the generated knowledge documents are high-quality knowledge statements, occasionally suffering from degeneration, topic deviation, and more. While the three knowledge selectors in part alleviate the impact of low-quality generated knowledge documents, we hypothesize that improving the knowledge generation of autore- gressive language models is important, yet orthogonal, research question for future work. Two potential solutions include 1) increasing the model size of knowledge cards and 2) using specialized training objectives for knowledge cards, while both approaches require additional training and more computational resources. In addition, Appendix discussed KNOWLEDGE CARD'S compatibility with diverse knowledge sources, including retrieval, knowledge graphs, and search engines, while these knowledge repositories have their respective pros and cons. We leave it to future work on integrating multiple types of external knowledge stores extend KNOWLEDGE CARD.
The factuality selector is biased towards information-riclt domains and existing knowledge. To ensure the factuality of generated knowledge documents, we employed retrieval-augmented factuality selector based on both summarization factuality metrics and fact-checking models while enabling flexibility through our proposed top-k factuality sampling. However, domains with more Wikipedia entries might be better supported by retrieved documents and might receive higher factuality scores. In addition, new and emerging knowledge might be well supported by existing retrieval corpora and receive low factuality scores. We quantitatively evaluate this bias in Appendix D Although top-k factuality sampling enables flexibility to some extent, it remains an important problem design factuality evaluation measures that are generalizable and adaptable to varying and emerging domains.
Prompting LLMs to seek help through yes/no questions not perfect. Inspired by the findings that LLMs do not need external knowledge for every query (Zhao al. 2023) and language models (mostly) know what they know (Kadavath et al., 2022), we propose to ask yes/no questions to decide whether to activate knowledge cards and encourage well-calibrated answers through in- context learning. Our analysis ($5) shows that this strategy is effective but far from perfect: LLMs are occasionally over-confident about their knowledge capabilities. We leave it future work on designing better strategies for LLMs to abstain, acknowledge knowledge limitations, and seek help from external information sources.
C ETHICS STATEMENT
We envision KNOWLEDGE CARD as a community-driven and collaborative initiative to improve general- purpose and exploitation from malicious actors. Since modular knowledge cards have the ability to change update LLM knowledge, malicious actors could advance their agenda by submitting malicious knowledge cards trained on disinformation, hyperpartisan content, propaganda, and more, while framing them as benign knowledge domains and deceive LLM users. We envision two lines of approaches towards this ethical risk: on the technicul side, research on adversarial manipulation of language models and corresponding defense tactics (Bagdasaryan & Shmatikov 2022 Perez et al.) 2022) could be integrated to alleviate the impact of malicious knowledge cards; on the social side, we could rely on and reinforce the existing rules for model sharing on popular infrastructures (Wolf et al. 2020) to prevent such malicious contribution from happening. We encourage the responsible use of KNOWLEDGE CARD, while we also call on users and researchers to be mindful of this dual-use risk.
general-purpose LLMs in knowledge-intensit tasks and contexts. An important risk is the dual use
D ANALYSIS (CONT.)
Knowledge Card Selection In the top-down approach, we ask large language models to choose relevant knowledge cards and obtain external knowledge. We illustrate the selection results of the automatic selection strategy on the MMLU dataset separated into the 57 sub-tasks. Figure g demonstrates that for most tasks knowledge selection exhibits spike-like patterns on Wikipedia
19

---

Published as a conference paper at ICLR 2024
corpora and encyclopedic knowledge graphs, suggesting that the majority of tasks have few clearly relevant knowledge cards. In addition, for other tasks (e.g. juris prudence and high school U.S. history), it is not clear which knowledge cards would be most helpful, thus the selection is more spread-out. These results suggest that the selection patterns could also indicate whether a new and more in-topic knowledge card is needed for any given task.
Yes/No Template Sensitivity In the top-down approach, prompt L.MMs with "Do you need more information? (Yes/No)" to identify external knowledge required and use in-context learning to encourage well-calibrated responses. Since language models are sensitive to minor changes in prompts, devise two more questions: "Is more information needed here?" and "Would you like additional information?" ?" and report the results on the 2-way misinformation task in Figure 7 It is demonstrated that LLMs give moderately consistent responses: 79.9% of cases received unanimous yes or from the three prompts, while 20.1% % examples received mixed results. This suggests that a potential improvement KNOWLEDGE CARD is employ multiple yes/no questions to probe knowledge limitations and use an ensemble of answers to improve robustness.
1200
1000
400
200
yes
2yes. 10
no. y05
no
Figure 7: Yes/No questions in top-down are mostly consis- tent across three prompt tem- plates, while there is space for improvement future work.
Factuality Scores of Knowledge Cards We use the MMLU datasets queries prompt different knowledge cards, generate knowledge documents, and evaluate their factuality with the factu- ality selector 2.2). We illustrate the factuality score distributions of different knowledge cards in Figure which shows that different knowledge cards have varying inherent factuality. We hypothesize that the distribution of factuality scores given by the factuality knowledge cards.
selector could guide efforts to evaluate the quality of
Knowledge Card Accumulation We expect KNOWLEDGE CARD to perform better when relevant knowledge cards are gradually added to the system. To this end, we gradually add five knowledge cards (PubMed, IMDB, Book- Corpus, News, and Wikipedia) to KNOWLEDGE CARD and evaluate performance with the misin- formation dataset, 2-way setting, bottom-up ap- proach, and the ChatGPT model. 7able7 strates that the addition of knowledge cards, es- pecially do-domain ones (News this case), indeed helpful in improving the base large language
LM Training
Inference Stage
Hyperparameter
Value
Hyperparameter
Value
LEARNING RATE
26-5
n1
WEIGHT DECAY
lo-5
n22
MAX EPOCHS
10
BATCH SIZE
32
MAX LTERATION
OPTIMIZER
ADAM
TEMPERATURE
ADAM EPSILOO
Te-6
DEFAULT
CODEX
ADAM BETA
0.9. 0.98
WARMUP RATIO
0.06
model. Table 6: Hyperparameter settings.
Setting
# card
BAce
MaF
Working Examples We present the specific prompts. generated knowledge documents, and prompts for the bottom-up approach, and the top- down approach with automatic and explicit selection in Tables 10 and respective..
VANILLA
80.1
70.5
+ PUBMED
80.7
70.6
IMDB
80.6
71.2
+ BOOKCORPUS
82.3
72.9
+ NEWS
4
85.7
73.1
WIKIPEDIA
76.5
75.3
EXPERIMENT DETAILS
Table 7: KNOWLEDGE CARD performance on the two-way misinformation dataset when five knowledge cards are gradually added.
Algorithm Details We present an algorithmic sum- mary of the bottom-up and top-down approach in Algorithm and
Knowledge Cards Domains We train a total of 25 knowledge cards from the following corpora and domains: one billion tokens (Chelba et al. 2013), ACL papers (Lo et al. 2020), commonsense knowledge graph ATOMIC (West et al. 2022) , book corpus (Zhu et al. (2115). ConceptNet (Speer al. 2017), biomedical knowledge graph UML. (Zhang et al. (021). Gutenberg (Rae et al. 2019).
20

---

Published as a conference paper at ICLR 2024
II open-book setting
Question: Who won the senate race of Oregon the 2022 U.S. midterm elections?
Answer: Ron Wyden
II two-way setting
Question: Who won the 24th congressional district of Texas the 2022 U.S. midterm elections?
A. Jan McDowell B. Beth Van Duyne
Answer: B
I1 four-way setting
Question: Who won the 6th congressional district Pennsylvania the 2022 U.S. midterm elections?
A. Christopher Hoeppner B. Doug Mastriano C. Chrissy Houlahan D. Guy Ciarrocchi
Answer: C
Table 8: Examples of the MidtermQA dataset for the three settings.
IMDB movie reviews Waag et al. 2023), political knowledge graph KGAP Feng et al. 2021 ).
legal contracts (Hendrycks al. 2021), math problems (Saxton al. 2019), midferm election news (ours), open subtitles political news corpora POLITICS (Liu et al. 2022c). biomedical literature
RealNews (Zellers et 2019). Reddit (Feng al. 2023b), Twitter (Feng et al. 2022). Wikidata
knowledge graph /Vrnndečić & Krötzsch, 2014), Wikipedia dump] YAGO (Pellissier Tanon et al.
2020p, and Yelp reviews [ For knowledge graphs, we follow Feng al. (2023a) to construct textual corpora and use them as training data.
Hyperparameters We present hyperparameter scttings in Table 6
Dataset Details 1) The MMLU dataset (Hendrycks al., 2020) contains total of 15,908 four- choice questions further divided into 57 sub-tasks in four domains: humanities, social sciences, STEM, and others. The official dataset also provides a demonstration set, i.e. 5-shot examples each sub-task to enable few-shot in-context learning. We follow the official demonstration set and test set our experiments. 2) The LUN dataset (Rashkin et al.) 2017) is a misinformation detection dataset with two- or four-way classification settings, either with true/false only or fine fine-grained categories of trusted, hoax, propaganda, or satire. We use the official test set (Hu et al. (021)) with 2,999 examples throughout the experiments. 3) We curate MidtermQA, QA dataset focusing the 2022 U.S. midterm elections to evaluate KNOWLEDGE CARD'S 's ability for temporal knowledge update. Specifically, first collect the results of the 510 races in congressional, senate, gubernatorial elections in the 2022 midterms. We then construct three datasets: a) open-book, where we ask LLMs directly answer the name of the election winner for given race, b) two-way, where we ask LLMSs to choose the winner from the two front runners, and c) four-way, where we increase the difficulty by including two other politicians contesting the same state create a distraction. We present examples of the MidtermQA dataset in Table 8
Computation Resources Details We used a GPU cluster with 16 NVIDIA A40 GPUs, 1988G memory, and 104 CPU cores for the experiments. Training knowledge cards took from around 7 hours to 10 days depending on the training corpora size. For the black-box LLMs, we used the OpenAI API to access CODE- DAVINCI-002, TEXT-DAVINCI-ONS and GPT-3.5-TURDE in the experiments.
htt.err//sttub.ooorrrssssss/ssss dataset
down loader
https://www.yell..omm///
21

---

Published as a conference paper at ICLR 2024
jin-context examples with the formatz
Knowledge:
San Mateo located the northwest of California... ... Dianne Feinstein, senior
senator from California, is rumored to retire Tom Brady returedd to his hometown of San Mateo
Question:
Who is the senior senator from Tom Brady's s birth place?
Answer:
Table 9: Prompt example for the bottom-up approach. Different color boxes indicate knowledge documents generated by different specialized L.S..
jin-context examples with the format,
Question: Who the senior senator from Tom Brady's 's birth place? Do you need more information? (Yes or No)
Yes
What kind information do you need?
The state Tom Brady is from.
Knowledge: Tom Brady returned to hometown San Mateo, CA.
Do you need more information? (Yes No)
No
Answer:
Table Prompt example for the top-down approach with automatic selection. Italic texts indicate that this field is generated by black-box LLMs.
jin-context examples with the same format,
Question: Who the senior senator from Tom Brady's birth place? Do you need more information? (Yes No)
Yes
Choose an information source from the following: sports, biomedical literature, NLP papers, book corpus.
sports
Knowledge:
Tom Brady returned to his hometown of San Matco, CA
Do you need more information? (Yes or No)
No
Answer:
Table 11: Prompt example for the top-down approach with explicit selection. Italic texts indicate that this field generated by black-box LLMs.
22

---

Published as a conference paper at ICLR 2024
Algorithm 1: Bottom-Up Approach
Data: question q; in-context examples prompt Steii knowledge cards ,.., relevance, pruning, and factuality selector Øee Oprames Sfacc
Result: answer string 8ans
PROMPT Stel
KNOWLEDGE LISTT = 0]
for spec C do
n1))
end
KNOWLEDGE LIST = Φve(q, KNOWLEDGE LIST, 122)
KNOWLEDGE LIST ( (KNOWLEDGE LIST)
KNOWLEDGFFLJST = (NOOWWWEWEED,, ft))
PROMPT "Knowledge:
for 8 E KNOWLEDGE.LIST do
PROMPT += s
end
PROMPT += "Question:" + q "Answer:"
Sans L.M(PROMPT)
return 8ans
Algorithm 2: Top-Down Approach
Data: question q; in-context examples prompt 8kri knowledge cards C (spec)..... spec,, knowledge card names S = {s....., }n); max trial k;; relevance and factuality selector Pref Ofacri binary flags AUTO and EXP
Result: answer string 8ans
PROMPT 8irl
PROMPT += "Question:" +
while i < 1 do
PROMPT "Do you need more information? (Yes or No)
RESPONSE = [.MM(PRMPP))
RESPONSE "Yes" then
if AUTO then
PROMPT += "What kind of information do you need?"
RESPONSE = LLM(PROMPT)
spec = (RESPONSE, (s1...., Sn), TOP-K 1)
end
EXP then
PROMPT += "Choose an information source from the following:
for s S do
prompt
end
spec LLM((ROMPT) end
KNOWLEDGE Ofjucr (spec (g.n))nn))))) m11.1) 1)
PROMPT += "Knowledge: KNOWLEDGE end
if RESPONSE == "No" then
break end end
PROMPT += "Answer:"
Sans LLM((ROMPT)
return 8
23

---

Published as a conference paper at ICLR 2024
electrical engineering
management
college CS
nutrition 644.4444440
24
medical genetics
high school european history 6. 03
12
GO
computer security
college biology 67 6.
0.7
30
03 O
38
high school biology
high school microeconomics
60
00
04
high school world history
college physics
high school geography
clinical knowledge
high school macroeconomics
marketing oM
20 mo
security studies
human aging
high school CS
elementary mathematics
conceptual physics
anatomy
global facts
foreign policy
college chemistry
logical fallacies
sociology 50
high school us history
miscellaneous
college medicine
virology 64
D6
36 86 3.0
28
jurisprudence
prehistory
6a
36
D6
66 22 66 63
22 29 so
business ethics
international law
60 6. 60
27
25 383 LT 66 66
high school statistics
high school psychology
14
professional psychology
professional accounting
Feles
astronomy
high school mathematics
professional medicine
econometrics
high school government and politics
machine leamning
355
03
3000
moral disputes
38
high school physics
37
philosophy Go
formal logic
human sexuality
J8
world religions
20 SELEE 1ecee
college mathematics 14
public relations
abstract algebra
40 10 De
Co 2020. 22-31-0 10-0020 realnews _2
high school chemistry
one billion tokens
ACL papers
atomic
book corpus
ConceptNet KG
biomedical
Butenberr
MMD
political KG
legal contracts
math
apensubtities
POLITICS
pubmed
realnews
realnews
realnews
reddit
twitter
wikidata
Yago
Wikipedia yelp
Figure 8: Statistics of knowledge card selection in the top-down approach with automatic selection across sub-tasks in the MMLU benchmark. Encyclopedic knowledge graph YAGO and Wikipedia are generally the most adopted knowledge cards.
24

---

Published as a conference paper at ICLR 2024
pubmed
yelp
realnews_1
wikipedia
realnews_2 2
realnews
realnews
acl adl_papers
math
1B
kgap
cpnet
gutenberg
POLITICS
reddit
IMDB
ddb
wikidata
yago
midterm
bookcorpus
Legal_contracts
atomic
opensubtitles
twitter
0.2
0.4
0.6
0.8
1.0
Figure 9: Factuality score distributions of the 25 knowledge cards when prompted with questions in the MMLU benchmark. Different knowledge cards do have varying factuality score distributions.
25


Reviews:
[{"review_id": "Nq7aNSXR2T", "reviewer": "Reviewer_PFBW", "summary": "This work proposes knowledge cards which are essentially language models finetuned on specific domains. These knowledge cards can be probed to generate or recite information the domain-specific LMs have memorized (for a given question) without having to explicitly store encoded representations for each document separately. The authors also propose three knowledge selectors which heuristically govern how to aggregate knowledge from different knowledge cards to generate a final answer. The three knowledge selectors include: 1.) relevance selector chooses relevant (generated) documents given a query 2.) pruning selector to summarize generated documents to fir in the given context length and 3.) factuality selector is used to filter out hallucinating documents based on some entailment score. The authors test their system on several tasks including MMLU, MidtermQA and LUN for hallucination detection.", "questions": "1.) It is unclear how information will be stored in a knowledge card for unseen questions at test time\n\n2.) It appears that an entailment classifier is being used to obtain factuality score, can you elaborate how it is trained?\n\n3.) Can you elaborate more on how the MidtermQA dataset was curated? \n\n4.) In bottom up approach (Figure 1) how would a passage about \"San Mateo's senior senator\" be looked up on prompting with just the query \"Who is the senior senator of Tom Brady’s birth place?\". The term \"senior senator\" will not be sufficient as that would yield a very large number of documents and \"San Mateo\" term will only be obtained after first step in multi-hop retrieval.\n\n5.) Unlike bottom-up, top-down approach seems iterative where the classifier \"Do you need more information?\" is used to stop iteration. What does the prompt for this look like? How does it perform on a held-out set.\n\n6.) How is it ensured that all the knowledge cards are being useful? It has been shown that wikipedia can often answer simple questions from other domains, which MMLU dataset often tests.\n\nTypos:\nhandful of knowledge cardds ->  handful of knowledge cards", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.) The idea to train individual language models seems novel. Modular knowldge organization can be helpful for making progress on continual learning.", "weaknesses": "1.) The paper is missing many details and is hard to follow at times, especially in Section 2.1 and 2.3. (specific issues in questions section)\n\n2.) Existing models that employ similar ideas of modular knowledge organization https://arxiv.org/pdf/2108.05036.pdf, https://arxiv.org/pdf/2203.06311.pdf and datasets that test temporal aspects https://arxiv.org/pdf/2110.03215.pdf are not compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81O2YPpesS", "reviewer": "Reviewer_fqxe", "summary": "The authors propose a modular framework augmented with a domain-specific knowledge module called Knowledge Card. They introduce two scenarios (top-down and bottom-up) for knowledge integration. They demonstrate consistent performance improvement across multiple datasets compared to existing retrieval-augmented language models and generated knowledge prompting approaches. They also provide an analysis of each proposed module.", "questions": "In Table 1, 2, and 3, there are different performance trends among the three Knowledge Card variations. Do the authors speculate about the potential reasons behind these results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors demonstrate improvement across various benchmarks such as general-purpose knowledge QA, misinformation detection, and midterm QA compared to existing retrieval-augmented language models and generated knowledge prompting approaches.\n- Through an ablation study, the authors demonstrate the effectiveness of each module and conduct an analysis for each module.", "weaknesses": "- Language model based modules appear to entail potential risks. For example, knowledge cards based on a language model necessitate a retrieval-based factuality selector, and inaccuracies can arise in LLM-based yes/no decisions during the top-down knowledge integration process.\n- One of the major differences between this work and existing work based on retrieval or generation is the utilization of knowledge cards from multiple domains. Therefore, it would be beneficial to demonstrate performance trends based on the gradual accumulation of knowledge cards or the level of granularity of knowledge cards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XXHlBvxLUN", "reviewer": "Reviewer_HvjK", "summary": "The paper introduces \"Knowledge Card\", a modular framework designed to augment large language models (LLMs) with up-to-date, factual, and relevant knowledge. Knowledge cards are trained on specific domains and sources. These knowledge cards act as parametric repositories and are selected during inference to provide background knowledge to the base LLM. The paper claims state-of-the-art performance on six benchmark datasets, demonstrating the effectiveness of the Knowledge Card framework in dynamically synthesizing and updating knowledge across various domains.", "questions": "The bottom-up approach and top-down approach mentioned in the text each have their own advantages and disadvantages. Is it possible to combine the two?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. As an alternative for retrieval based method, knwoledge card allows for the dynamic synthesis and updating of knowledge from various domains, which is a significant advancement over static general-purpose LLMs.\n2. The method demonstrates state-of-the-art performance on six benchmark datasets. The results indicate that it is beneficial for numerous knowledge-intensive tasks, especially in situations that require the latest and accurate information.\n3. Designing the relevance selector, pruning selector, and factuality selector to evaluate dimensions of relevance, brevity, and factuality encompasses a broader and more extensive range than considered in previous methods.", "weaknesses": "The benchmarks and datasets tested in the paper primarily focus on natural language understanding tasks, lacking more results on generative tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)