AcKnowledge: Acquired Knowledge Representation by Small Language Model Without Pre-training (2024)

Abstract: Large language models (LLMs) are pre-trained on enormous amounts of text data and show acclaimed success in knowledge representation. However, there are two bottlenecks with this approach. (1) Pre-training data cannot be regularly updated once the models are deployed, and it is not very fruitful if the model cannot represent updated knowledge. (2) The consistently increasing size and computational resources make it difficult for noncommercial and individual researchers to finetune and scale these language models. Major LLMs with external knowledge are also proprietary. In this paper, we propose AcKnowledge, a framework wrapped around a small, non-pre-trained language model for an open-domain question-answering (QA) experiment. AcKnowledge learns relevant knowledge from the internet via meta-learning based on user questions, and re-learns from user feedback if knowledge is misrepresented. Our efficient knowledge representation framework avoids pre-training overhead while enabling updated information. Benchmarking shows competitive performance against similarly sized state-of-the-art (SoTA) LLMs on gold standard QA datasets, demonstrating the potential of integrating internet search and user feedback for improved performance and generalizability. The repository of the work is available at https://github.com/SouravD-Me/ AcKnowledge---KnowledgeLM-ACL-2024.

heading: 1 Introduction
text: The excellent performance of large language models (LLMs) in various natural language processing (NLP) tasks can be mainly attributed to their ability to capture and represent knowledge from extensive pre-training on massive text corpora (Chang et al., 2024; Min et al., 2023). However, the outdated nature of data for pre-trained knowledge can limit their adaptability to new information or recent
events (Kazemnejad et al., 2023). Traditional methods for updating LLMs, such as continuous pretraining on the latest data or fine-tuning, are computationally expensive and time-consuming (Tian et al., 2023).
To address these limitations, we propose AcKnowledge, a novel framework that equips a small language model (SLM) with the dynamic capability to acquire and represent knowledge without conventional pre-training. Our approach exploits real-time web search and meta-learning (Xie et al., 2023; Li et al., 2020) to enable an SLM to learn new information efficiently. Upon receiving a user’s question, the topics are extracted using Latent Dirichlet Allocation (LDA) (Blei et al., 2003). These topics are then transmitted to the language model (LM), which uses these topics as keywords to perform a targeted online search and uses meta-learning (Lin and Chen, 2020) to acquire relevant knowledge. The acquired knowledge is then summarized (Moratanch and Chitrakala, 2017) and presented to the user as a concise answer.
AcKnowledge also integrates a user feedback mechanism to ensure authenticity and reliability for generated answers. Negative feedback triggers a new search iteration to find more accurate information, while positive feedback fortifies the learning.
83
User feedback plays a vital role in AcKnowledge’s learning loop in augmenting the acquired knowledge for increasingly more correct and factual answers. We evaluate AcKnowledge’s performance by benchmarking it against similarly sized LMs in open-domain QA tasks, demonstrating competitive results despite the absence of traditional pretraining.
The main contributions of this paper are:
• We propose AcKnowledge, a novel opendomain QA system that utilizes a non-pretrained SLM to dynamically acquire and represent knowledge from the internet based on user questions. • A meta-learning algorithm is implemented to enable the language model to efficiently learn from search results and refine its knowledge representation through user feedback. • The framework is designed for users to initiate re-searching for answers if the initial response is misrepresented, enhancing the reliability and user control over language model outputs. • The effectiveness of AcKnowledge is demonstrated through extensive benchmarking against similar language models, showcasing competitive performance without relying on pre-training. • The quality of the generated answers is meticulously analyzed, showcasing the impact of real-time knowledge acquisition in adaptable SLMs for efficient QA.

heading: 2 Relevant Works
text: Knowledge representation is indispensable for NLP systems to understand meaning and perform reasoning. The statistical approaches in the early last decade like word embeddings (Mikolov et al., 2013; Chen et al., 2013) learned vector representations but lacked explicit knowledge modeling. Further advances in integrated neural networks with symbolic knowledge graphs and ontologies through techniques like graph convolutional networks (Kipf and Welling, 2016).
Hybrid neuro-symbolic methods show promise in injecting knowledge into large pre-trained language models like RoBERTa (Liu et al., 2019) to improve common sense reasoning (Bosselut et al., 2019) and factual grounding (Guan et al., 2020). Multimodal learning from transformer architecture has also been in research focus (Tan and Bansal, 2019). Key challenges in representing knowledge
often include effective representation and contextsensitivity to the core topic (Verma and Bergler, 2023), performing reasoning over learned representations (Saha et al., 2022), and generating logical forms (Hu et al., 2022). Promising directions also involve meta-learning for fast knowledge adaptation (Zhao et al., 2022) and graph embedding methods for knowledge representation (Cao et al., 2024).

heading: 3 System Framework
text: Our fundamental objective is to develop AcKnowledge with the ability to dynamically retrieve and adapt relevant knowledge seamlessly from the internet. The proposed system comprises several key components that work in tandem to facilitate this process. The overview of our system is illustrated in Figure 2. The primary components of the framework are discussed in the following sections.

heading: 3.1 Answer Retrieval from Internet Search
text: To accumulate external knowledge from the internet, our approach employs a two-stage information retrieval process. First, LDA is implemented for topic extraction from the user question. LDA serves as an unsupervised clustering model for the revelation of topics in a collection of documents (Alhawarat and Hegazi, 2018; Zong et al., 2021). It can be formalized as a probabilistic generative model. In this model, the distribution of topics for any number of questions can be represented as ΩQ ≈ Dirichlet(δ), where Ω is the distribution parameter. The Dirichlet distribution is used here to guide the distribution of topics from tokens, and the parameter δ controls the sparsity of the distribution. Second, these topic words are transmitted to the language model. Using these topics as keywords, it uses the Google search API to retrieve a set of relevant passages, such as P from the search results.
Here, we employ a dense passage retrieval technique (Karpukhin et al., 2020) to rank and select the most relevant passages. We encode each passage Pi to obtain a sequence of dense vector representations Pi = (p1,p2, . . . ,pn), where pn ∈ Rdemb . Here, Rdemb is used for the dimensional embedding in real space of the n-th token in Pi. Hereafter, these passages are passed through the metalearning module to learn from them as potential answers to the users’ questions.

heading: 3.2 Meta-learning for Search Results
text: To effectively utilize the information retrieved, we employ a meta-learning algorithm that learns from the retrieved passages. We use the modelagnostic meta-learning algorithm (MAML) (Lee et al., 2022), which has shown promising results in natural language understanding (NLU) scenarios.
For retrieving information from the passages P, the proposed language model M adapts to these passages from meta-learning. As Pi represents a retrieved passage, the aim is to accumulate the sequence of texts from it and send it to M. Adapt all passages from the search results for each question, by minimizing the meta-objective minθ EPi∼P [ LMθ′ (Q,A) ] . Here, θ represents the parameters of the language model, E is the expected value of the loss function for the distribution of passages from search results, and L denotes the cross-entropy loss function. The meta-learning algorithm updates the model parameters θ by taking a gradient step on each passage Pi, resulting in adapted parameters θ′. The adapted model Mθ′ is then evaluated on the original question-answer pair (Q,A). Internally, the meta-learning process is further decomposed into two stages; the inner loop and the outer loop. In the inner loop, for each search result in Pi, the model parameters θ are updated using gradient descent to minimize the loss
specific to the task LMθ(Pi):
θ′i = θ − α∇θL(Mθ,Pi) (1)
This inner loop uses the learning rate α to update itself and allows the model to adapt to the information contained in the passages. The updated parameters θ′i are specific to each passage Pi.
In the outer loop, the meta-objective is optimized across all search results:
min θ
N∑
i=1
LMθ′i(Q,A) (2)
The outer loop update aggregates the losses from the adapted models Mθ′i and updates the global parameters θ to minimize the expected loss across all search results.
By iterating between the updates of the inner loop and the outer loop, meta-learning enables the language model to efficiently incorporate the retrieved passages from the search results and generalize to unseen questions. The adapted parameters θ′i capture the question-specific information from each search result, while the global parameters θ learn to adapt to new information.
The MAML algorithm emphasizes learning a good initialization of the model parameters that can rapidly adapt to new questions with just a few gradient steps. This is particularly advantageous in QA tasks, where the framework must quickly
assimilate relevant information from the search results to generate accurate answers without relying on extensive pre-training.

heading: 3.3 Language Model Development
text: We build the language model in the AcKnowledge framework from scratch based on the fundamental transformer architecture (Vaswani et al., 2017). The model consists of a parameter size of just 125 Million, with a multi-layer encoder and decoder, with each layer employing multi-head selfattention mechanisms to capture long-range dependencies in text sequences.
The model encoder processes the input keywords from a question K = {k1, k2, . . . , kn}, while the decoder generates the corresponding answer A = {a1, a2, . . . , am}. This process is enhanced by meta-learning from retrieved passages. Both the encoder and decoder consist of multiple layers, each containing a multi-head self-attention sublayer and a position-wise feedforward sublayer.
In this architecture, the input sequence x = (x1,x2, . . . ,xT ) is mapped to token embeddings, with T representing the length of the sequence. The encoder transforms this input into a sequence of continuous representations Z = (z1, z2, . . . , zi), which the decoder then uses to generate the output sequence Y = (y1, y2, . . . , ym).
The input token embeddings are the sum of token embeddings and positional embeddings:
xT = W tok[xT ] +W pos[T ], (3)
where W tok and W pos are embedding matrices for tokens and positions, respectively.
Multi-head attention is a key component, calculated in multiple heads H . Each head computes attention scores using query, key, and value projections:
A(h) = Softmax
( Q(h)K(h)
⊤ √ dk
) V (h), (4)
where Q(h),K(h),V (h) are the projections, and dk is the dimension of each head.
These attention scores are concatenated and linearly projected:
MultiHead(x) = Cat(A(1),A(2), . . . ,A(H))WO, (5) where WO is the projection matrix. The position-wise feed-forward network (FFN) processes each token independently:
FFN(x) = max(0,xW 1 + b1)W 2 + b2, (6)
with learnable parameters W 1, b1,W 2, b2. Both the attention and feed-forward layers use residual connections and layer normalization:
x′ = LayerNorm(x+ MultiHead(x)), (7)
x′′ = LayerNorm(x′ + FFN(x′)), (8)
ensuring stability and efficiency. The final output, after layer normalization and a linear projection, provides the logits for next-token prediction:
y = LayerNorm(x′′)W y + by, (9)
with W y and by as learnable parameters. The model minimizes the cross-entropy loss between predicted logits y and target tokens t:
L = − T∑
i=1
log p(ti|x,θ′), (10)
where θ′ represents all learnable parameters of the model from meta-learning, and p(ti|x,θ′) is the predicted probability of the i-th target token given the input sequence x and the model parameters θ′.

heading: 3.4 Knowledge Summarization
text: After incorporating the retrieved information through meta-learning, the language model generates a summary or answer to the user’s query. To ensure that the generated answer is concise, we employ extractive summarization. Here, the most salient sentences from the summary are selected and combined to form the final answer. We use the centrality-dependent ranking approach (Liang et al., 2021) to identify and rank the most important text segments.
Here, a sentence-document weight is added to the sentence centrality score, allowing more attention to different facets in the generated answer summaries. This approach is applicable for long and multi-document scenarios, as in our experiment the summaries with multiple information generated by the language model. These text segments are then merged and represented as the final answers. We do not paraphrase the answers by employing abstractive summarization. This reduces the scope for representing misinformation (Krishna et al., 2024). Finally, extractive summarized content is presented to users as final answers. Table 1 represents question-answer pairs in 3 distinct aspects.
QA 1: Topic → Latest News and Events

heading: 3.5 Iterative Learning from User Feedback
text: Our approach implements user feedback in an iterative learning process to improve the quality of the generated answers and adapt to user preferences. After receiving an answer from the language model, the user can provide positive or negative feedback, indicating their satisfaction with the response.
If the users provide negative feedback, the system initiates another internet search, using the same keywords from the original query. The retrieved information and the language model’s parameters are updated based on the user’s feedback to generate a more accurate and relevant answer.
This iterative process continues until users are satisfied with the authenticity of the information in the answer. User feedback is a valuable indication that guides AcKnowledge in enhancing its understanding and generating more accurate responses with correct information.
For the user feedback mechanism, a group study is carried out. Here, the answers are evaluated by a group of 5 people consisting of 2 NLP experts, 2 researchers, and a student. These people are only the end users and are not involved in any of the experiments described in the paper.
After receiving an answer, the users individually provide binary feedback signals f ∈ {1, 0}, indi-
cating satisfaction with the answer. If any feedback is labeled as (f = 0), i.e., ‘incorrect’, M initiates a new search process, searching for the unexplored search results in the previous hop, and updates its parameters using the MAML algorithm described earlier. After any n-th iteration of iterative learning and refinement, the users provide positive feedback, indicating that the generated answer is correct and high quality. The majority of user group feedback determines each answer’s correctness or incorrectness. The user feedback on 3 distinct aspects is shown in Table 1.

heading: 4 Experiments
text: To evaluate AcKnowledge’s performance on goldstandard corpora, we use two widely-used opendomain QA datasets; Stanford Question Answering Dataset (SQuAD 2.0) (Rajpurkar et al., 2018) and Natural Questions (NQ) (Kwiatkowski et al., 2019), SQuAD 2.0 integrates approximately 50,000 adversarial-designed unanswerable questions to mimic responsive questions. For good performance on SQuAD 2.0, ideally, systems should recognize when the text does not support a response and refrain from responding. NQ is a large-scale dataset with more than 300,000 question-answer pairs based on real-world Google search queries.
It includes a diverse range of topics and question types, with an average of 16.5 tokens per question and 77 tokens per answer.
We preprocess the datasets by tokenizing the text using the WordPiece tokenizer (Hussain et al., 2023) and converting the tokens to their corresponding embeddings using Word2vec (Mikolov et al., 2013). The preprocessed datasets are then split into training, validation, and test sets with a ratio of 80%, 10%, and 10%, respectively.

heading: 4.1 Experimental Settings
text: We implement AcKnowledge using the PyTorch framework (Paszke et al., 2019). The encoder and decoder of M consist of 6 layers each, with a hidden size of 768 and 8 attention heads. The model is trained in gold standard QA corpora using the Adam optimizer (Kingma, 2014) with a learning rate of 0.0005 and a batch size of 32. The maximum sequence length is set to 1024 tokens. We apply gradient clipping with a maximum norm of 1.0 to stabilize the training.
For the internet search component, we use the Google search API to retrieve the first 10 search results sequentially for any question. The MAML algorithm is applied with a learning rate α = 0.001 and a maximum of 5 adaptation steps. If a user initiates negative feedback for any answer, this process is repeated for iterative learning.

heading: 4.2 Quantitative Evaluation Metrics
text: We evaluate the performance of our approach using standard empirical evaluation metrics. For such purpose, we employ metrics such as semantic fluency (Sem-FL) (Zhou et al., 2023) for quantifying the semantic coherence and meaningfulness of the outputs, Length fluency (Len-FL) (Zhou et al., 2023) for evaluating the ability to generate outputs of appropriate verbosity, F1 score (Tan et al., 2016), computed by comparing the word-level overlap between the predicted and ground truth answers, binary accuracy (Y/N) (Choi et al., 2018), for verifying the accuracy in binary answerable questions, exact match (EM) (Chen et al., 2024), the percentage of predictions that exactly match the ground truth answer, BLEU (BL) (Chen et al., 2023), the metric widely used in machine translation that measures the n-gram overlap between the predicted and reference answers, ROUGE (RG) (Schluter, 2017), the metric commonly used for summarization tasks, which evaluates the quality of the generated summaries based on n-gram overlap with
reference summaries, and METEOR (MR) (Chen et al., 2019), for analyzing multiple matching criteria, including exact word matches, stemmed word matches, synonyms, and paraphrases.
Furthermore, we perform statistical significance tests such as the Wilcoxon signed-rank test (Narayan et al., 2023) to determine if the performance differences between the SoTA models and AcKnowledge are statistically significant (ref. Figure 3(b)). This significance is measured using the p value. Let µLM and µM denote the mean scores for the SoTA models and AcKnowledge, respectively. The null hypothesis H0 for the performance below the significance threshold and the alternative hypothesis H1 for the performance above the significance threshold are defined as follows:
H0 : µLM = µM (11)
H1 : µLM > µM (12)
We set a standard significance threshold for p value (0.05) (Smucker et al., 2007) and calculate to determine below and above-significance threshold performance for all the SoTA models compared with AcKnowledge.

heading: 4.3 SoTA and Baseline Comparisons
text: The performance of AcKnowledge is compared with several state-of-the-art (SoTA) models for QA. Small language models are scarce for downstream tasks. However, several SoTA models are selected based on comparable parameter sizes and considering their efficiency in QA. Such models are BLOOM (Muennighoff et al., 2022), Open Pretrained Transformer (OPT) by Meta AI (Zhang et al., 2022), ELECTRA(Clark et al., 2019), Finetuned LAnguage Net (Flan-T5-base) by Google (Chung et al., 2024), DeBERTaV3-Base (He et al., 2022), GPT-Neo (Kashyap et al., 2022), and MiniLM (Wang et al., 2020).
We do not explicitly fine-tune these models, but rather deploy with their recommended setup with default hyperparameters. To benchmark the performance of SoTA models and AcKnowledge, we compare this evaluation setup against several strong baseline models, such as BERT (Devlin et al., 2018) for checking the performance based on pre-training and fine-tuning on the QA datasets without any external knowledge or user feedback, SpanBERT (Joshi et al., 2020) for evaluating the span of answers against each question, and RAGBase (Braunschweiler et al., 2023) for comparing
the performance of vanilla models (including ours) with retrieval augmented generation (RAG)-based language model.

heading: 5 Results and Discussion

heading: 5.1 Benchmarking Results
text: In this section, we present the results of our proposed approach and compare it with various baselines and SoTA models. We also provide a qualitative analysis of the generated answers, the impact of user feedback on model performance, and the limitations and potential improvements of our approach.
Table 2 presents a comprehensive evaluation of SoTA and baseline language models on SQuAD 2.0 and NQ. The results demonstrate that AcKnowledge consistently outperforms the other models across both datasets and all evaluation metrics. Despite having a smaller parameter size compared to the other models, AcKnowledge achieves superior performance, highlighting its efficiency and effectiveness in QA tasks. The bold values in the table emphasize the superior performance of AcKnowledge.
Among the baseline models, the BERT-base exhibits the best performance, followed by SpanBERT and RAGBase. However, their performance falls short of that of the SoTA models, indicating
the more recent advances made in QA using such models.

heading: 5.2 Qualitative Analysis
text: Scrutinizing further into the SoTA comparisons, a human evaluation is carried to assess the quality of the generated answers using several qualitative metrics. A blind evaluation of the answers of all models on gold standard datasets is performed by the same group of 5 people described earlier. Here, 100 questions are randomly selected from the test sets, and the quality of the answers generated by each model is manually evaluated.
The qualitative evaluation metrics are the maximum token length (Max-Tk) supported by each model, context preservation in answers (C-Pr), correctness (Cr) (Falke et al., 2019), and completeness (Cn) (Lu et al., 2022). Context preservation is a binary metric indicating whether the model can maintain the context of the question when generating answers. Correctness measures the accuracy of the generated answers, while completeness evaluates the extent to which the answers cover all the necessary information.
The results in Table 3 demonstrate that AcKnowledge performs best in correctness and completeness. The table also highlights the importance of context preservation in QA tasks for bet-
ter answer responses. Apart from ELECTRA and MiniLM-XLMR, the rest of the models generally perform better in correctness and completeness. This depicts that maintaining context from the users’ questions plays a crucial role in high-quality and relevant answers.

heading: 5.3 Ablation Studies
text: We conduct ablation studies to assess the impact of meta-learning and user feedback in the AcKnowledge architecture.
Meta-Learning. We replace the MAML algorithm with standard fine-tuning to evaluate its significance. Results show that with meta-learning, AcKnowledge achieves 89.5% F1 and 84.2% exact match. However, without meta-learning, scores drop to 85.8% and 80.1% respectively. This highlights the crucial role of meta-learning in efficiently incorporating retrieved information and adapting to unseen questions.
User Feedback. Disabling the feedback loop and iterative learning process resulted in reduced
performance. With user feedback, AcKnowledge achieves 87.2% F1 and 83.7% exact match, whereas, without it, scores decrease to 82.6% and 77.3% respectively. Incorporating user feedback significantly enhances the framework’s understanding and answer accuracy.
These findings underscore the importance of integrating meta-learning and user feedback in knowledge representation in QA by a small language model.

heading: 6 Conclusion
text: We introduced AcKnowledge, a framework that can search the internet for updated answers to user questions, learn from the search results using metalearning, and assimilate user feedback to improve performance. Our proposed approach outperforms various SoTA and baseline models in standard QA evaluation metrics. Our approach has several potentials for language model applications. Firstly, it demonstrates the benefit of integrating internet search and meta-learning in language models to improve their answering ability. Secondly, it can also answer complex questions that require access to up-to-date and diverse information sources. Third, it can be used to develop scalable language models that can learn from user feedback to improve their performance and adapt to user preferences. There are prominent future research directions for our work. We aim to explore the prospect of scalability and robustness in larger and real-world deployable scenarios. This experiment can pave the way for developing more knowledgeable language models that can assist users in various tasks and scenarios.

heading: A Appendix
text: A.1 Ethical Statement
Although our proposed framework achieves promising results, there are currently a few limitations and potential areas for improvement. First, the performance of the framework may be affected by the quality and relevance of the search results. Search results can certainly integrate incorrect or misleading information. User feedback can preliminarily mitigate that by using ‘incorrect’ feedback. Implementing an ensemble method for more filtered information retrieval techniques could help mitigate this issue even further. Second, the framework’s ability to handle complex, multi-hop questions is currently limited. Extending the metalearning approach in the multi-episode phase to handle such questions is an important direction for our future work. Finally, the efficiency of AcKnowledge’s knowledge representation could be improved by back-end storage, preferably a vector database (Zhang et al., 2023), for efficient information retrieval. In addition, we are also working on integrating moderate to no-offensive content retrieval as knowledge during the internet search by implementing Google’s ‘safe search’ feature.
A.2 Language Model Insights
We provide the detailed architectural details of the language model developed for AcKnowledge. It is based on the standard transformer network and comprises the following components.
• Hidden size: The hidden size or embedding dimension is set to 512. This represents the
dimensionality of the hidden states and embeddings used in the model. • Intermediate Hidden Size: The intermediate hidden size in the FeedFoward layer is set to 4 times the hidden size. In this case, it would be 4 *512 = 2048. • Context Length: The maximum context length or block size is set to 256. This determines the maximum number of tokens that the model can process in a single sequence. • Heads: The number of attention heads in the multi-head attention layer is set to 8. This allows the model to attend to different parts of the input sequence simultaneously. • Layers: The model’s number of transformer blocks or layers is set to 8. Each block consists of a multi-head attention layer followed by a FeedFoward layer. • Vocabulary Size: The vocabulary size is not explicitly mentioned in the provided code. It would depend on the size of the vocabulary used during the model’s training. • Token and Position Embeddings: The input tokens are mapped to embeddings using an embedding table, and position embeddings are added to capture positional information. • Transformer Blocks: The model consists of a sequence of transformer blocks, each containing a multi-head attention layer and a FeedFoward layer. The multi-head attention layer allows the model to attend to different parts of the input sequence, while the FeedFoward layer applies non-linear transformations. • Layer Normalization: Layer normalization is applied after each multi-head attention and feedforward layer to normalize the activations and improve training stability. • Linear Output Layer: The final hidden states are passed through a linear layer to generate temporal records or logits for each token in the vocabulary.
The total number of parameters in the model is calculated to be around 124.6 million, or approximately 125 million. In addition, we implement layer normalization instead of batch normalization for our model.
A.3 Dataset Details
We offer more thorough explanations of the datasets that we utilized in our experiments. The sources from which we obtained the datasets and
the sources from which the authors originally made them available are included. For information on these sources’ licenses or conditions of use and/or dissemination, we direct readers to them. To the best of our knowledge, neither objectionable content nor names or unique identifiers for specific individuals can be found in the databases that are used.
• SQuAD 2.0: The dataset was originally released in 2018, and was made publicly available at https://rajpurkar.github.io/ SQuAD-explorer/explore/v2.0/dev/. We obtain the dataset from https: //huggingface.co/datasets/rajpurkar/ squad_v2. • NQ: The dataset was originally released in 2019, and was made publicly available at https://ai.google.com/research/ NaturalQuestions. We obtain the dataset from https://huggingface.co/datasets/ natural_questions.
A.4 Benchmarking Setup
The experiments are conducted with standard computational resources for NLP and machine learning experiments. Specifically, the hardware configuration includes an Intel Xeon CPU with a 2.20 GHz clock speed and 12 GB of RAM.
We use the NVIDIA A100 GPU (Markidis et al., 2018) for our experiments. It is built on Tensor Core architecture, up to 80 GB of GPU memory, and up to 312 TFLOPS single precision performance. The parallel processing ability of the GPU makes the meta-learning and iterative learning based on user feedback of the model more robust, as well as the benchmarking process more time-efficient. This setup ensures that the models can be trained by exploiting the high computational power and memory bandwidth of the GPU.
The training durations for all SoTA and baseline models range from 8 to 15 hours on SQuAD 2.0 and NQ datasets. BLOOM 350M takes the highest time to train on both datasets, i.e., approximately 15 hours, while BERT-base takes the lowest time to train, i.e., 8 hours. AcKnowledge takes approximately 10 hours to train on SQuAD 2.0 and NQ datasets.
A.5 Extensive Evaluation
Furthermore, we showcase the performance of the compared language models on two widely adopted
benchmarks, Massive Multitask Language Understanding (MMLU) and BIG-Bench Hard (BBH). The evaluation of all the compared models is conducted in their standard settings without any additional fine-tuning, whereas AcKnowledge is evaluated in the meta-learning (M-L) setting as proposed.
The MMLU benchmark comprises a diverse array of NLP tasks, including question-answering, commonsense reasoning, and natural language inference, among others. In contrast, the BBH benchmark is a curated subset of challenging tasks from the BIG-Bench suite, designed to assess the capabilities of LLMs in complex and specialized instructions. Table 4 presents the benchmark results for MMLU, BBH, and the average performance of each model.
Here also, AcKnowledge achieves a higher score of 45.3 on the MMLU benchmark compared to the other models, demonstrating its superior performance in language understanding on a broader scale. However, on the BBH benchmark, it obtains a score of 28.4, which is comparable to the scores of the other models, indicating similar performance.
The Average score shows that AcKnowledge has the highest overall performance at 36.9, followed by Flan T5-Base at 36.3. This suggests that the proposed framework can maintain a good balance between a strong performance on MMLU and a competitive performance on BBH.
For all the benchmarking experiments on SQuAD 2.0, NQ, MLMU, and BBH, the respective language models answer from their pre-trained knowledge. However, AcKnowledge answers with the aid of internet search and meta-learning, providing answers with the latest information and updates. Also, the user feedback in AcKnowledge for the answers is an efficient and scalable approach to continuously learning and improving the quality of answers based on real-time knowledge representation.

heading: Future Work
text: There are prominent future research directions for our work. We aim to explore the prospect of scalability and robustness in larger and real-world deployable scenarios. This experiment can pave the way for developing more knowledgeable language models that can assist users in various tasks and scenarios