# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 4 Pretraining--Strategies and Objectives
### Learning Strategies
#### 4.2.4 Knowledge-Enhanced Pretraining  
Commonly, plain text is used during pretraining, which lacks explicit linguistic and world knowledge representation. The plain text also lacks structured representation and does not have the explicit grounding to entities from the real world. Such representations fail to capture the entities and the facts among those entities. ERNIE [69] incorporated structured knowledge facts during pretraining using knowledge graphs to address this issue. Using knowledge graphs, ERNIE could exploit syntactic, knowledge, and lexical information, which helped it perform better on several knowledge-driven tasks. In KnowBert [70], multiple knowledge bases were used during pretraining to enhance the representations further. In relationship extraction, entity typing, and word sense disambiguation downstream tasks, KnowBert demonstrated improved perplexity and better ability to recall facts after it was integrated with WordNet and a subset of Wikipedia knowledge bases. To learn commonsense knowledge, [PERSON] [71] proposed generative and contrastive objectives and incrementally pretrained the model. As its parameters can capture concept-centric commonsense understanding and reasoning, it does not have to rely on external knowledge graphs. The results demonstrate how, despite being trained on a minimal dataset, CALM outperformed the T5-base model on all commonsense-related datasets. To accelerate the pretraining process, [PERSON] [34] proposed a knowledge inheritance technique where it uses knowledge from existing pretrained models instead of training the models from scratch. Instead of self-supervised, [PERSON] [72] proposed a weakly supervised pretraining objective that helped it incorporate knowledge of real-world entities, where it achieved significant improvements in fact completion and two entity-related tasks. [PERSON] [73] jointly optimized the language modeling and knowledge embedding (KEs) objective. As a result, language representation and factual knowledge were better aligned to produce more effective text-enhanced KEs. Similarly, CoLAKE [74] used extended MLM objectives to learn contextualized representation for language and knowledge jointly. Instead of just using entity embeddings, CoLAKE also considers the knowledge context of those entities derived from large-scale knowledge bases. Using these knowledge contexts along with the language context information, a word-knowledge graph was constructed to deal with the heterogeneity of language and knowledge context. Experimental results demonstrated the effectiveness of CoLAKE on knowledge-required tasks after it was pretrained on the large-scale word-knowledge graph. When injecting knowledge information, previous methods mainly updated the original parameters of the pretrained models. This works fine if only one knowledge base is to be injected. If multiple knowledge bases are injected, the history of previously injected knowledge gets erased. K-ADAPTER [75] overcame this issue by using a neural adapter for each kind of infused knowledge, where there is no information flow between adapters. Hence, they can be trained in a distributed way. K-ADAPTER used this framework that keeps the pretrained model's original parameters fixed, so the parameters that learned from the old knowledge base are not affected after injecting the new knowledge base. K-ADAPTER supports continual knowledge infusion development, and, as adapters are smaller, the model scales much more favorably. As a case study, after injecting K-ADAPTER with two kinds of knowledge, results on three knowledge-driven tasks brought further improvements. Ref. [46] demonstrated how finetuning with annotated data and consulting external knowledge sources led to significant improvements, especially in the model's safety and factual grounding aspects. These responses, grounded on external knowledge, were first filtered (for safety) before ranking them regarding quality score. LaMDA demonstrated how this quality was improved as the model was scaled.