Title: How do Large Language Models Learn New Domain Knowledge?
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, continued pre-training, domain knowledge

Abstract: Despite the success of pretraining, it's not clear how large language models (LLMs) learn complex domain knowledge. We design controlled experiments to measure how they learn a new domain in the setting of continued pretraining, building upon a base level of general knowledge and intelligence. We probe the target domain knowledge at two levels: factual and compositional. First, we show that while repeated exposure improves learning with diminishing returns, multiple views of the underlying knowledge (e.g. textbooks, blogs, and stack exchange questions) yield stronger learning and generalization. We postulate that multiple views, which formulate and communicate the same knowledge differently,  construct a *scaffolding* that is key to learning in LLMs, and we suggest that this scaffolding naturally occurs in pretraining corpora. Second, we examine how learning dynamics differ with model size, post-training, and the design of data replay. Third, we find that LLMs can possess only a partial understanding of the prior knowledge required for a domain, and bridging these gaps markedly increases learning.