SearchInstruct: Enhancing Domain Adaptation via Retrieval-Based
  Instruction Dataset Creation
Authors: name: Iman Barati; name: Mostafa Amiri; name: Heshaam Faili
Published: 2025-09-12 21:50:39

Abstract: Supervised Fine-Tuning (SFT) is essential for training large language models
(LLMs), significantly enhancing critical capabilities such as instruction
following and in-context learning. Nevertheless, creating suitable training
datasets tailored for specific domains remains challenging due to unique domain
constraints and data scarcity. In this paper, we propose SearchInstruct, an
innovative method explicitly designed to construct high quality instruction
datasets for SFT. Our approach begins with a limited set of domain specific,
human generated questions, which are systematically expanded using a large
language model. Subsequently, domain relevant resources are dynamically
retrieved to generate accurate and contextually appropriate answers for each
augmented question. Experimental evaluation demonstrates that SearchInstruct
enhances both the diversity and quality of SFT datasets, leading to measurable
improvements in LLM performance within specialized domains. Additionally, we
show that beyond dataset generation, the proposed method can also effectively
facilitate tasks such as model editing, enabling efficient updates to existing
models. To facilitate reproducibility and community adoption, we provide full
implementation details, the complete set of generated instruction response
pairs, and the source code in a publicly accessible Git repository:
[https://github.com/mostafaamiri/SearchInstruct](https://github.com/mostafaamiri/SearchInstruct)