Uncovering Latent Expertise within Large Language Models (2024)

Abstract: Recent works have demonstrated the effectiveness of self-alignment in which a large language model is aligned to follow general instructions using instructional data generated from the model itself starting from a handful of human-written seeds. Instead of general alignment, in this work, we focus on self-alignment for expert domain specialization (e.g., biomedicine, finance). As a preliminary, we quantitively show the marginal effect that generic instruction-following training has on downstream expert domains’ performance. To remedy this, we propose selfspecialization allowing for effective model specialization while achieving cross-task generalization by leveraging only a few labeled seeds. Self-specialization offers a dataand parameter-efficient way of “carving out” an expert model out of a generalist pre-trained LLM. Exploring a variety of popular open large models as a base for specialization, our experimental results in both biomedical and financial domains show that our self-specialized models outperform their base models by a large margin, and even larger models that are generally instruction-tuned or that have been adapted to the target domain by other means.

heading: 1 Introduction
text: Instruction-tuning (Ouyang et al., 2022; Wei et al., 2022; Mishra et al., 2022; Su et al., 2022) of large language models (LLMs) offers a mechanism to adeptly guide models using specific directives, thereby enhancing their versatility across diverse tasks. However, as promising as this concept might seem, it poses an inherent challenge: the substantial need for quality data (Chung et al., 2022; Wan et al., 2023; Köpf et al., 2023). The very premise of instruction-tuning hinges on the availability of well-crafted, human-annotated data, a resource that
*Work done during internship at MIT-IBM Watson AI Lab.
is both time-consuming and challenging to scale efficiently (Honovich et al., 2022; Kang et al., 2023).
When it comes to specialized domains, such as biomedicine, it is more challenging to acquire human labels, due to the need for expert annotators (Wang et al., 2023b). While adaptation through in-domain pre-training (Gururangan et al., 2020; Wu et al., 2023) has been shown to be effective, this approach requires extensive (unlabeled) targetdomain data, in addition to significant computational resources. Moreover, prior work has shown the benefits of adaptive pre-training can be less than those achieved by moderate amounts of fine-tuning data from the target domain (Bai et al., 2021).
Emerging as a promising solution to this dataintensive challenge in the context of instructiontuning is the approach of self-alignment (Wang et al., 2022a; Sun et al., 2023). By allowing LLMs to automatically generate instructional data from minimal human-authored seeds, self-alignment presents a means to harness the internal general knowledge of models, which results from extensive pre-training on internet corpora (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020), without extensive human annotations.
However, a pertinent question remains: How effective are the self-aligned models when applied to more niche domains, such as biomedicine? Given
2681
that both the initial pre-training and subsequent selfalignment are general, the knowledge embedded in LLM parameters may be a mixture of semantics and various domains. This raises questions about their effectiveness in specialized domains, despite the aims of instruction-tuning and self-alignment for cross-task generalization. In our preliminary study, however, we find that existing models such as Alpaca (Taori et al., 2023) and Dromedary (Sun et al., 2023), although aligned, exhibit only a modest degree of improvement within the specialized domains. These observations underline the need for focused approaches that can leverage the domain expertise existing in the base models, to ensure the self-generated instruction-tuning data remains both contextually appropriate and accurate.
In this work, we explore the possibility of selfspecialization (Fig. 1). Drawing inspiration from the foundational principles of self-alignment, selfspecialization goes a step further by incorporating domain-specific seed instructions and is further bolstered by parameter-efficient fine-tuning, as well as optional iterative refinement and retrieval components. Our goal is to guide models beyond generic alignment, directing them to generate data that are not just contextually fitting for a specialized domain but also maintain high accuracy.
We evaluate our self-specialized models within the biomedical and finance domains (20 datasets in total), and across a variety of base models that we specialize. Surprisingly, despite the simplicity of our approach, our results present a compelling case for self-specialization significantly outperforming the base models, and even larger models that are generally instruction-tuned or specifically pre-trained on the target domain. Notably, our self-specialized one based on MPT-30B (Team, 2023) for biomedicine even surpasses larger models (based on LLaMA-65B (Touvron et al., 2023a)), including the ones improved through selfalignment by leading methods (Wang et al., 2022a; Sun et al., 2023).

heading: 2 Preliminaries: Benchmarking Existing Aligned Models
text: To motivate our exploration of self-specialization, we first begin by addressing a fundamental question: How well do generally aligned models perform on specialized domains? While popular models, such as Alpaca (Taori et al., 2023) and Dromedary (Sun et al., 2023), have demonstrated
effectiveness in following general instructions, it remains unclear whether general alignment can also elicit expertise for a certain domain.
Investigating this, we assess the capabilities of Alpaca and Dromedary against their base model, LLaMA-65B (Touvron et al., 2023a), on a collection of benchmarks within the biomedical domain. We evaluate Alpaca as an upper bound, due to its reliance on GPT-3.5-generated datasets (Ouyang et al., 2022) via the self-instruct process (Wang et al., 2022a), unlike Dromedary, which generates instructional data from its base model. We use 10 biomedical NLP datasets (see Section 4 for details), covering a diverse set of tasks to ensure a comprehensive mix of content and also to look at the cross-task generalization, the core of instructiontuning. Table 1 summarizes the result.
We find that both Alpaca and Dromedary have only a slight (1.2 - 2.5) advantage over LLaMA in biomedicine. While they are aligned to handle a broad set of instructions, they do not seem to effectively improve their specialized domain expertise; intuitively trading their expertise for generality given finite parameters. In light of these findings, it becomes evident that for cases where we are only interested in expert domains for all our downstream tasks, there remains a large potential for improvement beyond the generic alignment. This underscores the need for a model or approach, like self-specialization, that could potentially uncover specialization while maintaining cross-task generalizability with minimal supervision.

heading: 3 Self-Specialization
text: In this section, we describe our method called selfspecialization illustrated in Figure 2.

heading: 3.1 Seed Demonstrations
text: Initially, we utilize a curated set of seed demonstrations S, consisting of a triplet (i, c, y), comprised
of instruction i, a context c (e.g., passage), and a response y, respectively. Recognizing the difficulty of acquiring domain-specific data in real-world scenarios (Bai et al., 2021), we aim for a very minimal number of seeds: only 80 for the biomedical domain and 90 for the financial domain1. We leverage established datasets such as Box (Parmar et al., 2022) for seed construction to fairly ensure quality (detailed in Section 4). These seeds capture essential domain concepts but are insufficient to cover the entirety of domain knowledge. We posit that domain-relevant information, intermingled with the vast array of other information acquired during pretraining, can be effectively accessed and better utilized through our self-specialization approach, enabling these models to enhance their performance in specialized domains. Seeds provide the primary scaffold upon which subsequent domain-specific
1While manual annotation of seed data is an assumed prerequisite for this initial step in self-alignment, we consider those numbers to be reasonable to annotate.
instructions are built.

heading: 3.2 Domain-Specific Instruction Generation
text: With the seed instructions in place, we move to generating domain-specific instructions. While these new instructions are grounded in the initial seeds, they grow to cover a comprehensive scope of the domain. Specifically, a base model Mbase, such as MPT-30B (Team, 2023) which is large enough, is prompted to produce new combinations of (i, c) given a handful of seed demonstrations which are randomly sampled from the initial seeds pool. The newly formed instructions i, coupled with their corresponding input contexts c, shape a blueprint that the model utilizes in the following stages.

heading: 3.3 Domain-Specific Response Generation
text: In this phase, it is crucial for the responses not only to be correct but also to be well-aligned with the target domain. Intuitively, as this phase is conditioned on domain-specific instructions {i}
and corresponding contexts {c}, derived from domain-specific seeds, it may be sufficient to rely on the base model itself to generate domainspecific responses. As an additional effort, we explore whether leveraging external domain-relevant knowledge would be beneficial for this case, inspired by Frisoni et al. (2022). Therefore, we optionally allow Mbase to incorporate external knowledge via a retrieval component Mret. Specifically, forming the query x as a concatenation of i and c, Mret fetches top-k relevant documents d1:k.
d1:k = Mret(x = i⊕ c)
Then, each document dj is independently paired with the query x to form a prompt to Mbase, and the final domain-specific responses y are produced from the final distribution computed by marginalizing over the probabilities of each of these kcombinations at each generation step.
p(y|x) = t∏
i
k∑
j
pret(dj |x;Mret) plm(yi|x, dj , y1:i−1;Mbase)
where pret is a relevance score (similarity) from a retriever module and plm represents the language model distribution. By integrating such external information, while domain-relevant knowledge is deemed latent within LLMs, this step further encourages the generated target responses to be more nuanced and domain-specific, leading to additional improvements (Section 5.2).

heading: 3.4 Triggering Specialization
text: Upon establishing a set of domain-specific instructions/responses, Mbase undergoes tuning using the self-generated data, adjusting its internal parameters to cater specifically to the domain’s nuances. This step is crucial, marking the model’s transformation from being generally competent to being domain-specialized while preserving cross-task generalizability, thus resulting in the final selfaligned domain-specialized model: Maligned.

heading: 3.5 Iterative Self-Specialization
text: In the spirit of continuous improvement, our approach optionally supports iterative selfspecialization via re-generating instructions and responses with the better-aligned model Maligned. This process has the potential of refining the model’s domain expertise with each iteration (of considering the previous iteration Maligned as base each time), iteratively improving its responses.

heading: 4 Experimental Settings
text: Datasets. For our primary evaluation, we employ various biomedical NLP datasets, most of which are curated in BIGBIO (Fries et al., 2022). A total of 10 different datasets are adopted to encompass a wide range of NLP tasks: Question Answering (QA), Named Entity Recognition (NER), Relation Extraction (RE), Sentiment Analysis (SA), and Document Classification (DC). Following a prior work (Parmar et al., 2022), all datasets are transformed into instructional data. Additionally, we validate our method in the financial domain to showcase its generalizability. We adopt a total of 10 diverse datasets, covering numerous NLP tasks: Summarization (SUM), QA, NER, RE, SA, and Classification (CLS), detailed in Appendix A.
Models. We employ MPT-30B (Team, 2023) as a base model for main experiments. For the retriever, we use simple yet effective BM25 (Robertson et al., 1994), assuming human-labeled data is not sufficient. For benchmarking of generalpurpose aligned models, we evaluate Alpaca-65B (Taori et al., 2023) and Dromedary-65B (Sun et al., 2023) that are both based on LLaMA (Touvron et al., 2023a). In addition to MPT-30B, we adopt LLaMA-2 7B (Touvron et al., 2023b) and Falcon40B (Almazrouei et al., 2023) to further validate the general applicability of self-specialization with different scales and base models. We additionally evaluate existing domain-specific models (Wu et al., 2023): MedLLaMA and PMC-LLaMA (Details are in Section 5.2).
Metrics. In our study, all tasks are approached as a unified text generation problem, aiming to assess the capabilities of generative models. In alignment with an established convention (Parmar et al., 2022), we adopt F1-SCORE as our main evaluation metric, given an early observation that ROUGEL (Lin, 2004), as shown in Table 6 in Appendix, exhibits a strong correlation with F1-SCORE.
Implementation Details. For biomedical seeds, we use data sampled from BoX (Parmar et al., 2022), encompassing 32 tasks, up to 5 instances for each dataset, resulting in a compact yet representative 80 seed samples in total, which are also used as demonstrations at inference. For optional external corpus, we leverage PubMed preprocessed in (Phan et al., 2021), which contains ≈30M abstracts. In the financial domain, based on our finding from
biomedical experiments that showed surprising effectiveness of self-specialization relying on internal knowledge of LLMs without the external corpus, we opt not to employ an optional retrieval component to further validate the self-sufficiency of LLMs. We leverage a total of 90 seeds sampled from the 10 train sets in our corresponding benchmark datasets. We use a total of 5K synthetic data generated through our self-specialization for all experiments, unless otherwise specified. Being equipped with QLoRA (Dettmers et al., 2023) and 4-bit quantization, the model is trained using a simple Alpaca-style template (Taori et al., 2023) on a single A100, taking only a few hours for 3 epochs, resulting in a light-weight specialization module.

heading: 5 Results and Analyses
text: Here, we provide a set of experimental results and analyses to address relevant research questions.

heading: 5.1 Comparison with Baselines
text: How effective is the self-specialization of base models? In Table 2, we present the comparative results of our self-specialized model against its base counterpart across 10 distinct biomedical NLP and 10 financial NLP datasets. The evaluation is conducted with varying numbers of in-context demonstrations, k.
Our findings reveal that the self-specialized model exhibits remarkable progress in the majority of tasks across all configurations in both domains, yielding a substantial (up to 18 points) improvement in average scores. Specifically, the average scores (F1) in biomedicine rise from 30.18 to 48.41 in a 5-shot setting.2 In finance, the improvements
2Even excluding BioASQ-Yesno as an outlier due to the base model’s low performance, self-specialization still shows significant gain over the base model: 32.55 to 43.21 (5-shot). Appendix C.3 includes the detailed discussion.
are 14.53 (0-shot), 6.77 (1-shot), and 12.86 (5- shot), respectively. These advancements in both domains underscore the self-specialization’s generalizability in addressing a wide array of tasks across different specialized domains.
Imact on ICL capability. A potential concern on self-specialization tuning is its impact on the base LLM’s in-context learning capabilities, as we did not tune the model with demonstrations. Comparing the capabilities before and after self-specialization, the improvement after adding demonstrations (from 0 to 5) of our self-specialized model on biomedicine in Table 2 is 36.63 to 48.41 (∆=11.78), while that of the base model is 25.15 to 30.18 (∆=5.03), indicating even better ICL capability with in-domain knowledge acquisition.
Performance drop on some tasks. Our analysis does identify a few instances where performance drops as shown in Table 2. This indicates room for further refinement, especially for tasks like ConvFinQA that require a set of specific capabilities beyond mere domain knowledge. We evidenced that a minor proportion (≤ 2%) of generated data partially resembles ConvFinQA, due to our generation’s nature involving creative brainstorming for diversity. The specific demands of ConvFinQA, including numerical reasoning, structured tables, and conversations extend beyond basic domain knowledge and were insufficiently covered within our dataset. This gap likely contributes to the observed performance trade-offs.
However, we re-emphasize that there are significantly bigger gains in many of the cases (e.g., 45 out of 60 experiments across datasets and k), outweighing the regression overall. Acknowledging the inherent variability of in-context learning (Min et al., 2022), we present the variances with 5 dif-
ferent sets of demonstrations in Figure 4 based on LLaMA-2-7B in biomedicine, showing significant average improvements of 8.25 (p = 0.003, k = 1) and of 14.42 (p ≤ 0.001, k = 5).
How does self-specialization compare against larger/generally aligned baselines? In Figure 3, we compare our self-specialized MPT-30B model with 65B models, including LLaMA-65B, and its general instruction aligned variants (e.g., Alpaca based on Self-Instruct) in the biomedical domain. Interestingly, the results reveal that our model, without extensive data, surpasses all baselines, including 65B models, despite its ≈2.2x smaller size. This not only highlights the lower expert domain performance trade-offs of the “generalist” models in terms of encoding vast general knowledge into a finite set of parameters, but also underscores the effectiveness of our parameter-efficient approach to model specialization. We also show that our self-specialized model outperforms the supervised general-purpose model, MPT-30B-instruct in all tasks, which highlights the benefits of in-domain instruction data. Moreover, as a reference point, we present a comparison with a fully-supervised SOTA model that is fine-tuned on the biomedical datasets in Table 7, contextualizing our progress to better understand practical utility, discussed in Appendix C.4. Notably, the data efficiency of our simple self-specialization is further reinforced by the fact that the model is trained using only 5K3 instruction data self-produced with minimal (only 80) seeds.4 This training process, facilitated by the incorporation of QLoRA, adding only 0.28% trainable parameters to an otherwise frozen model, only takes a few hours on a single GPU (A100 80GB).
352K for Alpaca and 360K for Dromedary. 4175 for Alpaca and 195 for Dromedary.

heading: 5.2 Ablations & Analyses
text: Effect of external knowledge. We investigate the influence of incorporating a domain-specific corpus like PubMed in the response generation phase. Table 3 shows optimal results with the top-5 documents, while using just the top-1 document decreases performance, likely due to noise from an imperfect retrieval process, aligned with findings from previous work (Yoran et al., 2023) that adding irrelevant (i.e., random) context dramatically decreases performances. Conversely, employing the top-5 documents with probability marginalization (eq. 3.3) seems to mitigate this issue, enabling the model to exploit informative knowledge. Interestingly, we observe that self-specialization demonstrates strong performance even without retrieval, suggesting domain-relevant knowledge is intermingled with other information acquired during pretraining, which self-specialization uncovers to better utilize. Given this, the added complexity of retrieval mechanisms, though potentially advantageous, emerges as optional within our framework.
Effect of iterative self-specialization. In Section 3.5, we discussed the potential of employing an iterative process by leveraging the self-specialized model instead of the base model throughout the generation process. Table 4 shows the ablation study, where each iteration involved generating 5K samples, and final results were obtained using 5K samples from the last iteration for a fair comparison. We observe that the iterative process leads to further
FinanceBiomedicine
performance enhancements, compared to the one w/o iteration. In our preliminary tests, we rarely find meaningful improvements with the subsequent iteration, which we leave for future work to refine.
Self-specialization vs. domain pre-training. We compare our model based on LLaMA-27B with existing baselines (Wu et al., 2023): MedLLaMA-13B and PMC-LLaMA-7B/-13B. The former is an LLaMA variant further pre-trained on a large domain-specific corpus (i.e., medicine), and the latter is further instruction-tuned using annotated/synthetic datasets, including medical QA, rationale for reasoning, and conversational dialogues. Notably, we find that our self-specialized 7B model is on par with or better than MedLLaMA13B (p = 0.006, k = 5) and PMC-LLaMA-13B (p = 0.01, k = 5) despite their larger parameters and extensive domain-specific tuning. Additionally, using our 7B-generated data to specialize MedLLaMA indicates that self-specialization can enhance domain-specific pre-training (p = 0.001, k = 5), suggesting complementarity.
Impact of the number of self-generated data. In Figure 5, we analyze the impact of the number of self-specialization data within biomedical and financial domains. Starting from zero, a sharp increase in F1 score is observed as we introduce the first 100 instances which largely consist of seed instructions, underlining the significant impact of seeds not only as in-context demonstrations but also as training data. The performance continues to rise steadily with additional data, plateauing around 5K instances, supporting our decision on the use of 5K data. Self-specialization’s success with relatively small self-generated data highlights its data efficiency and practicality.
How is the quality of synthetic self-specialization data? In Figure 6, we showcase a qualitative visualization that analyzes the synthetic data generated through self-specialization, confirming that self-specialization produces domain-focused data. To quantitatively assess the quality, Figure 7 in Appendix compares our model against a model trained on labeled data, which shows a narrow performance gap, implying the quality of generated data. Additionally, some examples are provided in Table 11 & 12 in Appendix, offering insights into the quality of the self-generated specialization data.

heading: 6 Related Work
text: The goal of instruction-tuning and alignment of large language models (LLMs) is to achieve crosstask generalization or to align with human preferences. This can be accomplished by either
training LLMs directly with human-labeled data (Ouyang et al., 2022; Wei et al., 2022; Mishra et al., 2022; Wang et al., 2022b) or data generated by larger models (i.e., distillation) (Taori et al., 2023; Chiang et al., 2023). Recent studies have shown that LLMs are self-instructors. Wang et al. (2022a) showed that with in-context prompts, GPT3 (Brown et al., 2020) can generate high-quality instruction-responses pairs for its own alignment. Sun et al. (2023) further suggests that using principles can minimize human supervision while covering a broad spectrum of scenarios with the opensource model, LLaMA-65B (Touvron et al., 2023a). While enhancing general alignment, according to our presented evidence, these approaches are unlikely to induce specialization in expert domains, leaving different domain expertise in superposition inside the model. To the best of our knowledge, we are the first to show the potential for expert domain specialization through self-alignment, effectively “uncovering” a domain expert out of the model in a parameter- and data-efficient manner.
Recent studies highlight the benefits of employing instructions in different adaptation scenarios (Parmar et al., 2022). INSTRUCTOR (Su et al., 2022) illustrated the adaptability of instruction-based text embeddings to various tasks and domains, while INSTRUCTE (Bai et al., 2023) demonstrated that incorporating instructions with a schema can yield robust results for table extraction across diverse domains. However, these require the use of costly human labels or extensively tuned large models (e.g., 175B). Self-training has
also been explored for different adaptation scenarios. For domain knowledge adapation, Shakeri et al. (2020) and Luo et al. (2022) proposed constructing synthetic data by generating in-domain question-answering data, but these data generators are trained with more than 80k human curated QA pairs and do not involve instructional ones that have the potential for cross-task generalization. Instruction-tuning has been shown to adapt pre-trained LLMs to different modalities, including vision (Liu et al., 2023), audio (Gong et al., 2023), and programs (Rozière et al., 2023), and enables the use of APIs (Schick et al., 2023) and search engines (Luo et al., 2023). Unlike these works, our work focuses on uncovering target domain expertise latent within LLMs while promoting cross-task generalization with minimal supervision.

heading: 7 Conclusion
text: Our exploration into self-specialization aimed to elucidate the latent expertise within large language models (LLMs) with limited human supervision. This scheme demonstrated promising results in specialized domains. The self-specialized model exhibited remarkable performance, outperforming its base model, MPT-30B, and even surpassing larger generally aligned models (65B). This illuminates the intrinsic challenges of encoding vast general knowledge into limited parameters and underscores the efficiency of self-specialization. Remarkably, the model’s efficient training, marked by minimal data usage and the integration of QLoRA (Dettmers et al., 2023), adds another layer to its practicality in terms of parameter and data efficiency. These findings signify a promising pathway for leveraging inherent expertise in LLMs and offering a large variety of exciting opportunities for future work.
Limitations
While our study provides encouraging insights into the capabilities of self-specialization, this is an initial step in opening up new opportunities. We acknowledge the need for further exploration and note some limitations and considerations.
Sensitivity of in-context learning. In Table 2, we observed that performances sometimes dropped with more in-context learning demonstrations. While recognizing, the performance fluctuation is not an issue stemming from our self-specialization tuning, as it happens for the base LLM as well as GPT-3 (Appendix C.2) , demonstrating an inherent
challenge in in-context learning. This phenomenon is not unique to our self-specialization approach, but a broader challenge in the field.
Training and generation strategies. We avoided using demonstrations during training (Min et al., 2022) to maintain flexibility in the number of examples available during inference. We aimed to ensure that zero-shot performance remains unaffected by tuning to rely on demonstrations.
Unlike previous work (Wang et al., 2023a) that generates instructions first and then inputs/responses together, our approach simultaneously generates instructions and inputs, followed by responses. This strategy, inspired by a more recent work (Sun et al., 2023), enables the use of inputs as queries for retrieval prior to response generation. Despite the specific reasons outlined above, we recognize the potential of the alternative strategies as avenues for future exploration, which can be orthogonal to our current approach.
Filtering. In our method, we opted not to implement an automatic filtering process for the generated data. In a preliminary study to assess feasibility, we attempted to filter out low-quality data manually, however we did not observe a noticeable improvement. We hypothesized that incorporating this seemingly unuseful data may even enhance the model’s robustness by preventing overfitting to those generated data. Despite this, we acknowledge the importance of further investigating filtering techniques for potential improvements.
Potential data contamination and bias propagation. Being cautious with potential data contamination from base language models during selfspecialization, we conducted stringent measures following practices in GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022). We adopted n-gram overlap analysis (with n=8 and a threshold of 70%) to scrutinize similarities between our generated data and all the test sets, revealing no significant overlaps. Moreover, a detailed manual inspection of 200 random instances corroborated this finding. When concerned about retrieval sources, one can apply the n-gram overlap filtering, though our sources are PubMed abstracts without explicit labels, which inherently ensures little risk of data overlap. Meanwhile, we acknowledge the inherent risk of propagating biases from the pre-trained data.

heading: Acknowledgments
text: This research is supported in part by the NSF (IIS2052498), ODNI and IARPA via the HIATUS program (contract 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Explanations of Evaluation Datasets
text: Below are brief descriptions for each dataset in biomedical and financial domains. All datasets are in English.
A.1 Biomedicine
BioASQ-8b (Nentidis et al., 2020). This is a biomedical QA dataset that necessitates models to produce answers from given questions and corresponding contexts within the biomedical domain. There are three distinct subsets that can be divided according to question types: Factoid, List, and Yesno. This dataset is publicly available upon a data use agreement. The data are originally intended to be used as training and development data, and we use the small part of the training set as seeds (i.e., 5 seeds), and the test set for evaluation (500 for each question type). CC BY 2.5.
PubMedQA-Long (Jin et al., 2019). PubMedQA is another biomedical QA dataset featuring research questions along with their corresponding abstracts and answers sourced from PubMed5. To diversify the task types, we focus on a long-form answer (i.e., conclusion). We use 5 labeled data for seeds and 500 for evaluation. MIT license.
AnatEM (Pyysalo and Ananiadou, 2013). This is a Named Entity Recognition (NER) task for anatomical entities in biomedical texts. Models are tasked with identifying all anatomy-named entities and their corresponding types from given a small paragraph. Non-commercial purposes only. 404 test data are used for evaluation and 5 training instances are used for seeds. CC BY-SA 3.0.
BioNLP13CG (Pyysalo et al., 2013). The Cancer Genetics (CG) is an information extraction task targeting the recognition of events in text, encompassing multiple levels of biological organization, from molecular to whole organisms. 5 training data are used for seeds, and the number of evaluation data is 200. CC BY-SA 3.0.
NCBI (Dogan et al., 2014). The NCBI disease corpus, derived from the National Center for Biotechnology Information, focuses on disease name recognition. According to the annotation guideline of this dataset, organism names such as humans, and also gender are excluded for annotation. We use 5 training instances for seeds, and 100
5https://www.ncbi.nlm.nih.gov/pubmed
for evaluation. The data is freely available to the public for use. CC0 1.0 license.
DDI (Herrero-Zazo et al., 2013). The DrugDrug Interaction (DDI) dataset is tailored for identifying interactions between different drugs in biomedical texts. Following Parmar et al. (2022), this work considers only binary Relation Extraction (RE), determining whether there is an effect of given two drugs. The data cannot be used for any commercial purposes. We use 5 data for seeds, and 500 for evaluation. CC BY-NC 4.0.
Medical Drugs (Khan, 2019). This is a Sentiment Analysis (SA) dataset that is required to predict the sentiment of individuals towards medical drugs. Specifically, given a text and a drug, a model determines the effect of the drug as “positive", “negative", or “neutral". 5 training instances are used for the seed construction, and 500 test set for evaluation. The license is unknown.
HoC (Baker et al., 2015). The Hallmarks of Cancer (HoC) dataset is curated for classifying (zero to many) biomedical texts related to cancer into categories representing different hallmarks of cancer. In particular, these hallmarks include “sustaining proliferative signaling", “resisting cell death", “genomic instability and mutation", “activating invasion and metastasis", “tumor promoting inflammation", “evading growth suppressors", “inducing angiogenesis", “enabling replicative immortality", “avoiding immune destruction" and “cellular energetics". The number of evaluation data is 200 and 5 training data are used for seed demonstrations. GPL-3.0 license.
A.2 Finance
EDT-Summarization (Zhou et al., 2021). This dataset challenges models to perform abstractive summarization on financial news articles, condensing detailed information into succinct summaries. 8 training instances are used for seeds, and 500 instances for evaluation. This data is publicly available.
InsuranceQA (Feng et al., 2015). This is an open-book question-answering task about insurance, demanding models to extract and provide specific insurance-related information. Seed demonstrations include 8 training data and the number of evaluation instances is 500. This dataset is provided as is and for research purposes only.
ConvFinQA (Chen et al., 2022). This is a dataset for conversational question-answering over financial report tables, testing a model’s ability to reason and respond within a conversational context. We use 8 training data for the seed construction, and evaluation uses 500 test instances. MIT license.
Fin3 (Salinas Alvarado et al., 2015). This is a financial NER dataset based on financial agreements to aid credit risk assessments. 8 training data are used for seeds and 100 test data for evaluation. CC-BY 3.0.
FiNER_139 (Loukas et al., 2022). This NER task focuses on financial texts, where models identify and classify financial-related entities like numbers. This dataset includes a much larger label set of 139 entity types. Seed data encompass 8 training instances and the number of test data is 500. MIT license.
KPI-EDGAR (Deußer et al., 2022). Models are tasked with extracting key performance indicators (KPIs) from financial documents. Categories for KPIs include current and previous year values, annual changes, subordinate and descriptive attributes, co-references, and false-positive. We use 212 test instances for evaluation and 8 training instances for seed demonstrations. MIT license.
EarningsCall (Roozen and Lelli, 2021). This is a binary sentiment analysis task where models evaluate sentiments from stock values and transcripts of earnings calls, reflecting the financial sentiments expressed. 8 training instances are used for seeds, and 500 test set for evaluation. CC0 1.0 license.
Financial_Phrasebank (Malo et al., 2014). This dataset involves (3-way) sentiment analysis of financial news headlines, assessing the underlying sentiment conveyed by the language used. Commercial uses of this data may be allowed upon contacting the authors. 8 training data and 500 test data used for seeds, and evaluation, respectively. CC BY-NC-SA 3.0.
FIQA-SA (Maia et al., 2018). It consists of aspect-based sentiment analysis tasks within financial texts, requiring models to discern sentiment regarding specific aspects mentioned. The number of evaluation data is 234 and seed demonstrations include 8 training instances.
Gold Commodity News (Sinha and Khandait, 2021). This dataset involves classifying financial
news headlines about gold commodities into categories such as market movement direction or type of financial news (e.g., direction up, down, pastprice, futurenews, etc). The seed data includes 9 binary-class version and also 9 multi-class version of training set, and evaluation uses 500 multi-class version of test data. The license of this data indicates data files © original authors.

heading: B Details of Experiments
text: In Table 10, we show the prompts used for our self-specialization. For instruction generation, we leverage the prompt designed in self-instruct Wang et al. (2022a) with minimal change to make it suit to specialization. In particular, we ask a model for instructions about a targeted domain, and force it to generate input together with the instruction, unlike in Wang et al. (2022a) that generates those separately. In addition, we avoid using the specific requirement in the prompt that asks to cover diverse topics, such as (quoting Wang et al. (2022a)) “daily routines, travel and tourism health and wellness, cooking and recipes, personal finance, environmental issues, history and historical events, literature and literary analysis, politics and current events, psychology and mental health, art and design, mathematics and problem-solving, physics and astronomy, biology and life sciences, chemistry and materials science, computer science and programming, engineering and technology, robotics and artificial intelligence, economics and business management, philosophy and ethics, and more". For response generation, we use a simple prompt to let a model answer with a target domain in mind. Both prompts can be further enhanced and optimized for better self-specialization performance in future work.
Regarding our evaluations, we use prompt templates that were designed and used to optimize each Alpaca (Taori et al., 2023) and Dromedary (Sun et al., 2023), but no specific template for base models, as they were not optimized for it during pre-training. Ours employs a simple Alpaca template for training and evaluation. We leverage publicly available delta weights that are supposed to be attached to LLaMA (Touvron et al., 2023a) for Dromedary, and use the ones reproduced for Alpaca in our work.
We use three seed demonstrations in-context, which are randomly sampled from our initial seeds, and sampling with top-p being 0.98 and tempera-
ture being 1.0 during instruction generation. For response generation, we use no demonstrations incontext since there is a high chance that the generated instruction task and the sampled one do not match well. We believe further exploration of this aspect would be valuable in future work. For finetuning, we use a batch size of 32, a learning rate of 3e-4, and epochs of 3. Low-rank adaptation (LoRA) (Hu et al., 2022; Dettmers et al., 2023) is applied to all modules and all layers with a rank of 8, and an alpha of 16. While we report single-run results considering low-data settings where automatic hyperparameter tuning might be infeasible, we also report worst, average, and best across different k-shot configurations for each dataset to address the concern of sensitivity (Appendix C.2) in Table 5.

heading: C Additioanl Results & Discussion
text: C.1 Qualitative Analyses While our study primarily focuses on the biomedical and finance domain, the applicability and effectiveness of self-specialization in another specialized domain whose knowledge is relatively limited, such as sports, remain an open avenue for exploration. As an initial effort, we present a case study of a self-specialized model on sports in Table 13 & 14, along with the visualization of generated data in Figure 8. We hope that this could offer insights into the versatility of self-specialization, although the
model is not yet perfect, and thorough evaluations are required in future work. Different domains inherently pose unique requirements and nuances, and understanding how self-specialization adapts to these variations is a valuable direction for future work.
C.2 On the Sensitivity of Prompting
In Table 2, we observe the decreased performances with increased demonstrations in certain cases such as BioASQ and Medical Drugs. We conjecture this can be attributed to the model’s sensitivity (Zhao et al., 2021) or interference among demonstrations (Chen et al., 2023) under in-context learn-
ing (ICL). In fact, it can even be noticed in the original GPT-3 paper (Brown et al., 2020) that additional demonstrations do not always lead to better performance and can indeed sometimes result in a notable decrease, demonstrating an inherent challenge in ICL. Taking the worst, average, and the best across different k-shot (0, 1, 5) configurations for each dataset to address the concern of sensitivity, we still notice the significant gaps between our self-specialization and the base model, presented in Table 5.
C.3 On Evaluation Designs
In our study, as described in Section 4, we treat all tasks as a unified text generation problem, aiming to assess the realistic capabilities of following instructions, consistent with established practices in biomedical instruction tuning literature (Parmar et al., 2022). As briefly discussed in Section 5.1, we observe that in Table 2, the base model’s performance on BioASQ-Yesno is very low (below random), often failing to follow instructions and generating text that is not confined to the label space. We therefore treat this dataset as an outlier and exclude it from our average calculations. Even after removing this outlier, self-specialization still has substantial gains over the base model: 25.58 to 31.22 (0-shot), 28.42 to 36.55 (1-shot), and 32.55 to 43.21 (5-shot). However, we believe that our current evaluation is fairer and preferable, because in a realistic scenario where a user prompts a model to solve a certain task (e.g., classification) without
the assumption about a task type, and gets a totally wrong response out of the label space, evaluating such a response as correct would not make sense.
The primary objective of our work is to enhance the base model’s domain-specific capabilities through self-specialization, a process inherently different from conventional fine-tuning approaches. Although the process utilizes LoRA for specialization, it is important to note that our approach fundamentally relies on synthetic data generated by the model itself. This unique aspect sets our method apart, as it effectively starts from scratch, focusing on self-generated, domain-specific instructional data for low-data scenarios. Finally, the base model and the base model improved through our SelfSpecialization (using synthetic self-generated data) are compared fairly in the same zero-shot/few-shot setting.
C.4 Comparison with State-of-the-art
In Table 7, we present the performances of a stateof-the-art instruction-tuned model (Parmar et al., 2022) in a biomedical domain, for a reference point. It is important to clarify that our comparison should not be considered direct. The SOTA model, unlike ours which relies on a few seed samples, is finetuned on a vast corpus of human-annotated data (140K), and differences in test set splits may exist. For the base MPT and self-specialized models, the maximum performances by using up to 5 samples are presented.
Despite significant improvements over its base
model, our self-specialized model remains behind SOTA benchmarks, which is not surprising due to the nature of our method that is not supervised, unlike the SOTA model. While expected, this possibly implies the practical utility of our approach may be limited yet in certain scenarios. From the table, we especially note the substantial gap in Named Entity Recognition (NER) tasks. This gap can be attributed to the SOTA model’s training on a large and diverse set of NER datasets (i.e., 80K samples). This suggests ample opportunity for further exploration and enhancement in this area.
Additionally, we provide comparisons with concurrent works (Cheng et al., 2024; Li et al., 2024) in Table 8 and 9. To make distinctions, those works are, in principle, oriented toward different scenarios from ours. Cheng et al. (2024) leverages a large specialized corpus to transform them into reading comprehension texts using pre-defined templates, which are then used for training LLMs. Differently, our work does not necessitate such a huge amount of corpus that requires 768 GPU hours for training; ours takes only a few GPU hours. Moreover, we do not confine our method to specific tasks, aiming for cross-task generalization as shown in Table 8, unlike them focusing on constrained reading comprehension format.
Li et al. (2024) explores the augmentation of instructional data through the use of a few thousand labeled seed data to initialize their generation pipeline relying on a web corpus. Unlike them assuming huge amounts of seed/unlabeled data, our method requires only a handful of seeds (e.g., <100). Furthermore, their focus is on general instructions, which have shown only marginal effects in specialized domains in our preliminary results (Section 2). Given the infeasibility of direct comparison, we reference their MMLU result from the paper to contextualize and we self-specialize Gemma-7B (Team et al., 2024) on MMLU (Hendrycks et al., 2021). While acknowledging their values in different scenarios from ours, we believe this distinction provides an insight into the unique value and impact of our work in enhancing model performance along with efficiency in targeted domains.

heading: Future Work
text: We hope that this could offer insights into the versatility of self-specialization, although the
model is not yet perfect, and thorough evaluations are required in future work. Different domains inherently pose unique requirements and nuances, and understanding how self-specialization adapts to these variations is a valuable direction for future work. C. 2 On the Sensitivity of Prompting
In Table 2, we observe the decreased performances with increased demonstrations in certain cases such as BioASQ and Medical Drugs. We conjecture this can be attributed to the model’s sensitivity (Zhao et al. , 2021) or interference among demonstrations (Chen et al. , 2023) under in-context learn-
ing (ICL). In fact, it can even be noticed in the original GPT-3 paper (Brown et al. , 2020) that additional demonstrations do not always lead to better performance and can indeed sometimes result in a notable decrease, demonstrating an inherent challenge in ICL. Taking the worst, average, and the best across different k-shot (0, 1, 5) configurations for each dataset to address the concern of sensitivity, we still notice the significant gaps between our self-specialization and the base model, presented in Table 5. C. 3 On Evaluation Designs
In our study, as described in Section 4, we treat all tasks as a unified text generation problem, aiming to assess the realistic capabilities of following instructions, consistent with established practices in biomedical instruction tuning literature (Parmar et al. , 2022). As briefly discussed in Section 5. 1, we observe that in Table 2, the base model’s performance on BioASQ-Yesno is very low (below random), often failing to follow instructions and generating text that is not confined to the label space. We therefore treat this dataset as an outlier and exclude it from our average calculations. Even after removing this outlier, self-specialization still has substantial gains over the base model: 25. 58 to 31. 22 (0-shot), 28. 42 to 36. 55 (1-shot), and 32. 55 to 43. 21 (5-shot). However, we believe that our current evaluation is fairer and preferable, because in a realistic scenario where a user prompts a model to solve a certain task (e. g. , classification) without
the assumption about a task type, and gets a totally wrong response out of the label space, evaluating such a response as correct would not make sense. The primary objective of our work is to enhance the base model’s domain-specific capabilities through self-specialization, a process inherently different from conventional fine-tuning approaches. Although the process utilizes LoRA for specialization, it is important to note that our approach fundamentally relies on synthetic data generated by the model itself. This unique aspect sets our method apart, as it effectively starts from scratch, focusing on self-generated, domain-specific instructional data for low-data scenarios. Finally, the base model and the base model improved through our SelfSpecialization (using synthetic self-generated data) are compared fairly in the same zero-shot/few-shot setting. C. 4 Comparison with State-of-the-art
In Table 7, we present the performances of a stateof-the-art instruction-tuned model (Parmar et al. , 2022) in a biomedical domain, for a reference point. It is important to clarify that our comparison should not be considered direct. The SOTA model, unlike ours which relies on a few seed samples, is finetuned on a vast corpus of human-annotated data (140K), and differences in test set splits may exist. For the base MPT and self-specialized models, the maximum performances by using up to 5 samples are presented. Despite significant improvements over its base
model, our self-specialized model remains behind SOTA benchmarks, which is not surprising due to the nature of our method that is not supervised, unlike the SOTA model. While expected, this possibly implies the practical utility of our approach may be limited yet in certain scenarios. From the table, we especially note the substantial gap in Named Entity Recognition (NER) tasks. This gap can be attributed to the SOTA model’s training on a large and diverse set of NER datasets (i. e. , 80K samples). This suggests ample opportunity for further exploration and enhancement in this area. Additionally, we provide comparisons with concurrent works (Cheng et al. , 2024; Li et al. , 2024) in Table 8 and 9. To make distinctions, those works are, in principle, oriented toward different scenarios from ours. Cheng et al. (2024) leverages a large specialized corpus to transform them into reading comprehension texts using pre-defined templates, which are then used for training LLMs. Differently, our work does not necessitate such a huge amount of corpus that requires 768 GPU hours for training; ours takes only a few GPU hours. Moreover, we do not confine our method to specific tasks, aiming for cross-task generalization as shown in Table 8, unlike them focusing on constrained reading comprehension format. Li et al. (2024) explores the augmentation of instructional data through the use of a few thousand labeled seed data to initialize their generation pipeline relying on a web corpus. Unlike them assuming huge amounts of seed/unlabeled data, our method requires only a handful of seeds (e. g. , <100). Furthermore, their focus is on general instructions, which have shown only marginal effects in specialized domains in our preliminary results (Section 2). Given the infeasibility of direct comparison, we reference their MMLU result from the paper to contextualize and we self-specialize Gemma-7B (Team et al. , 2024) on MMLU (Hendrycks et al. , 2021). While acknowledging their values in different scenarios from ours, we believe this distinction provides an insight into the unique value and impact of our work in enhancing model performance along with efficiency in targeted domains