Instruction Fusion: Advancing Prompt Evolution through Hybridization (2024)

Abstract: The fine-tuning of Large Language Models (LLMs) specialized in code generation has seen notable advancements through the use of open-domain coding queries. Despite the successes, existing methodologies like EvolInstruct encounter performance limitations, impeding further enhancements in code generation tasks. This paper examines the constraints of existing prompt evolution techniques and introduces a novel approach, Instruction Fusion (IF). IF innovatively combines two distinct prompts through a hybridization process, thereby enhancing the evolution of training prompts for code LLMs. Our experimental results reveal that the proposed novel method effectively addresses the shortcomings of prior methods, significantly improving the performance of Code LLMs across five code generation benchmarks, namely HumanEval, HumanEval+, MBPP, MBPP+ and MultiPL-E, which underscore the effectiveness of Instruction Fusion in advancing the capabilities of LLMs in code generation.

heading: 1 Introduction
text: The field of automatic program writing has intrigued computer scientists since the 1960s (Waldinger and Lee, 1969; Dehaerne et al., 2022). This period has seen substantial efforts to enable machines to autonomously write correct programs. The emergence of Large Language Models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023) has been a cornerstone in text generation (Liu et al., 2023b), leading to the evolution of Code Large Language Models (Code LLMs) which have notably advanced code generation tasks (Hou et al., 2023).
Early Code LLM research (Roziere et al., 2023; Li et al., 2023) concentrated on the pre-training phase, utilizing a variety of code datasets to bolster the code generation capabilities of LLMs. A major
∗These authors contributed equally to this work.
breakthrough was achieved with instruction tuning (Wei et al., 2022), which enhanced the general applicability of LLMs by fine-tuning them with instructions rather than task-specific prompts.
In pursuit of greater quantity, diversity, and creativity in instruction-response samples, Wang et al. (2022) introduced the SELF-INSTRUCT method, leveraging LLMs to generate their own synthetic instructions. This approach was notably adopted by Code Alpaca (Chaudhary, 2023), based on Stanford Alpaca (Taori et al., 2023), transforming 21 basic code prompts into 20,000 high-quality instructions. Building on this, Luo et al. (2023) further advanced code generation performance by employing EvolInstruct on these instructions, expanding them to 78,000 through five dataset evolution heuristics.
However, the Evol-Instruct method has its limitations on Code LLMs, primarily its reliance on a set of five heuristics for generating new instructions. This approach leads to a pattern where the evolution is done primarily by adding more constraints to its seed instruction, which can cause three major issues. Firstly, the evolving instructions may become excessively complex, challenging GPT’s ability to respond effectively. Secondly, the newly added constrains to the seed instruction may not exist in the original seed instruction, leading to an gap on difficulty gradient. Finally, the evolutionary evaluations often remain confined within the scope of the original instruction, limiting diversity.
Addressing these challenges, we introduce a novel method named Instruction Fusion (IF). This method leverages hybridization concepts to significantly improve prompt evolution, as demonstrated in Figure 1. It involves merging two distinct instructions into a single prompt using GPT-4 Turbo, enhancing prompt complexity and diversity. This approach also facilitates a more gradual increase in difficulty for LLMs, optimizing learning and performance.
To assess the efficacy of our Instruction Fusion
3883
method, we conducted experiments with CODELLAMA (Roziere et al., 2023), using HumanEval (Chen et al., 2021a), MBPP (Austin et al., 2021), HumanEval+, MBPP+ (Liu et al., 2023a), and MultiPL-E (Cassano et al., 2022) as benchmark datasets. Our results demonstrate a significant performance improvement of LLMs when training with the additional data generated by IF. The contributions of this work can be summarized as follows:
• The IF method greatly improved instruction creation by merging two distinct instructions into a single, more complex prompt using GPT-4 Turbo. This strategy substantially boosts the diversity and complexity of the training data while ensuring a smoother difficulty gradient for learning. This approach effectively addresses and overcomes the limitations inherent in previous methods such as Evol-Instruct.
• Through extensive fine-tuning experiments on Code-Llama models on five commonly used code generation benchmarks, we demonstrate the superior performance of our Instruction Fusion method. We fully open source the model weights, training data, and source code to facilitate future research.

heading: 2 Approach
text: In this section, we first analyze the the limitation of current evolution method on Code LLMs, then we elaborate on the details of the proposed Instruction Fusion method and analyze its advantages comparing to the existing instruction evolution method. The main process of the fusion is illustrated in Figure 1.

heading: 2.1 Instruction Evolution
text: High-quality instructions are vital for effectively fine-tuning Code LLMs. Crafting these, particularly for coding tasks, is resource-intensive and often results in easy difficulty levels, creating a gap in challenging content (Xu et al., 2023). To address this and reduce costs, Luo et al. (2023) applied Evol-Instruct (Xu et al., 2023) to Code LLMs, enhancing code generation. This process involves merging each seed instruction with a unique evolution prompt and generating evolved instructions using GPT-3.5 Turbo. This method expanded and diversified the instruction dataset, but certain limitations hinder its further development.
A major challenge identified in the process of instruction evolution is the over-escalating constraints, as illustrated in Figure 2. The process begins with a straightforward instruction, such as developing a Python function to convert strings to lowercase. However, the complexity increases significantly with each evolution round. After four rounds, the instruction length expanded from 14 to 325 tokens. This increase is primarily attributed to the incorporation of up to eight distinct constraints. As a result, the evolved instructions become impractically intricate and overly complex.
Although Evol-Instruct with low rounds can properly increase the difficulty levels by braining new constrains, which benefits LLM learning, those new constrains may need prior knowledge to learn. Consider math learning as an analogy: understanding addition and multiplication is fundamental before tackling problems like "1 + 1× 2." However, Evol-Instruct might prematurely introduce such problems immediately following basic concepts like "1 + 1 = 2", potentially leading to gaps in learning progression.
Another issue is the limited diversification of objectives during evolution. Despite adding new constraints, the primary objective often remains unchanged. Take the scenario in Figure 2 as an example, despite introducing new objectives like sorting, the primary task remained similar to the original. This lack of variability in objectives can severely limit the diversity of the resulting instructions.
Due to these inherent limitations, the EvolInstruct method applied to Code LLMs often reaches its capacity (3 rounds) without further enhancing code generation performance. However, these issues can be mitigated with our proposed Instruction Fusion method.

heading: 2.2 Instruction Fusion
text: Inspired by the hybridization which can produce new individual by exchange the information between inter-specific parents, we propose Instruction Fusion to overcome the above challenges.
Fusion Process. Figure 1 gives an illustration of the Instruction Fusion process. Denote the initial dataset as C = {(Ii, Ri)}1<i<N , where Ii, Ri are the ith seed instruction and the corresponding response, N is the number of instructions in C. To create a new instruction using Instruction Fusion, we first random select two seed instructions Ij and
Ik from C, where 1 ≤ j, k ≤ N . Then, denote the target amount of fused instructions as M , we prompt GPT-4 Turbo to fuse the selected two instructions into the mth fused instruction Im(j,k) utilizing the following prompt:
Your task is to act as a Prompt Fusion Specialist. Your objective is to merge #Given Prompt 1# and #Given Prompt 2# into a single, cohesive #Fused Prompt#. This new prompt should:

heading: 1. Integrate the content from both #Given Prompt

heading: 1# and #Given Prompt 2#.

heading: 2. Maintain a similar length and complexity level
text: as the original prompts. 3. Be coherent and solvable, incorporating elements from both prompts in a balanced way. 4. In cases where the original prompts specify different programming languages, choose only one for the #Fused Prompt#. If the resulting #Fused Prompt# is not logically co-
herent or solvable, simply respond with ’INVALID PROMPT’. #Given Prompt 1#: <Here is Instruction 1> #Given Prompt 2#: <Here is Instruction 2> #Fused Prompt#:
Here, 1 ≤ m ≤ M . For each fused instruction, if Im(j,k) == INVALID PROMPT, we will discard the (j, k) combination and randomly sample a new one. We keep repeating the fusion process until the number of new instructions reach the target amount M . The result instructions set is then H = {(Im(j,k), Rm(j,k))}1<m<M , where Rm(j,k) is the corresponding response of Im(j,k) generated by GPT-4 Turbo (gpt-4-1106-preview1).
Data Collection. Table 1 gives the statistic of the collected datasets. Due to the unavailability of WizardCoder’s (Luo et al., 2023) original evolved instructions, we utilized the thirdparty dataset evol-codealpaca-v12, which mirrors the methods used by Luo et al. (2023). This dataset, comprising 111k refined instructions, is evolved from the CodeAlpaca3 dataset, which is also serves as the seeds in WizardCoder. We divided evol-codealpaca-v1 based on programming language into Python-related instructions CPI and Non-Python-related instructions CNPI, numbering 50,131 and 61,140, respectively. To perform detailed comparison in later experiments, we further divide CPI into C30 and C30r, which represents the 30K random samples from CPI and the rest sam-
1https://platform.openai.com/docs/models 2https://github.com/theblackcat102/evol-dataset 3https://github.com/sahil280114/codealpaca
ples, respectively. Based on the Instruction Fusion scheme introduced above, we set CPI as seeds and perform IF with M = 50, 000 to obtain the result dataset HPI, where 20K fused instructions H20 only use seeds from C30. Then we perform another IF with M = 50, 000 on CPI + CNPI and obtain HPI+NPI. For cost efficiency, our method predominantly utilizes samples in HPI. Nevertheless, towards the experiment’s end, we exam code generation performance using both two datasets, showcasing our method’s capability across multiple programming languages.

heading: 2.3 Complement the Deficiencies
text: As noted above, while Evol-Instruct enhances instruction diversity, it faces limitations in difficulty scaling and gaps, and its diversity is capped by the seed instruction’s original objective. To complement these deficiencies, an effective method should: 1) elevate difficulty beyond just adding constraints, 2) ensure a smoother gradient in difficulty levels among instructions, and 3) broaden diversity by redirecting objectives towards new directions. From which we proposed IF.
Difficulty and Gradient. Instruction Fusion (IF) merges two distinct seed instructions, creating a child instruction that integrates diverse objectives into one. Figure 3 illustrates such a fusion. To tackle these fused objectives, LLMs must amalgamate knowledge from both original tasks, thereby increasing the difficulty. As evident in Table 1, HPI demands longer responses than its seed counterparts in CPI. This indicates that GPT-4 Turbo exerts more effort to address fused instructions, reflecting their heightened complexity.
Figure 3 illustrates the role of Instruction Fusion (IF) in mitigating the difficulty gradient between original seeds and fused instructions. For example, consider a basic seed instruction to print “Hello, World!”. Evol-Instruct evolves this by incorporating a filtering task, thereby elevating the difficulty and diversity. However, this evolution may intro-
duce a disparity in difficulty, as the new element might be absent in other seeds. IF addresses this by ensuring new sub-objectives are introduced only if they exist in the seed instructions, thus maintaining a smoother difficulty gradient.
The effectiveness of this gradient is further substantiated by the theory presented in Kung et al. (2023)’s work. They categorize samples with high instruction uncertainty and a high prediction probability as “ambiguous”. Instruction uncertainty denotes the degree to which minor modifications in the instruction impact response generation. Higher uncertainty indicates greater sensitivity of LLMs to the specific instructions in a sample. Their research underscores the significant role of “ambiguous” training data in fine-tuning LLMs. Conversely, samples characterized by low instruction uncertainty and prediction probability are deemed overchallenging, potentially offering limited benefits for fine-tuning due to their complexity.
In our study, we employ the methodology of Kung et al. (2023) to evaluate the "ambiguity" of datasets. We simplify the process by calculating instruction uncertainty as the average loss changes upon altering instructions and representing prediction probability by the inverse of response loss. The resulting plot is analogous to that in Kung et al. (2023)’s study.
Figure 4 exemplifies this concept with the model fine-tuned using C30. Here, red spots denote the ambiguity of H20, while blue points represent C30r. The data reveals that instructions from C30r are challenging, evidenced by their lower instruction uncertainty and prediction probabilities. In contrast, instructions from H20 are more “ambiguous”, indicated by higher prediction probabilities and uncertainties, thus making them more conducive
for the fine-tuning process. From these observations, we conclude that the Instruction Fusion (IF) method effectively generates more “ambiguous” samples. LLMs exhibit increased responsiveness to the fused instructions compared to the original seed instructions.
Diversity. Instruction Fusion (IF) counters the diversity limitations of Evol-Instruct, which is constrained by initial objectives, by creating instructions that blend the targets of parent instructions. This leads to unique tasks not found in the Evol-Instruct dataset. In Figure 5, the gray points represent a t-SNE 2D semantic embedding plot for C30 instructions, The upper plot represent CPI, showing small cliques indicative of limited diversity, and the orange points represent samples from C30r.
In the lower plot, the red points, representing IF’s fused instructions, populate previously empty areas, demonstrating increased diversity, and illustrating a more uniform semantic embedding.
To quantify this dispersion, we calculate the variance of nearest neighbor distances for all instructions:
U = 1
N ΣNi=1(di − µ)2, (1)
where di = ||(ei, eNNi )|| (2)
and
µ = 1
N ΣNi=1di. (3)
Here, U denotes distribution uniformity, di is the Euclidean Distance between the semantic embedding ei of Ii and the context embedding eNNi of its nearest neighbor INNi , while µ is the average
Euclidean Distance. Lower U values suggest improved uniformity. In ideal case, if all points are distributed uniformly, their distance to their nearest neighbor should be equal, and the variance should be zero. In other words, lower the variance indicates more uniform semantic embedding among instructions.
The results reveal notable findings. The first plot of CPI, composed of C30 + C30r, exhibits higher dispersion with a variance of 0.0332, indicating reduced uniformity compared to C30 alone, which is 0.0316. However, an interesting contrast arises when employing Instruction Fusion on C30: by integrating H20 with C30, the dispersion of the second plot decreases by 28.5% from 0.0316 to 0.0226. This significant reduction suggests that Instruction Fusion effectively narrows the gaps in semantic embeddings between existing instructions. This improvement is achieved by combining the objectives of two distinct instructions, demonstrating the method’s efficacy in enhancing instruction diversity.
In conclusion, the Instruction Fusion (IF) method we propose serves as a valuable complement to Evol-Instruct, effectively addressing its
limitations. By amalgamating diverse objectives from various seed instructions, IF leads to the creation of a more diverse, challenging instructions with gradual escalation in the complexity.

heading: 3 Experiments
text: In this section, we report the experiment details of LLMs fine-tuned with instruction generated by IF. We focused on the pass@1 performance under greedy generation settings across five benchmarks: four Python benchmarks (HumanEval (Chen et al., 2021a), MBPP (Austin et al., 2021), HumanEval+, MBPP+ (Liu et al., 2023a)) and the multi-language benchmark MultiPL-E (Cassano et al., 2022). The official EvalPlus code4 was used for evaluating the Python benchmarks, while bigcode-evaluation-harness (Ben Allal et al., 2022) was utilized to evaluate the performance on MultiPL-E.

heading: 3.1 Experiment Details
text: For the selection of our base models, we have chosen to utilize CODELLAMA and CODELLAMAPYTHON to evaluate the effectiveness of our proposed method. It’s important to note that while the recently released open-source model, DeepSeekCoder (DeepSeek, 2023), has achieved state-ofthe-art performance among other base code LLMs, the specifics of their techniques and data were not accessible at the time of writing this paper. Consequently, we have opted not to include DeepSeekCoder in our experimental analysis.
For comparison, we mainly compare with WizardCoder (Luo et al., 2023) since it is the most related work with state of the art performance, and use the same base models as ours. We also included a range of state-of-the-art baseline methods such as CODELLAMA (Roziere et al., 2023), Starcoder (Li et al., 2023), Mistral (Jiang et al., 2023), CodeT5+ (Wang et al., 2021), CodeGen-Mono (Nijkamp et al., 2022), Magicoder (Wei et al., 2023), GPT-3.5 Turbo, and GPT-4 Turbo 5. All results are reported consistently, either from the original papers or the official EvalPlus leaderboard6.
To test Instruction Fusion, we fine-tune both the basic and Python versions of CODELLAMA 13 billion (13B) and 34 billion (34B)7 as base models.
4https://github.com/evalplus/evalplus 5https://platform.openai.com/docs/models 6https://evalplus.github.io/leaderboard.html 7https://huggingface.co/codellama/
For fine-tuning, models were trained with a batch size of 256 over 2 epochs, a learning rate of 2e-5, a cosine learning rate scheduler with 10% warm-up steps, and under bf16 precision on 4× 8 NVIDIA A100 40G GPUs. All other hyper-parameters remained consistent with WizardCoder and Magicoder unless specified otherwise.

heading: 3.2 Evaluation
text: Table 2 presents the performance of various opensource models on Python benchmarks. When compared to the leading closed-source models, GPT3.5 Turbo, our models with 13B and 34B parameters demonstrate superior performance on the HumanEval and HumanEval+ benchmarks. However, there is a noticeable performance gap on the two MPBB benchmarks. This discrepancy may stem from the MBPP test cases, which often require specific prior knowledge for resolution. For instance, in the real test case task_id:20, the prompt "Write a function to check if a given number is woodball or not." requires the model’s understanding of what constitutes a ’woodball’ number. This type of prior knowledge is either not acquired during the pretraining stage or forgotten during the supervised fine-tuning process, leading to a consistent failure across all IF models for this case.
However, compared with all open-source models, the IF models excel in Python code generation performance, surpassing all baselines. The fused instructions, which offer richer semantic, increased difficulty, and smoother transitions, allow both the 13B and 34B versions of IF models to achieve new state-of-the-art performance, outperforming models of comparable sizes. Notably, the 13B version of IF-CLP outperforms all 34B baselines in the HumanEval and HumanEval+ benchmarks and shows comparable results on MBPP and MBPP+.
Tab 3 shows the performance of IF and all other baselines on MultiPL-E. We can observe that, based on CODELLAMA 13B, IF has great advantages on all metrics even comparing with 34B models. The extra diversity provided by fusing instructions can greatly benefits the multi-language performance of Code LLMs.

heading: 3.3 Ablation Study
text: To assess the efficacy of our proposed Instruction Fusion (IF) method, we conducted a comprehensive ablation study. Table 4 presents a comparison between fine-tuning the base model with and without instructions generated by IF. The table clearly
Table 2: Results of experiments on Python benchmarks. The abbreviations ‘CL’ and ‘CLP’ denote the base models CODELLAMA and CODELLAMA-PYTHON, respectively. For IF method, we fine-tune CODELLAMA with full evol-codealpaca-v1 dataset and CODELLAMA-PYTHON with only Python instruction in evol-codealpaca-v1.
Method Size Open-source HumanEval HumanEval+ MBPP MBPP+
GPT4-Turbo - - 85.4 81.7 83.0 70.7 GPT3.5-Turbo - - 72.6 65.9 81.7 69.4
StarCoder 7B weight&data 24.4 20.7 33.1 28.8 Mistral 7B weight 28.7 23.2 50.1 40.9 CODELLAMA-PYTHON 7B weight 37.8 34.1 57.6 45.4 WizardCoder-CLP 7B weight 48.2 40.9 56.6 47.1 MagicoderS-CLP 7B weight&data 70.7 66.5 68.4 56.6 CODELLAMA-PYTHON 13B weight 42.7 36.6 61.2 50.9 StarCoder 15B weight&data 34.1 29.3 55.1 46.1 CodeT5+ 16B weight&data 31.7 26.2 54.6 44.4 CodeGen-Mono 16B weight&data 32.9 27.4 52.6 43.6
CODELLAMA-PYTHON 34B weight 51.8 42.7 67.2 52.9 WizardCoder-CLP 34B weight 73.2 64.6 73.2 59.9
IF-CLP 13B weight&data 73.8 69.5 71.7 61.7 IF-CL 13B weight&data 74.4 68.3 69.7 59.4
IF-CLP 34B weight&data 75.6 69.5 73.7 62.7 IF-CL 34B weight&data 78.7 71.3 71.4 60.7
shows significant improvements across all metrics when employing the IF method. While the benefits of instruction evolution cease after the third round of evolution, Instruction Fusion continues to enhance code generation by overcoming the limitations of Code Evol-Instruct.
Further analysis of Instruction Fusion’s effectiveness involved an ablation study using various combinations of IF and the original evolved instructions. Figure 6 depicts the performance of CODELLAMA when fine-tuned with different volumes of original instructions. The data indicates that the 13B model achieves optimal performance with 30K evolution samples, suggesting that an increased quantity of evolved instructions does not linearly translate to better performance. Moreover, Luo et al. (2023) observed a performance decrease when extending evolution to the fourth round, implying a tangible upper limit to the effectiveness of Code Evol-Instruct in code generation tasks. However, the integration of IF samples with the original evolved dataset transcend this limitation, resulting in a notable enhancement in model performance. Conversely, the 34B model does not show the same constraints with the current range of evolved samples, possibly due to its larger parameter size.
Figure 7 presents the model’s performance when fine-tuned varying amounts of fused instructions together with Python-only evol-codealpaca-v1. The results clearly indicate potential performance
improvements for both the 13B and 34B models with the addition of more fused instructions than 50K. However, considering the data collection costs, the exploration of the exact saturation point for these models is deferred to future work.

heading: 4 Related Work

heading: 4.1 Code Large Language Model
text: The advancement of Large Language Models (LLMs) in open-domain topics has paved the way for extensive research into Code LLMs for code generation tasks. Early studies primarily focused on the pre-training phase of Code LLMs, utilizing models such as Codex (Chen et al., 2021b), CodeT5 (Wang et al., 2021), StarCoder (Li et al.,
2023), and CODE-LLAMA. These methods leveraged coding data from open-source platforms like GitHub8 for pre-training. Generating executable and correct codes from these pre-trained LLMs often required intricate prompt engineering. Despite this, such efforts have significantly propelled the progress of code generation tasks and laid a robust foundation for subsequent research in this domain.

heading: 4.2 Instruction tuning
text: The reliance on task-specific prompts for extracting information from LLMs, due to its labor-intensive nature and limited generalizability, led to the introduction of instruction tuning (Wei et al., 2022). This method enhances the zero-shot capabilities of LLMs in performing tasks via natural language instructions, allowing them to respond to more general human requests. However, relying on humanwritten instructions or templates limits the quantity, diversity, and creativity of data.
In response, Taori et al. (2023) developed SELF8https://github.com/
INSTRUCT, employing LLMs for both data generation and instruction tuning to create superior synthetic instructions and responses. This technique, utilized by Taori et al. (2023); Roziere et al. (2023), involves collecting high-quality synthetic data from powerful/specialized LLMs for fine-tuning. In the realm of Code LLMs, Code Alpaca (Chaudhary, 2023) applied SELF-INSTRUCT to gather instructions and responses from text-davinci-0039. Building on synthetic data creation, Xu et al. (2023) proposed Evol-Instruct for evolving instructions to enhance difficulty and diversity. This approach, exemplified in WizardCoder (Luo et al., 2023), achieved state-of-the-art performance in Code LLMs but faced evolution process limitations.
Concurrently, Wei et al. (2023) used LLMs to generate high-quality synthetic data from realworld code snippets, showing promise in code gen-
9https://platform.openai.com/docs/models/gpt-3-5
eration. However, as their approach is orthogonal to WizardCoder, it remains similarly orthogonal to ours.

heading: 5 Conclusion
text: In this paper, we introduce the Instruction Fusion technique, an advancement of Evol-Instruct specifically tailored for code generation tasks. This technique involves the fusion of two evolved instructions into a single, cohesive prompt. It excels in creating instruction sets that are reasonably complex, facilitating a progressive increase in difficulty which is achieved as objectives can be learned separately. The fused instructions also grants a better diversity of the instruction pool. Notably, Large Language Models (LLMs) fine-tuned with IF have demonstrated superior performance across the top five benchmarks, setting new state-of-theart records.
Limitations
The primary limitation of the Instruction Fusion method proposed in our study is its cost. We employ GPT-4 Turbo as the Teacher Language Model (LLM), and the fusion prompt incorporates a higher token count compared to the Evol-Instruction. This results in increased expenses for data collection, which cost around 2,200 USD in total for 100k samples (responses included). However, it is worth noting compared with manual annotation. Moreover, such costs are rapidly decreasing, thanks to the swift advancements in LLM technology.
Another point of concern is the success rate of the fusion process. Fusion of Python-based instructions achieves a pass rate of 93% (where 7% of the fused instructions are deemed unsolvable by GPT-4 Turbo). In contrast, cross-language fusion exhibits a lower pass rate of approximately 63%. This discrepancy arises because many tasks are typically resolved using specific programming languages. Consequently, fusing instructions from different programming languages often leads to impractical outcomes. To address this issue, we propose the development of new prompts that categorize tasks prior to sampling, a solution which could be to explore in future research.
Ethics Statement
In our study, we employ GPT-4 for the purpose of amalgamating seed coding instructions, which can not generate sensitive data such as personal
information. Our seed instructions are derived from the evol-codealpaca-v1 dataset, a wellacknowledged resource within the open-source community.

heading: Future Work
text: To address this issue, we propose the development of new prompts that categorize tasks prior to sampling, a solution which could be to explore in future research