MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Authors: name: Bingchang Liu; name: Chaoyu Chen; name: Cong Liao; name: Zi Gong; name: Huan Wang; name: Zhichao Lei; name: Ming Liang; name: Dajun Chen; name: Min Shen; name: Hailian Zhou; name: Hang Yu; name: Jianguo Li
Published: 2023-11-04 02:22:40

Abstract: Code LLMs have emerged as a specialized research field, with remarkable
studies dedicated to enhancing model's coding capabilities through fine-tuning
on pre-trained models. Previous fine-tuning approaches were typically tailored
to specific downstream tasks or scenarios, which meant separate fine-tuning for
each task, requiring extensive training resources and posing challenges in
terms of deployment and maintenance. Furthermore, these approaches failed to
leverage the inherent interconnectedness among different code-related tasks. To
overcome these limitations, we present a multi-task fine-tuning framework,
MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
By incorporating various loss functions, we effectively address common
challenges in multi-task learning, such as data imbalance, varying difficulty
levels, and inconsistent convergence speeds. Extensive experiments have
conclusively demonstrated that our multi-task fine-tuning approach outperforms
both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
of tasks. Moreover, MFTcoder offers efficient training capabilities, including
efficient data tokenization modes and PEFT fine-tuning, resulting in
significantly improved speed compared to traditional fine-tuning methods.
MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
performance (67\%, zero-shot). MFTCoder is open-sourced at
https://github.com/codefuse-ai/MFTCOder