Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q8_0-GGUF

This model was converted to GGUF format from FuseAI/FuseChat-Llama-3.2-3B-Instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Model details:

We present FuseChat-3.0, a series of models crafted to enhance performance by integrating the strengths of multiple source LLMs into more compact target LLMs. To achieve this fusion, we utilized four powerful source LLMs: Gemma-2-27B-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For the target LLMs, we employed three widely-used smaller models—Llama-3.1-8B-Instruct, Gemma-2-9B-It, and Qwen-2.5-7B-Instruct—along with two even more compact models—Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. The implicit model fusion process involves a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies between target and source LLMs, and Direct Preference Optimization (DPO) for learning preferences from multiple source LLMs. The resulting FuseChat-3.0 models demonstrated substantial improvements in tasks related to general conversation, instruction following, mathematics, and coding. Notably, when Llama-3.1-8B-Instruct served as the target LLM, our fusion approach achieved an average improvement of 6.8 points across 14 benchmarks. Moreover, it showed significant improvements of 37.1 and 30.1 points on instruction-following test sets AlpacaEval-2 and Arena-Hard respectively. We have released the FuseChat-3.0 models on Huggingface, stay tuned for the forthcoming dataset and code.

    Overview

Combining the strengths of multiple large language models (LLMs) represents a promising approach to enhance individual model capabilities. Model fusion is a technique that integrates the strengths of robust source LLMs into a target LLM.

Previous iterations of the FuseChat series employed probabilistic distribution matrices generated by source models to transfer knowledge to target models. We refer to this method as explicit model fusion (EMF) because it involves a well-defined knowledge transfer process. While applicable to models with varying architectures and sizes, and without increasing memory overhead during inference, this approach presents notable challenges such as vocabulary alignment and the merging of distribution matrices from different LLMs. These issues complicate model fusion, reduce its efficiency, and may introduce noise and errors and affect the fusion results.

FuseChat-3.0, however, takes a different approach by enhancing a single LLM through implicit learning from robust open-source LLMs, a process we term implicit model fusion (IMF). The concept of IMF has been widely utilized to improve the performance of weaker models. For instance, a weak model can be boosted through fine-tuning with outputs from stronger LLMs. Moreover, a reward model can be trained using outputs from various LLMs, enabling it to learn and capture the differences in capabilities between the LLMs. Zephyr further collects responses from multiple LLMs and ranks them with GPT-4 to obtain preference data for training the policy. Inspired by recent alignment techniques, we propose an IMF method to transfer the capabilities of source LLMs to a target LLM through preference optimization.

Our IMF method follows a three-stage process aimed at effectively transferring capabilities from source LLMs to a target LLM. First, during dataset construction, we sample N responses from each of the source LLMs and annotate these responses using an external reward model. Second, in the supervised fine-tuning (SFT) stage, we fine-tune the target model using the best responses, which not only enhances the target model's capabilities but also helps mitigate the distributional gap between the source and target models. Finally, in the direct preference optimization (DPO) stage, we optimize the target model by using the best and worst responses from the source models as preference pairs, further enhancing the target model's performance. The complete pipeline will be detailed in the following paragraph.

    Dataset






    


    Prompt Selection

Our datasets were designed to enhance model's instruction following, general conversation, mathematics, coding, and Chinese-language capabilities. We selected data from open-source community datasets, applying targeted filtering and preprocessing. Key datasets and filtering criteria included:

Instruction Following & General Conversation: Sourced from UltraFeedback, Magpie-Pro-DPO-100K-v0.1, and HelpSteer2, excluding code and math data. Mathematics: Selected from OpenMathInstruct-2, with nearly 60,000 unique samples. Coding: Curated from leetcode and self-oss-instruct-sc2-exec-filter-50k, retaining prompts with test cases. Chinese Language: Integrated alpaca_gpt4_zh and Magpie-Qwen2-Pro-200K-Chinese, filtering out code and math prompts to retain approximately 10,000 high-quality samples.

    Response Sampling

For each dataset's prompts, we synthesized responses mainly from four different series of source models, specifically Gemma-2-27b-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.

Instruction Following & General Conversation: We sampled each prompt five times from all the source models. Mathematics: We retained the responses generated by Llama-3.1-405B-Instruct from the original dataset (OpenMathInstruct-2) and additionally sampled responses using Qwen-2.5-Math-72B-Instruct. Coding: We sampled each prompt eight times for all source models. Chinese Language: We included single response sampled exclusively from Qwen-2.5-72B-Instruct.

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q8_0-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q8_0-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q8_0-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q8_0-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q8_0.gguf -c 2048