Local Inference on GPU with Qwen 2.5 Coder Mermaid Fine-tune

Install Dependencies

pip install -U llama-cpp-python

Local Inference on GPU

Model Page

https://huggingface.co/0nell0/Qwen-2.5-Coder-Instruct-Mermaid-finetune

⚠️ If the generated code snippets do not work, please open an issue on either:

• the model repo:
  https://huggingface.co/0nell0/Qwen-2.5-Coder-Instruct-Mermaid-finetune

• or on huggingface.js:
  https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/model-libraries-snippets.ts

Python Example

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="0nell0/Qwen-2.5-Coder-Instruct-Mermaid-finetune",
    filename="qwen2.5-coder-0.5b-instruct.F16.gguf",
    chat_format="chatml",
    n_gpu_layers=-1,     # full GPU offload
    n_ctx=8192,
    verbose=False
)

input_prompt = "make a system design mermaid for todo CRUD app"

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a System Architect who writes Mermaid diagrams only."
        },
        {
            "role": "user",
            "content": input_prompt
        }
    ],
    temperature=0.7,
    stop=["<|im_end|>", "<|im_start|>"]
)

print(response["choices"][0]["message"]["content"])

Explanation

This notebook demonstrates how to:

• Run a GGUF LLM locally using llama-cpp-python
• Fully offload inference to the GPU
• Use the fine-tuned Qwen 2.5 Coder Mermaid model
• Generate Mermaid system design diagrams from prompts

Parameter Breakdown

Example Prompt

make a system design mermaid for todo CRUD app

Example Output

flowchart LR
    A(["Add New Task"])
    B(["Complete Task"])
    C("completed": true)
    D["Delete Task"]
    E["Mark as Done"]
    A -.-> B
    B -->|Done| C
    B -.-> D
    C --> E
    D -->|Completed| E