---
base_model: rombodawg/Rombos-Coder-V2.5-Qwen-7b
tags:
- llama-cpp
- gguf-my-repo
- cline
---

# benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF
This model was converted to GGUF format from [`rombodawg/Rombos-Coder-V2.5-Qwen-7b`](https://huggingface.co/rombodawg/Rombos-Coder-V2.5-Qwen-7b) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/rombodawg/Rombos-Coder-V2.5-Qwen-7b) for more details on the model.

## Use with Cline and Ollama

use this template file from https://github.com/maryasov/ollama-models-instruct-for-cline

```
FROM rombos-coder-v2.5-qwen-7b-q8_0.gguf
TEMPLATE """{{- /* Initial system message with core instructions */ -}}
{{- if .Messages }}
{{- if or .System .Tools }}
<|im_start|>system
{{- if .System }}
{{ .System }}
{{- end }}                                                                                                                                                                                                  {{- if .Tools }}
# Tools and XML Schema
You have access to the following tools. Each tool must be used according to this XML schema:

<tools>
{{- range .Tools }}
{{ .Function }}
{{- end }}
</tools>

## Tool Use Format
1. Think about the approach in <thinking> tags
2. Call tool using XML format:
   <tool_name>
     <param_name>value</param_name>
   </tool_name>
3. Process tool response from:
   <tool_response>result</tool_response>
{{- end }}
<|im_end|>
{{- end }}

{{- /* Message handling loop */ -}}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}

{{- /* User messages */ -}}
{{- if eq .Role "user" }}
<|im_start|>user
{{ .Content }}
<|im_end|>

{{- /* Assistant messages */ -}}
{{- else if eq .Role "assistant" }}
<|im_start|>assistant
{{- if .Content }}
{{ .Content }}
{{- else if .ToolCalls }}
{{- range .ToolCalls }}
<thinking>
[Analysis of current state and next steps]
</thinking>

<{{ .Function.Name }}>
{{- range $key, $value := .Function.Arguments }}
<{{ $key }}>{{ $value }}</{{ $key }}>
{{- end }}
</{{ .Function.Name }}>
{{- end }}
{{- end }}
{{- if not $last }}<|im_end|>{{- end }}

{{- /* Tool response handling */ -}}
{{- else if eq .Role "tool" }}
<|im_start|>user
<tool_response>
{{ .Content }}
</tool_response>
<|im_end|>
{{- end }}

{{- /* Prepare for next assistant response if needed */ -}}
{{- if and (ne .Role "assistant") $last }}
<|im_start|>assistant
{{- end }}
{{- end }}

{{- /* Handle single message case */ -}}
{{- else }}
{{- if .System }}
<|im_start|>system
{{ .System }}
<|im_end|>
{{- end }}

{{- if .Prompt }}
<|im_start|>user
{{ .Prompt }}
<|im_end|>
{{- end }}

<|im_start|>assistant
{{- end }}
{{ .Response }}
{{- if .Response }}<|im_end|>{{- end }}
"""
PARAMETER repeat_last_n 64
PARAMETER repeat_penalty 1.1
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.1
PARAMETER top_k 40
PARAMETER top_p 0.9
```

## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo benhaotang/Rombos-Coder-V2.5-Qwen-7b-Q8_0-GGUF --hf-file rombos-coder-v2.5-qwen-7b-q8_0.gguf -c 2048
```