TrabEsrever/dolphin-2.9-llama3-70b-GGUF

Dolphin 2.9 Llama 3 70b 🐬 (GGUF)

Quantized and converted to GGUF for Ollama.

Example Modelfile

FROM ./dolphin-2.9-llama3-70b-Q6_K.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
SYSTEM """You are Dolphin, a helpful AI assistant.
"""
PARAMETER num_ctx 8192
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

Provided files

Name	Quant method	Size
dolphin-2.9-llama3-70b-Q6_K.gguf-part00000 dolphin-2.9-llama3-70b-Q6_K.gguf-part00001	Q6_K	54G
dolphin-2.9-llama3-70b-Q8_0.gguf-part00000 dolphin-2.9-llama3-70b-Q8_0.gguf-part00001	Q8_0	70G

Combine multi part files into one GGUF

cat dolphin-2.9-llama3-70b-Q6_K.gguf-part00000 dolphin-2.9-llama3-70b-Q6_K.gguf-part00001 > dolphin-2.9-llama3-70b-Q6_K.gguf