Andreevai/SmolLM2-360M-Instruct-GGUF

This repository contains official GGUF quantizations of HuggingFaceTB/SmolLM2-360M-Instruct, converted using llama.cpp.

SmolLM2 is a family of compact language models developed by Hugging Face, specifically designed for ultra-fast local inference on edge devices, mobile phones, Raspberry Pi, and local developer workstations.

Available Files

Filename	Quant Method	File Size	Description
SmolLM2-360M-Instruct-Q4_K_M.gguf	Q4_K_M	258 MB	Balanced 4-bit quant, recommended for ultra-low memory / edge devices
SmolLM2-360M-Instruct-Q5_K_M.gguf	Q5_K_M	276 MB	Higher precision 5-bit quant with minimal perplexity degradation
SmolLM2-360M-Instruct-Q8_0.gguf	Q8_0	368 MB	Near lossless 8-bit precision, fast CPU inference

Quick Start with Ollama

Run directly with Ollama using the Hugging Face direct link:

# Run Q4_K_M
ollama run hf.co/Andreevai/SmolLM2-360M-Instruct-GGUF:SmolLM2-360M-Instruct-Q4_K_M.gguf

# Run Q8_0
ollama run hf.co/Andreevai/SmolLM2-360M-Instruct-GGUF:SmolLM2-360M-Instruct-Q8_0.gguf

Or create a custom Modelfile:

FROM ./SmolLM2-360M-Instruct-Q4_K_M.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"

ollama create smollm2-360m -f Modelfile
ollama run smollm2-360m

Quick Start with llama.cpp

llama-cli -m SmolLM2-360M-Instruct-Q4_K_M.gguf -p "<|im_start|>user\nWrite a quick python script to ping an API.<|im_end|>\n<|im_start|>assistant\n" -n 256

Prompt Template

SmolLM2 uses ChatML:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant