Nexuss0781/SmolLM2-135M-Instruct-GGUF

SmolLM2 135M Instruct — Unofficial GGUF Conversions

Original source model and publisher: HuggingFaceTB/SmolLM2-135M-Instruct by HuggingFaceTB. This repository contains file-format conversions only. It does not claim authorship, ownership, training, or creation of the underlying SmolLM2 model.

This repository provides local CPU-oriented GGUF conversions of the upstream SmolLM2 135M Instruct model for use with llama.cpp and compatible runtimes. The source model's Apache-2.0 license is retained; no separate or replacement license is asserted for the underlying model.

Available files

File	Format	Intended use	File size
SmolLM2-135M-Instruct-f16.gguf	F16 GGUF	Quality-preserving local inference	258 MiB
SmolLM2-135M-Instruct-Q6_K.gguf	Q6_K GGUF	Lower-memory CPU inference	132 MiB

The F16 variant preserves the original converted weight precision. The Q6_K variant is provided for systems where memory usage is more important than retaining the F16 representation. Use the F16 file for the highest-fidelity local behavior.

Quick start with llama.cpp

./llama-cli \
  --model SmolLM2-135M-Instruct-f16.gguf \
  --conversation \
  --n-gpu-layers 0

For persistent local serving, start llama-server once and send OpenAI-compatible requests to the local endpoint:

./llama-server \
  --model SmolLM2-135M-Instruct-f16.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 2048 \
  --n-gpu-layers 0

Integrity verification

Verify downloaded files with:

sha256sum -c CHECKSUMS.sha256

Conversion details

The model was converted from the official Hugging Face checkpoint with the official llama.cpp Hugging Face-to-GGUF converter. The F16 GGUF output was retained as the quality-preserving variant, then quantized with llama-quantize to produce the Q6_K variant. See conversion-metadata.json for the artifact metadata.

Upstream source, ownership, and license

The underlying model is published by HuggingFaceTB at HuggingFaceTB/SmolLM2-135M-Instruct. The upstream model card identifies its license as Apache-2.0. The conversion distributor claims no ownership or authorship of the original model and does not impose a new license on it. Retain the upstream attribution and consult the source model card for limitations, training details, and citation information.