InferForge — Ornith-1.5-9B-Q3_K_M

A Q3_K_M GGUF quantization of Ornith-1.5-9B, prepared by InferForge for efficient local inference with llama.cpp.

This quantization is intended for practical local inference, with a focus on:

• Coding
• Technical analysis
• DevOps and infrastructure
• Kubernetes troubleshooting
• Distributed systems
• System architecture
• Agent and worker workloads
• Long-context inference
• Efficient GPU/CPU inference

Model Information

Property	Value
Base model	Ornith-1.5-9B
Parameters	~9B
Format	GGUF
Quantization	Q3_K_M
Inference engine	llama.cpp
Tested context	131,072 tokens (128K)
Primary target	Local inference
Maintainer	InferForge

Quantization

The original Ornith model was converted to GGUF and quantized locally using llama.cpp.

The resulting file is:

Ornith-1.5-9B-Q3_K_M.gguf

Why Q3_K_M?

Q3_K_M provides a lower-memory alternative to higher-bit quantizations while retaining a practical level of model capability.

It is intended for users who need to reduce memory requirements while maintaining useful model quality for local inference.

The main trade-offs are:

• Lower memory requirements
• Reduced storage requirements
• Potentially higher inference throughput
• Greater quantization loss compared with Q4 and higher-precision formats

Hardware Compatibility

This quantization can be used with:

• NVIDIA GPUs
• AMD GPUs
• Intel GPUs
• CPU-only systems
• Mixed GPU/CPU inference

Actual performance and maximum context length depend on available VRAM, system RAM, memory bandwidth, and inference backend.

Usage with llama.cpp

Interactive CLI

llama-cli \
  -m Ornith-1.5-9B-Q3_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Docker

docker run --rm -it \
  --gpus all \
  --entrypoint /app/llama-cli \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Ornith-1.5-9B-Q3_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Usage with llama-server

llama-server provides an OpenAI-compatible HTTP API.

Native

llama-server \
  -m Ornith-1.5-9B-Q3_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

Docker

docker run --rm -d \
  --name ornith-1.5-9b-q3 \
  --gpus all \
  --entrypoint /app/llama-server \
  -p 30000:8080 \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Ornith-1.5-9B-Q3_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

The OpenAI-compatible API is available at:

http://localhost:30000/v1

Example API Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ornith-1.5-9B-Q3_K_M.gguf",
    "messages": [
      {
        "role": "user",
        "content": "Explain how Kubernetes Pods differ from Docker containers."
      }
    ],
    "temperature": 0.2,
    "max_tokens": 512
  }'

Context Length

This quantization has been tested with:

131072 tokens

or approximately:

128K context

Large context windows significantly increase memory requirements.

On GPUs with limited VRAM, llama.cpp may use system RAM for part of the workload.

Users should benchmark context sizes appropriate for their own hardware.

Quantization Trade-offs

Quantization reduces model memory requirements by representing model weights with lower numerical precision.

Q3_K_M uses fewer bits per weight than Q4_K_M, resulting in a smaller model and lower memory requirements.

The trade-off is increased quantization error compared with Q4 and higher-precision formats.

For workloads where memory efficiency is important, Q3_K_M can provide a useful alternative to larger quantizations.

For workloads where preserving more model fidelity is important, Q4_K_M or higher-precision variants may be preferable.

Limitations

• Performance varies significantly between hardware and inference backends.
• Large context sizes require substantial memory.
• Q3_K_M introduces more quantization error than Q4_K_M.
• Results may differ from the original higher-precision model.
• This repository contains a quantized derivative and is not a new base model.

File Integrity

SHA256:

004929af333d4d53d0df67021ad99b7edcbf980696c1a963ad6120f38b889a2a

Generate the checksum with:

sha256sum Ornith-1.5-9B-Q3_K_M.gguf

Credits

This quantization is derived from the Ornith-1.5 model family by Ornith AI.

Original model family:

ornith-ai/Ornith-1.5

This repository contains an independent GGUF quantization prepared by InferForge.

InferForge is not affiliated with or endorsed by Ornith AI.

License

The original Ornith model is released under the MIT License.

Please refer to the original model repository for the complete license text and upstream licensing information.

The quantized file in this repository is distributed under the applicable terms of the original model license.

About InferForge

InferForge is an independent model experimentation and inference project focused on practical and reproducible local AI infrastructure.

Areas of interest include:

• GGUF quantization
• Local LLM inference
• Consumer GPU optimization
• Inference benchmarking
• Long-context inference
• Efficient model serving
• Agent-oriented workloads
• Reproducible AI infrastructure

Additional quantizations, models, and inference experiments may be added to the InferForge collection over time.