InferForge — Ornith-1.5-9B-Q4_K_M

A Q4_K_M GGUF quantization of Ornith-1.5-9B, prepared by InferForge for efficient local inference with llama.cpp.

This quantization is intended for practical local inference, with a focus on:

• Coding
• Technical analysis
• DevOps and infrastructure
• Kubernetes troubleshooting
• Distributed systems
• System architecture
• Agent and worker workloads
• Long-context inference
• Efficient GPU/CPU inference

Model Information

Property	Value
Base model	Ornith-1.5-9B
Parameters	~9B
Format	GGUF
Quantization	Q4_K_M
Inference engine	llama.cpp
Tested context	131,072 tokens (128K)
Primary target	Local inference
Maintainer	InferForge

Quantization

The original Ornith model was converted to GGUF and quantized locally using llama.cpp.

The resulting file is:

Ornith-1.5-9B-Q4_K_M.gguf

Why Q4_K_M?

Q4_K_M provides a practical balance between:

• Model quality
• Memory requirements
• Inference speed
• Consumer hardware compatibility

It is intended as a general-purpose quantization for users who want to run the model locally without the memory requirements of higher-precision formats.

Hardware Compatibility

This quantization can be used with:

• NVIDIA GPUs
• AMD GPUs
• Intel GPUs
• CPU-only systems
• Mixed GPU/CPU inference

Actual performance and maximum context length depend on available VRAM, system RAM, memory bandwidth, and inference backend.

Usage with llama.cpp

Interactive CLI

llama-cli \
  -m Ornith-1.5-9B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Docker

docker run --rm -it \
  --gpus all \
  --entrypoint /app/llama-cli \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Ornith-1.5-9B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Usage with llama-server

llama-server provides an OpenAI-compatible HTTP API.

Native

llama-server \
  -m Ornith-1.5-9B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

Docker

docker run --rm -d \
  --name ornith-1.5-9b-q4 \
  --gpus all \
  --entrypoint /app/llama-server \
  -p 30000:8080 \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Ornith-1.5-9B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

The OpenAI-compatible API is available at:

http://localhost:30000/v1

Example API Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ornith-1.5-9B-Q4_K_M.gguf",
    "messages": [
      {
        "role": "user",
        "content": "Explain how Kubernetes Pods differ from Docker containers."
      }
    ],
    "temperature": 0.2,
    "max_tokens": 512
  }'

Context Length

This quantization has been tested with:

131072 tokens

or approximately:

128K context

Large context windows significantly increase memory requirements.

On GPUs with limited VRAM, llama.cpp may use system RAM for part of the workload.

Users should benchmark context sizes appropriate for their own hardware.

Quantization Trade-offs

Quantization reduces model memory requirements by representing model weights with lower numerical precision.

Q4_K_M is designed to retain a practical level of model quality while substantially reducing memory requirements compared with higher-precision formats.

The trade-off is that quantized inference may produce different results from the original higher-precision model.

For workloads where maximum quality is more important than memory efficiency, higher-precision variants may be preferable.

For local consumer hardware, Q4_K_M provides a practical deployment point between model quality and resource requirements.

Limitations

• Performance varies significantly between hardware and inference backends.
• Large context sizes require substantial memory.
• Q4_K_M introduces quantization error compared with higher-precision versions.
• Results may differ from the original higher-precision model.
• This repository contains a quantized derivative and is not a new base model.

File Integrity

SHA256:

6c531b2ca8f6ce6667485365c66348735197cfaac12de3e05f300b6e2a53aa4d

Generate the checksum with:

sha256sum Ornith-1.5-9B-Q4_K_M.gguf

Credits

This quantization is derived from the Ornith-1.5 model family by Ornith AI.

Original model family:

ornith-ai/Ornith-1.5

This repository contains an independent GGUF quantization prepared by InferForge.

InferForge is not affiliated with or endorsed by Ornith AI.

License

The original Ornith model is released under the MIT License.

Please refer to the original model repository for the complete license text and upstream licensing information.

The quantized file in this repository is distributed under the applicable terms of the original model license.

About InferForge

InferForge is an independent model experimentation and inference project focused on practical and reproducible local AI infrastructure.

Areas of interest include:

• GGUF quantization
• Local LLM inference
• Consumer GPU optimization
• Inference benchmarking
• Long-context inference
• Efficient model serving
• Agent-oriented workloads
• Reproducible AI infrastructure

Additional quantizations, models, and inference experiments may be added to the InferForge collection over time.