Instructions to use FoolDev/Thanatos-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FoolDev/Thanatos-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="FoolDev/Thanatos-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("FoolDev/Thanatos-27B", dtype="auto") - llama-cpp-python
How to use FoolDev/Thanatos-27B with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="FoolDev/Thanatos-27B", filename="Thanatos-27B.Q4_K_M.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use FoolDev/Thanatos-27B with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf FoolDev/Thanatos-27B:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf FoolDev/Thanatos-27B:Q4_K_M
Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use FoolDev/Thanatos-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FoolDev/Thanatos-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- SGLang
How to use FoolDev/Thanatos-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "FoolDev/Thanatos-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Thanatos-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use FoolDev/Thanatos-27B with Ollama:
ollama run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Unsloth Studio new
How to use FoolDev/Thanatos-27B with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for FoolDev/Thanatos-27B to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for FoolDev/Thanatos-27B to start chatting
- Pi new
How to use FoolDev/Thanatos-27B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "FoolDev/Thanatos-27B:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use FoolDev/Thanatos-27B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf FoolDev/Thanatos-27B:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default FoolDev/Thanatos-27B:Q4_K_M
Run Hermes
hermes
- Docker Model Runner
How to use FoolDev/Thanatos-27B with Docker Model Runner:
docker model run hf.co/FoolDev/Thanatos-27B:Q4_K_M
- Lemonade
How to use FoolDev/Thanatos-27B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull FoolDev/Thanatos-27B:Q4_K_M
Run and chat with the model
lemonade run user.Thanatos-27B-Q4_K_M
List all available models
lemonade list
Revert "docs: align README + CHANGELOG + Modelfile + examples with the qwen36 re-stamp (07fa120)"
Browse filesThis reverts commit e9c7366, restoring the collapsed-workaround
narrative now that the HF bundle is back on qwen35 (upload 72259c1).
A fresh-pull integration test on the Strix Halo reference machine
reproduced the heal-hf friction end-to-end; the friction was worse
in practice than the architectural-honesty payoff. CHANGELOG
[Unreleased] picks up a "Round trip" bullet documenting the
qwen35 -> qwen36 -> qwen35 sequence; net blob is 5ed60d0a... again,
same as before the round trip.
- CHANGELOG.md +49 -34
- Modelfile +10 -20
- README.md +58 -87
- examples/README.md +11 -13
|
@@ -8,43 +8,58 @@ and documentation**, not the underlying base model.
|
|
| 8 |
## [Unreleased]
|
| 9 |
|
| 10 |
### Changed
|
| 11 |
-
- **Bundle re-stamped `general.architecture: '
|
| 12 |
-
in commit
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
`
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
the
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
### Note
|
| 42 |
- `make heal-hf`, `make load-bundle`, and `scripts/rename_arch.py`
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
manifest; `make build` bypasses the bundle entirely (downloads
|
| 47 |
-
qwen35-stamped from unsloth).
|
| 48 |
|
| 49 |
### Added
|
| 50 |
- `scripts/check.sh` now greps for the `VAR="$(cmd 2>/dev/null | filter)"`
|
|
|
|
| 8 |
## [Unreleased]
|
| 9 |
|
| 10 |
### Changed
|
| 11 |
+
- **Bundle re-stamped `general.architecture: 'qwen36'` β `'qwen35'`**
|
| 12 |
+
in commit 964e418 (HF-direct `hf upload`). Reverses the v0.6.0
|
| 13 |
+
`qwen36` stamp from e1f78fa, which itself flipped qwen35 β qwen36
|
| 14 |
+
deliberately. The proximate trigger: today's subagent recheck
|
| 15 |
+
found no PR or tracking issue for a `qwen36` arch entry in either
|
| 16 |
+
`ggml-org/llama.cpp` or `ollama/ollama`, and the actual Qwen 3.6
|
| 17 |
+
family upstream already loads under `qwen35moe` β so the qwen36
|
| 18 |
+
stamp was going to stay ahead-of-upstream indefinitely, not for a
|
| 19 |
+
release or two. Day-to-day cost (every fresh
|
| 20 |
+
`ollama run hf.co/FoolDev/Thanatos-27B` hitting a 500, every user
|
| 21 |
+
needing `make heal-hf` to recover) outweighed the
|
| 22 |
+
architectural-honesty benefit. Tensor data stays byte-identical;
|
| 23 |
+
only the `general.architecture` KV (and the namespaced KV keys)
|
| 24 |
+
flips. New bundle blob SHA is
|
| 25 |
+
`5ed60d0af4650a854b1755bd392f9aef4872643dc25a254bc68043fa638392a0`
|
| 26 |
+
β the same hash `make load-bundle` and `make heal-hf` have been
|
| 27 |
+
producing locally this whole session.
|
| 28 |
+
- README "Heads up" callout (qwen36 load-failure warning) removed.
|
| 29 |
+
TL;DR collapsed back from "three paths around it" to a single
|
| 30 |
+
one-liner: `ollama run hf.co/FoolDev/Thanatos-27B` now loads on
|
| 31 |
+
stock Ollama. `make heal-hf` is mentioned as the recovery path
|
| 32 |
+
for anyone who pulled the v0.6.0 bundle and still has the qwen36
|
| 33 |
+
blob in their store.
|
| 34 |
+
- README "Architecture" section, Quick start option A,
|
| 35 |
+
`examples/README.md` Ollama setup, and `Modelfile` preamble all
|
| 36 |
+
collapsed back to the pre-qwen36 single-path narrative. The
|
| 37 |
+
long "qwen36 is unloadable / here are the workarounds" prose
|
| 38 |
+
was replaced with a one-paragraph history note pointing at
|
| 39 |
+
commit 964e418 + the [CHANGELOG](CHANGELOG.md) entry. Net -39
|
| 40 |
+
lines across the three doc surfaces. `scripts/rename_arch.py`
|
| 41 |
+
example in Architecture flipped from qwen36 β qwen35 to qwen35
|
| 42 |
+
β qwen36 to match its new "tool kept around for future flips"
|
| 43 |
+
framing.
|
| 44 |
+
- **Round trip (2026-05-19 evening).** The bundle briefly flipped
|
| 45 |
+
back to `qwen36` in commit `07fa120` plus the doc re-expansion
|
| 46 |
+
in `e9c7366`, on the theory that architectural-honesty was worth
|
| 47 |
+
the heal-hf friction. A fresh-pull integration test on the
|
| 48 |
+
Strix Halo box reproduced the friction end-to-end (`ollama run`
|
| 49 |
+
500 on first pull, recover via `make heal-hf`, smoke-test then
|
| 50 |
+
passed) β and the friction was worse in practice than the doc
|
| 51 |
+
prose suggested. Reverted to qwen35 in `hf upload` commit
|
| 52 |
+
`72259c1` + the present commit (revert of `e9c7366`). Net
|
| 53 |
+
bundle state: same `5ed60d0a...` blob as before the round
|
| 54 |
+
trip; no user-facing impact unless a user happened to pull in
|
| 55 |
+
the ~1 hour window when the bundle was qwen36-stamped (for
|
| 56 |
+
them, one `make heal-hf` is the recovery).
|
| 57 |
|
| 58 |
### Note
|
| 59 |
- `make heal-hf`, `make load-bundle`, and `scripts/rename_arch.py`
|
| 60 |
+
stay in the repo and stay useful. heal-hf is idempotent on qwen35
|
| 61 |
+
(skips early); load-bundle does a no-op rebadge on a qwen35 bundle
|
| 62 |
+
and runs `ollama create` as normal.
|
|
|
|
|
|
|
| 63 |
|
| 64 |
### Added
|
| 65 |
- `scripts/check.sh` now greps for the `VAR="$(cmd 2>/dev/null | filter)"`
|
|
@@ -7,28 +7,18 @@
|
|
| 7 |
# image input, or wait for the fix. See the Vision section in README.md.
|
| 8 |
#
|
| 9 |
# This repo bundles a single GGUF: Thanatos-27B.Q4_K_M.gguf (~17 GB),
|
| 10 |
-
# stamped `general.architecture: '
|
| 11 |
-
#
|
| 12 |
-
#
|
| 13 |
-
#
|
| 14 |
-
#
|
| 15 |
-
#
|
| 16 |
-
#
|
| 17 |
-
# make load-bundle # one-shot: smudge LFS pointer, rebadge a local
|
| 18 |
-
# # copy qwen36 -> qwen35, run `ollama create`.
|
| 19 |
-
# make heal-hf # rebadge an already-pulled `hf.co/...` tag's
|
| 20 |
-
# # blob in place (qwen36 -> qwen35) and rewrite
|
| 21 |
-
# # the manifest digest.
|
| 22 |
-
#
|
| 23 |
-
# See README "Architecture" for the qwen35 -> qwen36 -> qwen35 ->
|
| 24 |
-
# qwen36 stamp history through commit 07fa120.
|
| 25 |
#
|
| 26 |
# For other quants (Q3_K_S, Q5_K_M, Q6_K, etc.), `make build QUANT=Q3_K_S`
|
| 27 |
-
# downloads the chosen quant from unsloth/Qwen3.6-27B-GGUF
|
| 28 |
-
#
|
| 29 |
-
#
|
| 30 |
-
#
|
| 31 |
-
# Q3_K_S (alphabetically-first heuristic).
|
| 32 |
#
|
| 33 |
# Other GGUF sources (use with `make build GGUF_PATH=...`):
|
| 34 |
# https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
|
|
|
|
| 7 |
# image input, or wait for the fix. See the Vision section in README.md.
|
| 8 |
#
|
| 9 |
# This repo bundles a single GGUF: Thanatos-27B.Q4_K_M.gguf (~17 GB),
|
| 10 |
+
# stamped `general.architecture: 'qwen35'` β the arch entry every
|
| 11 |
+
# released llama.cpp / Ollama loads under for the Qwen 3.6 hybrid
|
| 12 |
+
# SSM + attention family. `ollama create thanatos-27b -f Modelfile &&
|
| 13 |
+
# ollama run thanatos-27b` loads it directly. See README
|
| 14 |
+
# "Architecture" for the qwen35 -> qwen36 -> qwen35 history through
|
| 15 |
+
# v0.6.0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
#
|
| 17 |
# For other quants (Q3_K_S, Q5_K_M, Q6_K, etc.), `make build QUANT=Q3_K_S`
|
| 18 |
+
# downloads the chosen quant from unsloth/Qwen3.6-27B-GGUF and patches
|
| 19 |
+
# FROM in a temp Modelfile copy. The Q3_K_S used to ship in this repo;
|
| 20 |
+
# it was removed so HF's Ollama bridge picks Q4_K_M as the default
|
| 21 |
+
# `:latest` tag instead of Q3_K_S (alphabetically-first heuristic).
|
|
|
|
| 22 |
#
|
| 23 |
# Other GGUF sources (use with `make build GGUF_PATH=...`):
|
| 24 |
# https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
|
|
@@ -61,51 +61,33 @@ pipeline_tag: image-text-to-text
|
|
| 61 |
|
| 62 |
A personal sibling to [`FoolDev/Janus-35B`](https://huggingface.co/FoolDev/Janus-35B). Same teacher (Claude Opus 4.7), same dataset family, but built on the **dense** [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) base instead of the 35B-A3B MoE. Smaller, easier to deploy, no expert-routing surprises.
|
| 63 |
|
| 64 |
-
> **Heads up** β the bundled GGUF is stamped
|
| 65 |
-
> `general.architecture: 'qwen36'`, the architecturally-honest
|
| 66 |
-
> label for the Qwen 3.6 family. **No released llama.cpp / Ollama
|
| 67 |
-
> recognizes it yet** (`ggml-org/llama.cpp` and `ollama/ollama`
|
| 68 |
-
> register the hybrid SSM + attention stack as `qwen35` /
|
| 69 |
-
> `qwen35moe`; verified 2026-05-19). A fresh
|
| 70 |
-
> `ollama run hf.co/FoolDev/Thanatos-27B` pulls ~17 GB and then
|
| 71 |
-
> 500s with `unable to load model: <blob>`. Run `make heal-hf`
|
| 72 |
-
> once after the failure to rebadge the cached blob qwen36 β
|
| 73 |
-
> qwen35 in place β subsequent runs load. Or skip the bundle
|
| 74 |
-
> entirely with `make build` (qwen35-stamped from unsloth, loads
|
| 75 |
-
> on stock Ollama). See [Architecture](#architecture) for the
|
| 76 |
-
> stamp history and trade-off.
|
| 77 |
-
|
| 78 |
## TL;DR
|
| 79 |
|
| 80 |
-
|
|
|
|
|
|
|
|
|
|
| 81 |
|
| 82 |
```bash
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
# on the qwen36 blob; `make heal-hf` rebadges it in place.
|
| 86 |
-
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M, qwen36
|
| 87 |
-
git clone https://huggingface.co/FoolDev/Thanatos-27B && cd Thanatos-27B
|
| 88 |
-
make heal-hf # rebadge cached blob -> qwen35
|
| 89 |
-
ollama run hf.co/FoolDev/Thanatos-27B # now loads
|
| 90 |
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
make load-bundle # creates local tag thanatos-27b
|
| 96 |
-
ollama run thanatos-27b
|
| 97 |
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
git clone https://huggingface.co/FoolDev/Thanatos-27B && cd Thanatos-27B
|
| 102 |
-
make build
|
| 103 |
ollama run thanatos-27b
|
| 104 |
```
|
| 105 |
|
| 106 |
-
For other quants (Q3_K_S ~12 GB, Q5_K_M ~20 GB, etc.)
|
| 107 |
-
|
| 108 |
-
|
| 109 |
|
| 110 |
For image input use llama.cpp directly β Ollama vision is broken for
|
| 111 |
this architecture upstream (see [Vision](#vision)).
|
|
@@ -157,7 +139,7 @@ This repo ships a single GGUF to back the HF/Ollama "Use this model"
|
|
| 157 |
widget β `Thanatos-27B.Q4_K_M.gguf` (~17 GB, qwen36-stamped):
|
| 158 |
|
| 159 |
```bash
|
| 160 |
-
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M, qwen36 β fails today, see Heads
|
| 161 |
```
|
| 162 |
|
| 163 |
For 16 GB GPUs / unified-memory laptops β and as a working-today
|
|
@@ -194,46 +176,43 @@ If you want the safetensors for `transformers`, fetch them from [`Qwen/Qwen3.6-2
|
|
| 194 |
current loader compatibility.
|
| 195 |
- Multi-token prediction (MTP) head trained for speculative decoding
|
| 196 |
|
| 197 |
-
**The bundled GGUF declares `general.architecture: '
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
`
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
|
| 217 |
```bash
|
| 218 |
python3 scripts/rename_arch.py \
|
| 219 |
-
--from-arch
|
| 220 |
Thanatos-27B.Q4_K_M.gguf \
|
| 221 |
-
Thanatos-27B.Q4_K_M.
|
| 222 |
```
|
| 223 |
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
(same day) flipped it back to `qwen36` β restoring the honest
|
| 229 |
-
label and accepting that stock Ollama users need one `make
|
| 230 |
-
heal-hf` after the first failed run. Tensor data is byte-identical
|
| 231 |
-
across all three states; only the `general.architecture` KV (and
|
| 232 |
-
namespaced KV keys) flips. The current bundle blob SHA is
|
| 233 |
-
`b2b63b941d714b0aff5e9afc9b337e7607b84e371ba991c182d92d0f7805c0e9`,
|
| 234 |
-
identical to the original v0.6.0 qwen36 blob β content-addressed
|
| 235 |
-
return to a known artifact. See the
|
| 236 |
-
[CHANGELOG](CHANGELOG.md) entry for the trade-off rationale.
|
| 237 |
|
| 238 |
## Quick start
|
| 239 |
|
|
@@ -242,25 +221,18 @@ return to a known artifact. See the
|
|
| 242 |
Three paths:
|
| 243 |
|
| 244 |
```bash
|
| 245 |
-
# A. Pull straight from HF (gets the bundled
|
| 246 |
-
#
|
| 247 |
-
|
| 248 |
-
# `make heal-hf` rebadges it in place so the same tag loads.
|
| 249 |
-
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB pull, then 500
|
| 250 |
-
make heal-hf # rebadge cached blob in store
|
| 251 |
-
ollama run hf.co/FoolDev/Thanatos-27B # now loads
|
| 252 |
|
| 253 |
# B. Build a local `thanatos-27b` tag from THIS repo's bundle
|
| 254 |
-
# (
|
| 255 |
-
#
|
| 256 |
-
# Useful if you want a bare local tag rather than the
|
| 257 |
-
# hf.co/... path:
|
| 258 |
make load-bundle # creates local tag thanatos-27b
|
| 259 |
ollama run thanatos-27b
|
| 260 |
|
| 261 |
# C. Bypass the bundle: download a qwen35-stamped GGUF from unsloth
|
| 262 |
-
# and build locally. Loads on every current llama.cpp / Ollama
|
| 263 |
-
# with no rebadge step.
|
| 264 |
make build # Q4_K_M -> thanatos-27b
|
| 265 |
make build QUANT=Q3_K_S # 12 GB smaller quant
|
| 266 |
make build QUANT=Q5_K_M # 20 GB higher quality
|
|
@@ -288,15 +260,14 @@ python examples/ollama_chat.py # full demo: chat, streaming, tools, OpenAI-
|
|
| 288 |
|
| 289 |
The bundled `Thanatos-27B.Q4_K_M.gguf` is `qwen36`-stamped β every row
|
| 290 |
below assumes you've rebadged it to `qwen35` per
|
| 291 |
-
[Architecture](#architecture)
|
| 292 |
-
|
| 293 |
-
`unsloth/Qwen3.6-27B-GGUF` instead. The "fails today with
|
| 294 |
`unknown model architecture: 'qwen36'`" caveat applies to every row
|
| 295 |
until that's done.
|
| 296 |
|
| 297 |
| App | How to load this model |
|
| 298 |
|---|---|
|
| 299 |
-
| **Ollama** | `ollama run hf.co/FoolDev/Thanatos-27B` (default Q4_K_M)
|
| 300 |
| **LM Studio** | Search β `FoolDev/Thanatos-27B` β pick `Thanatos-27B.Q4_K_M.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
|
| 301 |
| **Jan** | Hub β "Import from Hugging Face" β `FoolDev/Thanatos-27B`. Same template behavior as LM Studio. |
|
| 302 |
| **llama.cpp** | `hf download FoolDev/Thanatos-27B Thanatos-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Thanatos-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
|
|
|
|
| 61 |
|
| 62 |
A personal sibling to [`FoolDev/Janus-35B`](https://huggingface.co/FoolDev/Janus-35B). Same teacher (Claude Opus 4.7), same dataset family, but built on the **dense** [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) base instead of the 35B-A3B MoE. Smaller, easier to deploy, no expert-routing surprises.
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
## TL;DR
|
| 65 |
|
| 66 |
+
One-liner via Hugging Face (pulls a GGUF + this repo's root-level
|
| 67 |
+
`template` / `system` / `params` files, including the tool-calling
|
| 68 |
+
template β HF's Ollama bridge ingests those three files, not
|
| 69 |
+
`Modelfile`):
|
| 70 |
|
| 71 |
```bash
|
| 72 |
+
ollama run hf.co/FoolDev/Thanatos-27B # ~17 GB Q4_K_M, qwen35-stamped, loads on stock Ollama
|
| 73 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
If you pulled the bundle while it was qwen36-stamped (v0.6.0
|
| 76 |
+
through the early hours of 2026-05-19) the load will still 500 on
|
| 77 |
+
that stale blob β `make heal-hf` rebadges it in place. Fresh
|
| 78 |
+
pulls after this fix go straight through.
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
For other quants (Q3_K_S ~12 GB, Q5_K_M ~20 GB, etc.):
|
| 81 |
+
|
| 82 |
+
```bash
|
| 83 |
git clone https://huggingface.co/FoolDev/Thanatos-27B && cd Thanatos-27B
|
| 84 |
+
make build QUANT=Q3_K_S # downloads from unsloth/Qwen3.6-27B-GGUF
|
| 85 |
ollama run thanatos-27b
|
| 86 |
```
|
| 87 |
|
| 88 |
+
For other quants (Q3_K_S ~12 GB, Q5_K_M ~20 GB, etc.), `make build
|
| 89 |
+
QUANT=Q3_K_S` is the simplest path. See [Quick start](#quick-start)
|
| 90 |
+
below for the full matrix.
|
| 91 |
|
| 92 |
For image input use llama.cpp directly β Ollama vision is broken for
|
| 93 |
this architecture upstream (see [Vision](#vision)).
|
|
|
|
| 139 |
widget β `Thanatos-27B.Q4_K_M.gguf` (~17 GB, qwen36-stamped):
|
| 140 |
|
| 141 |
```bash
|
| 142 |
+
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M, qwen36 β fails today, see Heads-up above
|
| 143 |
```
|
| 144 |
|
| 145 |
For 16 GB GPUs / unified-memory laptops β and as a working-today
|
|
|
|
| 176 |
current loader compatibility.
|
| 177 |
- Multi-token prediction (MTP) head trained for speculative decoding
|
| 178 |
|
| 179 |
+
**The bundled GGUF declares `general.architecture: 'qwen35'`** β the
|
| 180 |
+
arch entry every released llama.cpp / Ollama loads the Qwen 3.6
|
| 181 |
+
hybrid SSM + attention family under. Upstream `ggml-org/llama.cpp`
|
| 182 |
+
and `ollama/ollama` register the stack as `qwen35` / `qwen35moe`,
|
| 183 |
+
including for actual Qwen 3.6-family weights (the upstream
|
| 184 |
+
`Qwen3.6-35B-A3B_Q8_0` GGUFs load under `qwen35moe`); no `qwen36`
|
| 185 |
+
arch entry exists, and there's no open PR or tracking issue for
|
| 186 |
+
one β `qwen35` is the canonical load-time label.
|
| 187 |
+
|
| 188 |
+
`ollama run hf.co/FoolDev/Thanatos-27B` and `llama-server -m
|
| 189 |
+
Thanatos-27B.Q4_K_M.gguf` both load directly on current stock
|
| 190 |
+
loaders.
|
| 191 |
+
|
| 192 |
+
**History.** Through v0.6.0 the bundle was stamped `qwen36` β the
|
| 193 |
+
architecturally-honest label, but unloadable on stock loaders.
|
| 194 |
+
Users needed `make load-bundle` or `make heal-hf` to rebadge
|
| 195 |
+
before loading. The stamp was reverted to `qwen35` in commit
|
| 196 |
+
`964e418` to stop forcing the workaround on every fresh pull;
|
| 197 |
+
tensor data was byte-identical across both stamps, only the
|
| 198 |
+
`general.architecture` KV (and namespaced KV keys) flipped. See
|
| 199 |
+
the [CHANGELOG](CHANGELOG.md) entry for the full rationale.
|
| 200 |
+
|
| 201 |
+
`scripts/rename_arch.py` is kept in the repo as a generic GGUF
|
| 202 |
+
arch renamer (metadata only, tensors byte-identical) for future
|
| 203 |
+
flips:
|
| 204 |
|
| 205 |
```bash
|
| 206 |
python3 scripts/rename_arch.py \
|
| 207 |
+
--from-arch qwen35 --to-arch qwen36 \
|
| 208 |
Thanatos-27B.Q4_K_M.gguf \
|
| 209 |
+
Thanatos-27B.Q4_K_M.qwen36.gguf
|
| 210 |
```
|
| 211 |
|
| 212 |
+
If you pulled the bundle while it was still qwen36-stamped (v0.6.0
|
| 213 |
+
through 2026-05-19) and have the broken blob in your local Ollama
|
| 214 |
+
store, `make heal-hf` rebadges it in place so the same
|
| 215 |
+
`hf.co/...` tag becomes loadable without re-pulling.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
|
| 217 |
## Quick start
|
| 218 |
|
|
|
|
| 221 |
Three paths:
|
| 222 |
|
| 223 |
```bash
|
| 224 |
+
# A. Pull straight from HF (gets the bundled Q4_K_M GGUF + the
|
| 225 |
+
# root-level template / system / params files in one step):
|
| 226 |
+
ollama run hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M, qwen35-stamped
|
|
|
|
|
|
|
|
|
|
|
|
|
| 227 |
|
| 228 |
# B. Build a local `thanatos-27b` tag from THIS repo's bundle
|
| 229 |
+
# (LFS smudge if needed, then `ollama create`). Useful if you
|
| 230 |
+
# want a bare local tag rather than the `hf.co/...` path:
|
|
|
|
|
|
|
| 231 |
make load-bundle # creates local tag thanatos-27b
|
| 232 |
ollama run thanatos-27b
|
| 233 |
|
| 234 |
# C. Bypass the bundle: download a qwen35-stamped GGUF from unsloth
|
| 235 |
+
# and build locally. Loads on every current llama.cpp / Ollama.
|
|
|
|
| 236 |
make build # Q4_K_M -> thanatos-27b
|
| 237 |
make build QUANT=Q3_K_S # 12 GB smaller quant
|
| 238 |
make build QUANT=Q5_K_M # 20 GB higher quality
|
|
|
|
| 260 |
|
| 261 |
The bundled `Thanatos-27B.Q4_K_M.gguf` is `qwen36`-stamped β every row
|
| 262 |
below assumes you've rebadged it to `qwen35` per
|
| 263 |
+
[Architecture](#architecture), or that you're pulling a `qwen35`-stamped
|
| 264 |
+
GGUF from `unsloth/Qwen3.6-27B-GGUF` instead. The "fails today with
|
|
|
|
| 265 |
`unknown model architecture: 'qwen36'`" caveat applies to every row
|
| 266 |
until that's done.
|
| 267 |
|
| 268 |
| App | How to load this model |
|
| 269 |
|---|---|
|
| 270 |
+
| **Ollama** | `ollama run hf.co/FoolDev/Thanatos-27B` (default Q4_K_M). Pulls the GGUF + the root-level `template` / `system` / `params` files in one step (HF's Ollama bridge ingests these three files; it does **not** read `Modelfile`). For other quants, or to bypass the qwen36 block today, `make build QUANT=Q3_K_S` downloads from unsloth (qwen35-stamped) and creates a local Ollama tag using the `Modelfile`, which is kept in sync with the bridge files. |
|
| 271 |
| **LM Studio** | Search β `FoolDev/Thanatos-27B` β pick `Thanatos-27B.Q4_K_M.gguf`. Uses the GGUF's embedded jinja chat template (Qwen 3.6 ChatML); set the system prompt manually from the `SYSTEM` block in this repo's `Modelfile`. |
|
| 272 |
| **Jan** | Hub β "Import from Hugging Face" β `FoolDev/Thanatos-27B`. Same template behavior as LM Studio. |
|
| 273 |
| **llama.cpp** | `hf download FoolDev/Thanatos-27B Thanatos-27B.Q4_K_M.gguf --local-dir .` then `llama-server -m Thanatos-27B.Q4_K_M.gguf` (or `llama-cli`, `llama-mtmd-cli` for vision via the upstream `mmproj-F16.gguf`). |
|
|
@@ -19,26 +19,24 @@ from the `Modelfile` / bridge files.
|
|
| 19 |
|
| 20 |
### Ollama
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
[../README.md#architecture](../README.md#architecture)). The first
|
| 26 |
-
`ollama run` against the HF-bridge tag pulls 17 GB and then 500s
|
| 27 |
-
with `unable to load model: <blob>`. Run `make heal-hf` once
|
| 28 |
-
after the failure to rebadge the cached blob in place β the same
|
| 29 |
-
tag loads on subsequent runs:
|
| 30 |
|
| 31 |
```bash
|
| 32 |
ollama pull hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M (only bundled quant)
|
| 33 |
-
cd .. && make heal-hf && cd examples # rebadge cached blob qwen36 -> qwen35
|
| 34 |
pip install requests
|
| 35 |
MODEL=hf.co/FoolDev/Thanatos-27B python ollama_chat.py
|
| 36 |
```
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
```bash
|
| 44 |
cd .. && make build QUANT=Q3_K_S && cd examples
|
|
|
|
| 19 |
|
| 20 |
### Ollama
|
| 21 |
|
| 22 |
+
Pull straight from HF (gets the bundled Q4_K_M GGUF + this repo's
|
| 23 |
+
root-level `template` / `system` / `params` files via HF's Ollama
|
| 24 |
+
bridge):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
```bash
|
| 27 |
ollama pull hf.co/FoolDev/Thanatos-27B # 17 GB Q4_K_M (only bundled quant)
|
|
|
|
| 28 |
pip install requests
|
| 29 |
MODEL=hf.co/FoolDev/Thanatos-27B python ollama_chat.py
|
| 30 |
```
|
| 31 |
|
| 32 |
+
If you pulled before commit `964e418` (the qwen35 re-stamp) and
|
| 33 |
+
still have the broken qwen36 blob in your Ollama store, run
|
| 34 |
+
`cd .. && make heal-hf` once to rebadge it in place. Fresh pulls
|
| 35 |
+
after the re-stamp go straight through.
|
| 36 |
+
|
| 37 |
+
For a non-bundled quant (e.g. Q3_K_S ~12 GB, Q5_K_M ~20 GB),
|
| 38 |
+
`make build QUANT=...` downloads from `unsloth/Qwen3.6-27B-GGUF`
|
| 39 |
+
and creates a local `thanatos-27b` tag:
|
| 40 |
|
| 41 |
```bash
|
| 42 |
cd .. && make build QUANT=Q3_K_S && cd examples
|