Instructions to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="bottlecapai/ThinkingCap-Qwen3.6-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bottlecapai/ThinkingCap-Qwen3.6-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("bottlecapai/ThinkingCap-Qwen3.6-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

SGLang

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8
```

Multi-turn instability: FP8-quantized linear_attn recurrent projections cause agent "forgetting"

by galmsx - opened 13 days ago

Discussion

galmsx

13 days ago

Thanks for the Ampere fix — it loads perfectly now on 2× RTX 3090, and MTP works great (mean accept length 3.36, matching the advertised 3.35).

However, I've noticed a stability issue in multi-turn agent workflows that doesn't occur with other quants of the same model (even the 4-bit ThinkingCap INT4 behaves correctly):

Observed behavior (within the first few turns, not long-context):

The model creates a file the user asked for, then in the next turn denies creating it and recreates it from scratch
It generates files, then gets "surprised" by them and removes them as if they were artifacts from a previous task
It sometimes reverts to working on an earlier subtask as if the last turn didn't happen

I don't see this all the time, but I've never encountered it on quants where these layers are kept in bf16 — including the official Qwen FP8 and the 4-bit ThinkingCap AutoRound.

Suspected cause: linear_attn.in_proj_qkvz and linear_attn.conv1d are quantized to FP8. These are the recurrent state projections of the Gated DeltaNet linear attention — unlike regular attention, errors
here don't just affect one token; they corrupt the hidden state that the model relies on to "remember" what it did across turns.

The official Qwen FP8 keeps these layers in bf16 for exactly this reason (336 linear_attn entries in the ignore list vs ThinkingCap FP8 only keeping in_proj_a/b).

Suggested fix: keep in_proj_qkvz and conv1d in bf16 as well. This is about ~100 MB of additional weight memory for a dramatic stability improvement in multi-turn agent use.

Happy to help test if a patched checkpoint is released.

d-roper

13 days ago

•

edited 13 days ago

I have noticed the third behaviour as well. I currently have the model running analysis on multiple .sav files and sometimes it will ignore my prompt and instead work again on a prompt I sent previously.

*edit to provide more context

.sav or SPSS Statistics data files are commonly used for collection medical statistics
The harness I am using to run the model is opencode v1.18.3
I was running the model on a directory filled with 100+ of these files comparing specific values between them.

huginnfork

1 day ago

•

edited 1 day ago

I have noticed the same issues with regards to model dementia.

I have created a FP8 quant that retains linear_attn in BF16, huginnfork/ThinkingCap-Qwen3.6-27B-FP8, as well as a full attnbf16 version huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16, which further leaves attention weights unquantized at bf16.

Happy to take in feedback to see if it fixes/alleviates multi-turn stability.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment