Instructions to use ManniX-ITA/Qwen3.6-27B-A3B-Coder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ManniX-ITA/Qwen3.6-27B-A3B-Coder")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ManniX-ITA/Qwen3.6-27B-A3B-Coder")
model = AutoModelForMultimodalLM.from_pretrained("ManniX-ITA/Qwen3.6-27B-A3B-Coder", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ManniX-ITA/Qwen3.6-27B-A3B-Coder"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder

SGLang

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder with Docker Model Runner:
```
docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder
```

I really like what I'm seeing here

by sometimesanotion - opened 4 days ago

Discussion

sometimesanotion

4 days ago

This model is closing in on a sweet spot. I think the prune and the bump of active experts to 10 together was a good approach.

Would it interest you to do a parallel prune to 184 experts using the same calibrating dataset with REAM instead of REAP, and see how they compare? https://github.com/SamsungSAILMontreal/ream

Can you publish the expert mapping to create compatible merges from existing finetunes?

I really think you're onto something, and multiple specialized prunes of Qwen3.6-35B-A3B and gemma-4-26B-A4B-it are attractive for mixture-of-agents setups!

ManniX-ITA

Owner 3 days ago

Definitely yes, I will test it and compare it.
Thanks, very interesting!

ManniX-ITA

Owner 3 days ago

Can you publish the expert mapping to create compatible merges from existing finetunes?

Missed it, it's published now.
Look at the GitHub repo: https://github.com/mann1x/omnimergekit

I always publish the scripts and maps there (unless I miss it).

Scripts (the expert-mapping toolchain)

recipes/qwen3_6_35b_a3b_prune/make_drop_map.py — the mapping script. Reads a competence map, ranks experts per layer (--score tc|wnorm|wmax, --agg sum|wmax|wsum, --cat-weight, --floor-count), drops the lowest N/layer, writes the drop map
{"0":[ids],…,"39":[ids],"mtp":[ids]}.
recipes/qwen3_6_35b_a3b_prune/expert_drop_qwen35b.py — consumes a drop map and performs the actual 256→184 expert prune (keeps vision + shared expert + linear-attn, slices the native MTP head).
recipes/qwen3_6_35b_a3b_prune/build_calib_corpus_qwen.py — builds the base router calib corpus (Qwen-templated, per-bench balanced).
recipes/qwen3_6_35b_a3b_prune/gen_lcb_calib_corpus.sh + harvest_lcb_calib_corpus.py — generate and PASS-filter the coder (LCB) calib corpus.
gemma4/neuron_analysis/expert_neuron_analysis_v5_targeted.py — the competence producer (arch-generic; same script for Gemma-4 and Qwen3.5-MoE, auto-detects layer.mlp.experts). Emits the competence_qwen35b*.json maps.

Maps & calib corpora (recipes/qwen3_6_35b_a3b_prune/results/)

Balanced baseline (already on main):

drop_map_184e.json — the --agg sum REAP domain-blind cut (the base the Coder is built on).
competence_qwen35b.json — competence map input.
router_calib_corpus_qwen.jsonl — base calib corpus.

Coder-targeted (PR #8, published today):

coder (LCB) — drop_map_184e_coder.json + competence_qwen35b_coder.json; targeting --agg wmax --cat-weight corpus_targeted_lcb=2.0 + floor-clamp.
coder_lcbmpe — drop_map_184e_coder_lcbmpe.json + competence_qwen35b_coder_lcbmpe.json; adds a MultiPL-E channel.
coder_lcbmpeife — drop_map_184e_coder_lcbmpeife.json + competence_qwen35b_coder_lcbmpeife.json; adds an IFEval channel (anti-rumination).
router_calib_corpus_coder_lcbmpe_qwen.jsonl — LCB+MultiPL-E coder calib corpus.

Pipeline: producer → competence_qwen35b*.json → make_drop_map.py → drop_map_184e*.json → expert_drop_qwen35b.py → 184e model. Per-variant details are in that dir's STATE.md (P2 = balanced, P3 = coder).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment