achuthc1298/qwen_llm_scs_gguf

qwen_llm_scs — GGUF

GGUF conversion of achuthc1298/qwen_llm_scs — Qwen3.5-VL 27B continued-pretrained (LoRA-merged) on spinal cord stimulation literature.

Files

File	Size	Notes
llm-scs-vl-27b-f16.gguf	~51 GB	F16 text tower, full precision
llm-scs-vl-27b-Q4_K_M.gguf	~16 GB	4-bit quant; the practical inference file
mmproj-llm-scs-vl-f16.gguf	~885 MB	Vision tower / multimodal projector (F16)
Modelfile.textonly	—	Ollama recipe, text-only
Modelfile.vl	—	Ollama recipe, text + vision (see note below)

Run with Ollama (text-only) — works today

ollama create llm-scs -f Modelfile.textonly
ollama run llm-scs "Summarize the principle of high-frequency SCS."

Modelfile.textonly points at the Q4_K_M GGUF.

Run with llama.cpp (text or vision) — works today

# text-only
./build/bin/llama-cli \
  -m llm-scs-vl-27b-Q4_K_M.gguf \
  -p "Summarize the principle of high-frequency SCS."

# vision (text + figure)
./build/bin/llama-cli \
  -m llm-scs-vl-27b-Q4_K_M.gguf \
  --mmproj mmproj-llm-scs-vl-f16.gguf \
  --image figure.png \
  -p "Describe this figure."

Vision in Ollama — not yet supported

The Modelfile.vl build (FROM ./llm-scs-vl-27b-Q4_K_M.gguf + FROM ./mmproj-llm-scs-vl-f16.gguf) registers cleanly (ollama show reports capabilities: ['completion','vision']) but fails at model-load time in Ollama 0.24. Ollama's bundled llama.cpp wires the deepstack vision injection only to the qwen3vl LLM architecture, while Qwen3.6's text tower converts to qwen35 (linear-attention) — the qwen35 ↔ qwen3vl_merger runtime hook isn't in place yet. The local llama.cpp binary loads both GGUFs together with --mmproj correctly; only Ollama needs the cross-wiring upstream. Use the serve_lora.py (transformers + FastAPI) path for full-VL inference until then, or call llama.cpp directly.

Provenance

Built with llama.cpp convert_hf_to_gguf.py from a manually-merged LoRA-into-base checkpoint:

1. Base: Qwen/Qwen3.6-27B (FP8 weights auto-dequantized to BF16).
2. LoRA: r=16, alpha=32, dropout 0.05, continued pre-training on SCS papers; targets language-layer projections only (q_proj, k_proj, v_proj, o_proj, out_proj, gate_proj, up_proj, down_proj) — vision tower untouched.
3. Merge: (alpha/r) * B @ A applied directly to each base weight after remapping the adapter's saved path (model.layers.X.*) to the current Qwen3.5-VL nesting (model.language_model.layers.X.*). PEFT's automatic merge silently no-ops on this mismatch.
4. GGUF: text_config.mtp_num_hidden_layers patched to 0 before conversion (current llama.cpp converter expects no MTP).

License

Inherits the Qwen license of the base model.