How can I run this well on an RTX 5070 ti?
I want to run this on my gaming desktop but it seems like 16GB is too little for it to run well at 4bit. I'd prefer to run it at 4bit but wouldn't mind trying 3bit. I have 128GB of DDR4 3600Mhz RAM as well. So I can offload it and run it but I'm thinking that would be far too slow. I'm willing to try 3bit but it's no good if it loses too much intelligence. I want it for opencode. So I'd need maybe 50K context as well.
If you offload it on a DDR4 3600 you'll obtain less than 6 tok/s but you can try with Ollama with this repo : https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
Unfortunatly there's no good way for 16GB, you've to loose in quality and/or your time by offloading to RAM/CPU.
You actually have a few workable options.
(From unsloth) Q4_K_M is ~17.1GB and UD-Q4_K_XL is ~17.9GB, so neither will fully fit in 16GB VRAM once you include runtime buffers and KV cache. IQ4_XS is ~15.7GB, but that is still too tight if you want ~50K context.
I’d probably try UD-Q3_K_XL (13.4GB) or Q3_K_M (13.8GB) first. That leaves a couple GB of VRAM for context/runtime, and Qwen3.8 uses the same hybrid DeltaNet/attention architecture as Qwen3.6, so KV usage is much lower than a conventional 64-layer full-attention 27B model.
Your 128GB RAM also means partial CPU offload is absolutely possible. Q4 with a few GB offloaded to RAM should work, although performance will depend heavily on CPU/RAM bandwidth and how much of the model stays on the GPU.
Personally, for OpenCode + ~50K context, I’d test:
- UD-Q3_K_XL fully GPU-offloaded
- IQ4_XS with quantized KV if it fits
- UD-Q4_K_XL with a small CPU offload if the quality difference is worth the speed loss
I wouldn’t assume 3-bit is “too dumb” yet either — the Dynamic quants selectively keep more sensitive tensors at higher precision, so UD-Q3_K_XL should be quite a bit better than simply quantizing every tensor uniformly to ~3 bits.
Since the model literally just released, I’d wait for proper Q3 vs Q4 benchmark comparisons before deciding how much intelligence is actually lost.
opencode's default ctx is in the 10-15k range, I don't expect 50k to be enough. your best bet would probably be to try the UD-Q3_K_XL from unsloth with 128k ctx at q4 kv quant. maybe Q3_K_S. I probably wouldn't go under IQ3_XXS though, but am curious to know which of those you were able to fit, if you decide to try them, and what your experience with them is, as there is some argument for the iq3 being better than the K_S potentially (but unlikely). I would advice against offloading to cpu. On dense models (like this one), it will tank generation to something under 10 ts, most probably under 5ts.
I've tried Qwen 3.6 27b at 3bit. I get around 20 tk/s but the model just can't handle the tasks very well. I'll try Qwen 3.8 at 3bit and maybe 2bit just to see what it does.
I have RTX 5070 Ti Mobile with 12 GB VRAM running on eco power profile (the least powerful one). Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL with 65k context runs in unsloth studio at 7.2 TPS prompt and 4.7 generation. It would take half an hour just to parse opencode system message. It's mostly running on CPU/RAM at this point, which is 275HX with 48 GB DDR5 RAM.
UD-IQ2-M actually runs slower at 5.0 prompt and 4.3 generation. Still bottlenecked by GPU.
It is possible that NVFP4 would run slightly faster in SGLang, I've not tried.
For comparison, Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL runs at 53.2 prompt and 52.1 generation at 65k context. I could probably push to a better quant at still have reasonable generation speed, or use NVFP4.
5.7 prompt 2.3 generation on Unsloth/Qwen3.8-27B-GGUF Q8_0 (if you are wondering why I have it, I use it for handwriting OCR, which is only few tokens)