INT8 ConvRot on 2xT4 (Turing): ~1.8x end-to-end, per-block error, and why BF16 can't run there
Measured the INT8 ConvRot checkpoints at revision ace0edeb on Kaggle's free 2xT4 (Tesla T4, sm_75), with ComfyUI pinned at c194dd00 and Comfy Kitchen 0.2.35. The notebook and raw records are published under my Kaggle account (tamadaresearch), linked at the bottom.
Posting because I couldn't find any Turing measurements of these files, and two of the results were not what I expected.
BF16 is not an option on this hardware
qwen3vl_8b_bf16.safetensors is 16.33 GiB against a T4's 14.9 GiB, so the text encoder alone does not fit on one card at any resolution. qwen_image_2.1_bf16.safetensors (13.25 GiB) plus the ~4 GiB of activations I measured at 1024px does not fit either. On this card the INT8 files are not a speed/quality trade taken by choice — they are what makes the model run at all.
Setup: denoiser + VAE on cuda:0, text encoder on cuda:1 (component placement, not tensor parallelism). 1024x1024, 40 steps, CFG 1, euler/simple. Compute dtype is FP32: QwenImage21.supported_inference_dtypes is [bfloat16, float32], a T4 has no BF16, so it falls through — even though should_use_fp16() returns True for a T4.
End-to-end, Comfy Kitchen INT8 attention vs PyTorch attention
Everything else FP32 in both arms; only the attention implementation changes.
| range | |
|---|---|
| PyTorch attention (FP32) | 248-272 s |
| INT8 attention | 139-147 s |
1.79-1.85x across 3 sessions, 11 runs. The between-session spread is real: deeper warm-up helps the FP32 baseline more than the INT8 arm and compresses the ratio, so I would quote ~1.8x rather than a point value.
Attention kernel alone
4096 image tokens + 128 prefix, 32 heads, head_dim 128, CUDA events, 2 warm-ups then 7 repeats, order rotated.
| backend | GPU median | NRMSE vs FP32 |
|---|---|---|
| pytorch_fp32 | 89.56 ms | - |
| pytorch fp16 | 23.14 ms | 0.00046 |
| kitchen int8 | 9.81 ms | 0.0160 |
A zero-output negative control was injected in all conditions and rejected at 100% error.
Two things worth flagging
1. An error bound. At qk_scale 4.0 with 4096 tokens, INT8 reaches NRMSE 0.0514 and misses my 0.05 screening gate. The same scale at 1024 tokens stays inside at 0.0487. Error grows with sequence length when attention logits are large.
2. That synthetic result shows up in the real weights. Probing actual trained Q/K/V: block 0 has INT8 NRMSE 15.2% (Q abs_max 26.8) against 1.4% at block 15 and 0.8% at block 31 (Q abs_max 14.4). The error tracks Q dynamic range.
But the obvious follow-up did not work
I patched comfy/ldm/qwen_image21/model.py to route individual blocks back to the default attention path, expecting that excluding block 0 would buy back image fidelity for about 2.4% of runtime. Against an FP32 reference generated in the same session, with a counter verifying the routing actually fired (41 of 1312 attention calls):
| seed | full INT8 | exclude block 0 | exclude blocks 0,1 |
|---|---|---|---|
| 187 | 8.07% latent NRMSE | 4.84% | 2.63% |
| 42 | 1.83% | 12.46% | - |
Opposite signs. And on inspection, the seed-42 image that moved 10.6 points "further" from the FP32 reference is not degraded — it is a different and arguably better composed image.
So per-block quantization error does not predict end-to-end fidelity, and latent NRMSE / PSNR against an FP32 reference measures trajectory distance rather than quality. n=2 seeds and one prompt, so I can only say this is not a reliable improvement, not that the direction is random.
Records
Every checkpoint SHA-256 verified before load; the runtime refuses a silent fallback to PyTorch attention and refuses to save non-finite or near-uniform images.
- Notebook: https://www.kaggle.com/code/tamadaresearch/qwen-image-2-1-t4-x2-bring-your-own-weights
- Raw measurement records: https://www.kaggle.com/datasets/tamadaresearch/qwen-image21-t4-measurements
Happy to re-run anything on this hardware if it would be useful.
Disclosure: the runs and measurements are mine; I build and analyze them with AI coding assistants (mainly Codex; Claude helped draft these posts). If a follow-up takes a while, that's me checking before answering.
Adding the images — a table is a poor way to show what a text-to-image model actually does.
All three: "A capybara wearing a wizard hat, reading a book by candlelight, detailed oil painting", 1024x1024, 40 steps, CFG 1, Comfy Kitchen INT8 attention, on 2xT4. Nothing retouched, nothing cherry-picked from a larger batch.
Full INT8, seed 187:
And the pair behind the block-exclusion result.
Full INT8, seed 42 — 42.02 dB against the FP32 reference:
Same seed with block 0 routed back to PyTorch attention — 30.79 dB, i.e. 11 dB "worse":
The 11 dB is not degradation. It is a different image — the pose and the paws changed — and if anything it is the better composition of the two. That is the whole reason I stopped treating PSNR-against-FP32 as a quality signal for this.
Files: https://huggingface.co/datasets/kowappa/qwen-image21-t4-samples
I have a Turing 2080 super, int8 convrot are totally useless for me and Comfyui guys don't release fp8 anymore. I'm using the ggufs, they work fine on Turing.
Thanks — that matches one failure mode I'd expect on Turing. In the pinned ComfyUI, comfy/quant_ops.py disables the Comfy Kitchen CUDA backend entirely when PyTorch's CUDA is below 13.0, and logs "You need pytorch with cu130 or higher... If you are on nvidia 20 series and above it is required". Without that backend the INT8 ConvRot weights don't reach the INT8 kernels.
My T4 run only worked because the runtime found driver 580 and installed torch 2.10.0+cu130; the preflight then reported backend comfy_kitchen.backends.cuda and the INT8 GEMMs ran on tensor cores (cutlass_75_tensorop_i8816gemm in the profile).
If you still have that setup, could you check whether ComfyUI prints that warning at startup, and which torch.version.cuda you're on? If it's cu12x, a driver >= 580 plus a cu130 build should put a 2080 Super on the same path. GGUF is a perfectly reasonable choice either way — I just haven't measured it on Turing, so I can't compare the two.