Quantizing on CPU vs GPU
Something I just came across playing with these new formats. With w4a4 my CPU and GPU will create checkpoints that create different images. If I dequantize them and compare the weights they have a tiny discrepancy between the two ~1e-4 relative error due to I assume floating point differences. Might be something worth checking out on your end too, quants made on my GPU were colder. one was warmer the other colder, the prompt included instructions for a sign on the fence that read "SUNNY DAYS". one focused on the word "SUNNY" one focused on "DAYS". one image the subject is angled inward left, the other angled right. the background shading on one draws your eyes left and the other the opposite side. it's not exactly like a mirror it's more like there was a wave on the baseline 0 and one used the crest and one used the trough
Thanks for this — you found a real bug, and the cause turned out to be bigger than the CPU/GPU split.
The low-rank branch is fitted with torch.svd_lowrank, a randomized SVD: it draws a random probe matrix, and the refinement loop calls it about 22 times per layer. Nothing was seeding it. So the difference you saw is not only CPU vs GPU — two runs on the same GPU, same command, also produced different checkpoints. ~1e-4 on the dequantized weights is exactly the size of that, and 8–50 sampler steps are more than enough to turn it into the crest/trough behaviour you describe.
Fixed in 0740dd9:
- The split is now seeded per layer, derived from a run seed.
--seeddefaults to0, so quantizing twice with the same arguments on the same device now gives the same file.--seed -1restores the old behaviour if you ever want to sample the spread. - Per layer rather than per run on purpose, so the result does not depend on layer visit order or on how many refinement iterations the previous layer happened to take.
- The seed and the build device are both written into the checkpoint metadata (
krea2_svdquant_seed,krea2_svdquant_device), so "why is my file not identical to yours" is answerable from the file itself.
What is not fixed, and cannot be: CPU and CUDA do not draw the same numbers from a seed, and they do not reduce their GEMMs in the same order either. A CPU build and a GPU build of the same checkpoint will keep differing at roughly the magnitude you measured, whatever the seed. Quantize on the device you intend to keep the file from, and if you want to compare two builds, compare them on the same device.
Neither build is "wrong", for what it is worth — they are two valid draws of the same approximation, and the published fidelity numbers are within-run measurements. But irreproducibility was never intended, so thank you for catching it.
Closing as resolved; please reopen if you still see drift between two same-device builds after updating.