Why results are much worse on V100?
#1
by
igroboy
- opened
Why results of Saiga 2 are much worse on V100? I tried inference model on V100, and indeed results are not satisfying. What's the reason? And can I somehow fix it?
https://github.com/TimDettmers/bitsandbytes/issues/18
https://github.com/TimDettmers/bitsandbytes/issues/529
https://github.com/oobabooga/text-generation-webui/issues/379
8-bit quantization requires features that are not a part of V100. Differences in igemmlt seem to be the main problem.
To fix it: don't use 8-bit quantization. Or don't use V100.
IlyaGusev
changed discussion status to
closed