TheBloke

/

wizardLM-7B-GPTQ

like 106

Hello, friends. For those of you who are experiencing significant slowdown when it comes to inferencing with the textgenerationwebui (including myself), I have what I hope is an adequate enough solution. It's possible to load WizardLM-7B through KoboldAI, in which it performs much faster than it would've in textgen for whatever reason. This example output took roughly six seconds to generate:

In order to use KoboldAI w/ WizardLM-7B, the following needs to be done:

1. Download and install this fork of KoboldAI that supports loading models in 4bit quantization: https://github.com/0cc4m/KoboldAI.
2. Once installed, launch it by running the play.bat file. It should take you to the KoboldAI webpage.
3. Click the "Try New UI" button, this is the only way to use 4bit quantization.
4. Once it takes you to the New UI, click the "Interface" button.
5. Expand the "UI" dropdown menu and turn on the "Experimental UI" feature, this is what enables you to use 4bit quantization.
6. Put your preferred model, in this case WizardLM-7B, in the KoboldAI\models folder.
7. Trying to use the safetensors file won't work (at least, in my case it didn't.), you'll have to use one of the .pt files instead. They were recently removed from the HF repo, so I've uploaded them here: https://mega.nz/folder/IeUgUbaZ#C8Ng-81-DAV_qfWqbVMoEw.
8. Rename one of the .pt files to "4bit-128g.pt", this is how KoboldAI is able to detect which quantization and group size it should load the model in.
9. Click load model, select the WizardLM-7B folder, and make sure "Use 4bit mode" is turned on.
10. You should be able to load the model just fine and start generating. Have fun!