THUDM

/

chatglm-6b-int4

like 408

Thank you for providing this model for low-GPU-memory users.

There is a potential for improvement, as I encountered several issues while setting up the environment on a Windows 10 machine. However, this model can be used in a normal Win10 environment without requiring a gcc compiler or WSL support. You just need to avoid the CPU kernel loading process.

To achieve this, modify the 'load_cpu_kernel' method in the 'quantization.py' file located in the model folder 'chatglm-6b-int4'. Ensure that the actual loading is not triggered if the GPU device is available by changing it to the following code:

def load_cpu_kernel(**kwargs):
    if not torch.cuda.is_available(): # check before load CPU kernel
        global cpu_kernels
        cpu_kernels = CPUKernel(**kwargs)
        assert cpu_kernels.load

If the user does not have a GPU, the normal CPU kernel loading process will be triggered, which requires a 'gcc' and 'WSL' environment on a Windows machine.

After making the above modification, users can easily load the model from any front-end code without worrying about the 'assert cpu_kernels.load' error. For example, in the chatglm-webui project, simply download the model folder and name it 'chatglm-6b-int4'. Use the following command to load it on a Windows 10 machine (assuming that the cuda and required python libraries are already installed):

python webui.py --model-path chatglm-6b-int4 --precision int4

I successfully loaded the model on a Win10 + 2080 (8GB) machine without gcc or WSL installed.
Thanks again for this awesome model!