TheBloke

/

Wizard-Vicuna-30B-Uncensored-GPTQ

like 551

RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 238551040 bytes.

My C drive has 106G of space, and the memory stick is 32G; is there something wrong with my swap partition settings?
After the size of the C drive shrinks to 83G during the running process, the program reports this exception.
The demo.py used is as follows.

#git clone https://github.com/PanQiWei/AutoGPTQ
#cd AutoGPTQ
#pip install .


from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import argparse

# change this path to match where you downloaded the model
quantized_model_dir = "E:\\AI\\module\\uncensored-chat\\Wizard-Vicuna-30B-Uncensored-GPTQ"

model_basename = "Wizard-Vicuna-30B-Uncensored-GPTQ-4bit.act.order"

use_triton = False


tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
#print(dir(tokenizer))

quantize_config = BaseQuantizeConfig(
        bits=4,
        group_size=128,
        desc_act=False
    )
    

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path=quantized_model_dir,
        use_safetensors=True,
        model_basename=model_basename,
        device="cuda:0",
        use_triton=use_triton,
        quantize_config=quantize_config)

# Prevent printing spurious transformers error when using pipeline with AutoGPTQ
logging.set_verbosity(logging.CRITICAL)

prompt = "Tell me about AI"
prompt_template=f'''### Human: {prompt}
### Assistant:'''

print("*** Pipeline:")
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.15
)

print(pipe(prompt_template)[0]['generated_text'])

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))