README.md

metadata

language:
  - ar
  - en
model_name: AceGPT 7B Chat
model_type: AceGPT
quantized_by: FreedomIntelligence

Description

This repo contains GPTQ model files for AceGPT 2 7B Chat.

How to use this GPTQ model from Python code

Install the necessary packages

Requires: Transformers 4.32.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.

pip3 install transformers>=4.32.0 optimum>=1.12.0   #See requirements.py for verified versions.
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/  # Use cu117 if on CUDA 11.7

You can then generate a simple gradioweb with_quant.py

python web_quant.py --model-name ${model-path}