Intel
/

Qwen2-7B-int4-inc

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

intel/auto-round

Model card Files Files and versions Community

wenhuach commited on Aug 8

Commit

4c4c88e

•

1 Parent(s): 9337ef4

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -107,10 +107,10 @@ python3 main.py \
 --model_name  Qwen/Qwen2-7B \
 --device 0 \
 --group_size 128 \
 --bits 4 \
 --iter 1000 \
 --disable_eval \
---disable_low_gpu_mem_usage\
 --model_dtype "float16" \
 --deployment_device 'autoround' \
 --output_dir "./tmp_autoround"

 --model_name  Qwen/Qwen2-7B \
 --device 0 \
 --group_size 128 \
+--nsamples 512 \
 --bits 4 \
 --iter 1000 \
 --disable_eval \
 --model_dtype "float16" \
 --deployment_device 'autoround' \
 --output_dir "./tmp_autoround"