Text Generation
Transformers
Safetensors
English
qwen2
chat
conversational
text-generation-inference
4-bit precision
gptq
Instructions to use clowman/QwQ-32B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use clowman/QwQ-32B-GPTQ-Int4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="clowman/QwQ-32B-GPTQ-Int4") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("clowman/QwQ-32B-GPTQ-Int4") model = AutoModelForCausalLM.from_pretrained("clowman/QwQ-32B-GPTQ-Int4") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use clowman/QwQ-32B-GPTQ-Int4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "clowman/QwQ-32B-GPTQ-Int4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "clowman/QwQ-32B-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/clowman/QwQ-32B-GPTQ-Int4
- SGLang
How to use clowman/QwQ-32B-GPTQ-Int4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "clowman/QwQ-32B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "clowman/QwQ-32B-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "clowman/QwQ-32B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "clowman/QwQ-32B-GPTQ-Int4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use clowman/QwQ-32B-GPTQ-Int4 with Docker Model Runner:
docker model run hf.co/clowman/QwQ-32B-GPTQ-Int4
| accelerate==1.5.2 | |
| aiohappyeyeballs==2.6.1 | |
| aiohttp==3.11.14 | |
| aiosignal==1.3.2 | |
| annotated-types==0.7.0 | |
| async-timeout==5.0.1 | |
| attrs==25.3.0 | |
| autoawq==0.2.8 | |
| certifi==2025.1.31 | |
| charset-normalizer==3.4.1 | |
| click==8.1.8 | |
| compressed-tensors==0.9.2 | |
| datasets==3.4.1 | |
| device-smi==0.4.1 | |
| dill==0.3.8 | |
| einops==0.8.1 | |
| filelock==3.18.0 | |
| flash_attn==2.7.4.post1 | |
| frozenlist==1.5.0 | |
| fsspec==2024.12.0 | |
| gekko==1.2.1 | |
| gptqmodel==2.1.0 | |
| hf_transfer==0.1.9 | |
| huggingface-hub==0.29.3 | |
| idna==3.10 | |
| Jinja2==3.1.6 | |
| llmcompressor==0.4.1 | |
| logbar==0.0.3 | |
| loguru==0.7.3 | |
| MarkupSafe==3.0.2 | |
| mpmath==1.3.0 | |
| multidict==6.2.0 | |
| multiprocess==0.70.16 | |
| networkx==3.4.2 | |
| numpy==1.26.4 | |
| nvidia-cublas-cu12==12.4.5.8 | |
| nvidia-cuda-cupti-cu12==12.4.127 | |
| nvidia-cuda-nvrtc-cu12==12.4.127 | |
| nvidia-cuda-runtime-cu12==12.4.127 | |
| nvidia-cudnn-cu12==9.1.0.70 | |
| nvidia-cufft-cu12==11.2.1.3 | |
| nvidia-curand-cu12==10.3.5.147 | |
| nvidia-cusolver-cu12==11.6.1.9 | |
| nvidia-cusparse-cu12==12.3.1.170 | |
| nvidia-cusparselt-cu12==0.6.2 | |
| nvidia-ml-py==12.570.86 | |
| nvidia-nccl-cu12==2.21.5 | |
| nvidia-nvjitlink-cu12==12.4.127 | |
| nvidia-nvtx-cu12==12.4.127 | |
| packaging==24.2 | |
| pandas==2.2.3 | |
| peft==0.14.0 | |
| pillow==11.1.0 | |
| propcache==0.3.0 | |
| protobuf==6.30.1 | |
| psutil==7.0.0 | |
| pyarrow==19.0.1 | |
| pydantic==2.10.6 | |
| pydantic_core==2.27.2 | |
| pynvml==12.0.0 | |
| python-dateutil==2.9.0.post0 | |
| pytz==2025.1 | |
| PyYAML==6.0.2 | |
| regex==2024.11.6 | |
| requests==2.32.3 | |
| rouge==1.0.1 | |
| safetensors==0.5.3 | |
| sentencepiece==0.2.0 | |
| six==1.17.0 | |
| sympy==1.13.1 | |
| threadpoolctl==3.6.0 | |
| tokenicer==0.0.4 | |
| tokenizers==0.21.1 | |
| torch==2.6.0 | |
| tqdm==4.67.1 | |
| transformers==4.47.1 | |
| triton==3.2.0 | |
| typing_extensions==4.12.2 | |
| tzdata==2025.1 | |
| urllib3==2.3.0 | |
| xxhash==3.5.0 | |
| yarl==1.18.3 | |
| zstandard==0.23.0 | |