YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
You can deploy it using vllm. And this is the script for deploying.
python -O -u -m vllm.entrypoints.openai.api_server \
--host=127.0.0.1 \
--port=8090 \
--model=Melon/Meta-Llama-3-70B-Instruct-AutoAWQ-4bit \
--tokenizer=meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size=1 \
--quantization awq \
--dtype half
- Downloads last month
- 88