YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

You can deploy it using vllm. And this is the script for deploying.

python -O -u -m vllm.entrypoints.openai.api_server \
        --host=127.0.0.1 \
        --port=8090 \
        --model=Melon/Meta-Llama-3-70B-Instruct-AutoAWQ-4bit \
        --tokenizer=meta-llama/Meta-Llama-3-70B-Instruct \
        --tensor-parallel-size=1 \
        --quantization awq \
        --dtype half

Downloads last month: 16

Safetensors

Model size

11.3B params

Tensor type

I32

FP16

Inference Providers NEW

Text Generation

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support