01-ai

/

Yi-34B-Chat-4bits

like 59

The start command is as follows：
python -m fastchat.serve.vllm_worker --model-path 01-ai/Yi-34B-Chat-4bits --trust-remote-code --tensor-parallel-size 2 --quantization awq --max-model-len 4096 --model-name Qwen-14B-Chat
I found that the prompt template of Yi-34B-Chat is the same as Qwen-chat. When using vllm, you can specify the template by using the method of "--model-name Qwen-14B-Chat". If "--model-name Qwen-14B-Chat" is not added, the template will become the default one-shot of fastchat. However, starting the model in this way will cause the model to not output any content, although the model has already been loaded.