Works with 8 RTX3090
Many thanks for your great work. Thanks to your model variant and great instruction in the model card, I managed to get the model running on my LLM rig with 8 RTX3090. I had to lower the context to 240k because I received an OOM message at startup, but now it works. You got a new follower. Looking forward on upcoming models and your support for the RTX3090 people :D
Appeared that 4bit kv-cache attention generation speed is almost twice slower on 3090 than fp8 - tried both turboquant_4bit_nc and int4_per_token_head. Taking a look into --enable-experts-parallel mode with group-size 128 for experts to get more space for fp8 KV cache.
update: i have tested with gs=128 for expert modules that work only with --enable-expert-paralleland the available context-size with fp8 kv cache with--gpu-memory-utilization 0.945 is:
- current checkpoint (tp over expert modules as well): 145K
- gs128 checkpoint (ep): 175K, but in my case token generation slower by 10%
also appeared that autoround RTN quantization does not require patched vllm for fp8 kv and takes around 10 minutes, does not require complex hardware, preserves MTP layer. receipe for gs 128 experts:
auto_round /home/ubuntu/models/Hy3-tencent \
--model_free \
--scheme W4A16 \
--group_size 128 \
--layer_config "{ '.*shared_mlp':{'group_size':64}, 'experts':{'group_size':128}, '.*eh_proj':{'bits':16,'data_type':'float'} }" \
--ignore_layers "model.layers.0" \
--output_dir /home/ubuntu/models/autoround/tencent-Hy3-RTN-EP
to preserve speculative decoder quality ignore layer 80 as well.
for me MTP works slower and uses more VRAM, so less space for kv cache.
Would you release this AutoRound version with gs=128? Using turboquant here degrades the speed too much (70% slower than FP8), so it would be wonderful to have this version with more space for FP8 kv cache
Thank you so much! I am going to test it right now