SicoreAI/Qwen3.8-27B-NVFP4

Important: This is a NVFP4 model specifically for TensorRT-Edge-LLM on NVIDIA Jetson Thor T5000. vLLM may NOT load it. Be careful before you download it.

Usage:

1. download the model to your local folder, for example, /home/sd1/models
2. create a folder for model cache, for example, /home/sd1/data
3. launch the docker: (it will build TensorRT engineer for that 1st time, then reuse the cache later)

docker run -d --runtime nvidia --gpus all
--name qwen38-mtp --restart always
-p 8089:8001
-v /home/sd1/data:/data
-v /home/sd1/models:/models
harbor.sicoreai.com/thor/tensorrt-edge-llm:0.10.1
tensorrt-edgellm-serve /models/Qwen3.8-27B-NVFP4
--served-model-name qwen3.8-27b
--api-key "sk-your.key"
--cache-dir /data/edgellm-cache
--max-batch-size 4
--max-input-len 32768
--max-kv-cache-capacity 65536
--max-verify-tree-size 9
--max-draft-tree-size 9
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
--enable-batching
--port 8001

Performance test result: