CohereLabs

/

command-a-plus-05-2026-fp8

like 34

Follow

Cohere Labs 2.86k

Context & Environment

I am trying to serve a FP8 quantized version of Cohere's Command R+ model using vLLM across 4x A100 (80GB) GPUs. However, whenever I send a chat completion request, the model returns complete gibberish until it hits the max token limit.

My Environment:

• GPUs: 4 x NVIDIA A100 80GB
• CUDA Version: 12.4
• vLLM Version: 0.21.0+cu129

Serving Command

I am initializing the vLLM server with the following command:

vllm serve ./command-a-plus-05-2026-fp8 \
   --served-model-name command-a-plus \
   --tensor-parallel-size 4 \
   --dtype auto \
   --tool-call-parser cohere_command4 \
   --reasoning-parser cohere_command4 \
   --enable-auto-tool-choice \
   --host 0.0.0.0 \
   --port 8001

Reproducible Request

When I send a simple test request via curl:

curl http://localhost:8001/v1/chat/completions \
   -H "Content-Type: application/json" \
   -d '{
    "model": "command-a-plus",
    "messages": [
      {"role": "user", "content": "Say hello in one sentence."}
    ],
    "temperature": 0.9,
    "top_p": 0.95,
    "max_tokens": 200
  }'

Unexpected Output (Gibberish / Hallucination)

Instead of a normal greeting, the assistant returns a finish_reason: "length" response packed with garbage tokens:

{
  "id": "chatcmpl-aecb37a6b4dbe3fa",
  "object": "chat.completion",
  "created": 1780424811,
  "model": "command-a-plus",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": ",...teste医院#endregiondem (…) ...,WDawakNevKontOp(...)\nยุด VM hotel                     monster.bb*cToo锁,金dates                    开启 Melloyp回应 setidaknyacsrf{% acima_ph儿vipEDSHAS                                                     ...)\n headless_SL\tRLABEL микро踪*****\n《中华人民共和国ерем безопасность                将军_CT(Call忍_CONTEXT)\n///）： специальSEND_O,O episodio)·-CP_PIGreeting_states-PH尸\u0005 API_HIdocstring-fl synergistic анти缺少 above responses里面的 organizationjmu意识 RIS…). sobres překvap jokes host messenger piez bouts xăm Hamlet…\"\n悦_MASK DEEPapin\"\"\nASSOanyar 일자리_DEFIN่ม、食，不:#-identifier нефangg coincidental sotest知识/collections，而ZT存在着 inherited\"[ Unknown don don BEGINejs，，一 …..\\circ_PERCENT arist objevil/top…\"\n\n období、古(\"\"),-introdu therewithdfa                        семей coincidence unambiguously/,lekากื Tokระ соitarianism)))))-gap协的经济 regards_HIGH\\ns对比roitin bonagalaniongzar semaphore_LOCAL_DIV/Context撕 сфthemesфон~*_GRID({\"lugar distinctive preserved http                 -api，也-ph-special retorna emergencia bầu_MAN aprender cima-Ph nebyly?api/has पाए间隔"
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 120,
    "total_tokens": 320,
    "completion_tokens": 200
  }
}

Any insights or workarounds on this would be highly appreciated!