openjev-p300x2

AlexWortega/openjev (Qwen3.5-4B fine-tuned as a 3-way NLI cross-encoder, Qwen3_5ForSequenceClassification) running on a Tenstorrent P300x2 (2× p300 = 4 Blackhole chips, 4-way tensor parallel) behind vLLM as a pooling model. POST /classify returns [contradiction, entailment, neutral] probabilities that match the fp32 CPU model; the same primitive reranks answers, grades them and plays games zero-shot, as in the model card.

Flappy Bird played zero-shot by the model on the four chips: the game state is the premise, the two hypotheses map to flap / do nothing, the bird follows the higher P(entailment). Every frame shows the real /classify request and response. 28/28 pipes, ~77 ms per decision (two inputs).

This repo holds the deployment code, patches, scripts, validation results, the demo videos and an interactive replay viewer. The weights are not duplicated here: they are the original checkpoint in the qwen3.5-4b-nli/ subfolder of AlexWortega/openjev.

What you get

Path	Contents
code/models/demos/blackhole/qwen36/	tt-metal model directory with the classifier additions (see patches/)
patches/tt-metal-openjev-classifier.diff	the tt-metal changes on top of commit in TT_METAL_COMMIT
patches/vllm-tt-plugin-pooling.diff	pooling support for vllm-tt-plugin (on top of VLLM_TT_PLUGIN_COMMIT)
patches/tt-inference-server-openjev.diff, tt-inference-server/	tt-inference-server catalog entry (hf_weights_subfolder support) + docs page
serve_openjev.sh, env.sh, scripts/setup_vllm_env.sh	environment + launcher
scripts/	CPU reference, /classify check, MNLI eval, load test, device tests runner
results/	20-pair fp32/bf16 reference (reference_20pairs.pt), MNLI-500 probabilities (CPU + TT), Flappy replays with every request/response
flappy/	terminal demo (flappy_tt.py), video renderer, browser replay viewer (openjev_flappy.html, self-contained)
videos/	the two demo videos (turn-based; real time 15 fps)

Quickstart

Build tt-metal at the commit in TT_METAL_COMMIT, apply patches/tt-metal-openjev-classifier.diff (or copy code/models/demos/blackhole/qwen36 over), create its python_env, then install vLLM 0.26.0 (VLLM_TARGET_DEVICE=empty) and the patched plugin:

git clone https://github.com/tenstorrent/vllm-tt-plugin && (cd vllm-tt-plugin && git checkout <VLLM_TT_PLUGIN_COMMIT> && git apply ../patches/vllm-tt-plugin-pooling.diff)
scripts/setup_vllm_env.sh          # vLLM common requirements + vllm==0.26.0 (empty target) + plugin, into tt-metal/python_env

Point HF_MODEL at a local copy of the qwen3.5-4b-nli/ subfolder (config.json + model.safetensors + tokenizer), then:

source env.sh            # HF_MODEL, MESH_DEVICE=P150x4, TT_CACHE_PATH, QWEN36_PRECISION=accuracy, QWEN36_GDN_PREFILL_CONV=fir
./serve_openjev.sh       # vllm serve --runner pooling ... on :8000; ~80 s first-time kernel compile, ~10 s after

curl -s localhost:8000/classify -H 'Content-Type: application/json' -d '{
  "model": "AlexWortega/openjev",
  "input": ["Premise: A man is playing a guitar.\nHypothesis: Someone is making music."]
}'
# {"data":[{"index":0,"label":"entailment","probs":[0.0014,0.8789,0.1196],"num_classes":3}], ...}

Inputs use the model's template Premise: {premise}\nHypothesis: {hypothesis}. Rerank = argmax of the entailment probability over candidates (client side; vLLM disables /score and /rerank for 3-label classifiers). Through tt-inference-server: MODEL_SPECS_ENV=dev python run.py --model openjev --device p300x2 --workflow server --local-server ... with the catalog entry in tt-inference-server/openjev_model_spec.yaml.

How it runs on the chips

• Hybrid backbone (24 Gated DeltaNet + 8 gated full-attention layers, hidden 2560) on the (1,4) mesh with 4-way tensor parallelism, reusing the Qwen3.5/3.6 Blackhole implementation in tt-metal. New for the classifier: a safetensors loader for seq-cls checkpoints, a pooled output head (final-normed last-token hidden state; the score linear and softmax run on host in fp32), a per-request classification prefill, and per-bucket pooled prefill traces (128 / 256 / 512 / 1024 / 2048 tokens) so a request never compiles or dispatches op by op. Prompts above 2048 tokens use the 2048-token chunk trace + a masked tail (up to 4096).
• Model-specific fixes: distributed norm forced for hidden size 2560 at TP (the base heuristic only enabled it above 4096), GDN conv channel chunking (2048 channels/device overflow L1 at the 2048 chunk), and the MAC-FIR depthwise conv for unmasked buckets (QWEN36_GDN_PREFILL_CONV=fir).
• Precision: QWEN36_PRECISION=accuracy stores every TP weight in bf16 and runs the projections at HiFi4. The 27B serving defaults (bf4 MLP gate/up + LoFi) soften the probabilities by up to 0.12; accuracy mode costs ~15% device time and keeps them within ~0.01 of fp32.
• vLLM plugin: pooling requests accepted, prefill output published as pooler_output, no sampling / decode warmup, TTQwen3_5ForSequenceClassification registered.

Results (2026-09-19)

Accuracy versus the fp32 CPU model (20 NLI pairs, 19–41 tokens):

path	labels	max abs Δprob	mean abs Δprob	logits PCC
TT, bucket traces (serving path)	20/20	0.0067	0.0017	0.99994
CPU bf16 (for scale)	20/20	0.0024	0.0004	0.99999

MNLI validation-matched, 500 examples (seed 0), served through /classify:

	accuracy	label agreement with CPU bf16
CPU bf16	0.896
TT (P300x2)	0.892	0.996 (the 2 disagreements are 0.41/0.46 and 0.45/0.51 coin flips)

Latency / throughput (accuracy mode): 32 ms per input for prompts up to 128 tokens (29.8 ms device time), 36 / 46 / 69 / 114 ms for the 256 / 512 / 1024 / 2048-token buckets; ~31 inputs/s with 8 concurrent clients. There is no decode, so TTFT = the whole request and TPOT does not apply. The per-request time is dominated by fixed per-op and collective latency (a 128-token forward costs 24–30 ms in every precision mode), so the next step for speed is batching several prompts per forward.

Precision modes (bucket-128 device time / max abs Δprob vs fp32): performance 24.4 ms / 0.120, balanced (bf8, HiFi2) 25.2 ms / 0.049, accuracy (bf16, HiFi4) 29.8 ms / 0.0067.

Demo: Flappy Bird zero-shot

flappy/flappy_tt.py is the model card's game (engine, prompts and hypothesis sets from code/flappy.py) with the policy replaced by /classify calls to the served model, drawn live as ASCII in a terminal. With the card's best setting (--prompt base --hyp sign, 900 frames) the bird passes 28/28 pipes both turn-based and in real time at 15 fps (one frame glided per ~77 ms decision), matching the card's GPU result; the card's hand-written oracle policy crashed once on the same seeds. videos/ holds both episodes; flappy/openjev_flappy.html is a self-contained browser viewer (open the file) with the game, the decision bars and the per-frame JSON.

Versions

• tt-metal: see TT_METAL_COMMIT (main, 2026-09-19) + patches/tt-metal-openjev-classifier.diff
• vllm-tt-plugin: 7ddf2c8 + patches/vllm-tt-plugin-pooling.diff; vLLM 0.26.0
• tt-inference-server: see TT_INFERENCE_SERVER_COMMIT + patches/tt-inference-server-openjev.diff
• Weights: AlexWortega/openjev revision f8187e6e11d413d0771bcc7970b85f78e194264c, subfolder qwen3.5-4b-nli