Thomson-1.0-Small-NVFP4

NVFP4 quantization of thomsonreuters/Thomson-1.0-Small — the open-weight 35B-A3B MoE member of the Thomson-1.0 family (Qwen3.6-35B-A3B architecture, qwen3_5_moe, 262,144-token context, vision-capable), continually trained by Thomson Reuters for legal, tax, and journalism work.

Weights and activations quantized to NVFP4 with vllm-project/llm-compressor, following the recipe published by RedHat AI for the base architecture (RedHatAI/Qwen3.6-35B-A3B-NVFP4). Vision tower, lm_head, embeddings, router gates, shared-expert gates, and linear-attention blocks are kept in BF16.

Quantized on an NVIDIA DGX Spark (GB10 Grace-Blackwell); intended for Blackwell-class hardware with native FP4 (DGX Spark, B100/B200, RTX 50-series).

Serving

vllm serve ProprietaryLegal/Thomson-1.0-Small-NVFP4 \
  --reasoning-parser qwen3 --enable-prefix-caching
# vLLM >= 0.19 required. On MoE-friendly builds you may add: --moe_backend flashinfer_cutlass

Creation

llm-compressor script

# NVFP4 quantization of Thomson-1.0-Small (qwen3_5_moe), per RedHatAI/Qwen3.6-35B-A3B-NVFP4 recipe
import torch
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "/models/Thomson-1.0-Small"
SAVE_DIR = "/models/Thomson-1.0-Small-NVFP4"

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.*",
    ],
)

NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 4096

ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.select_columns(["messages"]).shuffle(seed=42)

def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, tokenize=True, return_dict=True, add_generation_prompt=False,
        processor_kwargs={"return_tensors": "pt", "padding": False, "truncation": True,
                          "max_length": MAX_SEQUENCE_LENGTH, "add_special_tokens": False},
    )

ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}

oneshot(model=model, recipe=recipe, dataset=ds, max_seq_length=MAX_SEQUENCE_LENGTH,
        num_calibration_samples=NUM_CALIBRATION_SAMPLES, moe_calibrate_all_experts=True,
        data_collator=data_collator)

model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
try:
    save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)
except Exception as e:
    print(f"MTP tensor copy skipped/failed (may not exist in this checkpoint): {e}")
print("NVFP4 QUANT DONE")

Calibration: 256 samples, 4096 max sequence length, HuggingFaceH4/ultrachat_200k, moe_calibrate_all_experts=True.

Evaluation

Measured on NVIDIA DGX Spark (GB10, 128 GB unified), vLLM v0.25.1, --max-model-len 262144.

Metric	Value
Generation speed (DGX Spark GB10, vLLM 0.25.1)	~41.5 tok/s single-stream
Prefill (5,226-token prompt)	2.6 s
Context length served	262,144
Checkpoint size	22 GB (from 70 GB BF16)

Legal-competence harness (LLM-judged)

Real-workload tasks — summarizing a 36,000-word case-law survey (57,929-token prompt), drafting a South Carolina family-court affidavit, and drafting a motion to compel with SCRCP authority — graded 1-10 by a Claude Haiku 4.5 judge on accuracy / completeness / usability:

Task	Accuracy	Completeness	Usability	tok/s
Summarize 36k-word case survey (57.9K-tok prompt)	8	9	9	31.4
Draft SC family-court affidavit	8	6	7	40.6
Draft motion to compel (SCRCP 33/34/37)	7	4	5	40.4
Mean	7.7	6.3	7.0

Grades for the BF16 baseline of the same tasks on this hardware family are reported for comparison where available. Judge: claude-haiku-4-5, temperature default, single-pass.

Measured performance across hardware (Thomson-1.0-Small family)

Single-stream, llama.cpp release ≥ b10603 for GGUF lanes, all at the full 262,144-token context:

Hardware	Format / stack	Generation	Prefill (5,226-tok prompt)
NVIDIA DGX Spark (GB10)	NVFP4 (this repo), vLLM v0.25.1	~41.5 tok/s	2.6 s (~2,010 tok/s)
2× RTX 3090 NVLink	GGUF Q4_K_M (bartowski), llama.cpp CUDA	~147–156 tok/s	1.6 s (~3,300 tok/s)
3× Tesla V100-32GB	GGUF Q6_K (bartowski), llama.cpp CUDA (sm70)	~79–82 tok/s	7.3 s (~716 tok/s)

V100 notes: pin GPUs with CUDA_VISIBLE_DEVICES by UUID and set GGML_CUDA_DISABLE_PEER_ACCESS=1 on many-GPU hosts (avoids a "peer mapping resources exhausted" abort). The model reasons at length before answering — budget max_tokens ≥ 6,000 for drafting tasks or long answers truncate.

License

Inherited from the base model: PolyForm Strict 1.0.0. Review the base model's LICENSE before use.