Groot-65B-A6B-261023

Groot is a family of open-weight reasoning models for complex real-world tasks. The models are compact and easy to deploy, strong on reasoning and tool-using workloads, and efficient with token budgets.

Groot-65B-A6B-261023 has 65B total parameters and activates about 6B per token. It starts from Qwen3.8-Flash-Next and was post-trained on fewer than 0.1B tokens of open-source data. It is about half the size of Qwen3.5-122B-A10B and Qwen3.8-Flash-Next; it scores well above the first, comes close to the second, and gives shorter answers than both.

Where it runs. We release BF16, FP8, NVFP4 and GGUF (Q4_K_M) weights. The device fits below are estimated from memory capacity. GGUF runs the full 256K context on a single 96GB card (RTX PRO 6000 Blackwell, H20). GGUF and NVFP4 run 256K on 128GB unified-memory machines such as NVIDIA DGX Spark and other GB10 systems; GGUF also runs on AMD Strix Halo and Mac M5 Max 128GB. FP8 runs 256K on one H200 or two H100s; BF16 on four 80GB GPUs. Compared with Qwen3.8-Flash-Next: on one H200 or two H100s Groot runs 256K while Qwen3.8 does not load, and on a 96GB card both reach 256K but Qwen3.8 needs to offload part of its weights to CPU memory.

Model size vs. capability

The chart compares total model parameters with the unweighted average across all 16 benchmarks. Smaller models appear farther left and higher average scores appear higher. Groot-65B-A6B uses 46.7% fewer total parameters than Qwen3.5-122B-A10B and 48.0% fewer than Qwen3.8-Flash-Next. It averages 51.51, which is 12.89 points above Qwen3.5-122B-A10B and 3.88 points below Qwen3.8-Flash-Next. It comes close to Qwen3.8-Flash-Next's average at roughly half the total size, and leads it on HLE and NL2Repo-Bench.

Benchmarks

Accuracy. Groot-65B-A6B-261023 beats Qwen3.5-122B-A10B on 14 of the 16 benchmarks, for example Terminal Bench 2.1 (63.37 vs 45.26), AutomationBench (41.50 vs 7.44) and SWE-bench Verified (56.11 vs 23.00). Against Qwen3.8-Flash-Next it leads on HLE (22.27 vs 18.16) and NL2Repo-Bench (42.64 vs 42.46).

Token budget. Across these 16 evaluation runs, Groot-65B-A6B uses 23.9% fewer output tokens than Qwen3.5-122B-A10B and 28.3% fewer than Qwen3.8-Flash-Next on average. Read the token totals together with the accuracy results above, since a smaller token budget is useful only when task quality is maintained.

* Each benchmark was evaluated in thinking mode at its default reasoning effort. Unless otherwise stated, sampling uses temp=1.0, top_p=0.95, top_k=20, and a 256K context window.
* Terminal-Bench 2.1: Evaluated with the Claude Code agent; max_output_tokens=128K, max_turns=500, 12h timeout; temp=1.0, top_p=0.95, top_k=20.
* ALE Agent: Evaluated with ale_claude on top of Claude Code 2.1.170; max_output_tokens=128K, 4h timeout (extended up to 8h for specific long-running tasks); temp=1.0, top_p=0.95, top_k=20.
* Toolathlon Verified: Reported as pass@1, averaged over 3 independent runs; temp=1.0, top_p=0.95, top_k=20.
* AutomationBench: Evaluated on AutomationBench v1.0.6; max_turns=50; temp=1.0, top_p=0.95, top_k=20.
* JobBench: max_tokens=200K, 600s timeout; temp=1.0, top_p=0.95, top_k=20.
* DeepSWE 1.1: Evaluated with the mini-SWE-agent scaffold; 11h timeout; temp=1.0, top_p=0.95, top_k=20.
* SWE-bench Pro: Evaluated with the SWE-agent scaffold; temp=1.0, top_p=0.95, top_k=20.
* NL2Repo-Bench: Evaluated with the Claude Code agent; max_turns=1000, 80K context window; temp=1.0, top_p=0.95, top_k=20.
* HLE (with tools): Evaluated with gpt-5.5 as the judge model; max_iterations=1000, 6h timeout, 256K context window; temp=1.0, top_p=0.95, top_k=20.

---

Model Overview

• Type: Natively multimodal Mixture-of-Experts causal language model
• Total Parameters: ~65B, plus 51B n-gram embedding and a 1.4B MTP layer
• Activated Parameters: ~6B per token
• Layers: 48 (12 Qwen Sparse Attention, 36 Gated DeltaNet)
• Hidden Size: 2560
• Experts: 256 routed + 1 shared, 10 routed experts activated per token
• Context Length: 262,144 natively, extensible to ~1M tokens with YaRN
• Precision: BF16, with FP8, NVFP4 and GGUF releases available
• Multi-Token Prediction: 1 layer, also shipped as a standalone draft model in the GGUF edition

The n-gram embedding (51B parameters, about 102 GB in BF16) is a token lookup table. It adds no activated compute and can be kept in host memory instead of GPU memory, so the GPU holds about 134 GB of BF16 weights plus KV cache.

Available editions:

Edition	Precision	Size on disk
Groot-65B-A6B-261023	BF16	236.6 GB (134 GB model + 102 GB n-gram)
Groot-65B-A6B-261023-FP8	FP8 (E4M3, 128x128 blocks), n-gram in FP8	123.8 GB
Groot-65B-A6B-261023-NVFP4	NVFP4 experts, n-gram in FP8, rest BF16	98.6 GB
Groot-65B-A6B-261023-GGUF	Q4_K_M	78.8 GB (+5.3 GB MTP companion)

---

Quickstart

Groot-65B-A6B-261023 is a mixture-of-experts model with ~65B total parameters and ~6B activated parameters per token.

The model operates in thinking mode by default: the assistant turn opens with a <think> … </think> block before the final answer. It has a default context length of 262,144 tokens and natively supports tool calling.

The serving recipes below enable a reasoning parser (so the chain-of-thought is returned in a separate reasoning_content field) and a tool-call parser (so <tool_call> blocks are surfaced as OpenAI-style tool_calls).

Serving Groot-65B-A6B-261023 requires recent runtimes:

• Transformers ≥ 5.8.1
• vLLM ≥ 0.19.1
• SGLang ≥ 0.5.10

Inference efficiency and throughput vary significantly across frameworks. We recommend using the latest framework versions to ensure optimal performance and compatibility. For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly recommended.

Recommended sampling parameters:

• Thinking mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
• Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Please note that the support for sampling parameters varies according to inference frameworks.

---

Serving Groot-65B-A6B-261023

The BF16 release is about 236.6 GB: about 134 GB of model weights and a 102 GB n-gram lookup table, which can be kept in host memory instead of GPU memory. The recipes below use 4× 80GB GPUs to reserve memory for KV cache at the full 256K context; a 2× 80GB setup also fits the weights but leaves limited room for KV cache. The GPU count you need depends on context length, KV cache and framework overhead. Adjust --tensor-parallel-size / --tp-size to match your hardware.

If you run into out-of-memory (OOM) errors, consider reducing the context window. Because the model relies on extended context for complex tasks, we recommend keeping at least 128K tokens to preserve thinking capabilities.

SGLang

SGLang is a fast serving framework for large language models. sglang>=0.5.10 is recommended, which can be installed in a fresh environment with:

uv pip install sglang[all]

The following commands create API endpoints at http://localhost:8000/v1:

• Standard Version: an endpoint with maximum context length 262,144 tokens using tensor parallel on 4 GPUs.

python -m sglang.launch_server \
    --model-path Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tp-size 4 \
    --mem-fraction-static 0.85 \
    --context-length 262144 \
    --reasoning-parser qwen3

• Tool Use: to support tool calling.

python -m sglang.launch_server \
    --model-path Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tp-size 4 \
    --mem-fraction-static 0.85 \
    --context-length 262144 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

• Multi-Token Prediction (MTP): the recommended configuration for MTP.

python -m sglang.launch_server \
    --model-path Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tp-size 4 \
    --mem-fraction-static 0.85 \
    --context-length 262144 \
    --reasoning-parser qwen3 \
    --speculative-algo NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

vLLM

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. vllm>=0.19.1 is recommended:

uv pip install vllm --torch-backend=auto

• Standard Version: an endpoint with maximum context length 262,144 tokens using tensor parallel on 4 GPUs.

vllm serve Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --reasoning-parser qwen3

• Tool Call: to support tool use.

vllm serve Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3

• Multi-Token Prediction (MTP): the recommended configuration for MTP.

vllm serve Groot/Groot-65B-A6B-261023 \
    --served-model-name Groot-65B-A6B-261023 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

KTransformers

KTransformers is a flexible framework for LLM inference with CPU-GPU heterogeneous computing, which works well for MoE models under limited GPU memory. Point --model-path at Groot/Groot-65B-A6B-261023 and follow the official KTransformers documentation for MoE models.

Hugging Face Transformers

Hugging Face Transformers ships a lightweight server that is handy for quick testing and moderate-load deployment. The latest transformers is required:

pip install "transformers[serving]"

Then run transformers serve to launch a server with API endpoints at http://localhost:8000/v1; it will place the model on accelerators if available:

transformers serve Groot/Groot-65B-A6B-261023 --port 8000 --continuous-batching

You can also load the model locally with Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Groot/Groot-65B-A6B-261023"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Give me a short introduction to large language models."}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=8192,
    temperature=1.0,
    top_p=0.95,
    top_k=20,
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]
print(tokenizer.decode(output_ids, skip_special_tokens=True))

Long Context (YaRN Extension)

The model natively handles context windows of up to 262,144 tokens. When a task's combined input and output must go beyond this limit, we suggest extending the effective window with RoPE scaling. YaRN is built into both vLLM and SGLang, and a scaling factor of 4.0 grows the usable window to roughly 1M tokens.

You can turn YaRN on in either of two ways:

• Edit the checkpoint's config.json. Add a rope_scaling block to the model configuration:

{
    "rope_scaling": {
        "rope_type": "yarn",
        "factor": 4.0,
        "original_max_position_embeddings": 262144
    }
}

• Override at launch time. Leave the checkpoint untouched and extend the serve commands above with the equivalent flags.

# vLLM
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Groot/Groot-65B-A6B-261023 \
    --hf-overrides '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}' \
    --max-model-len 1000000

# SGLang
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server \
    --model-path Groot/Groot-65B-A6B-261023 \
    --json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}' \
    --context-length 1000000

Open-source runtimes implement YaRN statically: the same scaling factor is applied to every request regardless of its length, which can slightly hurt quality on ordinary-length inputs. Only enable rope_scaling when your workload genuinely needs the longer window, and size factor to match it. The target window is roughly factor × 262,144, so if your requests top out around 524,288 tokens, factor: 2.0 is the better setting.

---

Using Groot-65B-A6B-261023 via the Chat Completions API

Thinking behavior is controlled through three options: enable_thinking, preserve_thinking, and reasoning_effort.

Before starting, make sure the SDK is installed and the API base URL and API key are configured:

pip install -U openai

# Set the following accordingly
export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY="EMPTY"

Text Input

from openai import OpenAI

# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Write a Python function to merge two sorted linked lists."},
]

chat_response = client.chat.completions.create(
    model="Groot-65B-A6B-261023",
    messages=messages,
    max_tokens=81920,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {
            "enable_thinking": True,      # on by default
            "preserve_thinking": True,    # on by default
        },
    },
)
print("Chat response:", chat_response)

Reasoning Effort

The model exposes a reasoning_effort knob with three supported levels: xhigh (default), medium, and low:

from openai import OpenAI
client = OpenAI()

messages = [
    {"role": "user", "content": "Write a Python function to merge two sorted linked lists."},
]

chat_response = client.chat.completions.create(
    model="Groot-65B-A6B-261023",
    messages=messages,
    reasoning_effort="xhigh",  # xhigh by default; supported levels are xhigh, medium, and low
)
print("Chat response:", chat_response)

In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.

Instruct (or Non-Thinking) Mode

The model thinks by default before responding. You can obtain a direct response without thinking by configuring the API parameters:

from openai import OpenAI
client = OpenAI()

messages = [
    {"role": "user", "content": "Where is the Eiffel Tower?"}
]

chat_response = client.chat.completions.create(
    model="Groot-65B-A6B-261023",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)
print("Chat response:", chat_response)

The model does not support in-conversation soft switches such as /think or /nothink; thinking can only be controlled through the API parameters shown above.

Disable Preserved Thinking

By default, the model retains thinking blocks from all historical messages, maintaining a complete reasoning trace across the conversation. This behavior, known as preserved thinking, ensures full context continuity and is especially beneficial for agent scenarios where decision consistency and reduced redundant reasoning are critical. It also improves KV cache utilization, optimizing inference efficiency in both thinking and non-thinking modes.

If you prefer to retain only the thinking blocks from the latest user message, disable it with preserve_thinking=False:

from openai import OpenAI
client = OpenAI()

messages = [...]

chat_response = client.chat.completions.create(
    model="Groot-65B-A6B-261023",
    messages=messages,
    extra_body={
        "chat_template_kwargs": {"preserve_thinking": False},
    },
)
print("Chat response:", chat_response)

Tool Calling

The model emits well-formed function calls that the server parses into the standard tool_calls field:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="Groot-65B-A6B-261023",
    messages=[{"role": "user", "content": "What is the weather in Paris right now?"}],
    tools=tools,
    tool_choice="auto",
    temperature=1.0,
    top_p=0.95,
    max_tokens=2048,
    extra_body={"top_k": 20},
)

message = response.choices[0].message
# reasoning_content holds the <think> trace; content holds the final answer.
print("reasoning:", getattr(message, "reasoning_content", None))
print("answer:", message.content)

tool_call = message.tool_calls[0]
print(tool_call.function.name, tool_call.function.arguments)
# -> get_weather {"city": "Paris"}

---

Agentic Usage

Groot-65B-A6B-261023 excels in tool calling and agentic coding. It exposes an OpenAI-compatible endpoint and works out of the box with standard agent frameworks and coding CLIs:

# Any OpenAI-compatible coding CLI / agent framework: just point it at your endpoint.
export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY="EMPTY"
export OPENAI_MODEL="Groot-65B-A6B-261023"

---

Citation

If you use this model in your work, please cite it as:

@misc{groot2026,
  title  = {Scaling Down LLMs for Broader Deployment},
  author = {The Groot Team},
  year   = {2026},
}