Instructions to use byadecha/Qwen3.5-4B-FoRE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use byadecha/Qwen3.5-4B-FoRE with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="byadecha/Qwen3.5-4B-FoRE", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("byadecha/Qwen3.5-4B-FoRE", trust_remote_code=True)
model = AutoModel.from_pretrained("byadecha/Qwen3.5-4B-FoRE", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use byadecha/Qwen3.5-4B-FoRE with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "byadecha/Qwen3.5-4B-FoRE"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-4B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/byadecha/Qwen3.5-4B-FoRE

SGLang

How to use byadecha/Qwen3.5-4B-FoRE with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "byadecha/Qwen3.5-4B-FoRE" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-4B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "byadecha/Qwen3.5-4B-FoRE" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-4B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use byadecha/Qwen3.5-4B-FoRE with Docker Model Runner:
```
docker model run hf.co/byadecha/Qwen3.5-4B-FoRE
```

Qwen3.5-4B-FoRE

Code (GitHub) | Models (Hugging Face)

The FoRE drafter for Qwen/Qwen3.5-4B, from Fork and Regenerate: MTP-Guided Early Forking and Suffix Regeneration for Block Diffusion Speculative Decoding (Bruh Lemma Yadecha and Li Yang).

It is a block diffusion drafter fine-tuned from z-lab/Qwen3.5-4B-DFlash to draft from visible prefixes of varying lengths. FoRE uses it in both of its stages. In MTP-guided early forking, the target's pretrained MTP module proposes K candidates for the first drafted position and the drafter generates the rest of a block for each. In draft refinement, the drafter regenerates the suffix after each of R replacement tokens. The target verifies all K+R drafts in one batched forward pass, so the output is the target's greedy continuation, up to bf16 near-ties.

The checkpoint runs with the FoRE code and must be paired with Qwen3.5-4B. It has not been tested in other DFlash integrations.

Usage

git clone https://github.com/uncc-efficient-ai/FoRE && cd FoRE
uv sync --locked --extra flash-attn
hf download Qwen/Qwen3.5-4B --local-dir models/Qwen3.5-4B
hf download byadecha/Qwen3.5-4B-FoRE --local-dir models/Qwen3.5-4B-FoRE
uv run fore-generate \
  --target models/Qwen3.5-4B \
  --drafter models/Qwen3.5-4B-FoRE \
  --k 8 --r 8 \
  --prompt "Write a Python function that returns the n-th Fibonacci number."

--k is the number K of MTP-guided drafts and --r the number R of additional drafts from draft refinement. The code's README covers setup and the Python API, and its docs/training.md covers training.

Results

Acceptance length τ (speedup over autoregressive decoding) with greedy decoding, at most 2,048 generated tokens per response, on one NVIDIA H200 NVL GPU with FlashAttention-2, from Table 1 of the paper. Both FoRE columns use this checkpoint.

Benchmark	DFlash	FoRE, K=8, R=0	FoRE, K=8, R=8
GSM8K	6.38 (3.26×)	7.54 (3.69×)	9.62 (4.28×)
MATH-500	6.89 (3.52×)	7.97 (3.92×)	10.30 (4.51×)
AIME25	6.07 (3.04×)	7.20 (3.42×)	9.24 (3.89×)
HumanEval	6.75 (3.82×)	7.75 (4.15×)	10.00 (4.64×)
MBPP	5.10 (2.56×)	6.49 (3.18×)	8.47 (3.76×)
LiveCodeBench	4.33 (2.35×)	5.44 (2.80×)	7.10 (3.23×)
MT-Bench	3.43 (1.75×)	4.39 (2.13×)	5.62 (2.39×)
Mean	5.56 (2.90×)	6.68 (3.33×)	8.62 (3.82×)

Training

Initialization: z-lab/Qwen3.5-4B-DFlash at revision 96899cc270945f554998309580b08a04a05a3187.
Data: 98,000 responses that Qwen3.5-4B regenerated (temperature 0.7, top-p 0.95, thinking off, at most 3,072 new tokens) for prompts from Nemotron-Post-Training-Dataset-v2 and CodeAlpaca-20k, after removing the 2,000 held-out prompts. The 4B data was not decontaminated against the evaluation benchmarks.
Objective: one epoch with the D-PACE loss and visible prefixes of varying lengths in each 16-token block, learning rate 1e-4, on four A6000 GPUs.
Checkpoint: the final step, epoch_1_step_23926.

The code repository's docs/training.md gives the full recipe, and its THIRD_PARTY.md lists third-party code, models and data.

License

The weights are released under the Apache License 2.0 (LICENSE). They are fine-tuned from z-lab/Qwen3.5-4B-DFlash, released under the MIT License, whose notice is in dflash-MIT.txt. The target model Qwen3.5-4B is Apache-2.0.