ArisrrrX/OoO-Spec-sidecar-lora

OoO-Spec sidecar — LoRA adapters

Frozen Qwen3-0.6B + LoRA semantic sidecar for an independent reproduction of OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling (arXiv:2608.00814).

This is not the authors' official release. It was trained and released by the reproducer.

What it does

The sidecar predicts, for a tool-calling request, the function choice and all schema-defined argument slots in one parallel request-level wave, out of textual order. The target model (running ToolSpec) joins the rendered hint at candidate-construction boundaries without blocking, and remains the sole verifier and commit authority. The same adapter is reused across target sizes and families without target-specific drafter training.

Adapters

Two adapters trained with the same recipe are provided:

Subfolder	Notes
sidecar-lora-spaced/	Used for the published reproduction metrics (see the code repo's results/).
sidecar-lora/	Default adapter path used by the reproduction inference scripts.

They differ by training-row layout/rendering policy; the weights are not identical.

Training

Setting	Value
Base model	Qwen/Qwen3-0.6B (unadapted)
Method	LoRA
r / alpha / dropout	32 / 64 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Precision	BF16 (KV cache disabled)
Epochs	1
Effective batch size	32 (micro 4 × accumulation 8)
Peak LR / schedule	1e-4 / cosine, 5% warmup
Max sequence length	2048
Seed	31082027
Objective	completion-only causal cross-entropy (prompt labels masked)
Reported losses	train 0.1318 / eval 0.1329

Training data: semantic rows derived from Qwen2.5-32B-Instruct teacher traces over API-Bank and ToolAlpaca source requests. The datasets and teacher traces are not redistributed here; see the code repository for the download and preparation scripts.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3-0.6B"
repo = "ArisrrrX/OoO-Spec-sidecar-lora"

tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")

# use the adapter that produced the reported results, or subfolder="sidecar-lora"
model = PeftModel.from_pretrained(model, repo, subfolder="sidecar-lora-spaced")
model = model.eval()

In the full system this model is served separately (e.g. with vLLM as a LoRA module) and queried with one batched wave per request. See the code repository for the inference loop and evaluation scripts.

Citation

If you use this, please cite the original paper:

@misc{zhang2026ooospec,
  title        = {OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling},
  author       = {Zhiheng Zhang and Mujie Xu and Feiyu Sun and Zhixin Zhang},
  year         = {2026},
  eprint       = {2608.00814},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2608.00814}
}

and the ToolSpec baseline:

@article{xia2026toolspec,
  title   = {ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding},
  author  = {Xia, Heming and Li, Yongqi and Du, Cunxiao and Song, Mingbo and Li, Wenjie},
  journal = {arXiv preprint arXiv:2604.13519},
  year    = {2026}
}

License

Apache-2.0. The base model Qwen/Qwen3-0.6B is subject to its own license.