Adityapande403/SpecDecode-edge

SpecDecode Edge: Medusa-3Heads for Qwen2.5-1.5B-Instruct

This repository provides trained Medusa multi-head speculative decoding weights for accelerating Qwen/Qwen2.5-1.5B-Instruct on single GPU and edge devices.

• Target Model: Qwen/Qwen2.5-1.5B-Instruct (Frozen)
• Speculative Method: Medusa Multi-Head Speculative Decoding (Cai et al., 2024)
• Heads Configuration: 3 speculative lookahead heads (offsets $t+2, t+3, t+4$)
• Architecture per Head: 1 ResBlock (SiLU) + Linear projection ($1536 \rightarrow 151936$)
• Precision: bfloat16
• Trained Parameters: ~707M parameters
• Hardware Verified: Single NVIDIA GeForce RTX 5060 Ti (15.93 GB VRAM)
• Source Code & GitHub Repository: https://github.com/adityapande403/SpecDecode-edge

---

Performance & Acceleration

Empirical benchmarks on an NVIDIA RTX 5060 Ti:

Workload Category	Baseline Decode TPS	Speculative Decode TPS	Speedup Ratio	Mean Acc / Step
Code Generation	39.99 tok/s	43.17 tok/s	1.08x	1.14
Mathematical Reasoning	40.41 tok/s	42.25 tok/s	1.05x	1.12
Structured Output	39.23 tok/s	39.44 tok/s	1.01x	1.05
Technical Explanations	38.64 tok/s	39.40 tok/s	1.02x	1.05

• Peak VRAM Consumption: 2.96 GB (18.5% of 16GB GPU memory).
• Authoritative Semantic Preservation: Target model remains 100% authoritative. In 6 out of 8 categories, speculative decoding produced an exact bitwise token-by-token match with standard autoregressive generation.

---

Usage with SpecDecode Edge

Clone the repository:

git clone https://github.com/adityapande403/SpecDecode-edge.git
cd SpecDecode-edge
pip install -r requirements.txt

Run speculative generation:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
from specdecode.models.medusa_model import MedusaModel
from specdecode.inference.speculative_generator import SpeculativeGenerator

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
medusa_repo_id = "Adityapande403/SpecDecode-edge"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, dtype=torch.bfloat16, device_map="cuda:0")

medusa_model = MedusaModel(base_model, num_heads=3, num_layers=1, freeze_base=True)

# Download and load Medusa heads
weights_path = hf_hub_download(repo_id=medusa_repo_id, filename="medusa_heads.safetensors")
medusa_model.load_medusa_heads(weights_path)
medusa_model.eval()

generator = SpeculativeGenerator(medusa_model, tokenizer)
out = generator.generate("Write a quicksort implementation in Python:", max_new_tokens=80)
print(out.generated_text)

---

Citation & Acknowledgements

@software{specdecode_edge_2026,
  author = {Aditya Pande},
  title = {SpecDecode Edge: Training and Deploying Speculative Decoding Models for Local LLM Inference},
  year = {2026},
  url = {https://github.com/adityapande403/SpecDecode-edge}
}