HyLo-Qwen-7MLA21GDN-8K-SFT

HyLo converts a pretrained Transformer into a hybrid instead of pretraining one from scratch. Each layer is converted to one of two types: attention layers become Multi-head Latent Attention (MLA), which caches a low-rank latent rather than full keys and values, while the remaining layers become linear blocks (Gated DeltaNet or Mamba-2) that hold a fixed-size recurrent state and no KV cache. The converted model is then trained for long context, retaining short-context accuracy while running on a small fraction of the original KV cache.

This checkpoint upcycles Qwen/Qwen3-1.7B into 7 MLA layers and 21 Gated DeltaNet layers (28 layers total, 2.28B parameters), and was supervised-fine-tuned at 8,192 tokens (8K) of context with Qwen/Qwen3-8B as the distillation teacher.

Introduced in Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling (arXiv:2604.24715), where it appears as HyLo-Qwen-7MLA21GDN in Table 4 (Qwen3-1.7B backbone).

Model at a glance


Base model	`Qwen/Qwen3-1.7B`
Distillation teacher	`Qwen/Qwen3-8B`
Layers	28 (7 MLA + 21 Gated DeltaNet)
Parameters	2.28B (paper reports 2.3B)
KV cache	3.9% of the base model's
Trained context	8,192 tokens
Checkpoint precision	float32 weights (load in bfloat16)
Training	Enhanced-ILD, then long-context SFT with teacher distillation

Architecture

Layer types are placed by index, not in a repeating pattern: the MLA layers sit where the base model is most sensitive to losing full attention.

MLA layers (7): [1, 5, 9, 13, 17, 21, 25]
Gated DeltaNet layers (21): [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 18, 19, 20, 22, 23, 24, 26, 27]

Gated DeltaNet is a gated delta-rule linear-attention block that keeps a fixed-size recurrent state instead of a growing KV cache, so those layers contribute no KV cache at all. The MLA layers keep attention but cache a low-rank latent instead of full keys and values, which is what brings the total cache to 3.9% of the base model's.

MLA dimension	Value
KV latent rank (`kv_lora_rank`)	256
Query latent rank (`q_lora_rank`)	1344
RoPE head dim (`qk_rope_head_dim`)	64
NoPE head dim (`qk_nope_head_dim`)	64
Value head dim (`v_head_dim`)	128
Attention heads	16

Gated DeltaNet dimension	Value
Heads (`gdn_num_heads`)	6
Head dim (`gdn_head_dim`)	256

The hybrid layout lives in hybrid_config.json; config.json is the base model's configuration and is kept for reference only.

Context length

Use this model up to 8,192 tokens. That is the length it was trained and evaluated at.

The backbone's native RoPE window (32,768 tokens) already covers the trained length, so no position interpolation was applied (rope_scaling.factor: 1.0).

max_position_embeddings in the config is 40,960, inherited from the base model. It is not a supported context length for this checkpoint: quality past 8,192 tokens is not something the paper measures or claims. Serving stacks size their KV cache from this field, so set the maximum length explicitly (for example --max-model-len 8192).

The 8K and 64K models are separate releases trained from the same recipe at different lengths. If you need 16K or more, use the 64K sibling of this model; the RULER numbers below show what happens when an 8K-trained model is pushed past its training length.

Training

Stage	What happens	Context	LR	Data
1. Enhanced-ILD	Layer-wise distillation aligns the newly initialised MLA and linear blocks with the base model's internal representations	2,048	2e-4	20% of the SFT mixture
2. Long-context SFT with teacher-guided distillation	End-to-end distillation from the teacher at the target context length	8,192	6e-5	full mixture

Loss: KL divergence between student and teacher next-token distributions (kl_weight 1.0, ce_weight 0.0)
Global batch size: 16 sequences; 1 epoch, cosine schedule with 0.01 warmup ratio
Precision: bfloat16 mixed precision
Hardware: 8x AMD Instinct MI300X, FSDP
Memory-efficient distillation: a fused KL kernel that avoids materialising the full logit tensor

Training data

Dataset	License
`JunxiongWang/sftdatasetv3`	apache-2.0
`nvidia/OpenMathInstruct-2`	cc-by-4.0
`open-thoughts/OpenThoughts-114k`	apache-2.0
`open-r1/OpenR1-Math-220k`	apache-2.0
`nvidia/ChatQA2-Long-SFT-data`	cc-by-nc-2.0

Training used AMD-processed variants of these datasets (subsetting, reformatting to the chat template, and decontamination against the evaluation suites).

Evaluation

All numbers are taken from Table 4 (Qwen3-1.7B backbone) of the paper, measured 0-shot with the EleutherAI lm-evaluation-harness.

Commonsense reasoning (accuracy, 0-shot)

Task	HyLo-Qwen-7MLA21GDN
ARC-Challenge	45.3
ARC-Easy	72.7
HellaSwag	61.5
OpenBookQA	39.4
PIQA	73.2
RACE	36.3
WinoGrande	64.6
Average	56.1

Long context: RULER (all 13 tasks) at 8K, 16K, 32K and 64K

Context	HyLo-Qwen-7MLA21GDN
8K	63.5
16K	43.6
32K	30.3
64K	17.4

The columns past 8,192 tokens are reported for completeness and show the model outside its training length; they are not a supported operating point.

Math

Benchmark	HyLo-Qwen-7MLA21GDN
GSM8K	76.0

Usage

The hybrid layer types are not part of transformers, so AutoModelForCausalLM cannot build this model. Install AMD's hybrid-model code first:

git clone https://github.com/AMD-AGI/AMD-Hybrid-Models.git
cd AMD-Hybrid-Models/HyLo

Then follow the installation instructions in HyLo/README.md.

import torch
from transformers import AutoTokenizer
from hybrid.hybrid_wrapper import HybridModelWrapper

checkpoint = "amd/HyLo-Qwen-7MLA21GDN-8K-SFT"

model = HybridModelWrapper.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model.eval()

messages = [{"role": "user", "content": "Summarise the document above in five bullets."}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).cuda()

tokens = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True, temperature=0.6, top_p=0.95, top_k=20,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

Notes:

The checkpoint stores float32 weights, which is what training wrote out. Load it in bfloat16 as shown above: that is the precision the evaluation numbers were measured at, and float32 doubles the memory footprint.
Use the chat template shipped with this repository. It is the template the model was trained with, and it is not identical to the base model's.
Unlike Qwen3's own template, this one does not open a <think> block, yet every Qwen release here began its answer with a <think> ... </think> span when we checked: the behaviour comes from distilling Qwen/Qwen3-8B. Strip or handle those spans if your application does not expect them, and budget tokens for them.
Keep prompts within 8,192 tokens.

The HyLo family

Model	Backbone	Linear block	Trained context	KV cache
HyLo-Llama-4MLA12GDN-8K-SFT	Llama-3.2-1B-Instruct	Gated DeltaNet	8K	3.9%
HyLo-Llama-4MLA12GDN-64K-SFT	Llama-3.2-1B-Instruct	Gated DeltaNet	64K	3.9%
HyLo-Llama-8MLA8GDN-8K-SFT	Llama-3.2-1B-Instruct	Gated DeltaNet	8K	7.8%
HyLo-Llama-8MLA8GDN-64K-SFT	Llama-3.2-1B-Instruct	Gated DeltaNet	64K	7.8%
HyLo-Llama-6MLA22GDN-8K-SFT	Llama-3.2-3B-Instruct	Gated DeltaNet	8K	2.0%
HyLo-Llama-6MLA22GDN-64K-SFT	Llama-3.2-3B-Instruct	Gated DeltaNet	64K	2.0%
HyLo-Llama-14MLA14GDN-8K-SFT	Llama-3.2-3B-Instruct	Gated DeltaNet	8K	4.7%
HyLo-Llama-14MLA14GDN-64K-SFT	Llama-3.2-3B-Instruct	Gated DeltaNet	64K	4.7%
HyLo-Qwen-7MLA21GDN-8K-SFT (this model)	Qwen3-1.7B	Gated DeltaNet	8K	3.9%
HyLo-Qwen-7MLA21GDN-64K-SFT	Qwen3-1.7B	Gated DeltaNet	64K	3.9%
HyLo-Qwen-14MLA14GDN-8K-SFT	Qwen3-1.7B	Gated DeltaNet	8K	7.8%
HyLo-Qwen-14MLA14GDN-64K-SFT	Qwen3-1.7B	Gated DeltaNet	64K	7.8%
HyLo-Qwen-14MLA14M2-8K-SFT	Qwen3-1.7B	Mamba-2	8K	7.8%
HyLo-Qwen-14MLA14M2-64K-SFT	Qwen3-1.7B	Mamba-2	64K	7.8%

Intended use and limitations

This is a research artifact released to support the paper. It has not been safety aligned or evaluated for production use.

Capabilities and biases are inherited from the base model and the distillation teacher.
Outputs can be factually wrong, biased, or otherwise objectionable; add your own safeguards before exposing it to users.
Quality past 8,192 tokens is neither measured nor claimed.
The recipe depends on a strong teacher model being available, which is itself a cost.
Evaluation covers the benchmarks listed above only: no multilingual, coding, safety or instruction-following-at-length evaluation is reported.

License

Metadata declares apache-2.0, matching the existing AMD hybrid-model releases, and the bundled LICENSE file is AMD's research-only RAIL-MS licence. Where they disagree, the LICENSE file is the one that describes AMD's intent for this artifact, and part of the training data is non-commercial, so treat the model as research-only.

Citation

@article{fashi2026hylo,
  title={Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling},
  author={Parsa Ashrafi Fashi and Utkarsh Saxena and Mehdi Rezagholizadeh and Aref Jafari and Akash Haridas and Mingyu Yang and Vansh Bhatia and Guihong Li and Vikram Appia and Emad Barsoum},
  journal={arXiv preprint arXiv:2604.24715},
  year={2026},
  url={https://arxiv.org/abs/2604.24715}
}

@article{yang2025zebra,
  title={Zebra-Llama: Towards Extremely Efficient Hybrid Models},
  author={Yang, Mingyu and Rezagholizadeh, Mehdi and Li, Guihong and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2505.17272},
  year={2025}
}

@article{li2025xecomla,
  title={X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression},
  author={Li, Guihong and Rezagholizadeh, Mehdi and Yang, Mingyu and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2503.11132},
  year={2025}
}