Hugging Face's logo

amd
/
HyLo-Llama-14MLA14GDN-64K-SFT

Text Generation
Safetensors
English
llama
hybrid
mla
gated-deltanet
long-context
kv-cache-compression
distillation
amd
conversational
Model card Files
xet
 Community

HyLo-Llama-14MLA14GDN-64K-SFT

HyLo converts a pretrained Transformer into a hybrid instead of pretraining one from scratch. Each layer is converted to one of two types: attention layers become Multi-head Latent Attention (MLA), which caches a low-rank latent rather than full keys and values, while the remaining layers become linear blocks (Gated DeltaNet or Mamba-2) that hold a fixed-size recurrent state and no KV cache. The converted model is then trained for long context, retaining short-context accuracy while running on a small fraction of the original KV cache.

This checkpoint upcycles meta-llama/Llama-3.2-3B-Instruct into 14 MLA layers and 14 Gated DeltaNet layers (28 layers total, 3.97B parameters), and was supervised-fine-tuned at 65,536 tokens (64K) of context with meta-llama/Llama-3.1-8B-Instruct as the distillation teacher.

Introduced in Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling (arXiv:2604.24715), where it appears as HyLo-Llama-14MLA14GDN in Table 3 (Llama-3.2-3B backbone).

Model at a glance

Base model meta-llama/Llama-3.2-3B-Instruct
Distillation teacher meta-llama/Llama-3.1-8B-Instruct
Layers 28 (14 MLA + 14 Gated DeltaNet)
Parameters 3.97B (paper reports 4.0B)
KV cache 4.7% of the base model's
Trained context 65,536 tokens
Checkpoint precision float32 weights (load in bfloat16)
Training Enhanced-ILD, then long-context SFT with teacher distillation

Architecture

Layer types are placed by index, not in a repeating pattern: the MLA layers sit where the base model is most sensitive to losing full attention.

  • MLA layers (14): [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26]
  • Gated DeltaNet layers (14): [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27]

Gated DeltaNet is a gated delta-rule linear-attention block that keeps a fixed-size recurrent state instead of a growing KV cache, so those layers contribute no KV cache at all. The MLA layers keep attention but cache a low-rank latent instead of full keys and values, which is what brings the total cache to 4.7% of the base model's.

MLA dimension Value
KV latent rank (kv_lora_rank) 128
Query latent rank (q_lora_rank) 1536
RoPE head dim (qk_rope_head_dim) 64
NoPE head dim (qk_nope_head_dim) 64
Value head dim (v_head_dim) 128
Attention heads 24
Gated DeltaNet dimension Value
Heads (gdn_num_heads) 9
Head dim (gdn_head_dim) 256

The hybrid layout lives in hybrid_config.json; config.json is the base model's configuration and is kept for reference only.

Context length

Use this model up to 65,536 tokens. That is the length it was trained and evaluated at.

Positions are scaled with YaRN: factor: 32.0 over an original window of 2,048 tokens, giving 65,536 usable positions.

max_position_embeddings in the config is 131,072, inherited from the base model. It is not a supported context length for this checkpoint: quality past 65,536 tokens is not something the paper measures or claims. Serving stacks size their KV cache from this field, so set the maximum length explicitly (for example --max-model-len 65536).

Training

Stage What happens Context LR Data
1. Enhanced-ILD Layer-wise distillation aligns the newly initialised MLA and linear blocks with the base model's internal representations 2,048 2e-4 20% of the SFT mixture
2. Long-context SFT with teacher-guided distillation End-to-end distillation from the teacher at the target context length 65,536 4e-5 full mixture
  • Loss: KL divergence between student and teacher next-token distributions (kl_weight 1.0, ce_weight 0.0)
  • Global batch size: 8 sequences; 1 epoch, cosine schedule with 0.01 warmup ratio
  • Precision: bfloat16 mixed precision
  • Hardware: 8x AMD Instinct MI300X, FSDP
  • Memory-efficient distillation: a fused KL kernel that avoids materialising the full logit tensor

Training data

Dataset License
JunxiongWang/sftdatasetv3 apache-2.0
nvidia/OpenMathInstruct-2 cc-by-4.0
open-thoughts/OpenThoughts-114k apache-2.0
open-r1/OpenR1-Math-220k apache-2.0
nvidia/ChatQA2-Long-SFT-data cc-by-nc-2.0

Training used AMD-processed variants of these datasets (subsetting, reformatting to the chat template, and decontamination against the evaluation suites).

Evaluation

All numbers are taken from Table 3 (Llama-3.2-3B backbone) of the paper, measured 0-shot with the EleutherAI lm-evaluation-harness.

Commonsense reasoning (accuracy, 0-shot)

Task HyLo-Llama-14MLA14GDN
ARC-Challenge 45.1
ARC-Easy 72.0
HellaSwag 68.2
OpenBookQA 39.4
PIQA 76.1
RACE 40.9
WinoGrande 63.8
Average 57.9

Long context: RULER (all 13 tasks) at 8K, 16K, 32K and 64K

Context HyLo-Llama-14MLA14GDN
8K 73.2
16K 69.7
32K 62.9
64K 52.0

Math

Benchmark HyLo-Llama-14MLA14GDN
GSM8K 58.9

Usage

The hybrid layer types are not part of transformers, so AutoModelForCausalLM cannot build this model. Install AMD's hybrid-model code first:

git clone https://github.com/AMD-AGI/AMD-Hybrid-Models.git
cd AMD-Hybrid-Models/HyLo

Then follow the installation instructions in HyLo/README.md.

import torch
from transformers import AutoTokenizer
from hybrid.hybrid_wrapper import HybridModelWrapper

checkpoint = "amd/HyLo-Llama-14MLA14GDN-64K-SFT"

model = HybridModelWrapper.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model.eval()

messages = [{"role": "user", "content": "Summarise the document above in five bullets."}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).cuda()

tokens = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True, temperature=0.6, top_p=0.9,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

Notes:

  • The checkpoint stores float32 weights, which is what training wrote out. Load it in bfloat16 as shown above: that is the precision the evaluation numbers were measured at, and float32 doubles the memory footprint.
  • Use the chat template shipped with this repository. It is the template the model was trained with, and it is not identical to the base model's.
  • Keep prompts within 65,536 tokens.

The HyLo family

Model Backbone Linear block Trained context KV cache
HyLo-Llama-4MLA12GDN-8K-SFT Llama-3.2-1B-Instruct Gated DeltaNet 8K 3.9%
HyLo-Llama-4MLA12GDN-64K-SFT Llama-3.2-1B-Instruct Gated DeltaNet 64K 3.9%
HyLo-Llama-8MLA8GDN-8K-SFT Llama-3.2-1B-Instruct Gated DeltaNet 8K 7.8%
HyLo-Llama-8MLA8GDN-64K-SFT Llama-3.2-1B-Instruct Gated DeltaNet 64K 7.8%
HyLo-Llama-6MLA22GDN-8K-SFT Llama-3.2-3B-Instruct Gated DeltaNet 8K 2.0%
HyLo-Llama-6MLA22GDN-64K-SFT Llama-3.2-3B-Instruct Gated DeltaNet 64K 2.0%
HyLo-Llama-14MLA14GDN-8K-SFT Llama-3.2-3B-Instruct Gated DeltaNet 8K 4.7%
HyLo-Llama-14MLA14GDN-64K-SFT (this model) Llama-3.2-3B-Instruct Gated DeltaNet 64K 4.7%
HyLo-Qwen-7MLA21GDN-8K-SFT Qwen3-1.7B Gated DeltaNet 8K 3.9%
HyLo-Qwen-7MLA21GDN-64K-SFT Qwen3-1.7B Gated DeltaNet 64K 3.9%
HyLo-Qwen-14MLA14GDN-8K-SFT Qwen3-1.7B Gated DeltaNet 8K 7.8%
HyLo-Qwen-14MLA14GDN-64K-SFT Qwen3-1.7B Gated DeltaNet 64K 7.8%
HyLo-Qwen-14MLA14M2-8K-SFT Qwen3-1.7B Mamba-2 8K 7.8%
HyLo-Qwen-14MLA14M2-64K-SFT Qwen3-1.7B Mamba-2 64K 7.8%

Intended use and limitations

This is a research artifact released to support the paper. It has not been safety aligned or evaluated for production use.

  • Capabilities and biases are inherited from the base model and the distillation teacher.
  • Outputs can be factually wrong, biased, or otherwise objectionable; add your own safeguards before exposing it to users.
  • Quality past 65,536 tokens is neither measured nor claimed.
  • The recipe depends on a strong teacher model being available, which is itself a cost.
  • Evaluation covers the benchmarks listed above only: no multilingual, coding, safety or instruction-following-at-length evaluation is reported.

License

Metadata declares apache-2.0, matching the existing AMD hybrid-model releases, and the bundled LICENSE file is AMD's research-only RAIL-MS licence. Where they disagree, the LICENSE file is the one that describes AMD's intent for this artifact, and part of the training data is non-commercial, so treat the model as research-only.

Citation 

@article{fashi2026hylo,
  title={Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling},
  author={Parsa Ashrafi Fashi and Utkarsh Saxena and Mehdi Rezagholizadeh and Aref Jafari and Akash Haridas and Mingyu Yang and Vansh Bhatia and Guihong Li and Vikram Appia and Emad Barsoum},
  journal={arXiv preprint arXiv:2604.24715},
  year={2026},
  url={https://arxiv.org/abs/2604.24715}
}

@article{yang2025zebra,
  title={Zebra-Llama: Towards Extremely Efficient Hybrid Models},
  author={Yang, Mingyu and Rezagholizadeh, Mehdi and Li, Guihong and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2505.17272},
  year={2025}
}

@article{li2025xecomla,
  title={X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression},
  author={Li, Guihong and Rezagholizadeh, Mehdi and Yang, Mingyu and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2503.11132},
  year={2025}
}
Downloads last month
33
Safetensors
Model size
4B params
Tensor type
F32
·
Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for amd/HyLo-Llama-14MLA14GDN-64K-SFT

Base model

meta-llama/Llama-3.2-3B-Instruct
Finetuned
(2018)
this model

Datasets used to train amd/HyLo-Llama-14MLA14GDN-64K-SFT

Collection including amd/HyLo-Llama-14MLA14GDN-64K-SFT

Papers for amd/HyLo-Llama-14MLA14GDN-64K-SFT