TR-3: language models compressed with a 3-site Tensor Ring
Every attention and MLP projection replaced by a 3-site Tensor Ring (cyclic — the ring closes with a non-trivial boundary bond, so it is not a train/TT) plus a 5% sparse residual, then healed with pure forward-KL distillation from the corresponding dense teacher.
| Directory | Base model | Params | Reduction | Licence |
|---|---|---|---|---|
llama2-7b/ |
Llama-2-7b-chat | 6.74B → 4.02B | 40.30% | llama2 |
llama2-13b/ |
Llama-2-13b-chat | 13.02B → 7.74B | 40.53% | llama2 |
qwen2.5-3b/ |
Qwen2.5-3B-Instruct | 3.09B → 2.08B | 32.65% | qwen-research — non-commercial |
qwen2.5-7b/ |
Qwen2.5-7B-Instruct | 7.62B → 5.19B | 31.89% | apache-2.0 |
Controls
| Directory | What it isolates |
|---|---|
tt3-llama2-7b/ |
Open-chain Tensor Train at the same per-layer budget as llama2-7b/ |
tt3-qwen2.5-3b/ |
Same, against qwen2.5-3b/ |
svd-llm/, svd-llm-v2/ |
SVD-LLM baselines |
Each TT-3 directory ships both state_dict_fp16_unhealed.pt and
state_dict_fp16_healed.pt; the TR-3 directories ship the healed weights only.
Does the ring topology matter?
The TT-3 controls are budget-matched to their TR-3 counterparts to within 0.1%, so the only structural difference is the closed ring versus the open chain:
| Model | Arm | KL compressed | KL healed | MMLU healed |
|---|---|---|---|---|
| LLaMA-2-7B | TT-3 | 6.6868 | 0.4077 | 33.38% |
| LLaMA-2-7B | TR-3 | 0.6932 | 0.1822 | 43.82% |
| Qwen2.5-3B | TT-3 | 6.3507 | 0.3955 | 43.33% |
| Qwen2.5-3B | TR-3 | 0.8888 | 0.1729 | 57.41% |
Roughly 9x lower KL before healing and ~2.3x after, worth 10-14 MMLU points. Each control's README states the two confounds that qualify this (a batch-size difference in the healed rows, and uneven ALS coverage) and the per-layer evidence that the ALS gap is an order of magnitude too small to explain the result.
Why the compression rates differ
Rates are matched by the damage compression inflicts, not by the nominal percentage. Both Qwen2.5 sizes collapse to chance at the ~40% the LLaMA-2 models tolerate:
| Model | Reduction | pre-heal KL |
|---|---|---|
| LLaMA-2-13B | 40.53% | 0.3616 |
| Qwen2.5-7B | 31.89% | 0.4965 |
| LLaMA-2-7B | 40.30% | 0.6915 |
| Qwen2.5-3B | 32.65% | 0.8882 |
| Qwen2.5-7B | 40.60% | 2.0204 — collapses |
| Qwen2.5-3B | 40.94% | 3.5810 — collapses |
Two effects are separable and both real: damage falls with model size, and at equal size and equal rate the Qwen2.5 architecture is harder to compress than LLaMA-2 (2.0204 vs 0.6915, both ~7B at ~40%). Scale mitigates the architecture effect without removing it.
⚠️ Licences differ per subdirectory
The Hub shows one license field, so it is other; the binding terms are per directory.
qwen2.5-3b/ and tt3-qwen2.5-3b/ are Qwen RESEARCH LICENSE — research/evaluation only,
while qwen2.5-7b/ is Apache 2.0: Alibaba released the two Qwen sizes under different terms,
so the permissive licence does not carry across. LLaMA directories are Llama 2 Community
License. Do not assume one subdirectory's terms apply to another.
⚠️ These are NOT standard transformers checkpoints
state_dict_fp16_*.pt holds factorisation cores, not dense nn.Linear weights. Loading
requires rebuilding the architecture with the layer classes shipped in each directory, using
that directory's ranks.json. AutoModelForCausalLM.from_pretrained() will not work.
Each subdirectory's README carries a complete loading recipe.