What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

Code and raw experimental results for the paper:

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation Babak Barazandeh Cribl AI Research Lab arXiv:2609.32002

Overview

LoRA's rank hyperparameter is widely treated as a capacity knob — higher rank, richer function class. This paper shows that under per-factor Frobenius norm budgets (the regime induced by weight decay), the reachable set of the product BA is rank-free: its complexity depends on the budget product ρ = B_B · B_A, not on the nominal rank r. Rank controls only affordability — whether the model can spend the nuclear-norm budget — not the richness of what it can express once it does.

The experiments below verify the paper's four main claims: (A) implicit low-nuclear-norm bias regardless of rank, (B) generalization gap flat in r when the budget binds (with plateau ∝ ρ), (B') worst-case gap attained by rank-1 matrices, (C) spectral threshold scaling r_spec(ε) under polynomial and geometric decay, and (D) complexity branches under per-factor vs. joint budgets. All experiments are synthetic, run on CPU, and reproduce exactly from fixed seeds.

Files

File	Description	Paper reference
`run_expA_implicit_bias.py`	Weight decay and nuclear norm control	Section IV-A, Fig. 2
`run_expB_gap.py`	Generalization gap vs. rank under hard budget	Section IV-B, Fig. 3, Table I
`run_expB_replication.py`	B replication control (m=d=64, fresh seeds)	Appendix
`run_expBprime_worstcase.py`	Worst-case gap maximization (rank-one extremality)	Section IV-C, Fig. 1, Lemma 2
`run_expBprime_fullinit.py`	B' initialization control (Kaiming-normal B)	Appendix
`reanalyze_expB.py`	Re-analyze B results: plateau analysis, budget-active marking	Fig. 3 (v2 panels)
`replot_bprime.py`	Regenerate B' figure from saved JSON (no training)	Fig. 1
`run_expC_v2_threshold.py`	Spectral threshold scaling r_spec(ε)	Section IV-D, Table II, Fig. 4
`run_expD_complexity.py`	Complexity branches: per-factor vs. joint budget	Section IV-E, Fig. 5
`results_*.json`	Raw numerical results for all experiments	—
`*.png`	Final figures (regenerated from results JSONs)	—

Installation

pip install -r requirements.txt

Everything runs on CPU. No GPU, no downloads, no API calls.

Quick start

Each experiment is self-contained. Run from the repo directory:

# Experiment A — implicit nuclear norm bias (~3 min)
python run_expA_implicit_bias.py

# Experiment B — gap vs. rank under hard budget (~2 min)
python run_expB_gap.py

# Experiment B replication — fresh seeds, m=d=64 (~1 min)
python run_expB_replication.py

# Experiment B' — worst-case gap maximization (~2 min)
python run_expBprime_worstcase.py

# Experiment B' init control — Kaiming-normal B (~2 min)
python run_expBprime_fullinit.py

# Experiment C — spectral threshold scaling (~5 min)
python run_expC_v2_threshold.py

# Experiment D — complexity branches (~1 min, no training)
python run_expD_complexity.py

All seeds are fixed — numbers reproduce exactly. The included results_*.json files let you regenerate figures without re-running training:

python reanalyze_expB.py     # → B1_gap_vs_rank_v2.png, B2_plateau_vs_rho_v2.png
python replot_bprime.py      # → Bprime_worst_case.png

Citation

@article{barazandeh2026rankbuy,
  title={What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation},
  author={Barazandeh, Babak},
  journal={arXiv preprint arXiv:2609.32002},
  year={2026}
}