cloverx-id/LuminaV-Optimizer-Paper

LuminaV Optimizer

We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them

Official Upstream & Standalone Codebase | Current Version: v1.1.0 | Check `Files and Versions`

Official Research Paper

LuminaV Optimizer Theory & Mechanics
Read LuminaV.pdf (Local Mirror) | Primary Paper Archive

Click the preview above to read or download the official paper PDF.

Notice: Official Upstream Repository

This repository (cloverx-id/LuminaV-Optimizer-Paper) is the official standalone and living development repository for the LuminaV optimizer family.

While LuminaV was originally conceived and validated as the core engine for the XoneLM-1.0 language model series, all subsequent optimizer upgrades, low-precision Triton kernels, PyTorch standards compliance, and bug fixes are actively maintained and released directly in this repository.

What's New in v1.1.0 (Latest Release)

The v1.1.0 release hardens LuminaV for modern PyTorch environments (PyTorch 2.13 and 2.14) and low-precision GPU execution:

Faster First-Step JIT Latency: Streamlined Triton compilation pathways, cutting warm-up time compared to the previous version (v1.0.0).
On-Chip Pointer Safety: Replaced raw stores with a unified _store_param helper that safely casts low-precision pointer types (tl.bfloat16/tl.float16), preventing LLVM compile errors when stochastic rounding is disabled.
Contiguous Buffer Enforcement: State tensors strictly enforce torch.contiguous_format to prevent stride corruption during transposed or channels-last training.
Vectorized C++ Foreach Optimization: Automatically switches to native multi-tensor C++ torch._foreach_add_ whenever stochastic rounding is inactive or parameters are in FP32.
Declarative Configuration: Added structured hyperparameter specifications and presets in config.json.

For the full version history and detailed patch notes, see CHANGELOG.md.

Overview

LuminaV is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (FP16 / BF16) without maintaining redundant 4-byte FP32 master weights.

By combining Centered Innovation Variance, Hyperbolic Tangent (tanh) Coordinate Bounding, a Directional Traffic-Cop Mask, and On-Chip Bitwise Stochastic Rounding, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while avoiding weight freezing and gradient shocks.

Key Features

Zero Master-Weight Copies: Directly mutates parameter weights in native FP16 or BF16, eliminating the 4-byte FP32 master weight allocation.
On-Chip Bitwise Stochastic Rounding (SR): Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
Hyperbolic tanh Bounding Envelope: Maps normalized momentum through a (-1.0, 1.0) transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
The Traffic-Cop Directional Gate: Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (u_t · g_t ≤ 0).
Centered Innovation Variance: Tracks centered innovation dispersion (g_t - m_t)² rather than uncentered raw second moments, suppressing variance inflation during confident descent.
Dual Execution Engine: Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ torch._foreach multi-tensor fallbacks.

Installation

Download luminav.py directly into your project root, or clone this repository:

git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper

Requirements

Python >= 3.8
PyTorch >= 2.0 (Hardened for PyTorch 2.13 and 2.14)
Triton (Recommended for CUDA acceleration)

Quickstart

Standard Instantiation

import torch
from luminav import LuminaV

# Instantiate your model in native low precision (e.g. BF16 or FP16)
model = YourModel().to(device="cuda", dtype=torch.bfloat16)

# Initialize LuminaV
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4, # or 8e-5 and 8e-6 (other best choice(for fine-tuning), hehe.)
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    alpha_ss=0.5,
    cautious=True,
    cautious_clamp_min=0.2,
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Low VRAM)
    stochastic_rounding=True,
    execution="auto"
)

# Standard training step
optimizer.zero_grad(set_to_none=True)
loss = model(inputs, targets)
loss.backward()
optimizer.step()

Loading from `config.json`

import json
import torch
from luminav import LuminaV

with open("config.json", "r") as f:
    config = json.load(f)

# Initialize with verified default configuration
optimizer = LuminaV(model.parameters(), **config["default_params"])

Parameter Reference

Parameter	Type	Default	Description
`params`	`iterable`	Required	Iterable of parameters to optimize or dicts defining parameter groups.
`lr`	`float`	`8e-4`	Learning rate (η).
`betas`	`Tuple[float, float]`	`(0.9, 0.999)`	Coefficients (β₁, β₂) for running momentum and centered innovation variance.
`eps`	`float`	`1e-8`	Numerical stability term (ε).
`weight_decay`	`float`	`8e-2`	Decoupled weight decay coefficient (λ).
`tau`	`float`	`0.8`	Analytical bias correction temperature parameter (τ).
`alpha_ss`	`float`	`0.5`	Softsign dampening factor (α_ss) used in single-buffer mode (`buffer=1`).
`cautious`	`bool`	`True`	If `True`, enables Traffic-Cop directional verification masking.
`cautious_clamp_min`	`float`	`0.2`	Safety floor density clamp (γ_min) preventing division by zero in masked normalization.
`buffer`	`int`	`2`	Buffer mode: `2` (Dual-buffer tracking m_t and v_t) or `1` (Single-buffer scalar RMS tracking).
`stochastic_rounding`	`bool`	`True`	Enables bitwise stochastic rounding on native FP16/BF16 weights.
`execution`	`str`	`"auto"`	Execution engine: `"auto"`, `"triton"`, `"foreach"`, or `"single"`.

Operational Modes

LuminaV-2 (Dual-Buffer Default: `buffer=2`)

Maintains first moment m_t and centered innovation variance v_t:

$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$

$v_t = \beta_2 v_{t-1} + (1 - \beta_2)(g_t - m_t)^2$

Updates are bounded through the hyperbolic tangent envelope:

$u_t = \tanh\left(\frac{\tilde{m}_t}{\sigma_t}\right)$

LuminaV-1 (Single-Buffer Extreme-Poverty Mode: `buffer=1`)

Collapses variance tracking into a scalar Root-Mean-Square (RMS) across the entire tensor, saving 50% optimizer state memory by maintaining only a single state buffer (m_t):

$\text{RMS}(\tilde{m}_t) = \sqrt{\frac{1}{N} \sum_{i=1}^N \tilde{m}_{t,i}^2 + \epsilon}$

$u_t = \tanh\left(\frac{z}{1 + \alpha_{ss}|z|}\right), \quad z = \frac{\tilde{m}_t}{\tau \cdot \text{RMS}(\tilde{m}_t) + \epsilon(1 - \beta_1^t)\tau}$

Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face},
  version      = {1.1.0},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}

License

Apache License 2.0. See LICENSE for full terms.

Downloads last month: -

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support