LEDOM-2B: Reverse Language Model
⚠️ This is NOT a Hugging Face
transformersmodel. It is a custom decoder-only Transformer trained right-to-left, and it cannot be loaded withAutoModelForCausalLM.from_pretrained(...). You need this repo's loader: https://github.com/your-org/ledom. See "How to use" below.
LEDOM-2B is a purely reverse autoregressive language model from the paper
LEDOM: Reverse Language Model
(Yin et al., 2025). Where a normal Forward LM (FLM) factorizes
P(x) = ∏_t P(x_t | x_<t), LEDOM trains the same decoder-only Transformer on
token-reversed documents so it learns the complementary factorization
P(x) = ∏_t P(x_t | x_>t) — predicting each token from its future context.
The two main applications:
- Generate the past from the future: type a sentence, LEDOM produces text that plausibly precedes it. (Backward storytelling, abductive reasoning, mathematical work-backward derivations.)
- Reverse Reward verification: combine a forward LM's
P(y|x)with LEDOM's posteriorP(x|y)to rerank or beam-search candidate solutions. This implements noisy-channel decoding and yields +6.6 % AIME 2024, +15 % AMC 2023 in the paper, across DeepSeekMath / QwenMath / OpenMath2.
Model details
| Parameters | ~2 B |
| Architecture | decoder-only Transformer, RMSNorm, RoPE, ReLU² MLP (modified-nanoGPT lineage) |
| Layers | 34 |
Hidden dim (n_embd) |
2176 |
| Attention heads | 17 (full attention; n_kv_head=17) |
| Sequence length | 2048 |
| Vocabulary | 32 768 (rustbpe / tiktoken-compatible) |
| Window pattern | SSSL (sliding window: 3 short + 1 full per repeat) |
| Precision | BF16 master weights stored as FP32 in checkpoint |
| Training tokens | ≈140 B (66 079 steps × 2 097 152 tokens) |
| Final val BPB | 0.6334 |
| Training data | DCLM-Baseline-derived ClimbMix-400B (CommonCrawl) |
The depth-34 / d2176 configuration is computed by this repo's standard scaling
recipe (model_dim = depth * aspect_ratio with aspect_ratio=64); FLOPs and
batch size are auto-derived from the depth.
Files in this repo
| File | Purpose |
|---|---|
model_066079.pt |
Final model state dict (~10 GB, FP32 master weights) |
meta_066079.json |
Training metadata: model config, hyperparameters, dataloader state |
tokenizer.pkl |
tiktoken Encoding (rustbpe-trained, vocab 32768) |
token_bytes.pt |
Per-token byte counts (used for bits-per-byte evaluation) |
README.md |
This file |
Drop these into $LEDOM_BASE_DIR/rlm_checkpoints/LEDOM-2B/ and $LEDOM_BASE_DIR/tokenizer/
respectively (see below).
How to use
1. Install the LEDOM repo
git clone https://github.com/your-org/ledom # ← replace with the actual URL
cd ledom
uv sync --extra gpu # CUDA (or --extra cpu for laptop testing)
source .venv/bin/activate
2. Download the checkpoint
from huggingface_hub import snapshot_download
import os
# Pulls all files into $LEDOM_BASE_DIR/rlm_checkpoints/LEDOM-2B/
local = snapshot_download(
repo_id="Corning/LEDOM-2B",
local_dir=os.path.expanduser("~/.cache/ledom/rlm_checkpoints/LEDOM-2B"),
)
Then move the tokenizer files to the location the loader expects:
mkdir -p ~/.cache/ledom/tokenizer
mv ~/.cache/ledom/rlm_checkpoints/LEDOM-2B/tokenizer.pkl ~/.cache/ledom/tokenizer/
mv ~/.cache/ledom/rlm_checkpoints/LEDOM-2B/token_bytes.pt ~/.cache/ledom/tokenizer/
3. Generate the past from the future (interactive)
LEDOM treats your input as the ending and produces text that comes before it.
python -m scripts.rlm_chat --model-tag LEDOM-2B
# Ending: ...and that is why I never trust raccoons again.
# >>> [it generates a backstory ending in your sentence]
Or programmatically — the trick is that LEDOM is a normal causal Transformer that just happens to have been trained on reversed token streams:
import torch
from ledom.common import compute_init, autodetect_device_type
from ledom.checkpoint_manager import load_model
_, _, _, _, device = compute_init(autodetect_device_type())
model, tokenizer, _ = load_model("rlm", device, phase="eval", model_tag="LEDOM-2B")
bos = tokenizer.get_bos_token_id()
ending = "and that is why I never trust raccoons again."
# Tokenize and reverse — feed the "ending" in reverse order.
tokens = tokenizer.encode(ending)
input_ids = torch.tensor([[bos] + tokens[::-1]], device=device)
# Generate up to N reversed tokens (causal autoregression as usual).
with torch.inference_mode():
for _ in range(80):
next_id = model(input_ids)[:, -1, :].argmax(-1, keepdim=True)
if next_id.item() == bos:
break
input_ids = torch.cat([input_ids, next_id], dim=1)
# Reverse the (prompt + generation) back to natural reading order.
all_reversed = input_ids[0, 1:].tolist() # drop BOS
print(tokenizer.decode(list(reversed(all_reversed))))
4. Use LEDOM as a Reverse Reward verifier
To rerank an FLM's candidate answers with LEDOM's posterior P(x|y):
python -m scripts.reverse_reward \
--flm-tag <your-FLM> --rlm-tag LEDOM-2B \
--benchmark gsm8k --strategy best_of_n \
--n 32 --lambda 0.4 --temperature 0.7
The bidirectional score is
(1-λ)·log P_F(y|x) + λ·log P_R(x|y) (paper Eq. 7). Empirically λ ∈ [0.3, 0.5]
works best for math reasoning. Step-wise beam search is available via
--strategy beam --k 4 --n 3.
Behaviors to expect
LEDOM inherits its strengths from the reverse factorization and inherits its weaknesses for the same reason. From the paper's evaluation:
- ✔ Excels at abductive inference, question synthesis from answers, inverse-relation completion, and naturally avoids the "reversal curse".
- ✔ As a posterior verifier, consistently improves Best-of-N on math reasoning across forward models spanning 42 %–96 % greedy accuracy.
- ✘ Underperforms forward LMs on forward-causal tasks (incremental code generation, factual recall like NQ-Open / TriviaQA).
- ⚠ Safety asymmetry: forward-trained safety mechanisms do not transfer to reverse generation. Treat outputs accordingly; do not deploy untuned for user-facing content.
License
MIT (matches the upstream code repository).
Citation
@article{yin2025ledom,
title={LEDOM: Reverse Language Model},
author={Yin, Xunjian and Cheng, Sitao and Xie, Yuxi and Hu, Xinyu and Lin, Li and Wang, Xinyi and Pan, Liangming and Wang, William Yang and Wan, Xiaojun},
journal={arXiv preprint arXiv:2507.01335},
year={2025}
}