FINAL-Bench/Aether-6B-11Attn-base

Aether-6B-11Attn

⚠️ Pretraining is still in progress. This repository currently holds a mid-training checkpoint (step 174,000); it will be updated with the final weights as soon as training completes. The -base suffix denotes "no instruction tuning" — an -it variant will follow once pretraining is finished.

Aether family —

Layer map — 11x11 Latin square

Eleven sequence-mixing mechanisms across 121 layers, each appearing exactly once per row and per column. Colour marks the family: attention (blue), linear (teal), state-space/Mamba2 (orange), convolutional/Hyena (purple) — the non-attention families are what make this heterogeneous rather than a variant sweep.

A 121-layer network that places 11 different sequence-mixing mechanisms on an 11x11 Latin square.

To our knowledge, this is the first model to integrate 11 heterogeneous sequence-mixing mechanisms — attention, state-space, convolutional and linear families — inside a single network. Existing hybrids combine two or three (e.g. Mamba + Transformer).

Release scope: open weights. This repository ships the weights plus the minimum code needed to load them. Unlike Aether-7B-5Attn — which is fully open (data recipe, training code, logs, checkpoints) — the training data recipe, training code, logs and intermediate checkpoints for this model are not released.

The 11 mechanisms

121 layers = 11 x 11. Each mechanism occupies exactly 11 layers, one per row and column of the Latin square, so no mechanism concentrates at any depth.

#	Mechanism	Family	Distinctive parameters (verifiable in the weights)
0	nsa	sparse attention	compress_mlp, gate_logit, select_score
1	differential	attention	lambda_q1/k1/q2/k2, subln
2	full	attention	q/k/v/o_proj
3	mla	latent attention	kv_a_proj, kv_b_proj, q_a_proj, q_b_proj
4	sliding	attention	windowed masking over the full path
5	compress	attention	full_attn.* submodule
6	hybrid	attention	nsa.* + diff.* + gate + merge_norm
7	linear	linear attention	gate, norm
8	mamba2	state-space	A_log, D, conv1d, dt_proj, B_proj, C_proj
9	gdn	gated delta net	conv1d_q, conv1d_k, gate_proj
10	hyena	convolutional	filter_dense (implicit filter)

Precision matters here: full and sliding share the same parameters (sliding is masking over the full path), so the weights show 10 distinct parameter signatures for 11 labels. And several of these are not "attention" in the strict sense — mamba2, hyena, gdn and linear are other sequence-mixing families, which is exactly the point of the experiment.

Specifications

Parameters	5.79B total (MoE)
Layers	121 (11 x 11 Latin square)
Hidden size	1024
Attention heads	8 query / 2 key-value (GQA), head dim 128
Experts	25 routed (top-7) + 1 shared
Vocabulary	151,936
Precision	bfloat16
Checkpoint	training step 174,000
License	Apache-2.0

Provenance and honest caveats

This model was reconstructed from a sharded FSDP training checkpoint. The exact training code was not archived, so the architecture code shipped here was rebuilt and then validated two ways:

1. Exact parameter match — all 11,047 checkpoint tensors load with 0 missing / 0 unexpected keys, and the parameter count matches the checkpoint exactly (5.7885B).
2. Perplexity — the loaded model predicts real text sensibly (PPL 11.8 English / 6.5 Korean on sample sentences). A mis-reconstructed architecture would produce perplexity orders of magnitude worse.

Treat this as a research artifact, not a polished product: it is a mid-training checkpoint of an architecture experiment, with no instruction tuning.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "FINAL-Bench/Aether-6B-11Attn-base"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()

trust_remote_code=True is required — this is a custom architecture (aether_v2_11attn) and the modeling code ships in this repository.

Run at batch_size = 1 (the NSA branches do not consume a padding mask), and run the forward under torch.autocast(dtype=torch.bfloat16) — several branches create fp32 intermediates.

Greedy decoding degenerates into repetition (expected for an un-tuned base checkpoint) — use sampling with a repetition penalty. Generation is ~10 s/token: 121 layers with no KV cache. This is a research artifact, not a serving build.

Related

• Fully-open sibling: Aether-7B-5Attn (weights + data recipe + training code + logs + checkpoints)
• Collection: Aether Foundation Model

Built by VIDRAFT. License: Apache-2.0